Best unofficial Apache Server developers community
Username
Forgot password?
Sign in with Twitter account
Sign in with Facebook account

Hive - create a table from zip file.

⇧
0
⇩
61 views

I have bunch of zip files of CSVs, that I want to create Hive table from. I'm trying to figure out what's the best way to do so.

  • Unzip the files, upload them to HDFS.
  • Is there a way to copy the files to HDFS, unzip the
  • Or is there any other better / recommended way?
hadoop hive hdfs
asked June 16, 2011 5:09 pm CDT
Jeeyoung Kim
posted via StackOverflow

1 Answers

⇧
0
⇩
 

It's common practice to convert CSV files to tab separated or Ctrl A , or Ctrl B delimited and then upload it to Hadoop/Hive.

To upload files to HDFS you can use following command -

hadoop fs -put file_to_uplload hdfs_path

I assume you would like to automate this. In that case following instructions will be helpful.

  1. Create hive table with columns mapping to CSV files fileds.(you can remove unnecessary fields at this step). Choose your delimiter in hive create table statement.

  2. Convert csv files to delimited format (Ctrl A or Ctrl B)

  3. Upload files to Hive table location.

You can automate about steps using python batch processing scripts/framework.

For further reading : http://wiki.apache.org/hadoop/Hive/GettingStarted

answered June 19, 2011 2:23 am CDT
Reko

Your answer

Join with account you already have

Sign in with Twitter account
Sign in with Facebook account
Sign in with Google Friend Connect

Preview
Similar questions
Create temporary table in Hive?
March 21, 2011
Hive - How can I write a create statement for a variable length, existing, hdfs file??
May 12, 2011
Using Apache Hive as a MapReduce Input Format and/or Scraping Hive Metadata
April 14, 2011
Hive, hadoop, and the mechanics behind hive.exec.reducers.max
February 17, 2011
How to transfer mysql table to hive?
January 7, 2011
Importing data from HDFS to Hive table
April 22, 2011
What are the consequences of adding a column to an exiting HIVE table ?
February 21, 2011
Can I change the datatype for the "Create Table" field in the Show Create Table function in MySQL?
January 10, 2011
Hive installation issue while following Hive apache wiki step by step
June 14, 2011
Create temporary table from select statement without using Create Table
May 2, 2011
PostgreSQL: Can you create an index in the CREATE TABLE definition?
June 4, 2011
Create array on c# to create many table on mysql
June 7, 2011
Where do I create the file .htaccess, in order to serve my HTML5 cache manifest file correctly?
June 13, 2011
Create table from select then alter table by adding an auto increment column in mysql
January 11, 2011
Mysql create table based on another table including indexes
January 25, 2011
Tried to repair large MySQL table table that throws "126: Incorrect key file for table" error only with Update queries
March 10, 2011
MySQL create table containing values from a column in another table.
January 28, 2011
Create a enum table and access it from other table in mysql
March 21, 2011
Create war file from configuration file
April 20, 2011
Create new table from two tables with multiple columns in first table replaced by columns in second table
April 10, 2011
How i can create file "remotely" by vim
January 6, 2011
Create a .war file from gwt-project
March 23, 2011
Using php to create an xml file from a mysql db
March 15, 2011
Using php to create an xml file from a mysql db
March 18, 2011
How to create a file using postgresql functions ?
January 17, 2011
How to create global configuration file?
May 11, 2011
IPhone doesn't create sqlite db file
February 26, 2011
"no such file to load -- pg" when trying rake db:create
December 30, 2010
Why I cannot create this table?
April 22, 2011
Create this table using my sql
February 1, 2011