Best unofficial Apache Server developers community

Answers

Importing data from HDFS to Hive table

⇧

⇩

79 views

I have my data in data/2011/01/13/0100/file in HDFS, each of thes file contain data in tab separated, say name, ip , url.

I want to create a table in Hive and import the data from hdfs, table should contain time,name, ip and url.

How can I import these using Hive ? r the data should be in some other format so that I can import the time as well ?

hadoop hdfs hive

asked April 22, 2011 6:22 pm CDT

AlgoMan

posted via StackOverflow

2 Answers

Newest Most Voted

⇧

⇩

Hi,
To do this you have to use partitions, read more about them here: http://wiki.apache.org/hadoop/Hive/LanguageManual/DDL#Add_Partitions
partition column in hive

answered April 24, 2011 2:43 pm CDT

Wojtek

⇧

⇩

Hey AlgoMan,

You need to create the table to load the files into and then use the LOAD DATA command to load the files into the Hive tables. See the Hive documentation for the precise syntax to use.

Regards, Jeff

answered May 2, 2011 7:54 pm CDT

Jeff Hammerbacher

Your answer

Similar discussion threads

do i need to define a table before importing csv data using copy?
February 14, 2011 05:01:45 PM
Hi, I have several csv files each from whose many columns I only require a small subset. How can I avoid the trouble of specifying tables with columns I won't need just to import all this data? Any suggestions using command line or pgadmin III…

hive newbie - importing data into hive
December 14, 2010 10:23:39 PM
just wondering if i have a pipe delimited file, how i can just import this data into hive: basically i am using the microlens database, which is pipe separated. for example: user id | age | gender | occupation | zip code translates to 123 | 24…

Best approach for data mashup... Import external Data from MySQL to a Hive table or use Custom Map u
November 16, 2010 02:12:12 PM
Hello all, I'm using Hive to do mashups of data that is currently on HDFS and data is that is on a MySQL database... I'm considering this scenario instead of bringing the data from MySQL to HDFS because it is a matter verifying the relating data…

Even After removing the hdfs blocks from Data Nodes, I could still able to query the table.
December 24, 2010 08:16:30 AM
Hello Users, One interesting observation, I experienced. I'm running a cluster of 2 DN and I NM / SNM and 1 HM and 2 RS. Replication Strategy is 2. I created a table say temp via HBase shell and put on entry(row) into that, I then…

Hive not reflecting hdfs data
March 10, 2011 05:35:13 AM
Hi, I am a hive newbie.I am managing a setup where data is regularly fed into HDFS using flume.However, hive does not show the data that is recently added to the HDFS.It used to earlier,but somehow its not updating now.The queries i fire all …

Importing/Appending to Existing Table
February 1, 2011 06:38:34 PM
I have an existing table with 15,492 rows and want to add additional rows from a .csv file. If I use 'COPY <tablename> from <filename> with delimiter as ":" csv quote as "'" ' will this overwrite existing rows in the table or append…

Load data into HIVE from HDFS Clarification
December 9, 2010 01:54:29 PM
When one loads data to hive , if the file is in hdfs, it is moved into the Hive-controlled file system namespace. and deleted from hdfs. Is it possible to change it so , file wouldn't be moved, but copied?

Created] (HIVE-2086) Data loss with external table
March 31, 2011 01:06:44 PM
Data loss with external table

Importing Data From a CSV File
January 11, 2011 07:28:56 AM
Hi there I'm a CouchDB-newbie newbie (yes, that's two newbies). How can I import data from a CSV file into a CouchDB database? I need to create an automated way to do this. The CSV file is produced from an MS Access database and each file will…

Importing versions data
December 15, 2010 04:19:44 AM
Hi All, I am trying to export/import complete repository. Export and import is working fine except the versions data. I am not able to import the jcr:systems node as it is already existing in the repository and I am getting UUID already exist…

Created: (HIVE-1968) data corruption with multi-table insert
February 7, 2011 02:03:20 PM
data corruption with multi-table insert

Updated: (HIVE-86) drop table should not delete data for external tables
July 23, 2010 11:50:54 AM
[ https://issues.apache.org/jira/browse/HIVE-86?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-86:

Review Request: HIVE-2086. Data loss with external table
April 14, 2011 08:35:12 PM

Mass data loading(importing) into hypertable cluster
February 13, 2011 10:58:57 PM
Hi All, I have successfully set up hypertable cluster in 5 nodes. Now, we are onto our real work of analyzing large amounts of data. We have large amount of data (size > 140GB) that would be entered into hypertable. Currently, we have data in…

Updated: (HIVE-230) While loading a table from a query that returns empty data results in null point
July 23, 2010 01:04:51 PM
[ https://issues.apache.org/jira/browse/HIVE-230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-230:

trouble loading from raw data table to data table
January 31, 2011 08:19:41 AM
Hello, I just started hive today. Following instructions I did set it up, and made it work to play with my web server log files. I created two tables: CREATE TABLE item_view(view_time BIGINT, ip_number STRING, session_id STRING, session_cookie…

Closed: (DERBY-449) importing data using SYSCS_UTIL.SYSCS_IMPORT_DATA generates wrong message
June 16, 2010 11:06:23 AM
[ https://issues.apache.org/jira/browse/DERBY-449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Knut Anders Hatlen closed DERBY-449.

Updated: (HIVE-41) Expose various hive table options through create table query.
January 24, 2011 03:34:12 PM
[ https://issues.apache.org/jira/browse/HIVE-41?page=com.atlassian.jira. plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-41:

Created: (HDFS-1707) Hdfs Federation: Failure in browsing data on new namenodes
March 2, 2011 06:57:59 PM
Hdfs Federation: Failure in browsing data on new namenodes

Commented: (HIVE-307) "LOAD DATA LOCAL INPATH" fails when the table already contains a fil
February 10, 2011 04:22:23 AM
[ https://issues.apache.org/jira/browse/HIVE-307?page=com.atlassian.jira .plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12 992979#comment-12992979 ] Carl Steinbach commented on HIVE-307:

Questions and Answers are licensed under cc-wiki license.

Importing data from HDFS to Hive table

2 Answers

Your answer

Join with account you already have