Best unofficial Apache Server developers community
Username
Forgot password?
Sign in with Twitter account
Sign in with Facebook account

Hive / Hadoop / Flatfile: What is an efficient way to combine and concat rows

⇧
0
⇩
41 views
 
id col1 col2 ... coln
---------------------
foo barA barB ...
foo barD barX
boo barA barC
foo barC barC

I'd like to combine this into 'collapsed' rows which look like this:

foo barA;barD;barC barB;barX;barC
boo barD barC

At the moment the source document is a hive 'table', [which is essentially the same as a flat text file I suppose] - and I am wondering what is the most efficient way to accomplish this?

java hadoop hive
asked May 5, 2011 2:56 am CDT
malangi
posted via StackOverflow

1 Answers

⇧
0
⇩
 

If you are loading the data into hive from a mapreduce job you may be able to adjust that MR to do the transformation on the data for you, and load it into the table as you want it (array or ; delimited, ect)

If you are looking to be able to update/adjust the data, HIVE probably isn't the best option for that. You may want to look at HBase and do an 'aggregation' to generate the data as you want it to load into HBase. Anytime the same Key/ColumnFamily/Column value is generated, it overwrites it if it exists so it will 'update' the value. I use this in production to generate data throughout the day that constantly updates.

In either case, to manipulate the structure of large quantities of data, you will probably want to use a mapreduce job and have it do the restructuring for you.

answered May 12, 2011 4:18 pm CDT
Nija

Your answer

Join with account you already have

Sign in with Twitter account
Sign in with Facebook account
Sign in with Google Friend Connect

Preview
Similar questions
Hive, hadoop, and the mechanics behind hive.exec.reducers.max
February 17, 2011
Hadoop/hive metastore
August 7, 2009
Even data distribution on hadoop/hive
July 6, 2010
How should I partition data in s3 for use with hadoop hive?
December 15, 2010
Ability to limit maximum reducers for a hadoop hive mapred job?
February 7, 2011
How do I concat grouped rows as a single row in MySQL?
April 15, 2011
MySQL concat multiple rows into a single row
February 9, 2011
Using Apache Hive as a MapReduce Input Format and/or Scraping Hive Metadata
April 14, 2011
Mysql combine rows?
April 5, 2011
How to combine two table rows where a series of conditions are met
March 15, 2011
Can I combine the results of one tables rows into a single row (with columns)?
February 10, 2011
What's the most efficient way to count rows in a table?
March 10, 2011
Efficient way to delete multiple rows in HBase
January 6, 2011
Efficient way to ensure unique rows in SQLite3
March 3, 2011
In mysql, what is an efficient way to get only rows in a column by a distinct value in one of that table's columns?
February 4, 2011
Efficient insert of multiple rows with SQLAlchemy/SQLite3 when duplicate entries exist.
April 20, 2011
Efficient method of finding database rows that have *one or more* qualities from a list of seven qualities
January 13, 2011
Hive with Lucene
January 31, 2011
Concat in php codeigniter
January 7, 2011
Create temporary table in Hive?
March 21, 2011
What is the advantage of integrating Hbase and Hive
January 6, 2011
How to transfer mysql table to hive?
January 7, 2011
Convert Data from HIVE to noSQL
March 6, 2011
Order by within group concat
February 22, 2011
Concat to text field
December 31, 2010
MYSQL Concat not working
February 24, 2011
Mysql group concat
April 3, 2011
How to concat two fields in Postgresql
April 17, 2011
How to setup Concat within SP in MySQL
April 26, 2011
Is there a CONCAT() like Index-Option?
February 24, 2011