Deployment architecture for Hadoop, HBase & Hive recommendations?

Hi,

I've checked the Release Notes of HDFS 0.21 and saw two fixes from hadoop-
append included, other two not, but still some more that have to do with
sync 
stuff.
Is Hadoop-append for HBase made obsolete with HDFS 0.21?

Thank you,

Thomas Koch, http://www.koch.ro

Hello folks,

I created a branch for doing the append/sync support for Hadoop 0.20. You
can fetch the branch via

http://svn.apache.org/repos/asf/hadoo...anch-0.20-append/

If you feel that there are some JIRAS that need to go into this branch,
please update the fix-version of those JIRAS with the tag
"0.20-append</jira/secure/IssueNavigator.jspa?reset=true&mode=hide&sorter/order=DESC&sorter/field=priority&resolution=-1&pid=12310942&fixfor=12315103>
".

thanks,
dhruba

On Mon, May 10, 2010 at 11:35 PM, Dhruba Borthakur
<dhru### @gmail.com> wrote:


 @Allen: we are definitely behind 0.21 release. Tom White is guiding
that
 release and most developers are committed to removing blockers for
that
 release. Todd rightly mentions that the work being done for 0.20
benefits
 0.21 as well.

 @Jay: Thanks for summing it up so well. I completely agree with your
 viewpoint.

 thanks
 dhruba


 On Mon, May 10, 2010 at 2:06 PM, Jay Booth <jayb### @gmail.com>
wrote:

> Given that the 0.20-append branch pretty much already exists
> unofficially, via IRC, IM and email forwarded patchsets, it seems
like
> giving it an official home is just recognizing the status quo.
> Especially since 0.21 probably won't be getting rolled out into
> production everywhere the first day it's officially released.  If
the
> work's going on anyways, I don't see how giving people a shared
home
> hurts matters, if anything it gives them a better shared
touchpoint
> for forward-porting bugfixes to 0.21.
>
> A case could be made that by making it more painful to run
> 0.20-append, more momentum is created towards 0.21 but since Tom
is
> already on top of 21 and seemingly doing an excellent job, and
since
> the HBase community will probably be some of the first people to
move
> to 0.21 anyways, I don't see why having 0.20-append will damage
0.21's
> momentum at this point.
>
>
>
> On Mon, May 10, 2010 at 4:21 PM, Michael Segel
> <michae### @hotmail.com> wrote:
> >
> >
> >
> >> From: to### @cloudera.com
> >> Date: Mon, 10 May 2010 10:45:13 -0700
> >> Subject: Re: Hadoop support for hbase
> >> To: gene### @hadoop.apache.org
> >>
> >
> >> > The above is a fallacious setup.  How does a branch
in 0.20 detract
> >> > from the 0.21 momentum (The append feature that
we'd work on in 0.20
> >> > branch has little relation to how append works in
0.21).
> >>
> >> For what it's worth, though, the majority of the size of
the 0.20
> >> append patch is made up of additional unit tests. I have
started
> >> forward-porting these new tests to the trunk append and
it's already
> >> exposed a number of bugs. So while it's tempting to say
that the 0.20
> >> append is "wasted effort", it really is benefiting the
entire
> >> community and the 0.21 release as well.
> >>
> >> -Todd
> >>
> >
> > Sometimes you have to slow down to go faster.
> >
> >
> >
> > 

I've found examples using the older mapred interface but not the newer
mapreduce interface. I want to write a mapper that is configured to only
pull out specific rows(which are the mapper's keys) and a specific column's
value(which is the mapper's value).


Is there any examples of something like this available?

James Kilbride

Hey,

We're using stock CHD2 without any patches so I'm not sure if we have
hdfs630 or not. For HBase we're currently on 0.20.3 and will be testing
and
moving to 0.20.5 soon

What I did with this rollout of just config changes was take one region
server down at a time and restart the datanode on the same server. So what
I
gather I should have done was shutdown all the region servers before
restarting any of the data nodes?

I guess if I split it into different parts it would be :-

- HBase Rolling update for point/config releases is supported
  - Update masters first
  - Then update region servers in turn

- HDFS Data nodes don't support rolling updates? (Maybe better in the hdfs
list I guess)
  - Take down HBase
  - Take down datanodes
  - Update all the datanodes code/configs
  - Start datanodes
  - Start HBase

Would you be able to let me know which of these I've got right/wrong?

Thanks,

On 29 June 2010 15:50, Michael Segel <michae### @hotmail.com>
wrote:


 Dan,

 I don't think you can do that because your 'new/updated' node will
clash
 with the rest of the cloud.
 (We're talking code and not just cloud tuning parameters.) [Read
different
 jars...]

 If you're going to push an update out, then it has to be an 'all or
 nothing' push.

 Since we're using Cloudera's release, moving from CDH2 to CDH3
represents a
 full backup, down the cloud, remove the software completely, and then
then
 install new CDH3. Outside of that major switch, if we were going from
one
 sub release to another, it would be just a $> yum update
hadoop-0.20 call on
 each node.
 Again, you have to take the cloud down to do that.

 So the bottom line... if you're going to do upgrades, you'll need to
plan
 for some down time.

 HTH

 -Mike

 > From: dan.ha### @mendeley.com
 > Date: Tue, 29 Jun 2010 14:43:26 +0100
 > Subject: Rolling out Hadoop/HBase updates
 > To: us### @hbase.apache.org
 >
 > Hey,
 >
 > I've been thinking about how we do out configuration and code
updates for
 > Hadoop and HBase and was wondering what others do and what is
the best
 > practice to avoid errors with HBase.
 >
 > Currently we do a rolling update where we restart the services
on one
 node
 > at a time, so shutting down the region server then restarting
the
 datanode
 > and task trackers depending on what we are updating and what has
change.
 But
 > with this I have occasional found errors with the HBase cluster
 afterwards
 > due to corrupt META table which I think could have been caused
by
 restarting
 > the datanode, or maybe not waiting long enough for the cluster
to sort
 out
 > loosing a region server before moving on to the next.
 >
 > The most resent error upon restarting a node was :-
 >
 > 2010-06-29 10:46:44,970 ERROR
 > org.apache.hadoop.hbase.regionserver.HRegionServer: Error
closing
 > files,3822b1ea8ae015f3ec932cafaa282dd211d768ad,1275145898366
 > java.io.IOException: Filesystem closed
 >         at
org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:230)
 >
 > 2010-06-29 10:46:44,970 FATAL
 > org.apache.hadoop.hbase.regionserver.HRegionServer: Shutting
down
 > HRegionServer: file system not available
 > java.io.IOException: File system is not available
 >         at
 >

org.apache.hadoop.hbase.util.FSUtils.checkFileSystemAvailable(FSUtils.java:129)
 >
 >
 > Followed by this for every region being served :-
 >
 > 2010-06-29 10:46:44,996 ERROR
 > org.apache.hadoop.hbase.regionserver.HRegionServer: Error
closing
 > documents,082595c0-6d01-11df-936c-0026b95e484c,1275676410202
 > java.io.IOException: Filesystem closed
 >         at
org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:230)
 >
 >
 > After updating all the nodes all the region server shut down
after a
 > few minutes reporting the following :-
 >
 > 2010-06-29 11:21:59,508 WARN org.apache.hadoop.hdfs.DFSClient:
Error
 > Recovery for block blk_-1437671530216085093_2565663 bad
datanode[0]
 > 10.0.11.4:50010
 >
 > 2010-06-29 11:22:09,481 FATAL
org.apache.hadoop.hbase.regionserver.HLog:
 > Could not append. Requesting close of hlog
 > java.io.IOException: All datanodes 10.0.11.4:50010 are bad.
Aborting...
 >         at
 >

org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2542)
 >
 >
 > 2010-06-29 11:22:09,482 FATAL
 > org.apache.hadoop.hbase.regionserver.LogRoller: Log rolling
failed with
 > ioe:
 > java.io.IOException: All datanodes 10.0.11.4:50010 are bad.
Aborting...
 >         at
 >

org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2542)
 >
 > 2010-06-29 11:22:10,344 ERROR
 > org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to
close log
 in
 > abort
 > java.io.IOException: All datanodes 10.0.11.4:50010 are bad.
Aborting...
 >         at
 >

org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2542)
 >
 >
 > This was fixed by restarting the master and starting the region
servers
 > again, but it would be nice to know how to roll out changes
cleaner.
 >
 > How do other people here roll out updates to HBase / Hadoop?
What order
 do
 > you restart services in and how long do you wait before moving
to the
 next
 > node?
 >
 > Just so you know we currently have 5 nodes and are getting
another 10 to
 add
 > soon.
 >
 > Thanks,
 >
 > --
 > Dan Harvey | Datamining Engineer
 > www.mendeley.com/profiles/dan-harvey
 >
 > Mendeley Limited | London, UK | www.mendeley.com
 > Registered in England and Wales | Company Number 6419015

 

I am trying to start and stop stargate rest server. I get
ClassNotFoundException intermittently.
I did perform these steps :
?  Place the Stargate jar in either the HBase installation root
directory or lib/ directories.
?  Copy the jars from contrib/stargate/lib/ into the lib/ directory of
the HBase installation.


:/usr/local/hbase-0.20.3 hadoop$./bin/hbase
org.apache.hadoop.hbase.stargate.Main -p 8080
2010-07-03 04:32:39.593::INFO:  Logging to STDERR via
org.mortbay.log.StdErrLog
2010-07-03 04:32:39.633::INFO:  jetty-6.1.14
2010-07-03 04:32:39.908::INFO:  Started SocketC### @0.0.0.0:8080


^Z
[1]+  Stopped                 ./bin/hbase
org.apache.hadoop.hbase.stargate.Main -p 8080
:/usr/local/hbase-0.20.3 hadoop$bg
[1]+ ./bin/hbase org.apache.hadoop.hbase.stargate.Main -p 8080 &
:/usr/local/hbase-0.20.3 hadoop$./bin/hbase-daemon.sh start
org.apache.hadoop.hbase.rest.Main -p 8080
starting org.apache.hadoop.hbase.rest.Main, logging to
/var/hbase/logs/hbase--org.apache.hadoop.hbase.rest.Main-phxradar03.out
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/hbase/rest/Main
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.hbase.rest.Main
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
Could not find the main class: org.apache.hadoop.hbase.rest.Main.  Program
will exit.

Hi there,

I was wondering if anyone out there had any good recommendations for a
good thorough JMeter training course.  This can be either online or in a
classroom (prefer US based).   Topics of interest include passing
parameters (multiple user login and other params), testing in a
load-balanced environment, simulating a real-time experience, and stress
testing.

Any help is appreciated.

Thanks,
Natalie

Hello,

In my current application environment, I need to have two HBase
clusters running in two different racks, to form a fault-tolerant
group to tolerate power failure. Then I have an HBase client, which is
sitting outside of these two clusters,  to make invocation to the
these two HBase clusters.

In my previous work, I just need to simply use the class of “HTable”,
and passed in an instance of HBaseConfiguration. And To construct the
HBaseConfiguration instance, I just need to pass in the path
information of the “hbase-site.xml”. And in the hbase-site.xml, there
is only one parameter called “hbase.rootdir” that need to configure.

Before HBase0.20.0, there used to be a parameter called “hbase.master”
that I can specify. But in HBase0.20.0, I found that it does not work
any more, likely because that the HBase master is managed by the
Zookeeper, and the master node now becomes dynamic.

Could you show me what are the APIs that I need to use, in order for
me to specify the end-point address of the HBase cluster, for the
HBase client invocation?

Regards,

Jun

Hi All,

This is my first mail in the apache mailing list... please bear with me as
I
am absolutely new to Hadoop and its family.

This is my question... I have some data on my hdfs in the following form.

(number:int,word:chararray, word2:chararray,somethingelse:int)

I want to get this data into a neatly formed HBase Table. I chose the
simpler way instead of writing my own udf. I wanted to do this....

register ../hbase/hbase-0.20.4.jar;
register ../hbase/hbase-0.20.4-test.jar;
A = Load '/some_data';
B = STORE A into 'hbase://something' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage;
dump B;

but this is the error I get when I do that

2010-07-22 16:38:35,041 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
Connecting
to hadoop file system at: hdfs://MyMachine01:9000
2010-07-22 16:38:35,550 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
Connecting
to map-reduce job tracker at: MyMachine01:9001
2010-07-22 16:38:35,868 [main] ERROR org.apache.pig.tools.grunt.Grunt -
ERROR 2998: Unhandled internal error.
org/apache/hadoop/hbase/mapreduce/TableInputFormat

I have checked my hbase-0.20.4.jar file and it does have a
TableInputFormat
class. I added the right path to hadoop-env.sh in the CLASSPATH field. I
added the conf folder to the classpath and also the test jar.

I don't know why it wouldn't work. My HBase installation went really
smooth.
I am able to check the status of the HBase in the hbase shell and still I
get this error. I am totally lost at this point. I would really appreciate
any help in this regard.

Thanks a bunch.
V.

     [
https://issues.apache.org/jira/browse...nels:all-tabpanel
]

Venkatesh S updated HIVE-1264:

    [
https://issues.apache.org/jira/browse...7#action_12890877
] 

Carl Steinbach commented on HIVE-1264:

    [
https://issues.apache.org/jira/browse...2#action_12891002
] 

Venkatesh S commented on HIVE-1264:

     [
https://issues.apache.org/jira/browse...nels:all-tabpanel
]

Venkatesh S updated HIVE-1264:

    [
https://issues.apache.org/jira/browse...7#action_12892517
] 

Ashish Thusoo commented on HIVE-1264:

     [
https://issues.apache.org/jira/browse...nels:all-tabpanel
]

Ashish Thusoo reassigned HIVE-1264:

     [
https://issues.apache.org/jira/browse...nels:all-tabpanel
]

Carl Steinbach updated HIVE-1264:

     [
https://issues.apache.org/jira/browse...nels:all-tabpanel
]

Carl Steinbach updated HIVE-1355:

Todd Lipcon wrote in this article,
http://www.cloudera.com/blog/2010/07/...-in-cdh3-b2-hbase
on  Hbase and Hive that in the next release writes from hive back into
hbase will be
supported. Currently you can only use hbase as an input table. Does
that mean an output writable for hbase is coming to hive rather than
using a map/reduce to load up the resultset? This would be fantastic!
I could not find a Jira ticket number to watch for this feature.

Thanks!
Wade Arnold

Hi Ray,

Apologies for my very slow response.

Here is a draft of a doc which explains how I think we can tackle this:

http://wiki.apache.org/hadoop/Hive/FilterPushdownDev

Maybe you can work on translation from ExprNodeDesc -> HBase scan
object?  If you can get that working in isolation in unit tests, I can help
with the rest of the parts for plumbing the filter through from Hive's
optimizer.

JVS

On Jul 1, 2010, at 2:57 PM, Ray Duong wrote:

Thanks John,

Can you provide me with some pointers?.  My team can try to work on it.

Our workaround right now is to call the Thrift API from within Hive using
a UDF.

Thanks,
-ray


On Thu, Jul 1, 2010 at 1:19 PM, John Sichi
<jsi### @facebook.com<mailto:jsi### @facebook.com>> wrote:
On Jul 1, 2010, at 10:36 AM, Ray Duong wrote:

 Is there away to do a hbase key lookup using the Hive-Hbase
integration without doing a full scan?

 Since I'm specifying the key='foo' in the where condition, shouldn't
it be a fast lookup?

Hi Ray,

Pushing down filters to HBase is one of our roadmap items.

https://issues.apache.org/jira/browse/HIVE-1226

If you'd like to work on it, let me know and I'll give you some pointers.

JVS

     [
https://issues.apache.org/jira/browse...nels:all-tabpanel
]

Carl Steinbach updated HIVE-812:

     [
https://issues.apache.org/jira/browse...nels:all-tabpanel
]

John Sichi updated HIVE-1229: