HBASE/HADOOP Examples

All,

The examples in the hbase examples, and on the hadoop wiki all reference
the deprecated interfaces of the mapred package. Are there any examples of
how to use hbase as the input for a mapreduce job, that uses the mapreduce
package instead? I'm looking to set up a job which will read from a hbase
table based on a row value passed into the job, and which starts the map
the row values(as the value key) and the column names(or value) as the map
values.

James Kilbride

Hi,

I've checked the Release Notes of HDFS 0.21 and saw two fixes from hadoop-
append included, other two not, but still some more that have to do with
sync 
stuff.
Is Hadoop-append for HBase made obsolete with HDFS 0.21?

Thank you,

Thomas Koch, http://www.koch.ro

Hello folks,

I created a branch for doing the append/sync support for Hadoop 0.20. You
can fetch the branch via

http://svn.apache.org/repos/asf/hadoo...anch-0.20-append/

If you feel that there are some JIRAS that need to go into this branch,
please update the fix-version of those JIRAS with the tag
"0.20-append</jira/secure/IssueNavigator.jspa?reset=true&mode=hide&sorter/order=DESC&sorter/field=priority&resolution=-1&pid=12310942&fixfor=12315103>
".

thanks,
dhruba

On Mon, May 10, 2010 at 11:35 PM, Dhruba Borthakur
<dhru### @gmail.com> wrote:


 @Allen: we are definitely behind 0.21 release. Tom White is guiding
that
 release and most developers are committed to removing blockers for
that
 release. Todd rightly mentions that the work being done for 0.20
benefits
 0.21 as well.

 @Jay: Thanks for summing it up so well. I completely agree with your
 viewpoint.

 thanks
 dhruba


 On Mon, May 10, 2010 at 2:06 PM, Jay Booth <jayb### @gmail.com>
wrote:

> Given that the 0.20-append branch pretty much already exists
> unofficially, via IRC, IM and email forwarded patchsets, it seems
like
> giving it an official home is just recognizing the status quo.
> Especially since 0.21 probably won't be getting rolled out into
> production everywhere the first day it's officially released.  If
the
> work's going on anyways, I don't see how giving people a shared
home
> hurts matters, if anything it gives them a better shared
touchpoint
> for forward-porting bugfixes to 0.21.
>
> A case could be made that by making it more painful to run
> 0.20-append, more momentum is created towards 0.21 but since Tom
is
> already on top of 21 and seemingly doing an excellent job, and
since
> the HBase community will probably be some of the first people to
move
> to 0.21 anyways, I don't see why having 0.20-append will damage
0.21's
> momentum at this point.
>
>
>
> On Mon, May 10, 2010 at 4:21 PM, Michael Segel
> <michae### @hotmail.com> wrote:
> >
> >
> >
> >> From: to### @cloudera.com
> >> Date: Mon, 10 May 2010 10:45:13 -0700
> >> Subject: Re: Hadoop support for hbase
> >> To: gene### @hadoop.apache.org
> >>
> >
> >> > The above is a fallacious setup.  How does a branch
in 0.20 detract
> >> > from the 0.21 momentum (The append feature that
we'd work on in 0.20
> >> > branch has little relation to how append works in
0.21).
> >>
> >> For what it's worth, though, the majority of the size of
the 0.20
> >> append patch is made up of additional unit tests. I have
started
> >> forward-porting these new tests to the trunk append and
it's already
> >> exposed a number of bugs. So while it's tempting to say
that the 0.20
> >> append is "wasted effort", it really is benefiting the
entire
> >> community and the 0.21 release as well.
> >>
> >> -Todd
> >>
> >
> > Sometimes you have to slow down to go faster.
> >
> >
> >
> > 

Hey,

We're using stock CHD2 without any patches so I'm not sure if we have
hdfs630 or not. For HBase we're currently on 0.20.3 and will be testing
and
moving to 0.20.5 soon

What I did with this rollout of just config changes was take one region
server down at a time and restart the datanode on the same server. So what
I
gather I should have done was shutdown all the region servers before
restarting any of the data nodes?

I guess if I split it into different parts it would be :-

- HBase Rolling update for point/config releases is supported
  - Update masters first
  - Then update region servers in turn

- HDFS Data nodes don't support rolling updates? (Maybe better in the hdfs
list I guess)
  - Take down HBase
  - Take down datanodes
  - Update all the datanodes code/configs
  - Start datanodes
  - Start HBase

Would you be able to let me know which of these I've got right/wrong?

Thanks,

On 29 June 2010 15:50, Michael Segel <michae### @hotmail.com>
wrote:


 Dan,

 I don't think you can do that because your 'new/updated' node will
clash
 with the rest of the cloud.
 (We're talking code and not just cloud tuning parameters.) [Read
different
 jars...]

 If you're going to push an update out, then it has to be an 'all or
 nothing' push.

 Since we're using Cloudera's release, moving from CDH2 to CDH3
represents a
 full backup, down the cloud, remove the software completely, and then
then
 install new CDH3. Outside of that major switch, if we were going from
one
 sub release to another, it would be just a $> yum update
hadoop-0.20 call on
 each node.
 Again, you have to take the cloud down to do that.

 So the bottom line... if you're going to do upgrades, you'll need to
plan
 for some down time.

 HTH

 -Mike

 > From: dan.ha### @mendeley.com
 > Date: Tue, 29 Jun 2010 14:43:26 +0100
 > Subject: Rolling out Hadoop/HBase updates
 > To: us### @hbase.apache.org
 >
 > Hey,
 >
 > I've been thinking about how we do out configuration and code
updates for
 > Hadoop and HBase and was wondering what others do and what is
the best
 > practice to avoid errors with HBase.
 >
 > Currently we do a rolling update where we restart the services
on one
 node
 > at a time, so shutting down the region server then restarting
the
 datanode
 > and task trackers depending on what we are updating and what has
change.
 But
 > with this I have occasional found errors with the HBase cluster
 afterwards
 > due to corrupt META table which I think could have been caused
by
 restarting
 > the datanode, or maybe not waiting long enough for the cluster
to sort
 out
 > loosing a region server before moving on to the next.
 >
 > The most resent error upon restarting a node was :-
 >
 > 2010-06-29 10:46:44,970 ERROR
 > org.apache.hadoop.hbase.regionserver.HRegionServer: Error
closing
 > files,3822b1ea8ae015f3ec932cafaa282dd211d768ad,1275145898366
 > java.io.IOException: Filesystem closed
 >         at
org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:230)
 >
 > 2010-06-29 10:46:44,970 FATAL
 > org.apache.hadoop.hbase.regionserver.HRegionServer: Shutting
down
 > HRegionServer: file system not available
 > java.io.IOException: File system is not available
 >         at
 >

org.apache.hadoop.hbase.util.FSUtils.checkFileSystemAvailable(FSUtils.java:129)
 >
 >
 > Followed by this for every region being served :-
 >
 > 2010-06-29 10:46:44,996 ERROR
 > org.apache.hadoop.hbase.regionserver.HRegionServer: Error
closing
 > documents,082595c0-6d01-11df-936c-0026b95e484c,1275676410202
 > java.io.IOException: Filesystem closed
 >         at
org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:230)
 >
 >
 > After updating all the nodes all the region server shut down
after a
 > few minutes reporting the following :-
 >
 > 2010-06-29 11:21:59,508 WARN org.apache.hadoop.hdfs.DFSClient:
Error
 > Recovery for block blk_-1437671530216085093_2565663 bad
datanode[0]
 > 10.0.11.4:50010
 >
 > 2010-06-29 11:22:09,481 FATAL
org.apache.hadoop.hbase.regionserver.HLog:
 > Could not append. Requesting close of hlog
 > java.io.IOException: All datanodes 10.0.11.4:50010 are bad.
Aborting...
 >         at
 >

org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2542)
 >
 >
 > 2010-06-29 11:22:09,482 FATAL
 > org.apache.hadoop.hbase.regionserver.LogRoller: Log rolling
failed with
 > ioe:
 > java.io.IOException: All datanodes 10.0.11.4:50010 are bad.
Aborting...
 >         at
 >

org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2542)
 >
 > 2010-06-29 11:22:10,344 ERROR
 > org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to
close log
 in
 > abort
 > java.io.IOException: All datanodes 10.0.11.4:50010 are bad.
Aborting...
 >         at
 >

org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2542)
 >
 >
 > This was fixed by restarting the master and starting the region
servers
 > again, but it would be nice to know how to roll out changes
cleaner.
 >
 > How do other people here roll out updates to HBase / Hadoop?
What order
 do
 > you restart services in and how long do you wait before moving
to the
 next
 > node?
 >
 > Just so you know we currently have 5 nodes and are getting
another 10 to
 add
 > soon.
 >
 > Thanks,
 >
 > --
 > Dan Harvey | Datamining Engineer
 > www.mendeley.com/profiles/dan-harvey
 >
 > Mendeley Limited | London, UK | www.mendeley.com
 > Registered in England and Wales | Company Number 6419015

 

I am trying to start and stop stargate rest server. I get
ClassNotFoundException intermittently.
I did perform these steps :
?  Place the Stargate jar in either the HBase installation root
directory or lib/ directories.
?  Copy the jars from contrib/stargate/lib/ into the lib/ directory of
the HBase installation.


:/usr/local/hbase-0.20.3 hadoop$./bin/hbase
org.apache.hadoop.hbase.stargate.Main -p 8080
2010-07-03 04:32:39.593::INFO:  Logging to STDERR via
org.mortbay.log.StdErrLog
2010-07-03 04:32:39.633::INFO:  jetty-6.1.14
2010-07-03 04:32:39.908::INFO:  Started SocketC### @0.0.0.0:8080


^Z
[1]+  Stopped                 ./bin/hbase
org.apache.hadoop.hbase.stargate.Main -p 8080
:/usr/local/hbase-0.20.3 hadoop$bg
[1]+ ./bin/hbase org.apache.hadoop.hbase.stargate.Main -p 8080 &
:/usr/local/hbase-0.20.3 hadoop$./bin/hbase-daemon.sh start
org.apache.hadoop.hbase.rest.Main -p 8080
starting org.apache.hadoop.hbase.rest.Main, logging to
/var/hbase/logs/hbase--org.apache.hadoop.hbase.rest.Main-phxradar03.out
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/hbase/rest/Main
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.hbase.rest.Main
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
Could not find the main class: org.apache.hadoop.hbase.rest.Main.  Program
will exit.

Hi All,

This is my first mail in the apache mailing list... please bear with me as
I
am absolutely new to Hadoop and its family.

This is my question... I have some data on my hdfs in the following form.

(number:int,word:chararray, word2:chararray,somethingelse:int)

I want to get this data into a neatly formed HBase Table. I chose the
simpler way instead of writing my own udf. I wanted to do this....

register ../hbase/hbase-0.20.4.jar;
register ../hbase/hbase-0.20.4-test.jar;
A = Load '/some_data';
B = STORE A into 'hbase://something' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage;
dump B;

but this is the error I get when I do that

2010-07-22 16:38:35,041 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
Connecting
to hadoop file system at: hdfs://MyMachine01:9000
2010-07-22 16:38:35,550 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
Connecting
to map-reduce job tracker at: MyMachine01:9001
2010-07-22 16:38:35,868 [main] ERROR org.apache.pig.tools.grunt.Grunt -
ERROR 2998: Unhandled internal error.
org/apache/hadoop/hbase/mapreduce/TableInputFormat

I have checked my hbase-0.20.4.jar file and it does have a
TableInputFormat
class. I added the right path to hadoop-env.sh in the CLASSPATH field. I
added the conf folder to the classpath and also the test jar.

I don't know why it wouldn't work. My HBase installation went really
smooth.
I am able to check the status of the HBase in the hbase shell and still I
get this error. I am totally lost at this point. I would really appreciate
any help in this regard.

Thanks a bunch.
V.

Hello,

In my current application environment, I need to have two HBase
clusters running in two different racks, to form a fault-tolerant
group to tolerate power failure. Then I have an HBase client, which is
sitting outside of these two clusters,  to make invocation to the
these two HBase clusters.

In my previous work, I just need to simply use the class of “HTable”,
and passed in an instance of HBaseConfiguration. And To construct the
HBaseConfiguration instance, I just need to pass in the path
information of the “hbase-site.xml”. And in the hbase-site.xml, there
is only one parameter called “hbase.rootdir” that need to configure.

Before HBase0.20.0, there used to be a parameter called “hbase.master”
that I can specify. But in HBase0.20.0, I found that it does not work
any more, likely because that the HBase master is managed by the
Zookeeper, and the master node now becomes dynamic.

Could you show me what are the APIs that I need to use, in order for
me to specify the end-point address of the HBase cluster, for the
HBase client invocation?

Regards,

Jun

All,
  I have followed the instructions on
http://wiki.apache.org/hadoop/EclipseEnvironment to download the latest
trunk source code and build .jar for common, hdfs and mapred. but how
should
I proceed to configure and start a hadoop cluster(psudo distributed)  with
these latest .jar? I knew how to configured/start the hadoop cluster with
formal hadoop package(hadoop-*.tar.gz with all stuff of common, hdfs and
mapred there). I  googled but didn't find the related information, most
information I got after compile is to run unit test.

  Can anyone help?  Thanks for the help.

Best Regards,
Fred

Good Afternoon,

Apologies for starting a whole bunch of threads in one afternoon, but
I'm desperately after a bit of information from the community ;(

I wondered if anyone has written some deployment examples for various
sizes of infrastructure, or if this is something Puppet Labs plans to
address as part of documentation?  There are a number of ways of
slicing the apple, including WEBrick for "smaller scale" deployments
through to mod_proxy or mod_passenger for "larger scale" deployments
and some seemingly dated notes in the Wiki about potentially splitting
serving of manifests and files?

I appreciate everyone has a home-cooked and favourite way to make
things work, but think we'd all benefit if initial deployment was a
bit easier, better documented, and standardized?

I've got in mind about three different topologies:

    1) Fifty nodes, managing 400 resources
    2) Two hundred nodes, managing 1000 resources
    3) One thousand nodes, managing 2000 resources
    4) Other suggestions from anyone here?

Optionally you might mix in the steps necessary to get Dashboard
working as an external node classifier with the setup instructions?
If the hardware and complexity involved in coping with #3 isn't vastly
greater than solving #1 then perhaps that should become the new
'Recommended' architecture for deploying any non-trivial install?

If we're just talking about installing Apache, dropping in Puppet via
Rack and mod_passenger, it should be fairly easy to write this from
the context of being distribution neutral?

Please feel free to point me at documentation or blogs if there is
anything particularly suitable out there!

Many Thanks,
  - Alex

Hi Peter,

can we take your examples and commit them into our SVN ? We would use them
for integration testing.

http://github.com/phoet/buildr-examples

Thanks,

Antoine

My problematic code is like this:

 

VersionManager vm = getSession().getWorkspace().getVersionManager();


vm.checkout(path);                  

doMyVersioning();

 

// I have to same before checking in, what if checkin failed? How can I
rollback?

getSession().save();                

 

Version v = vm.checkin(path);

 

 

From: Niu, Xuetao 
Sent: Thursday, July 01, 2010 1:08 PM
To: us### @jackrabbit.apache.org
Subject: Transactional versioning examples?

 

Hello,

 

I failed finding any transaction examples of using Jackrabbit2.0. Could
any of you provide one?

 

I need to have versioning behaviors transactional. My problem is that
before I checkout a node, I need to first save the current session. But
what if the checkin failed after my changes are saved? How can I
rollback the previously saved session changes?

 

Thanks a lot!

Xuetao

Hello,

 

I failed finding any transaction examples of using Jackrabbit2.0. Could
any of you provide one?

 

I need to have versioning behaviors transactional. My problem is that
before I checkout a node, I need to first save the current session. But
what if the checkin failed after my changes are saved? How can I
rollback the previously saved session changes?

 

Thanks a lot!

Xuetao

Hi,

I have ApacheDS 1.5.7 installed.  I also have sources.

I've found two similar examples on the ApacheDS site for running the
server embedded
in a servlet.  I can't get either one to work...

---
One example is at:
http://directory.apache.org/apacheds/...-application.html

I've been unable to build the example.  The first problem was that it uses
org.apache.directory.server.core.DefaultDirectoryService
which is not a public class.  So I changed the StartStopListener package
to org.apache.directory.server.core,
one compile problem solved.

The example also uses:
    org.apache.directory.server.ldap.LdapServer
    org.apache.directory.server.protocol.shared.transport.TcpTransport
I cannot find these classes anywhere, in 1.5.7  they seem to be a part of
1.5.6 only.  BUT
I cannot build 1.5.6 (which would load the jars to my local repository)
because the project
parent pom no longer exists, seems not to be in svn, or in some maven repo
somewhere..


btw.  The parent pom for the apacheds-archetype-webapp also does not seem
to exist...  ahg...


---
Another example is at:
https://cwiki.apache.org/DIRxSRVx10/e...-application.html

I've been able to build this example, however it cannot be started in
Tomcat.   The example uses javax.naming.directory.InitialDirContext
and Tomcat 5.5.29 explodes with exceptions trying to use this class.


+++
I can send stack traces etc... but first am I using the correct list?

Should these examples work?  What release of ApacheDS ?   How to build ?

Thanks
Roy

I am using spring and active mq using the embedded broker.now I want to
persists messages in oracle database are there any exmaples for oracle
activemq ?

Integrate NestedCayenneForm from examples into CayenneForm

When I extracted the examples from the current Camel package then I see
always two
build config files: pom.xml (for Maven) and build.xml (for Ant).

Do I have to run BOTH to get the examples working or is this a exclusive
choice up to the
user which one he wants to run?

Assume I want to use Ant (only) can I delete pom.xml?

Ben

StringUtils JavaDoc contains erroneous examples

Author: kkolinko
Date: Tue Jun 29 19:04:30 2010
New Revision: 959074

URL: http://svn.apache.org/viewvc?rev=959074&view=rev
Log:
Correct broken link on the index.html page of the jsp examples.
CTR

Modified:
    tomcat/tc6.0.x/trunk/webapps/examples/jsp/index.html

Modified: tomcat/tc6.0.x/trunk/webapps/examples/jsp/index.html
URL:
http://svn.apache.org/viewvc/tomcat/t...074&view=diff

Author: kkolinko
Date: Tue Jun 29 19:06:43 2010
New Revision: 959075

URL: http://svn.apache.org/viewvc?rev=959075&view=rev
Log:
Correct broken link on the index.html page of the jsp examples.

Modified:
    tomcat/trunk/webapps/examples/jsp/index.html

Modified: tomcat/trunk/webapps/examples/jsp/index.html
URL:
http://svn.apache.org/viewvc/tomcat/t...075&view=diff

Author: markt
Date: Fri Jul 16 14:55:01 2010
New Revision: 964816

URL: http://svn.apache.org/viewvc?rev=964816&view=rev
Log:
Make the bean Serializable so I can use the example to test/demo session
persistence on restart
Add required getters/setters
Indent using 4 spaces
Fix FindBugs warning

Modified:
    tomcat/trunk/webapps/examples/WEB-INF/classes/num/NumberGuessBean.java

Modified:
tomcat/trunk/webapps/examples/WEB-INF/classes/num/NumberGuessBean.java
URL:
http://svn.apache.org/viewvc/tomcat/t...816&view=diff