Hbase 0.89-hadoop version mismatch errors..

Hello!

I try to get the tutorial of pig0.7.0 running in mapreduce mode. But I 
always get IOExceptions. Looking into the HDFS-Logfiles I found a 
message "Incorrect header or version mismatch from ... got version2 
expected version 3". I don't know what to do. Is it a bug in the tutorial?

For completness: I'm running Hadoop Version 0.20.2 in a clustered setup.

Thanks, Rico.

pkg-config library version mismatch

Hi,

I've checked the Release Notes of HDFS 0.21 and saw two fixes from hadoop-
append included, other two not, but still some more that have to do with
sync 
stuff.
Is Hadoop-append for HBase made obsolete with HDFS 0.21?

Thank you,

Thomas Koch, http://www.koch.ro

Hi,

We recently upgraded our QA cluster to Cloudera Version 3 (CDH3) which has
Hbase 0.89. Our cluster is running on JDK 1.6.0_18 version. On trying to
start up Hbase it basically gives an error “you're running jdk 1.6.0_18
which has known bugs” even though Pig and Hive seems to work fine with
the version of JDK.

Any thoughts on why I am seeing this error ?

If there is a bug in this JDK version then what is recommended, upgrading
JDK to 19 or 20 or 21 (21 release this month) or downgrade the jdk version
?



Thanks for the support.



Regards

-SW



#java -version

java version "1.6.0_18"

Java(TM) SE Runtime Environment (build 1.6.0_18-b07)

Java HotSpot(TM) 64-Bit Server VM (build 16.0-b13, mixed mode)






 		 	   		  

Hello folks,

I created a branch for doing the append/sync support for Hadoop 0.20. You
can fetch the branch via

http://svn.apache.org/repos/asf/hadoo...anch-0.20-append/

If you feel that there are some JIRAS that need to go into this branch,
please update the fix-version of those JIRAS with the tag
"0.20-append</jira/secure/IssueNavigator.jspa?reset=true&mode=hide&sorter/order=DESC&sorter/field=priority&resolution=-1&pid=12310942&fixfor=12315103>
".

thanks,
dhruba

On Mon, May 10, 2010 at 11:35 PM, Dhruba Borthakur
<dhru### @gmail.com> wrote:


 @Allen: we are definitely behind 0.21 release. Tom White is guiding
that
 release and most developers are committed to removing blockers for
that
 release. Todd rightly mentions that the work being done for 0.20
benefits
 0.21 as well.

 @Jay: Thanks for summing it up so well. I completely agree with your
 viewpoint.

 thanks
 dhruba


 On Mon, May 10, 2010 at 2:06 PM, Jay Booth <jayb### @gmail.com>
wrote:

> Given that the 0.20-append branch pretty much already exists
> unofficially, via IRC, IM and email forwarded patchsets, it seems
like
> giving it an official home is just recognizing the status quo.
> Especially since 0.21 probably won't be getting rolled out into
> production everywhere the first day it's officially released.  If
the
> work's going on anyways, I don't see how giving people a shared
home
> hurts matters, if anything it gives them a better shared
touchpoint
> for forward-porting bugfixes to 0.21.
>
> A case could be made that by making it more painful to run
> 0.20-append, more momentum is created towards 0.21 but since Tom
is
> already on top of 21 and seemingly doing an excellent job, and
since
> the HBase community will probably be some of the first people to
move
> to 0.21 anyways, I don't see why having 0.20-append will damage
0.21's
> momentum at this point.
>
>
>
> On Mon, May 10, 2010 at 4:21 PM, Michael Segel
> <michae### @hotmail.com> wrote:
> >
> >
> >
> >> From: to### @cloudera.com
> >> Date: Mon, 10 May 2010 10:45:13 -0700
> >> Subject: Re: Hadoop support for hbase
> >> To: gene### @hadoop.apache.org
> >>
> >
> >> > The above is a fallacious setup.  How does a branch
in 0.20 detract
> >> > from the 0.21 momentum (The append feature that
we'd work on in 0.20
> >> > branch has little relation to how append works in
0.21).
> >>
> >> For what it's worth, though, the majority of the size of
the 0.20
> >> append patch is made up of additional unit tests. I have
started
> >> forward-porting these new tests to the trunk append and
it's already
> >> exposed a number of bugs. So while it's tempting to say
that the 0.20
> >> append is "wasted effort", it really is benefiting the
entire
> >> community and the 0.21 release as well.
> >>
> >> -Todd
> >>
> >
> > Sometimes you have to slow down to go faster.
> >
> >
> >
> > 

I've found examples using the older mapred interface but not the newer
mapreduce interface. I want to write a mapper that is configured to only
pull out specific rows(which are the mapper's keys) and a specific column's
value(which is the mapper's value).


Is there any examples of something like this available?

James Kilbride

Hey,

We're using stock CHD2 without any patches so I'm not sure if we have
hdfs630 or not. For HBase we're currently on 0.20.3 and will be testing
and
moving to 0.20.5 soon

What I did with this rollout of just config changes was take one region
server down at a time and restart the datanode on the same server. So what
I
gather I should have done was shutdown all the region servers before
restarting any of the data nodes?

I guess if I split it into different parts it would be :-

- HBase Rolling update for point/config releases is supported
  - Update masters first
  - Then update region servers in turn

- HDFS Data nodes don't support rolling updates? (Maybe better in the hdfs
list I guess)
  - Take down HBase
  - Take down datanodes
  - Update all the datanodes code/configs
  - Start datanodes
  - Start HBase

Would you be able to let me know which of these I've got right/wrong?

Thanks,

On 29 June 2010 15:50, Michael Segel <michae### @hotmail.com>
wrote:


 Dan,

 I don't think you can do that because your 'new/updated' node will
clash
 with the rest of the cloud.
 (We're talking code and not just cloud tuning parameters.) [Read
different
 jars...]

 If you're going to push an update out, then it has to be an 'all or
 nothing' push.

 Since we're using Cloudera's release, moving from CDH2 to CDH3
represents a
 full backup, down the cloud, remove the software completely, and then
then
 install new CDH3. Outside of that major switch, if we were going from
one
 sub release to another, it would be just a $> yum update
hadoop-0.20 call on
 each node.
 Again, you have to take the cloud down to do that.

 So the bottom line... if you're going to do upgrades, you'll need to
plan
 for some down time.

 HTH

 -Mike

 > From: dan.ha### @mendeley.com
 > Date: Tue, 29 Jun 2010 14:43:26 +0100
 > Subject: Rolling out Hadoop/HBase updates
 > To: us### @hbase.apache.org
 >
 > Hey,
 >
 > I've been thinking about how we do out configuration and code
updates for
 > Hadoop and HBase and was wondering what others do and what is
the best
 > practice to avoid errors with HBase.
 >
 > Currently we do a rolling update where we restart the services
on one
 node
 > at a time, so shutting down the region server then restarting
the
 datanode
 > and task trackers depending on what we are updating and what has
change.
 But
 > with this I have occasional found errors with the HBase cluster
 afterwards
 > due to corrupt META table which I think could have been caused
by
 restarting
 > the datanode, or maybe not waiting long enough for the cluster
to sort
 out
 > loosing a region server before moving on to the next.
 >
 > The most resent error upon restarting a node was :-
 >
 > 2010-06-29 10:46:44,970 ERROR
 > org.apache.hadoop.hbase.regionserver.HRegionServer: Error
closing
 > files,3822b1ea8ae015f3ec932cafaa282dd211d768ad,1275145898366
 > java.io.IOException: Filesystem closed
 >         at
org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:230)
 >
 > 2010-06-29 10:46:44,970 FATAL
 > org.apache.hadoop.hbase.regionserver.HRegionServer: Shutting
down
 > HRegionServer: file system not available
 > java.io.IOException: File system is not available
 >         at
 >

org.apache.hadoop.hbase.util.FSUtils.checkFileSystemAvailable(FSUtils.java:129)
 >
 >
 > Followed by this for every region being served :-
 >
 > 2010-06-29 10:46:44,996 ERROR
 > org.apache.hadoop.hbase.regionserver.HRegionServer: Error
closing
 > documents,082595c0-6d01-11df-936c-0026b95e484c,1275676410202
 > java.io.IOException: Filesystem closed
 >         at
org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:230)
 >
 >
 > After updating all the nodes all the region server shut down
after a
 > few minutes reporting the following :-
 >
 > 2010-06-29 11:21:59,508 WARN org.apache.hadoop.hdfs.DFSClient:
Error
 > Recovery for block blk_-1437671530216085093_2565663 bad
datanode[0]
 > 10.0.11.4:50010
 >
 > 2010-06-29 11:22:09,481 FATAL
org.apache.hadoop.hbase.regionserver.HLog:
 > Could not append. Requesting close of hlog
 > java.io.IOException: All datanodes 10.0.11.4:50010 are bad.
Aborting...
 >         at
 >

org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2542)
 >
 >
 > 2010-06-29 11:22:09,482 FATAL
 > org.apache.hadoop.hbase.regionserver.LogRoller: Log rolling
failed with
 > ioe:
 > java.io.IOException: All datanodes 10.0.11.4:50010 are bad.
Aborting...
 >         at
 >

org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2542)
 >
 > 2010-06-29 11:22:10,344 ERROR
 > org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to
close log
 in
 > abort
 > java.io.IOException: All datanodes 10.0.11.4:50010 are bad.
Aborting...
 >         at
 >

org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2542)
 >
 >
 > This was fixed by restarting the master and starting the region
servers
 > again, but it would be nice to know how to roll out changes
cleaner.
 >
 > How do other people here roll out updates to HBase / Hadoop?
What order
 do
 > you restart services in and how long do you wait before moving
to the
 next
 > node?
 >
 > Just so you know we currently have 5 nodes and are getting
another 10 to
 add
 > soon.
 >
 > Thanks,
 >
 > --
 > Dan Harvey | Datamining Engineer
 > www.mendeley.com/profiles/dan-harvey
 >
 > Mendeley Limited | London, UK | www.mendeley.com
 > Registered in England and Wales | Company Number 6419015

 

I am trying to start and stop stargate rest server. I get
ClassNotFoundException intermittently.
I did perform these steps :
?  Place the Stargate jar in either the HBase installation root
directory or lib/ directories.
?  Copy the jars from contrib/stargate/lib/ into the lib/ directory of
the HBase installation.


:/usr/local/hbase-0.20.3 hadoop$./bin/hbase
org.apache.hadoop.hbase.stargate.Main -p 8080
2010-07-03 04:32:39.593::INFO:  Logging to STDERR via
org.mortbay.log.StdErrLog
2010-07-03 04:32:39.633::INFO:  jetty-6.1.14
2010-07-03 04:32:39.908::INFO:  Started SocketC### @0.0.0.0:8080


^Z
[1]+  Stopped                 ./bin/hbase
org.apache.hadoop.hbase.stargate.Main -p 8080
:/usr/local/hbase-0.20.3 hadoop$bg
[1]+ ./bin/hbase org.apache.hadoop.hbase.stargate.Main -p 8080 &
:/usr/local/hbase-0.20.3 hadoop$./bin/hbase-daemon.sh start
org.apache.hadoop.hbase.rest.Main -p 8080
starting org.apache.hadoop.hbase.rest.Main, logging to
/var/hbase/logs/hbase--org.apache.hadoop.hbase.rest.Main-phxradar03.out
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/hbase/rest/Main
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.hbase.rest.Main
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
Could not find the main class: org.apache.hadoop.hbase.rest.Main.  Program
will exit.

Hello,

We're setting up a data warehouse environment that includes Hadoop, HBase,
Hive and our own in-house MR jobs.
I would like with your permission to discuss the architecture we should
choose for this.

Today we process ~10GB of data per day.
Trying to balance between performance & consolidation, would you
consider
the following setup reasnoble?


EC2 m1.large (amd64bit, 7.5GB RAM, 400GB HD).
EC2 m1.small (intel x86, 1.7GB RAM, 160GB HD).


Cluster components:

1:[NameNode], 1:[SecondaryNameNode], 1:[JobTracker], n:[DataNode]
n:[TaskTracker], 1:[HBaseMaster], n:[HBaseRegionServer], 2*n+1:[ZooKeeper]


Planned setup :

m1.large NodeM1 "master" : [NameNode], [SecondaryNameNode], [HBaseMaster]
m1.small NodeZ1 "zoo1" : [ZooKeeper]
m1.small NodeZ2 "zoo2" : [ZooKeeper]
m1.small NodeZ3 "zoo3" : [ZooKeeper]
m1.large NodeS1 "slave1" : [DataNode], [TaskTracker], [HBaseRegionServer]
m1.large NodeS2 "slave2" : [DataNode], [TaskTracker], [HBaseRegionServer]
m1.large NodeS3 "slave3" : [DataNode], [TaskTracker], [HBaseRegionServer]
m1.large NodeS4 "slave4" : [DataNode], [TaskTracker], [HBaseRegionServer]
m1.large NodeS4 "slave5" : [DataNode], [TaskTracker], [HBaseRegionServer]

I'm having second thoughts about:

- Zookeepers on separate machines (why not run them on a slave1, slave3,
slave4 for ex.) ?
- Do I really need the SecondaryNameNode? Can I disable it completely or
should I get another 1 instance and perhaps run it with a zoo keeper
(while
the other 2 zoo keepers will remain small instances) ?
- Is it wise to run Hadoop & HBase slaves on the same instance or
should
I separate them ?

Also, how much resources (RAM, I/O) should I be giving each resource? Some
things are clear like: Make the Data directory of Hadoop on several block
devices for efficient I/O but others are not: Is HBase CPU of RAM bound?
Will hadoop benefit from lots of RAM?

Architecture references will be highly appreciated :)

Thank you for reading,
Would love to hear your thoughts on this.

Maxim.

JSON gem as of version 1.4.6 is causing server errors

    [
https://issues.apache.org/jira/browse...3#action_12897003
] 

Leo Alekseyev commented on HIVE-1192:

Hello,

In my current application environment, I need to have two HBase
clusters running in two different racks, to form a fault-tolerant
group to tolerate power failure. Then I have an HBase client, which is
sitting outside of these two clusters,  to make invocation to the
these two HBase clusters.

In my previous work, I just need to simply use the class of “HTable”,
and passed in an instance of HBaseConfiguration. And To construct the
HBaseConfiguration instance, I just need to pass in the path
information of the “hbase-site.xml”. And in the hbase-site.xml, there
is only one parameter called “hbase.rootdir” that need to configure.

Before HBase0.20.0, there used to be a parameter called “hbase.master”
that I can specify. But in HBase0.20.0, I found that it does not work
any more, likely because that the HBase master is managed by the
Zookeeper, and the master node now becomes dynamic.

Could you show me what are the APIs that I need to use, in order for
me to specify the end-point address of the HBase cluster, for the
HBase client invocation?

Regards,

Jun

Hi All,

This is my first mail in the apache mailing list... please bear with me as
I
am absolutely new to Hadoop and its family.

This is my question... I have some data on my hdfs in the following form.

(number:int,word:chararray, word2:chararray,somethingelse:int)

I want to get this data into a neatly formed HBase Table. I chose the
simpler way instead of writing my own udf. I wanted to do this....

register ../hbase/hbase-0.20.4.jar;
register ../hbase/hbase-0.20.4-test.jar;
A = Load '/some_data';
B = STORE A into 'hbase://something' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage;
dump B;

but this is the error I get when I do that

2010-07-22 16:38:35,041 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
Connecting
to hadoop file system at: hdfs://MyMachine01:9000
2010-07-22 16:38:35,550 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
Connecting
to map-reduce job tracker at: MyMachine01:9001
2010-07-22 16:38:35,868 [main] ERROR org.apache.pig.tools.grunt.Grunt -
ERROR 2998: Unhandled internal error.
org/apache/hadoop/hbase/mapreduce/TableInputFormat

I have checked my hbase-0.20.4.jar file and it does have a
TableInputFormat
class. I added the right path to hadoop-env.sh in the CLASSPATH field. I
added the conf folder to the classpath and also the test jar.

I don't know why it wouldn't work. My HBase installation went really
smooth.
I am able to check the status of the HBase in the hbase shell and still I
get this error. I am totally lost at this point. I would really appreciate
any help in this regard.

Thanks a bunch.
V.

can't generate hadoop-metircs by chukwa-0.3.0 version

HDFS side of HADOOP-6904: first step towards inter-version communications
between dfs client and NameNode

All,
  I have followed the instructions on
http://wiki.apache.org/hadoop/EclipseEnvironment to download the latest
trunk source code and build .jar for common, hdfs and mapred. but how
should
I proceed to configure and start a hadoop cluster(psudo distributed)  with
these latest .jar? I knew how to configured/start the hadoop cluster with
formal hadoop package(hadoop-*.tar.gz with all stuff of common, hdfs and
mapred there). I  googled but didn't find the related information, most
information I got after compile is to run unit test.

  Can anyone help?  Thanks for the help.

Best Regards,
Fred

I have been wondering for quite sometime now. The region counts in the
master UI page dosent really add up. The snippets from the page are below.
When hbase starts up , the second number grows as regionservers report are
identified/report.
But, the second number platues as some point and always seems to be lower
from the first one by a couple of percentages. What does it mean ?
Master Attributes
Regions On FS

13870

Region Servers
Total:

servers: 8



requests=2044, regions=13583

Hello,

Late to the party but I thought I'd mention that I have seen this as
well, with a difference however.

I've seen it on two different servers, both using SNI hosts. It happens
to me a lot when logging into joomla administrator in Firefox. Since I
use no other browser I cannot comment on is it a Firefox bug or not. I
can however state that on both servers, there is a redirect to force
http to https.

My sneaky suspicion was that the redirect is causing the problem but I 
am currently unable to get it to fail at the moment. My suspicion 
arrives out of the fact I have download manager that does not have the 
redirect on it and I just use https instead of http. I cannot recall 
that I have ever had it happen when using it, so I was thinking that 
when you login in joomla, the script calls http, the redirect forces it 
to https and I get a warning about the cert not matching. Viewing the 
cert it is the _default_ SSL hosts certificate.

However, that is not the case either since I made a script to do just 
that for testing an I get a "sent over unecrypted" warning doing that, 
so this is not the case either as I do not get that warning logging into 
Joomla.

It could be cgi related since joomla is php and the other times on the 
other server it was a perl script. I'm however not sure if it has 
happened on static content since, I do not recall it happening but some 
days I cannot recall what I did the prior day. It doesn't have a trigger 
that I can find to pull and be able to give you a test case for it. It's 
a head scratcher since it does not happen every time.

Either the OP is not crazy or we both are crazy. I do not plan on 
subscribing to the user list or I would ask the OP about browser/s, 
static vs. dynamic content and redirects.

Regards,

Gregg







Kaspar Brand wrote:
> On Sun, May 16, 2010 at 3:14 PM, Eric Covener
<cov### @gmail.com> wrote:
>> User has a non-NVH on 10.137.1.104:9902 (CN=aaa.de)and
insists SNI is
>> choosing the SSL configuration from a different VH that
(CN=aaa.at)
>> comes earlier and b) has a matching servername.
 
 I can't reproduce/confirm this behavior with 2.2.15. Did the user
 doublecheck that the www.aaa.at.crt and www.aaa.de.crt files really
have
 the proper contents?
 
>> I think that 10.137.1.104 was sent, but i'm not sure if any
SNI
>> hostname was sent. I called it like this: openssl s_client
-connect
>> 10.137.1.104:9902
 
 openssl s_client doesn't send any SNI extension by default (needs to
be
 specified with -servername, if desired).
 
 The code in mod_ssl which possibly switches to a different
certificate
 (through OpenSSL's SSL_set_SSL_CTX) is only reached from
 ssl_callback_ServerNameIndication(). And this callback is not
executed
 if there's no SNI extension in the ClientHello (at APLOG_DEBUG,
mod_ssl
 will log the outcome of ap_vhost_iterate_given_conn, but my
prediction
 is that the user won't see any such messages if he's using s_client
w/o
 the servername switch).
 
 Kaspar
 

On 14.05.10 22:51, Reinhard Vicinus wrote:
 is the following behaviour of apache 2.2.15 (debian unstable) a
feature  
 or a bug?

 Listen 10.0.0.1:81
 <VirtualHost 10.0.0.1:81>
   SSLEngine on
   SSLCertificateFile /etc/apache2/conf/aaa.crt
   SSLCertificateKeyFile /etc/apache2/conf/aaa.key

   ServerName aaa
 </VirtualHost>

 Listen 10.0.0.2:81
 <VirtualHost 10.0.0.2:81>
   SSLEngine on
   SSLCertificateFile /etc/apache2/conf/bbb.crt
   SSLCertificateKeyFile /etc/apache2/conf/bbb.key

   ServerName aaa
 </VirtualHost>


 > curl https://bbb:81
  SSL: certificate subject name 'aaa' does not match target host name
'bbb'

does 'bbb' point to 10.0.0.2?

 > curl https://10.0.0.2:81
  SSL: certificate subject name 'aaa' does not match target host name 

 '10.0.0.2'

you can expect this one, when accessing server via IP address.