Best unofficial Apache Server developers community
Username
Forgot password?
Sign in with Twitter account
Sign in with Facebook account
List archives

Issue with Hive remote connection to mysql metastore

Hive basic questions
(37 lines)
Dose block size determine the number of map task
(14 lines)
May 31, 2011
Sanjeev Sagar
Sanjeev Sagar
Hello All, I'm having an issue while connecting hive on (client machine) to
Hadoop server running mysql based metastore.

My client side configuration for hive-site.xml file are below. I'm not sure
about the value i'm using for hive.metastore.uris

I did not find much info on this. I saw mails using value with thrift or
file. I was not sure if either one of them is valid for using mysql
metastore remotely.

<!-- Hive client parameters for remote metastore --> <property>

   <name>hive.metastore.warehouse.dir</name>

   <value>/user/hive/warehouse</value>

   <description>location of default database for the
warehouse</description>
</property>



<property>

  <name>hive.metastore.uris</name>

  <value>thrift://testh1:3306</value>

  <description>controls whether to connect to remote metastore server
</description> </property>



<property>

  <name>hive.metastore.local</name>

  <value>false</value>

  <description>controls whether to connect to remove metastore server
</description> </property>

Message in the hive log file:


2011-05-31 17:29:24,414 ERROR exec.DDLTask

(SessionState.java:printError(343)) - FAILED: Error in metadata:

org.apache.thrift.transport.TTransportException

org.apache.hadoop.hive.ql.metadata.HiveException:

org.apache.thrift.transport.TTransportException

        at

org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1028)

        at

org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1013)

        at

org.apache.hadoop.hive.ql.exec.DDLTask.showTables(DDLTask.java:1691)

        at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:289)

        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:130)

        at

org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)

        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1063)

        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:900)

        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:748)

        at

org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:164)

        at

org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:241)

        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:456)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at

sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

        at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at org.apache.hadoop.util.RunJar.main(RunJar.java:186)

Caused by: org.apache.thrift.transport.TTransportException

        at

org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)

        at

org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)

        at

org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:354)

        at

org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215)

        at

org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMetastore.java:351)

        at

org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:336)

        at

org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:634)

        at

org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1024)

        ... 16 more



2011-05-31 17:29:24,418 ERROR ql.Driver

(SessionState.java:printError(343)) - FAILED: Execution Error, return
code 1 from org.apache.hadoop.hive.ql.exec.DDLTask


Can some one assist me that what config i need to connect hive on clinet to
hadoop cluseter server with mysql metastore? Do i need to start the thrift
server by using hive --service metastore ?? if yes then where on client or
server???

Highly appreciate it.





Reply
Tags: hivexml fileuris
hive-0.6 don't connection mysql in metastore
August 30, 2010 12:31:37 AM
I use hive-0.6 an use mysql as metasore, but hive don't connection the mysql. 2010-08-30 13:28:24,982 ERROR [main] util.Log4JLogger(125): Failed initialising database. Invalid URL: jdbc:mysql://127.0.0.1:3306/hive6?createDatabaseIfNotExist=true…
Updated: (HIVE-1428) ALTER TABLE ADD PARTITION fails with a remote Thrift metastore
August 12, 2010 09:27:57 PM
[ https://issues.apache.org/jira/browse/HIVE-1428?page=com.atlassian.jir a.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-1428:
Commented: (HIVE-1428) ALTER TABLE ADD PARTITION fails with a remote Thirft metastore
July 13, 2010 04:14:51 PM
[ https://issues.apache.org/jira/browse/HIVE-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888011#action_12888011 ] Paul Yang commented on HIVE-1428:
Updated: (HIVE-1428) ALTER TABLE ADD PARTITION fails with a remote Thrift metastore
August 12, 2010 09:31:54 PM
[ https://issues.apache.org/jira/browse/HIVE-1428?page=com.atlassian.jir a.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-1428:
Updated: (HIVE-1428) ALTER TABLE ADD PARTITION fails with a remote Thrift metastore
July 14, 2010 07:48:50 PM
[ https://issues.apache.org/jira/browse/HIVE-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1428:
Commented: (HIVE-1428) ALTER TABLE ADD PARTITION fails with a remote Thirft metastore
July 13, 2010 04:21:50 PM
[ https://issues.apache.org/jira/browse/HIVE-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888018#action_12888018 ] Pradeep Kamath commented on HIVE-1428:
Updated: (HIVE-1428) ALTER TABLE ADD PARTITION fails with a remote Thrift metastore
August 12, 2010 09:29:45 PM
[ https://issues.apache.org/jira/browse/HIVE-1428?page=com.atlassian.jir a.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-1428:
Updated: (HIVE-1428) ALTER TABLE ADD PARTITION fails with a remote Thrift metastore
July 13, 2010 04:27:52 PM
[ https://issues.apache.org/jira/browse/HIVE-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1428:
Commented: (HIVE-1428) ALTER TABLE ADD PARTITION fails with a remote Thirft metastore
July 13, 2010 04:27:51 PM
[ https://issues.apache.org/jira/browse/HIVE-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888020#action_12888020 ] John Sichi commented on HIVE-1428:
Updated: (HIVE-1428) ALTER TABLE ADD PARTITION fails with a remote Thrift metastore
August 13, 2010 08:08:41 PM
[ https://issues.apache.org/jira/browse/HIVE-1428?page=com.atlassian.jir a.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1428:
Reopened: (HIVE-1428) ALTER TABLE ADD PARTITION fails with a remote Thrift metastore
August 12, 2010 09:27:43 PM
[ https://issues.apache.org/jira/browse/HIVE-1428?page=com.atlassian.jir a.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach reopened HIVE-1428:
Commented: (HIVE-1428) ALTER TABLE ADD PARTITION fails with a remote Thrift metastore
August 13, 2010 02:33:41 PM
[ https://issues.apache.org/jira/browse/HIVE-1428?page=com.atlassian.jir a.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1 2898384#action_12898384 ] John Sichi commented on HIVE-1428:
hwo to connection metastore server with hive JDBC client
September 18, 2010 12:01:42 PM
I use ./hive --service metastore command to start metastore server, how to connection metastore server with hive JDBC client?
metastore with mysql
February 23, 2011 06:29:37 AM
Hello, I am running hive with a hadoop mini cluster and want to switch to mysql as a metastore because I need multiple connections at same time (not sure if this will work at all). I tried the local metastore config. The DB and two tables in mysql…
MySQL Metastore migration
January 5, 2011 04:32:05 PM
Hi all, We've been running hive in the default derby single user mode for a while. Now we've got more users interested in Hive and so would like to change the metastore to run off of mysql. I was able to find a pretty easy tutorial for doing…
Trouble using mysql metastore
March 2, 2011 08:16:13 AM
I've checked the mysql connection with a separate java file with the same string. Also, I've checked the code works by running it against the original derby metastore. Thanks, Ajo. Some of the variables set: javax.jdo.option.ConnectionURL =…
Re: MySQL Metastore migration
January 6, 2011 12:48:32 PM
Chris, I've recently converted a metastore from derby to MySQL. It wasn't pain free, but we made it through without major issues. Here is what I recommend you try; though I make no promises it'll work for anyone else: 1. Backup your metastore…
Mysql Connection Pool asks for closing connection?
July 23, 2010 03:51:53 AM
Hi I'm using glassfish server, and config a connection pool with MysqlConnectionPoolDataSource implementation. I find that when I close session, the connection is closed. (in log: java.sql.connection - xxx connection closed) I'm wondering if the…
connection timeout problem, javax.net.ssl.SSLHandshakeException: Remote host closed connection durin
October 1, 2010 12:25:35 AM
Hi, When I try to set the connection timeout in any of the following methods, I get javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake. If I dont set the connection timeout it works fine.Thanks in advance for…
don't use DBCP to connection mysql
March 13, 2011 09:44:47 PM
I know hive mestore use DBCP to connection mysql, I want to use short connection to connection mysql, how can I close DBCP configuration? Thanks, LiuLei**
Hadoop/hive metastore
August 7, 2009
where do people place their multi user meta store? I'm going to use mysql but I don't know were I should stick it. on the name node or on its own…
SSL Certificate Installation Issue with Remote Desktop Connection
April 8, 2011
Using Remote Desktop Connection to connect from Win7 to Win2008 Server R2, what is the difference in installing a self-signed certificate (for…
How to remote ftp connection from remote server via WinSCP or a windows program?
April 18, 2011
Hi, I have a new server and my hosting gives me a ftp account to backing up my real server. Problem is, i can connect to this ftp account from only…
Establishing a connection between R and a Hive (Hadoop) database
May 19, 2011
Does anyone know how to achieve that? I am assuming that RJDBC would help; but from my (likely naive) understanding, a bit of tweaking is necessary…
Mysql connection from remote PC
May 23, 2011
What options do I have to enable in the my.ini of I want to connect my database from a remote PC. This is the my.ini file content [client] port=3306…
Using Apache Hive as a MapReduce Input Format and/or Scraping Hive Metadata
April 14, 2011
Our environment is heavy into storing data in hive. I find myself currently working on something that it outside the scope though. I have a…
Remote mysql database connection
March 9, 2011
if php uses the following server name mysqlxx.90 and the username ouxxxxxxch to connect successfully to the local database and the server IP is…
Error on remote connection to mysql
January 17, 2011
Hi, I ve been trying to connect my dedicated server mysql db from my computers localhost my code is here; $dbhost = 'domain.com'; $dbuser =…
Hive, hadoop, and the mechanics behind hive.exec.reducers.max
February 17, 2011
In context of this other question here Using hive.exec.reducers.max directive has truely baffled me. From my perspective I thought hive worked on…
How does the MySQL connection on remote server works?
February 6, 2011
When I connect to my MySQL database locally I use: $host = 'localhost'; $db = 'myDB'; $user = 'root'; $pass = ''; When my site resides on remote…
Apache/php/mysqli connection to remote db fails but IIS and MySQL workbench connects
March 22, 2011
Hi, I am developing locally and connecting to a remote mysql server. I had been using IIS on my local machine (WinXP) but am now using Apache…
MySQL -- enable connection to remote server via local /tmp/mysql.sock
March 11, 2011
Hey all, I run a shared hosting provider and we're looking to move to a High Availability (replicated across multiple datacenters) setup for our…
MySQL connection issue when migrating from webserver to MAMP
January 10, 2011
OK so I've recently moved to MAMP for a project using PHP and mySQL. I've set up the database and copied all the data but now the script doesn't…
Windows 7 Port forwarding remote desktop issue
February 17, 2011
I have been banging my head against the wall for a long time now trying to get remote desktop working on my Win 7 machine. Here are the things I…
The remote host closed the connection.
March 28, 2011
We saw "The remote host closed the connection" in our error logs with: The error code is 0x80070040 The error code is 0x800703E3.(most of) We are…
When I quit remote SSH connection,tomcat will exit .So why?
January 29, 2011
When I quit remote SSH connection,tomcat will exit.So why? btw: No any errors in log files. Help me please!Thanks
Lithium Mongodb error on remote connection
April 8, 2011
I have the following issue with Lithium. When I connect to mongodb on my localhost all works well, but when I connect to my remote mongodb on EC2 it…
Eclipse:IWAV0171E Could not create a socket connection to remote vm.
March 4, 2011
Hi, I had my Eclipse nicely setup for writing simple Java desktop applications untill today when I tried to open a class in Visual Editor I`m…
Create a direct connection between client pc and remote frp server.
April 21, 2011
Good day to everyone.. I'm devoloping an Asp.Net Web Application in which I have to implement file upload, but with a little complication: To better…
Python Multiprocessing: socket error timeout while remote connection to a Manager
March 29, 2011
Hi all, newcomer and first ever question here. I am using the multiprocessing module of Python which is currently creating a Manager and a couple…