Best unofficial Apache Server developers community
Username
Forgot password?
Sign in with Twitter account
Sign in with Facebook account
List archives

random search errors from one node of three on a previously working cluster

Error in script
(45 lines)
Re: Error in script
(41 lines)
Jul 6, 2011
Ian
Ian
Hi,

I'm getting some random errors that started happening today. This is
one error:-

{"error":"ReduceSearchPhaseException[Failed to execute phase [fetch],
[reduce] ]; nested: ArrayIndexOutOfBoundsException[1]; ","status":500}

and this is another:-

{"error":"ReduceSearchPhaseException[Failed to execute phase [fetch],
[reduce] ; shardFailures {SearchContextMissingException[No search
context found for id [36813]]}{RemoteTransportException[[Aardwolf]
[inet[/10.206.38.97:9300]][search/phase/fetch/id]]; nested:
SearchContextMissingException[No search context found for id
[55197]]; }]; nested: IndexOutOfBoundsException[index (0) must be less
than size (0)]; ","status":500}

I am load testing my cluster by sending pseudo-random generated search
requests. I can take the search requests that fail, and replay them
manually and they work. Strange, as this test was working for the past
few days....  I have found exceptions in my logs on only one node, I
have copied them here:-

https://gist.github.com/1066838

Any ideas what could be going wrong? Shall I raise a bug?

cheers

Ian


Reply
Tags: random errorserror error
cluster throwing errors when new or existing node joins
May 31, 2010 08:21:58 AM
Hi I have a setup of 4 nodes, whenever I am restarting any of the nodes, even after deleting the data directories and commit log I get the following error ERROR 18:46:41,296 Fatal exception in thread Thread[COMMIT-LOG-WRITER,5,main]…
Re: LDAP node search stopped working after Puppetmaster upgrade
June 30, 2010 09:13:59 AM
On Jun 4, 2010, at 12:41 AM, jc.listmail wrote: > I recently upgrade from 0.24 to 0.25. In my 0.24 config, I had LDAP > working such that Puppet was matching nodes in my directory to > determine what classes to apply. I upgraded to 0.25…
Problems manually copying a set of indices from a 3-node cluster to a 1-node cluster
June 15, 2011 07:52:17 AM
Previous posts have suggested that it's possible to backup/migrate/ restore indexes from one cluster to another via manual copy. I tested this yesterday and ran into some problems. Setup: My test/reference cluster ("infinite-aws") consists of 3…
How report errors and random errors in clients
August 30, 2010 06:07:51 AM
Hello, I am using puppet 2.6.1rc3 in a test environment with AIX, Suse, Debians. Right now there are 5 clients... I am running puppet from cron each 30m (using random minute per host). I have prepared a configuration ready to deploy puppet in…
Updated] (AMQ-2205) AMQ 5.2 SSL implementation broke SSL Broker network previously working on AMQ 5.
April 1, 2011 06:28:03 AM
[ https://issues.apache.org/jira/browse/AMQ-2205?page=com.atlassian.jira .plugin.system.issuetabpanels:all-tabpanel ] Gary Tully updated AMQ-2205:
Updated: (AMQ-2205) AMQ 5.2 SSL implementation broke SSL Broker network previously working on AMQ 5.
December 2, 2010 07:13:40 AM
[ https://issues.apache.org/jira/browse/AMQ-2205?page=com.atlassian.jira .plugin.system.issuetabpanels:all-tabpanel ] Dejan Bosanac updated AMQ-2205:
Updated: (AMQ-2205) AMQ 5.2 SSL implementation broke SSL Broker network previously working on AMQ 5.
September 18, 2010 11:48:04 AM
[ https://issues.apache.org/activemq/browse/AMQ-2205?page=com.atlassian. jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Snyder updated AMQ-2205:
copy data from multi-node cluster to single node
July 4, 2011 10:06:06 AM
Hi, I am using Cassandra 0.7.5 on Linux machines. I am trying to backup data from a multi-node cluster (3 nodes) and restore it into a single node cluster that has a different name (for development testing). The multi-node cluster is backed up…
cluster size, several cluster on one node for multi-tenancy
February 16, 2011 10:43:58 AM
Hi, We are interested in a multi-tenancy environment, that may consist of up to hundreds of data centers. The current design requires cross rack and cross DC replication. Specifically, the per-tenant CFs will be replicated 6 times: in three racks, …
Random errors in AIX-Zlinux clients
September 2, 2010 03:29:41 AM
>>>> The errors appear once or two times, randomly, and if I execute the >>>> client again it works properly. They are usually these kind of errors: >>> You will probably find life improved by deploying…
Node xxx not part of the cluster Cluster , ignoring...
June 1, 2011 02:10:18 AM
I started two elasticsearch nodes on a remote machine with ip: 10.18.102.134 Then i connect it using TransportClient(java) from ip:10.18.106.36, then the console begins to print warnings as below: [Payne, Frank] Node…
Expanding single node to 2 node cluster
April 27, 2011 02:08:32 AM
Hi, I had a 2 node cassandra cluster with replication factor 2 and OrderPreservingPartitioner but we did not provide InitialToken in the configuration files. One of the node was affected in the recent AWS EBS outage and had been partitioned from…
server errored on one node of a 8-node cluster
January 31, 2011 04:51:07 PM
I have an 8-node Voldemort cluster in production. It seems to run smoothly. How can I find out if the server shuts down on one of these nodes? 1. Is there any recommended tool ? I need this urgently. 2. Can Voldemort publish its metrics to Ganglia…
Cassandra node is not blanced Rf=2 Random Partitioner
May 13, 2011 01:47:17 AM
My cluster is unbalanced.One have 99 GB Data and other have 87 GB can any one explain why this is happening. [roo### @cassandra2 conf]# /root/cassandra/bin/nodetool -h 10.0.0.4 ring Address Status Load Range …
Created: (JCR-2857) Support sequential (non-random) node ids
January 4, 2011 09:45:31 AM
Support sequential (non-random) node ids
Taking down a node in a 3-node cluster, RF=2
November 28, 2010 08:32:54 AM
For the vast majority of my data usage eventual consistency is fine (i.e. CL=ONE) but I have a small amount of critical data for which I read and write using CL=QUORUM. If I have a cluster with 3 nodes and RF=2, and CL=QUORUM does that mean that a…
How to add a new node to cluster?
March 5, 2011 06:14:44 AM
Hi,everyone I am using apache version of hadoop0.20.2 to do my job. now i want to add a new machine to my cluster as TaskTracker and DataNode. How can i do that without restart the whole hadoop system? or just run some commands on the new machine?…
100-node EC2 cluster
April 4, 2011 08:40:02 AM
I've written a blog post about my experience launching a 100-node Riak cluster on EC2 and I thought it might be of interest to the list. All feedback welcome. http://reiddraper.com/100-node-riak-cluster/ Reid Draper
how far can I go with a 1 node cluster
February 13, 2011 06:59:08 PM
Hello, So all my statistics is finally being calculated, results being processed etc, i have a 1 node cluster. Mainly taking 3 aggreate logs from my apache logs. How far this setup will go? I have another machine ready to be hooked up to my…
Zookeeper cluster not working
March 16, 2011 09:02:30 AM
Hi All, I am using zookeeper for my one of the projects. I encountered zookeeper cluster not working in the following scenario, Our zookeeper cluster has size 3 say z1,z2, and z3 1) Start the zookeeper cluster. 2) Make client connections…
Mysql ndb cluster - node restart.
December 29, 2010
Hi guys! I just setup a mysql cluster on a fairly decent baby (IBM x3650 M3) with 24GB memory, xeon 6core, SAS 6Gbps HDD. Running Debian Lenny 5.…
How much data per node in Cassandra cluster?
January 23, 2011
Hi, Where are the boundaries of SSTables compaction (major and minor) and when it becomes ineffective? If I have major compaction couple of 500G…
Hadoop multi node cluster installation in 2 machies
April 19, 2011
hello in multinode cluster installation.. when the command start-all.sh.. that it when it try to start the various nodes. it dispalys permission…
Hadoop multi node cluster installation in 2 machies
April 19, 2011
hello scp error: "Permission denied (publickey). lost connection"
Running the Sort example on Hadoop (single-node cluster)
March 31, 2011
Hello, I have installed Hadoop single-node cluster 0.20.2 on Ubuntu 10.04 and run an example using the material of the tutorial I found on this…
Hadoop running sort example on single-node cluster
April 5, 2011
Hello, I am trying to run sort example on Hadoop single-node cluster. First of all, I start the deamons: hadoop@ubuntu:/home/user/hadoop$…
MySQL InnoDB random data corruptions: hardware or software errors?
May 24, 2011
Hi! I'm an administrator of a social game which uses MySQL(Percona 5.1.56 to be precise) for data storage(all tables have InnoDB type). There are…
How to connect my Java based Cassandra client to a Cassandra cluster, rather than a specific node?
May 25, 2011
Greetings, I am trying to learn Cassandra and have setup a 4 node Cassandra cluster. Ihave written a client in Java using Hector, which currently…
How to search the random words from a phrase using PHP?
May 2, 2011
My orginal Pname in the table 'english' is "The Digital Santa Monica Mug".If users try to search using "Digital Mug", its not returning productwith…
SQLite Random not working in browser.
February 2, 2011
Hey everyone, I'm using jQuery and HTML to write a web app. Currently all it does is create a database, add data to the database (checked and…
Use REGEXP in MySQL to match keywords for search engine in random order
April 17, 2011
Hi, I'm trying to use a regular expression to match a user entered search string to a title of an entry in my MySQL database. For example I have the…
Mongoose node.js, query with $lt and $gt not working
April 11, 2011
I want to get all the pupils whose last mark is between 15 and 20. To do so, I perform the following query in my mongoDB using mongoose: The models…
Rabbitmq 2.2 not working on Snow Leopard (SSL errors)
January 7, 2011
When I start rabbitmq, I see the following message. I have Activating RabbitMQ plugins ... WARNING Undefined function crypto:des3_cbc_decrypt/5…
Converted from HTTP to HTTPS... errors & log in forms not working - PHP?
March 8, 2011
We have converted our site from HTTP to HTTPS, and when our login forms are being submitted, we are getting the message: "Although this page is…
Incrementing Object not working ( mongoose, node.js, express, mongodb )
May 14, 2011
I have a commit of this app up here : https://github.com/varunsrin/Simple-Redirect The basic problem is, in server.js @ line 234 I have a method…
MAMP Apache server suddenly stopped working - no errors in the logs
November 27, 2010
It worked fine and then it just stopped working. version 1.9.4
MySQL: How to retrieve a random row or multiple random rows?
January 3, 2011
I have a MySQL database table that stores the URLs of photos. I need to pull 5 random records from the database of a particular type. I can pull 5…
PostgreSQL - use previously computed value from same query
February 21, 2011
I have the following sql query: SELECT (SELECT ...) AS X, (SELECT ...) AS Y from my_table 'X' is quite hard to compute, and it's used as an input…
Solr - Search Not Working
March 22, 2011
I am trying to run solr, but when i search for something, it wont show any result and just the name of the searched text. I have enclosed my…
How to understand if a row has been queried by a select query previously?
April 22, 2011
At my application there is a table and I am inserting new rows that table. There is a Java class that gets a row(by a select query) from that table…