Best unofficial Apache Server developers community
Username
Forgot password?
Sign in with Twitter account
Sign in with Facebook account
List archives

elasticsearch sharding

where to find the Java Doc?
(10 lines)
Finding the min and max value for a float/double field
(20 lines)
Feb 14, 2011
Mbx
Mbx
Hi,
for my application i've tested Lucene, but quickly had problems with
10 million documents in one index on one server.
Query latency growed up to several seconds and for some keywords
result sets had been very large - up to out of memory.
Now i'm looking for a better solution.
During my research it seems that sharding is state of the art (Solr,
Elasticsearch) to work with large indices ( >20 Mio docs).

My Questions:
Why do i need sharding?
If i want to search over several shards (e.g. jan2010 upto dec2010)
i've to merge the results. Isn't it the same work than searching in a
large 2010-index?

How does sharding work for search? I think understood how the
documents are hashed and distributed to different shards but where are
they merged?

Sorry for the beginner questions.
Tank you!
-mbx


Reply
Tags: result setsserver query10 millionmemory
Messages in this thread
elasticsearch sharding
reply Re: elasticsearch sharding
(38 lines) Feb 14, 2011 12:51
reply Re: elasticsearch sharding
(47 lines) Feb 15, 2011 20:03
Search the http://elasticsearch.org website with elasticsearch itself
February 11, 2011 08:53:09 AM
Hi, I've spent couple of hours during the last days with implementing an ElasticSearch-backed search for the ES website (so it could be “self-hosted”, so to speak), as we have been talking about it on IRC the other day. First, you can try the…
Sharding setup and strategy for sharding key in a collection
November 10, 2010 02:07:41 PM
Hi all, A few questions about sharding setup and assigning sharding keys within a collection. 1. We have 2 shared machines allocated for mongoDB ( Shared meaning, other team members may use those servers for running hadoop or some ad-hoc…
From Compass to ElasticSearch
January 11, 2011 09:19:18 AM
(I hope this post didn't sent twice..) Hello, I would like to migrate an app I wrote using Compass to work with ElasticSearch (the Java API). I did some searches and experiments, but failed to find the answers, so can you point me to relevant…
Anyone using ElasticSearch with Mongodb?
November 28, 2010 04:49:53 AM
I would be very interested to hear the paths that you guys have chosen for full text search on Mongodb documents. Anyone using ElasticSearch for the same? If yes can you throw some light on your experience as well as how you have done it. Would…
ElasticSearch and GeoIpSearch
October 26, 2010 11:03:07 AM
Oh... hey there ElasticSearchers :) Ok, so I'm thinking about using ES for looking up geo info about IP addresses. And I'm just wondering if someone has had some experience they would like to share. Or, if you have some insights on how this…
ElasticSearch 0.14.2 Released
January 5, 2011 03:51:05 PM
Hi, 0.14.2 is out, fixing a major bug in highlighting: https://github.com/elasticsearch/elasticsearch/issues/closed#issue/600 . There are also bug fixes and minor features added to this release, all found here:…
ElasticSearch 0.13.1 Released
December 3, 2010 01:46:16 PM
Hi, ElasticSearch 0.13.1 released with one major change, upgrading the Lucene version from 3.0.2 to 3.0.3. The new Lucene version fixes two important bugs, the first is a memory leak that happens when running a long indexing session, and the…
ElasticSearch Geo and rabbitMQ
December 30, 2010 08:47:03 AM
hi all, i'm new to elasticsearch and i'm looking for some directions. i would like to integrate geo searches into rabbitmq and since elasticsearch has integration for both i thought it could be the solution i am looking for. although there are some…
ElasticSearch 0.14.1 Released
December 29, 2010 06:12:05 AM
Hi, Just released a bug fix for 0.14.0 version that was just released. It fixed a major regression in the REST create index API. Details of the regression are here: https://github.com/elasticsearch/elasticsearch/issues/closed#issue/578 . sorry…
ElasticSearch 0.11.0 Released
September 28, 2010 03:44:32 PM
Hi, 0.11 is out, more details here: http://www.elasticsearch.com/blog/2010/09/29/0.11.0-released.html. -shay.banon
ElasticSearch 0.14 Released
December 28, 2010 04:00:15 AM
Hi, ElasticSearch 0.14 released, see more here: http://www.elasticsearch.com/blog/2010/12/27/0.14.0-released.html. cheers, -shay.banon
ElasticSearch.pm v 0.23 is out - big performance boost
October 19, 2010 05:12:17 PM
Hi all ElasticSearch.pm (the Perl API to ElasticSearch) is out here: http://search.cpan.org/~drtech/ElasticSearch-0.23/lib/ElasticSearch.p m It comes with a large performance boost, from bulk() indexing, and a different backend. You can read…
elasticsearch-head – new GUI up on GitHub
January 16, 2011 10:03:42 PM
Hi, just thought I'd pass on some info, one our our intrepid UI engineers at Aconex is building for us a useful in house tool to combine with our upcoming use of Flume->ElasticSearch, and we've licensed it under ASL 2. …
Elastica - PHP client for elasticsearch
October 20, 2010 06:49:41 AM
Hi I published my PHP client for elasticsearch named Elastica to github. The client is still in development but it I'm already using it in two installations. The difference to the PHP client from nervetattoo is that it integrates easily with Zend…
Few Querys related to ElasticSearch
November 29, 2010 05:08:48 AM
Hi Would appreciate if any of you can share your experience / thoughts on below questions: 1. REST Api vs Java api - Have read that Java api is much faster as it works at a lower level protocol. Do you guys have any comparison? 2. What approach do…
Hosting and securing ElasticSearch.
January 31, 2011 11:59:04 AM
This is a multi-part message in MIME format. I've been looking at ElasticSearch and I'm a huge fan of what I'm seeing so far. I'm currently using Lucene.NET on my site (sf4answers.com) to perform searching, but I feel that I'd…
org.elasticsearch.action.UnavailableShardsException
January 7, 2011 12:59:43 AM
Hi, Before doing any configuration my example was eorking well. I configured my elasticsearch.yml as follows: node: data: true index : number_of_shards : 3 number_of_replicas : 2 I am getting the following error: Exception in thread…
pyes - Python ElasticSearch 0.12.1 Released
October 20, 2010 03:03:39 AM
Hi, The new release target elasticsearch 0.12 or above. pyes is a connector to use elasticsearch from python. Web: http://pypi.python.org/pypi/pyes/ Source: http://github.com/aparo/pyes/ Features: - Thrift/HTTP protocols - Bulk…
Prototype use of ElasticSearch Twitter River
November 10, 2010 08:59:49 AM
Mozilla Metrics has created a five node ElasticSearch cluster on some test machines that is using the Twitter River functionality to automatically retrieve a filtered set of documents from Twitter's streaming API and index them. I've just created…
CouchDB ElasticSearch Integration Howto
October 15, 2010 12:03:09 AM
Hi, ElasticSearch has added a feature which directly indexes couchdb documents. ElasticSearch listens on the CouchDB _changes interface and indexes the docs. This feature has undergone some changes in the current development series. So,…
How to Verify Sharding?
January 6, 2011
HI, Hey, Buddy.I have clearly mentioned that i have done with the sharding configuration and i have configured some ordinary backups in-order to…
SHARDING CONFIGURATION HELP...........
January 10, 2011
Hi, I am new to mongo-db,I tried to configure sharding by following the procedure in the mongo-db web-site..But i am getting some errors...... I…
Sharding Problem........
January 12, 2011
Hi, I am new to mongodb, I am undergoing testing based on sharding in single machine...I have two shards running on the machine...One config server…
Multiple tables sharding with mysql
January 15, 2011
Hi, I'm making a GPS app that will deals with 200 millions records in a table. My initial thought is to divide the table into multiple tables like…
Mongodb:Can sharding improve query performance?
December 30, 2010
Hi I have a table with very large data,can the sharding improve the query performance?
MongoDB sharding using hashed based keys
January 3, 2011
I'm using a simple hashing algorithm that generates short hashes that I'm planning to use as IDs instead of the UUID generated by MongoDB. The…
Clustering, Sharding or simple Partition / Replication
January 4, 2011
Hello Everyone! Happy new year! I need some advice from you experts on this subject. The thing is that we have created a facebook application some…
For 32-bit sharding whats the size limit of each shard?
January 31, 2011
Hi,I am using 32-bit mongodb for sharding.I configured two shards on two 32-bit machines.Each machine has one shard...The max size of all shards is…