Practical Limits of ElasticSearch + Cassandra

⇧

⇩

47 views

I am planing on using ElasticSearch to index my Cassandra database. I am wondering if anyone has seen the practical limits of ElasticSearch. Do things get slow in the petabyte range? Also, has anyone has any problems using ElasticSearch to index Cassandra?

cassandra limits practical elastic-search

asked June 15, 2011 9:31 am CDT

user578730

posted via StackOverflow

1 Answers

⇧

⇩

See this thread, which mentions ElasticSearch configurations with 1700 shards each of 200GB, which would be in the 1/3 petabyte range. I would expect that the architecture of ElasticSearch would support almost limitless horizontal scalability, because each shard index works separately from all other shards.

The practical limits (which would apply to any other solution as well) include the time needed to actually load that much data in the first place. Managing a Cassandra cluster (or any other distributed datastore) of that size will also involve significant workload just for maintenance, load balancing etc.

answered June 22, 2011 12:56 pm CDT

DNA

Your answer

Similar discussion threads

Hard and practical limits for # of files in a hadoop directory.
May 19, 2011 04:51:32 PM
Hi, I have a question about practical limit on number of files per hdfs directory. (what's the hard limit btw?) What is a practical limit on a # of files in a hadoop directory so that glob selection still works efficiently (by efficiently i mean…

Re: ElasticSearch + Cassandra
April 6, 2011 03:41:24 PM
On 6 Apr., 14:21, Shay Banon <shay.b### @elasticsearch.com> wrote: > Yes, agreed, Solandra and Lucandra are pretty cool in terms of concepts, but I think they lack when it comes to how they use Lucene (putting the solr aspect aside).…

ElasticSearch + Cassandra
April 4, 2011 12:13:06 PM
Hi Everyone, Has anyone done/attempted to integrate ElasticSearch with Cassandra like was described at http://www.elasticsearch.org/blog/2010/02/25/nosql_yessearch.html ? I don't mean using Cassandra as a Gateway for ES, but using ES to query the…

Created: (AMQ-2953) Disk limits not observed when memory limits exceeded for non-persistent messagin
September 30, 2010 03:23:27 PM
Disk limits not observed when memory limits exceeded for non-persistent messaging

Search the http://elasticsearch.org website with elasticsearch itself
February 11, 2011 08:53:09 AM
Hi, I've spent couple of hours during the last days with implementing an ElasticSearch-backed search for the ES website (so it could be “self-hosted”, so to speak), as we have been talking about it on IRC the other day. First, you can try the…

Re: mod_dav - practical use
October 10, 2010 01:59:02 AM
>> It's not so much a trick.. You reverse-proxy DAV (write) requests to a >> back-end which is running on an unprivileged port, as an unprivileged >> user, who has the permission to do writes on the FS. ahh, sure... but i would…

A practical suggestion.
June 18, 2010 01:25:12 PM
Let me start out by saying that this maybe more of a Dev issue, then a users issue. However, given that to do things like change the location of the repository, you need to rebuild Sling, I think there's a reasonable case to be made for posting…

Re: mod_dav - practical use
October 12, 2010 03:12:31 AM
Hello, >> http://wiki.apache.org/httpd/ExtendingPrivilegeSeparation ahh,thanks a lot for your help. now i can go on... Thanks, Hajo

Re: mod_dav - practical use
October 10, 2010 01:16:57 PM
Thanks for your help. >> The obvious answer, of course, is to run httpd ;) may be i misunderstood something... at first line is apache which redirects dav-requests coming on special port or alias by reverse-proxy to a backend which is able…

mod_dav - practical use
October 8, 2010 08:35:36 AM
Hello List, a question to mod_dav. Some providers offer mod_dav to edit files which are also editable/writeable by ftp-user? In most cases ftp-users/apacheuser are different to avoid security problems. Whats the trick to make this possible…

Regarding Sharding Limits
April 11, 2011 04:51:52 PM
Hey guys, I'm reading this http://www.mongodb.org/display/DOCS/Sharding+Limits and the third point is: "All (non-multi)updates, upserts, and inserts must include the current shard key. This may cause issues for anyone using a mapping library…

Re: limits with lists
January 10, 2011 03:23:43 AM
How about something like this: var row, limit = 10; while(row = getRow() && limit-- != 0) { send(row.value); } Miche On Sun, Jan 9, 2011 at 9:11 PM, Chris Johnson <csj7### @yahoo.com> wrote: > I have a large…

Re: limits with lists
January 10, 2011 03:38:26 AM
surely you can pass ?limit= through the list function to the view call? list is 'just' for transforming the output of a view call to another form. B. On Mon, Jan 10, 2011 at 9:22 AM, Michel Legnered <michel.l### @gmail.com> wrote: >…

Limits on HBase
September 6, 2010 03:57:56 PM
Hi folks, I know this question may have been asked many times, but I am wondering if there is any update on the optimized cell size (in megabytes) and row size (in megabytes)? Many thanks. William

Cache limits
January 25, 2011 04:43:17 AM
Hello, currently I try to understand the Cayenne design and implementation. Can you explain, how you limit the number of objects, which are inside the cache? On an application with a huge number of data rows, the most important point is the…

Link limits, and more...
September 19, 2010 10:56:37 PM
Hi, I was wondering if there is a practical limit to the number of links between objects - for example, if a User object has a 'friend' link to many other User objects. Are these links indexed somehow within Riak to speed up any queries on these…

Re: limits with lists
January 10, 2011 08:32:08 AM
If I use the limit value such as .../_list/sort/view?limit=10 the limit is applied to the view, not the list. what I am wanting to do is pass the full view to the list and then limit the output of the list. Right now I am accomplishing it via a…

Query limits
January 27, 2011 11:49:18 AM
Hi list, we are planning to make queries with a lot of property tests and boolean operators. Is there a theoretical and practical limit to the size of the request? Could there be some serious slow-down if the query is too big? I'm currently using…

Any experiences with gridfs limits ?
December 17, 2010 06:55:50 AM
Hi everyone, Well, we (the company were i work) are actually using mongodb for several purposes... and feel very happy with it. One if these purposes is using mongodb as cdn storage for product images on a Dual intel xeon 2.6ghz box, with 12gb…

Sharding scale limits ?
August 26, 2010 12:09:13 AM
Hi there, According to this document http://www.mongodb.org/display/DOCS/Sharding+Limits: " Goal is support of systems of up to 1,000 shards. Testing so far has been limited to clusters with a modest number of shards (e.g., 20). More…

Questions and Answers are licensed under cc-wiki license.

Practical Limits of ElasticSearch + Cassandra

1 Answers

Your answer

Join with account you already have