Scanning through 12k rows, performance tuning suggestions?

Weijunli,

I also have an environment that has similar very large datasets with
strict
latency.  Can you please elaborate on the custom changes you added to
cassandra to meet these sla, either code or configuration.  i am very
interested in learning more about the internal workings of cassandra and
performance.

Thanx,
Artie

On Thu, May 6, 2010 at 10:06 AM, Weijun Li <weiju### @gmail.com>
wrote:

 Our use case is a little different: our server is a typical high
volume
 transaction server that processes more than half billion requests per
day.
 The write/read ratio is close to 1, and the cluster needs to serve
10k
 write+read with strict latency (<20ms) otherwise the client will
treat it as
 failure. Plus we have hundreds of millions of keys so the generated
sstable
 files are much bigger that the ram size. In this case using mmap will
cause
 Cassandra to use sometimes > 100G virtual memory which is much
more than the
 physical ram, since we are using random partitioner the OS will be
busy
 doing swap.

 I have finally customized cassandra to meet the above requirement by
using
 cheap hardware (32G ram + SATA drives): one thing I learned is that
you have
 to carefully avoid swapping, especially when you need to cache most
of the
 keys in memory, swap can easily damage the performance of your in
memory
 cache. I also made some optimizations to reduce memory/disk
consumption and
 to make it easier for us to diagnose issues. In one word: cassandra
is very
 well written but there's still huge potential for you to improve it
to meet
 your special requirements.

 -Weijun


 On Wed, May 5, 2010 at 9:43 AM, Jordan Pittier
<jordan.p### @gmail.com>wrote:

> I disagree. Swapping could be avoided. I don't know Cassandra
internals
> mechanisms but what I am expecting is that whenever I want to
read rows that
> are not in RAM, Cassandra load them from hard drive to RAM if
space is
> available, and, if RAM is full to reply my query without saving
rows in RAM.
> No need for swapping.
>
> I have no try yet to change DiskAccessMode to standard, I hope it
will
> help me.
>
> Another thing : please dont post your benchmark figures without
any
> explanation on the work load generator or your cluster settings.
It really
> doesn't make any sense...
>
>
> On Wed, May 5, 2010 at 6:16 PM, Weijun Li
<weij### @gmail.com> wrote:
>
>> When you have much more data than you can hold in memory, it
will be
>> difficult for you to get around of swap which will most
likely ruin your
>> performance. Also in this case mmap doesn't seem to make much
sense if you
>> use random partitioner which will end up with crazy swap too.
However we
>> found a way to get around read/write performance issue by
integrating
>> memcached into Cassandra: in this case you need to ask
memcached to disable
>> disk swap so you can achieve move than 10k read+write with
milli-second
>> level of latency. Actually this is the only way that we
figured out that can
>> gracefully solve the performance and memory issue.
>>
>> -Weijun
>>
>>
>> On Wed, May 5, 2010 at 8:19 AM, Ran Tavory
<rant### @gmail.com> wrote:
>>
>>> I'm still trying to figure out where my slowness is
coming from...
>>> By now I'm pretty sure it's the reads are slow, but not
sure how to
>>> improve them.
>>>
>>> I'm looking at cfstats. Can you say if there are better
configuration
>>> options? So far I've used all default settings, except
for:
>>>
>>>     <Keyspace Name="outbrain_kvdb">
>>>       <ColumnFamily CompareWith="BytesType"
Name="KvImpressions"
>>> KeysCached="50%"/>
>>>
>>> 
<ReplicaPlacementStrategy>org.apache.cassandra.locator.RackAwareStrategy</ReplicaPlacementStrategy>
>>>      
<ReplicationFactor>2</ReplicationFactor>
>>>
>>> 
<EndPointSnitch>org.apache.cassandra.locator.EndPointSnitch</EndPointSnitch>
>>>     </Keyspace>
>>>
>>>
>>> What does a good read latency look like? I was expecting
10ms, however
>>> so far it seems that my KvImpressions read latency is
30ms and in the system
>>> keyspace I have 800ms :(
>>> I thought adding KeysCached="50%" would improve my
situation but
>>> unfortunately looks like the hitrate is about 0. I
realize that's
>>> application specific, but maybe there are other magic
bullets...
>>>
>>> Is there something like adding cache to the system
keyspace? 800 ms is
>>> pretty bad, isn't it?
>>>
>>> See stats below and thanks.
>>>
>>>
>>> Keyspace: outbrain_kvdb
>>>         Read Count: 651668
>>>         Read Latency: 34.18622328547666 ms.
>>>         Write Count: 655542
>>>         Write Latency: 0.041145092152752985 ms.
>>>         Pending Tasks: 0
>>>                 Column Family: KvImpressions
>>>                 SSTable count: 13
>>>                 Space used (live): 23304548897
>>>                 Space used (total): 23304548897
>>>                 Memtable Columns Count: 895
>>>                 Memtable Data Size: 2108990
>>>                 Memtable Switch Count: 8
>>>                 Read Count: 468083
>>>                 Read Latency: 151.603 ms.
>>>                 Write Count: 552566
>>>                 Write Latency: 0.023 ms.
>>>                 Pending Tasks: 0
>>>                 Key cache capacity: 17398656
>>>                 Key cache size: 567967
>>>                 Key cache hit rate: 0.0
>>>                 Row cache: disabled
>>>                 Compacted row minimum size: 269
>>>                 Compacted row maximum size: 54501
>>>                 Compacted row mean size: 933
>>> ...
>>> 

1. Is there a difference in performance when you have keys that are int
versus string/datetime?

2. When designing for multi-tenancy, what suggestions do you guys have?
    My idea thus far would be do prefix keys with the website like:
      com.example.keyname

  Seeing as keyspaces can't be built dynamically, this is probably the
only
approach correct?

     [
https://issues.apache.org/jira/browse...nels:all-tabpanel
]

Carl Steinbach updated HIVE-256:

We have an existing product sitting on the hbase/hadoop ecosystem. We have
laid our object model on HBase: we have one table with a row per object,
and a separate table with composite index rows. Works great. We can
efficiently find our objects based on their type, relationships, etc. by
scanning the index table. We *never* scan the main table (except when
rebuilding the index).

 

A new requirement just came in: get a list of all objects that have been
modified since <timestamp>. This has to happen "quickly" (user
time).

 

If we scan the main table with a timestamp restriction, will that be
efficient? Or do we have to introduce a new composite index that has the
last modified timestamp as part of it and scan that? 

 

Thanks,

Mark

 

Hi Gurus,

Any advise on initial tuning values for apache MinSpareServers,
MaxSpareServers, and StartServers, tcp tunings, ulimits, sysctl....

I'm running svn 1.6 over apache2 pre-fork. System load goes high as
much as 10 during heavy usage.

What would be your recommended MaxRequestsPerChild for heavy svn usage?

Thanks,


West

[Rumen] TraceBuilder should provide recursive input folder scanning

Hi,

I'm using Tomcat 6.0.26, Java 1.6 and wondering what tools/strategies you
use to tune your garbage collection parameters?

Further, does anyone know how to read entries in the garbage collection
log? 
Entries in my log look like

Desired survivor size 10944512 bytes, new threshold 1 (max 15)
 [PSYoungGen: 129311K->3232K(136512K)] 558882K->434085K(585920K),
0.0090900
secs]

Thanks, - Dave

Hi All,

I noticed using Firefox with a plugin yslow that it recommends using only
4 virtual hosts instead of the 16 I have for serving images. I am using a 6
core system and want to make sure I take advantage of it using Apache APR
so I setup more than 4 virtual hosts.

I am using multiple virtual hosts for serving images since I heard that
can improve user response time where they have browsers with 4 or more
concurrent connections supported. I have a screenshots page where I have 24
thumbnails being loaded into the browser and I am trying to tune APR for
the best user response time.

So any best practice for this you all would recommend?

Thanks,
-Tony


      

Hi,

asking the people who probably use WebDAV clients all the time: any
suggestions for a good WebDAV client? (both Windows and Linux apply)

I'm downloading and trying clients myself of course, but not knowing yet
what to expect I also don't know yet what to look for.

Cheers,
-Tako

Hi all,

I've made a few improvements to the CLI (specifically support for add,
rename, and drop for keyspaces and column families) and as soon as
someone commits them, I'd like to make a few more.

So:

1. Can I get a code review for this:
https://issues.apache.org/jira/browse...nels:all-tabpanel

2. Does anyone have suggestions for more CLI improvements while I'm in
there?

Thanks!

--mason

I'm trying to figure out the best way to handle publishing artifacts to our
Ivy repository using ssh.  We can't prompt the user for the username and
password since the publication is usually done by Hudson.  We can't embed
the username or password as a job configuration property because we can't
have those in cleartext; similarly, we can't use a standard user with a
well-known password in cleartext because of security concerns.

I'm leaning towards using a keystore, but we'd need to use one without a
password for the same reasons above (can't prompt, don't want to embed),
but
a keystore without a password makes the security group twitchy.

I'm looking for any ideas or suggestions that might help; practical
experience with real examples would be best, but I'll consider anything.

Thanks,
Rich

Hello,

Although,  I've done a bit of reading about Cassandra's data model and
I've set up a Cassandra pair, I'm still unsure as to what might be
best for my purposes.

Briefly, I've got a set of strings A, B, and C. If needed, A could be
represented as an integer. Each A is associated with exactly one B or
C (but not both). A also has a number of parameters associated with it
which change over time. These changes, however, are reported with the
B or C identifier.

Currently, in mysql, I have three tables, A', B', and C' each using A,
B, and C as keys. When an update arrives, the code searches for the
key (B or C) in the appropriate table, determines the associated A
(using a foreign key), and then updates the values in the table row in
A' with key=A.

Anyone have ideas about how to model this in Cassandra?

Thanks!

Hello,

I'm going to be diving into buildr this week and once I play around with
it and get a feel for what it can do, I'm going to write up a blog entry on
it. What I want to do is help people see alternatives to Ant and Maven.
When I first saw Ant, I thought, interesting a nice step forward and two
backwards. Then came Maven and I thought WTF. And then came Maven2 and I
thought OMG WTF. And now Maven3 is upon us.

I'm going to port an open source java project from maven to buildr and
then blog about the effort, results, and long-term benefits.

Are there any particular java projects you would suggest as a candidate
for this build system port?

Regards -- Kramer
http://www.qbalsoftware.com

I am trying to use the ActiveMQ and Camel (running in broker) on .NET
using
Spring and NMS. In general, these components work brilliantly. I have a
camel route for send/receive, and some code in .NET to handle this - works
like a charm.

The issue I am trying to resolve is for a request-reply integration
pattern.
Camel support this, and I have this route in my config:

  <route>
    <from uri="activemq:foo"/>
    <to uri="activemq:bar" pattern="InOut"/>      
  </route>

In theory, Camel is supposed to set the NMSReplyTo and CorrelationID in
such
a route so the reply knows where to go back to.

I followed 
http://remark.wordpress.com/articles/...activemq-and-nms/
this VERY helpful article  to understand how to wire things together in
.NET. In the article, the code explicitly sets the location of the reply,
since it is not using Camel.

My question is how do I take advantage of the Camel route, so I don't have
to essentially duplicate it in code (which would defeat the purpose of a
config based integration pattern). I tried removing the NMSReplyTo and
CorrelationID in my code, but the reply never gets back to the sender.
It's
almost like the Camel route is not 'internalized' by the application.

Thoughts? Suggestions?

Thanks, Chet 

I wrote a code that iterate on all the rows by using get_range_slices.
for the first call I use KeyRange from "" to "".
for all the others I use from <the last key that I got in the previous
iteration> to "".
I always get the same rows that I got in the previous iteration. I tried
changing the batch size but I still gets the same results.
I tried it both in single node and a cluster.
I use RP with version 0.6.3 and Hector.

Does anyone know how this can be done?

Shimi

This is a multi-part message in MIME format.
Is there any way at all (In Java) to get all the data from a
ColumnFamily?

 

I've inserted data into Cassandra and I don't seem to have a way to
browse what's in there.

 

Anthony Ikeda

Java Analyst/Programmer

Cardlink Services Limited

Level 4, 3 Rider Boulevard

Rhodes NSW 2138

 

Web: www.cardlink.com.au | Tel: + 61 2 9646 9221 | Fax: + 61 2 9646 9283

 

 


**********************************************************************
This e-mail message and any attachments are intended only for the use of
the addressee(s) named above and may contain information that is privileged
and confidential. If you are not the intended recipient, any display,
dissemination, distribution, or copying is strictly prohibited.   If you
believe you have received this e-mail message in error, please immediately
notify the sender by replying to this e-mail message or by telephone to
(02) 9646 9222. Please delete the email and any attachments and do not
retain the email or any attachments in any form.
**********************************************************************

Reading URLs from CSV file.
Doing http request.
Summary report shows only HTTP Request and TOTAL.

I want to see lines for each URL. is it possible ?

sincerely,
Martin

Hi,

I am trying to limit the result set with a 'select top <value>'
statement. We would like to inject the value for maximum rows  dynamically
in the mapper interface instead of using a hardcoded value direct in the
sql statement.

public List<SomeObject> selectSomeObject(int somecriteria, int
maxRows);

<select id="selectSomeObjects" resultMap="someObjectResultMap">
	SELECT TOP ${1} ColumnA, ColumnB FROM Table WHERE ColumnB = #{0}
</select>

...doesn't work. The value '1' is set in the statement not the value of
the parameter maxRows.

Has anyone figured out how to do that?

Thanks in advance!

Björn

Hi,

Continuing with testing HBase suitability in a high ingest rate
environment, I've come up with a new stumbling block, likely
due to my inexperience with HBase.

We want to keep and purge records on a time basis: i.e, when
a record is older than say, 24 hours, we want to purge it from
the database.

The problem that I am encountering is the only way I've found
to delete records using an arbitrary but strongly ordered over
time row id is to scan for rows from lower bound to upper
bound, then build an array of Delete using

for Result in ResultScanner
     add new Delete( Result.getRow( ) ) to Delete array.

This method is far too slow to keep up with our ingest rate; the
iteration over the Results in the ResultScanner is the bottleneck,
even though the Scan is limited to a single small column in the
column family.

The obvious but naive solution is to use a sequential row id
where the lower and upper bound can be known.  This would
allow the building of the array of Delete objects without a scan
step.  Problem with this approach is how do you guarantee a
sequential and non-colliding row id across more than one Put'ing
process, and do it efficiently.  As it happens, I can do this, but
given the details of my operational requirements, it's not a simple
thing to do.

So I was hoping that I had just missed something.  The ideal
would be a Delete object that would take row id bounds in the
same way that Scan does, allowing the work to be done all
on the server side.  Does this exists somewhere?  Or is there
some other way to skin this cat?

Thanks

Thomas Downing

Are 6..8 seconds to read 23.000 small rows - as it should be?

I have a quick question on what I think is bad read performance for this
simple setup:

 <ColumnFamily Name="Dashboard"
    ColumnType="Super"
    CompareWith="UTF8Type"
    CompareSubcolumnsWith="UTF8Type" />

SCF:Dashboard
  key:username1 -> { 
  	SC:uniqStr1 -> { col1:val1, col2: val2, ... col8:val8 },
  	SC:uniqStr2 -> { col1:val1, col2: val2, ... col8:val8 },
  	SC:uniqStr3 -> { col1:val1, col2: val2, ... col8:val8 },
  	SC:uniqStr4 -> { col1:val1, col2: val2, ... col8:val8 },
  	... up to 23.000 "rows"
  key:username2 -> { 
  	SC:uniqStr5 -> { col1:val1, col2: val2, ... col8:val8 },
  	SC:uniqStr6 -> { col1:val1, col2: val2, ... col8:val8 },
  	SC:uniqStr7 -> { col1:val1, col2: val2, ... col8:val8 },
  	SC:uniqStr8 -> { col1:val1, col2: val2, ... col8:val8 },
  	...

A given key "username1" has e.g. 23.000 super column unique (rows). When I
try and simply raw-read all these rows, it takes what I think isn't pretty
fast - approximately 6-8 seconds. I know, there are a millions things that
affect this, but I would just like to have a yes or no if this really can
be as it should be?

My cassandra is a pretty unchanged v0.6.1.

I read using this code:

 ColumnParent parent = new ColumnParent("Dashboard");

 SlicePredicate predicate = new SlicePredicate();
 SliceRange sliceRange = new SliceRange();
 sliceRange.setCount(Integer.MAX_VALUE);
 sliceRange.setStart(toRawValue(""));
 sliceRange.setFinish(toRawValue(""));
 predicate.setSlice_range(sliceRange);

 // timing this takes 6-8 secs.
 return client.get_slice(
   "keyspace", 
   "theusername", 
   columnParent, 
   slicePredicate, 
   ConsistencyLevel.QUORUM
 );
 
My replication factor is 1 and I had two nodes setup in cluster when doing
the reads.

Shouldn't this be what cassandra can do dead-fast?