improving random read performance

Greetings Cassandra Developers!

We've been trying to benchmark Cassandra performance and have 
developed a test client written in C++ that uses multiple threads to 
send out a large number of write and read requests (as fast as the
server can handle them).   

One of the results we're seeing is a bit surprising, and I'm hoping
someone here can help shed some light on the topic - as far as I can
tell, it hasn't been discuseed on the mailing list.

Most of the requests return in a reasonable amount of time (10s or
100s of milliseconds), but every once in a while, the server seems to
just "stop" for up to several seconds.   During this time, all the 
reads and writes will take several seconds to complete and network traffic
in an out of the system drops off to nearly zero.   When plotted on a 
graph, these appear as very larges spikes every few minutes.  (Though
without
any particular pattern to how often those spikes occur).   Even though
the average response time is very good (and therefore we get a reasonable
number of requests/sec) these occasional outliers are a showstopper for
our potential applications.

We've experimented with a number of different machines of different 
capabilities including a range of physical machines, and clusters of
machines on Amazon's EC2.  We've also used different numbers of nodes
in the cluster and different values for ReplicationFactor.   All are 
qualitatively similar, though the numbers vary as expected (i.e. 
fast machines improve both the average and maximum numbers, but the 
max values are still on the order of seconds)

I know Cassandra has lots of configuration parameters that can be
tweaked, but most of the other parameters are left at the default
values of Cassandara-0.6.2 or 0.6.3.

Has anyone else seen nodes "hang" for several seconds like this?  I'm
not sure if this is a Java VM issue (e.g. garbage collection) or something
specific to the Cassandra application.   I'll be happy to share more 
details of our experiments either on the mailing list, or with interested
parties offline.  But I thought I'd start with a brief description and 
see how consistent it is with other experiences.   I'm sort of expecting
to see "Well, of course you'll see that kind of behavior because you
didn't change..."

I'm also interested in comparing notes with anyone  else that has been
doing
read/write throughput benchmarks with Cassandara.

Thanks in advance for any information or suggestions you may have!

We are using Hbase 0.20.5 drop with latest cloudera Hadoop distribution.

- We are hitting 3 nodes Hbase cluster from a client which has 10
threads each with thread local copy of HTable client object and
established connection to server.
- Each of 10 threads issuing 10,000 read requests of keys randomly
selected from pool of 1000 keys. All keys are present on HBase and
table is pinned in memory (to make sure we don't have any disk seeks).
- If we run this test with 10 threads we get avg latency as seen by
client = 8ms (excluding initial 10 connection setup time) . But if we
increase # threads to 100, 250 to 500 we get increasing latency
numbers like 26ms, 51ms, 90ms.
- We have enabled HBase metrics on RS and we see "get_avg_time" on all
RS between 5-15ms in all tests, consistently.

Is this expected? Any tips to get consistent performance below 20ms?

https://issues.apache.org/bugzilla/show_bug.cgi?id=49433

           Summary: improving info in server status
           Product: Apache httpd-2
           Version: 2.2.15
          Platform: PC
        OS/Version: All
            Status: NEW
          Severity: enhancement
          Priority: P2
         Component: mod_fcgid
        AssignedTo: bu### @httpd.apache.org
        ReportedBy: erno.### @freemail.hu


One more column in the server status list of fcgid processes would be very
helpful when you serve php scripts, it is the request file and hostname.
Atm if
a php-cgi process is eating up cpu one cant really know which script is
currently being worked on.

Improving Plan Exporter Tool 

Improving implementation of loading LDIF files during startup

Separating hadoop metrics log types at source for improving primary key
lookup

I want to run a couchdb node which allows public read access so that  
replication can occur but then protect writes behind an application  
server.  Should I set up a firewall or proxy rule that disallows  
anything but GETs on the public IP and then have an application server  
running on the localhost interface which has full HTTP verbs allowed?   
Or is there another topology I should consider?

Thanks
Chris

Chris Dawson
971-533-8335

I am running Tomcat 6.0.26 on a Solaris 10 system.  The Tomcat server is
configured to listen to HTTPS communications on port 8443.  When browsing
to the Tomcat server remotely using the following syntax everything works
as expected:
https://10.10.10.10:8443/

If however we accidentally leave out the "s" in https like this:
http://10.10.10.10:8443/

The Tomcat server responds with a 7 byte .exe file to download.  Each time
we make the request again it generates a new .exe file with a different
name (cd64dni2.exe or z0v8671g.exe for example).  The exe fail fails to
execute on a windows system.   The contents of all of the exe files are
exactly the same (binary data)

If I run an od on the file I get the following:
$od cd64dni2.exe
0000000 001425 000001 001002 000012
0000007


Can anyone explain what this file is and why it is getting generated?

Hi,

I have a test plan with about 20 pages. I have set up 250 threads for 
testing. I like to know if I can perform random testing in the following 
manner:

1. put the urls for the  20 pages in a file
2. have 250 threads randomly go to one of the urls on the file 

Thank you.


Patty

This is a multi-part message in MIME format.
Thomas,

That was indeed the source of the problem. I naively assumed that the
token range would help me avoid retrieving duplicate rows.

If you iterate over the keys, how do you avoid retrieving duplicate keys?
I tried this morning and I seem to get odd results. Maybe this is just a
consequence of the random partitioner. I really don't care about the order
of the iteration, but only each key once and that I see all keys is
important.

-Adam


-----Original Message-----
From: th.he### @gmail.com on behalf of Thomas Heller
Sent: Fri 8/6/2010 7:27 AM
To: us### @cassandra.apache.org
Subject: Re: error using get_range_slice with random partitioner
 
Wild guess here, but are you using start_token/end_token here when you
should be using start_key? Looks to me like you are trying end_token
= ''.

HTH,
/thomas

On Thursday, August 5, 2010, Adam Crain
<adam.### @greenenergycorp.com> wrote:
 Hi,

 I'm on 0.6.4. Previous tickets in the JIRA in searching the web
indicated that iterating over the keys in keyspace is possible, even with
the random partitioner. This is mostly desirable in my case for testing
purposes only.

 I get the following error:

 [junit] Internal error processing get_range_slices
 [junit] org.apache.thrift.TApplicationException: Internal error
processing get_range_slices

 and the following server traceback:

 java.lang.NumberFormatException: Zero length BigInteger
         at
java.math.BigInteger.<init>(BigInteger.java:295)
         at
java.math.BigInteger.<init>(BigInteger.java:467)
         at
org.apache.cassandra.dht.RandomPartitioner$1.fromString(RandomPartitioner.java:100)
         at
org.apache.cassandra.thrift.CassandraServer.getRangeSlicesInternal(CassandraServer.java:575)

 I am using the scala cascal client, but am sure that get_range_slice
is being called with start and stop set to "".

 1) Is batch iteration possible with random partioner?

 This isn't clear from the FAQ entry on the subject:

 http://wiki.apache.org/cassandra/FAQ#iter_world

 2) The FAQ states that start argument should be "". What should the
end argument be?

 thanks!
 Adam

This is a multi-part message in MIME format.
David,

This much like the behavior I saw... I thought that I might be doing
something wrong, but I haven't had the time to check out other clients
iteration implementations. What client are you using?

-Adam


-----Original Message-----
From: David McIntosh [mailto:dav### @radiotime.com]
Sent: Thu 8/12/2010 6:51 PM
To: us### @cassandra.apache.org
Subject: RE: error using get_range_slice with random partitioner
 
I'm also seeing an issue with not being able to iterate over all keys in
Cassandra 0.6.4.  In my unit test I create 20 keys (0-19) and iterate with
a
batch size of 6.  This is what I get.

 

Cassandra 0.6.4

start key: ""

9, 14, 4, 15, 11, 18

start key: 18

18, 7, 17, 7, 17

start key:17

17

 

Cassandra 0.6.3

start key: ""

3, 6, 5, 19, 10, 0

start key: 0

0, 8, 2, 16, 13, 1

start key: 1

1, 12, 9, 14, 4, 15

start key: 15

15, 11, 15, 11, 18, 7

start key: 7

7, 17, 7, 17

 

In both versions I get duplicates but in 0.6.4 I don't get the complete
set
of keys back.  The complete set is returned in 0.6.3.

It's an error to do this if I read the doc correctly, because shows and
lists functions should be idempotent. It's specified in the wiki.

Mickael

----- Mail Original -----
De: "Jan Prieser" <j.pr### @hotornot.de>
À: use### @couchdb.apache.org
Envoyé: Lundi 28 Juin 2010 18h37:38 GMT +01:00 Amsterdam / Berlin / Berne
/ Rome / Stockholm / Vienne
Objet: Re: selecting a random subset of a view

hi Mickael,

i've had the same problem and used the lists-feature from couch.

my list-function looks like this:

function(head,req) {
   function shuffle(ary) {
     function randOrd(){ return Math.round(Math.random()) - 0.5; }
     ary.sort( randOrd );
   }
   body={};
   eval('body='+req.body);
   out = head;
   out.rows = [];
   if(out.total_rows > out.offset) {
     while (row = getRow()) {
         out.rows.push(row);
     }
   }
   shuffle(out.rows);
   if (body && body.rlimit) {
     out.rows = out.rows.slice(-body.rlimit);
   }
   return toJSON(out) + '\n';
}

maybe you could use a range with startkey and endkey, if the number of 
rows is to big. I didn't test the performance with bigger datasets.


Am 28.06.2010 15:29, schrieb mickael.### @free.fr:
 Hello couchers,

 how would you do to select a random subset of a view result (a simple
view with map only).

 Example (I don't write the full view response array for clarity)

 When called normally, my view returns :

 {
 ...
 rows: [
 {id: aa1},
 {id: aa2},
 {id: aa3},
 {id: aa4},
 {id: aa5},
 {id: aa6},
 {id: aa7},
 {id: aa8},
 {id: aa9}
 ]
 }

 And I want only three of those rows, randomly chosen. So I launch the
magic "get three random rows" feature, and it gives me :

 {
 ...
 rows: [
 {id: aa5},
 {id: aa3},
 {id: aa6}
 ]
 }

 The second time I launch the same magic "get three random rows" I
got:
 {
 ...
 rows: [
 {id: aa7},
 {id: aa1},
 {id: aa5}
 ]
 }

 Thanks for your advices

 Mickael

I find it a really great idea. If it can be set per design doc, of course.

Mickael

----- Mail Original -----
De: "J Chris Anderson" <jch### @apache.org>
À: us### @couchdb.apache.org
Envoyé: Lundi 28 Juin 2010 21h34:16 GMT +01:00 Amsterdam / Berlin / Berne
/ Rome / Stockholm / Vienne
Objet: Re: selecting a random subset of a view


On Jun 28, 2010, at 12:28 PM, mickael### @free.fr wrote:

 It's an error to do this if I read the doc correctly, because shows
and lists functions should be idempotent. It's specified in the wiki.
 

It would be fine to have a config option to suppress the Etags on show and
list. Then you'd have no risk of improperly caching the output here. For
the time being, you can probably configure an http proxy to ignore and
strip the etags.

Chris

 Mickael
 
 ----- Mail Original -----
 De: "Jan Prieser" <j.pri### @hotornot.de>
 À: us### @couchdb.apache.org
 Envoyé: Lundi 28 Juin 2010 18h37:38 GMT +01:00 Amsterdam / Berlin /
Berne / Rome / Stockholm / Vienne
 Objet: Re: selecting a random subset of a view
 
 hi Mickael,
 
 i've had the same problem and used the lists-feature from couch.
 
 my list-function looks like this:
 
 function(head,req) {
   function shuffle(ary) {
     function randOrd(){ return Math.round(Math.random()) - 0.5; }
     ary.sort( randOrd );
   }
   body={};
   eval('body='+req.body);
   out = head;
   out.rows = [];
   if(out.total_rows > out.offset) {
     while (row = getRow()) {
         out.rows.push(row);
     }
   }
   shuffle(out.rows);
   if (body && body.rlimit) {
     out.rows = out.rows.slice(-body.rlimit);
   }
   return toJSON(out) + '\n';
 }
 
 maybe you could use a range with startkey and endkey, if the number
of 
 rows is to big. I didn't test the performance with bigger datasets.
 
 
 Am 28.06.2010 15:29, schrieb mickael.### @free.fr:
> Hello couchers,
> 
> how would you do to select a random subset of a view result (a
simple view with map only).
> 
> Example (I don't write the full view response array for clarity)
> 
> When called normally, my view returns :
> 
> {
> ...
> rows: [
> {id: aa1},
> {id: aa2},
> {id: aa3},
> {id: aa4},
> {id: aa5},
> {id: aa6},
> {id: aa7},
> {id: aa8},
> {id: aa9}
> ]
> }
> 
> And I want only three of those rows, randomly chosen. So I launch
the magic "get three random rows" feature, and it gives me :
> 
> {
> ...
> rows: [
> {id: aa5},
> {id: aa3},
> {id: aa6}
> ]
> }
> 
> The second time I launch the same magic "get three random rows" I
got:
> {
> ...
> rows: [
> {id: aa7},
> {id: aa1},
> {id: aa5}
> ]
> }
> 
> Thanks for your advices
> 
> Mickael

This is a multi-part message in MIME format.
Hi,

I'm on 0.6.4. Previous tickets in the JIRA in searching the web indicated
that iterating over the keys in keyspace is possible, even with the random
partitioner. This is mostly desirable in my case for testing purposes only.

I get the following error:

[junit] Internal error processing get_range_slices
[junit] org.apache.thrift.TApplicationException: Internal error processing
get_range_slices

and the following server traceback:

java.lang.NumberFormatException: Zero length BigInteger
	at java.math.BigInteger.<init>(BigInteger.java:295)
	at java.math.BigInteger.<init>(BigInteger.java:467)
	at
org.apache.cassandra.dht.RandomPartitioner$1.fromString(RandomPartitioner.java:100)
	at
org.apache.cassandra.thrift.CassandraServer.getRangeSlicesInternal(CassandraServer.java:575)

I am using the scala cascal client, but am sure that get_range_slice is
being called with start and stop set to "".

1) Is batch iteration possible with random partioner?

This isn't clear from the FAQ entry on the subject:

http://wiki.apache.org/cassandra/FAQ#iter_world

2) The FAQ states that start argument should be "". What should the end
argument be?

thanks!
Adam

Thanks for all your answers

I forgot to tell I try to avoid the "skip" param for performances reasons.

The first method Ian suggest imply to read all my doc ids... In this case
I can easily use the client random function (client is PHP). But reading
all doc ids is not really performance friendly.

The second method Ian speak about, is the one that others (Sebastian,
Robert) are proposing : some random parameter in the document. But as Ian
tells us, "If you use the MD5 or SHA function, then the order will be
repeatable if 
the data is not changed."... not so random :-)

My app is a music jukebox, I want to provide a way to fill the playlist
with random songs. Each song document have an _id composed of 'aa'+ a
random sha1 .

I scratched my head already but I think the random feature should be a
couchdb server feature, and can't be implemented client-side, but by
reading all documents ids and using client-side random function, which is,
once again, not really performance-friendly.

Any idea welcome...

Mickael

----- Mail Original -----
De: "Ian Hobson" <i### @ianhobson.co.uk>
À: us### @couchdb.apache.org
Envoyé: Lundi 28 Juin 2010 16h04:09 GMT +01:00 Amsterdam / Berlin / Berne
/ Rome / Stockholm / Vienne
Objet: Re: selecting a random subset of a view

On 28/06/2010 14:29, mickael### @free.fr wrote:
 Hello couchers,

 how would you do to select a random subset of a view result (a simple
view with map only).

    
Hi Mickael,

You want a random sample of predetermined size, from a list that you can 
only (best) access randomly. To do this you must know the number of 
records in the database.

Note - I know the stats side MUCH better than the couchdb so this might 
not be implementable.

Here are two methods.

Method 1.

Say, for example, you want 3 from 11.

Take the first index with a probability of 3/11  by computing a random 
number in range 0 to 1, and taking the record if rand < 3/11 (using
real 
math, not integer).

If you take that record, adjust the number required down by 1.
Reduce the number remaining by 1.

Take the next record with a probability or 2/10 or 3/10

Continue in like manner until you either

a) Take the last record with a probability of 1/1 or
b) Have all you want, and take the remaining records with a probability 
of 0/n

To do this with couchdb I would read all the IDs into the client and 
filter them there, and then read each records separately.

Method 2 -

Allocate a random number to each record (from a large range - we don't 
want duplicates). This could be a sha or MD5 of the actual data.
Sort by the random number allocated.
Read the first N records that you need.

I think this sort of index could be set up on the server, so the client 
needs only create the index, and read the first N records and remove the 
index. The work on the server will be much greater that method 1 though.

If you use the MD5 or SHA function, then the order will be repeatable if 
the data is not changed.

Regards

Ian

I've never changed the partitioner from the default random. Other ideas?

I can insert and do column queries using a single key but not range on CF.

-Adam

-----Original Message-----
From: Jonathan Ellis [mailto:jbe### @gmail.com] 
Sent: Thursday, August 05, 2010 11:33 AM
To: use### @cassandra.apache.org
Subject: Re: error using get_range_slice with random partitioner

Yes, you should be able to use get_range_slices with RP.

This stack trace looks like you changed your partitioner after the
node already had data in it.

On Thu, Aug 5, 2010 at 10:06 AM, Adam Crain
<adam.### @greenenergycorp.com> wrote:
 Hi,

 I'm on 0.6.4. Previous tickets in the JIRA in searching the web
indicated
 that iterating over the keys in keyspace is possible, even with the
random
 partitioner. This is mostly desirable in my case for testing purposes
only.

 I get the following error:

 [junit] Internal error processing get_range_slices
 [junit] org.apache.thrift.TApplicationException: Internal error
processing
 get_range_slices

 and the following server traceback:

 java.lang.NumberFormatException: Zero length BigInteger
         at
java.math.BigInteger.<init>(BigInteger.java:295)
         at
java.math.BigInteger.<init>(BigInteger.java:467)
         at

org.apache.cassandra.dht.RandomPartitioner$1.fromString(RandomPartitioner.java:100)
         at

org.apache.cassandra.thrift.CassandraServer.getRangeSlicesInternal(CassandraServer.java:575)

 I am using the scala cascal client, but am sure that get_range_slice
is
 being called with start and stop set to "".

 1) Is batch iteration possible with random partioner?

 This isn't clear from the FAQ entry on the subject:

 http://wiki.apache.org/cassandra/FAQ#iter_world

 2) The FAQ states that start argument should be "". What should the
end
 argument be?

 thanks!
 Adam

This is a multi-part message in MIME format.
Thanks Dave. I'm using 0.6.4 since I say this issue in the JIRA, but I
just discovered that the client I'm using mutates the order of keys after
retrieving the result with the thrift API... pretty much making key
iteration impossible. So time to fork and see if they'll fix it :(.

I'll review yours as soon as I get the client fixed that I'm using.

Adam


-----Original Message-----
From: davev### @gmail.com on behalf of Dave Viner
Sent: Fri 8/6/2010 11:28 AM
To: use### @cassandra.apache.org
Subject: Re: error using get_range_slice with random partitioner
 
Funny you should ask... I just went through the same exercise.

You must use Cassandra 0.6.4.  Otherwise you will get duplicate keys.
 However, here is a snippet of perl that you can use.

our $WANTED_COLUMN_NAME = 'mycol';
get_key_to_one_column_map('myKeySpace', 'myColFamily', 'mySuperCol',
QUORUM,
\%map);

sub get_key_to_one_column_map
{
    my ($keyspace, $column_family_name, $super_column_name,
$consistency_level, $returned_keys) = @_;


    my($socket, $transport, $protocol, $client, $result, $predicate,
$column_parent, $keyrange);

    $column_parent = new Cassandra::ColumnParent();
    $column_parent->{'column_family'} = $column_family_name;
    $column_parent->{'super_column'} = $super_column_name;

    $keyrange = new Cassandra::KeyRange({
            'start_key' => '', 'end_key' => '', 'count' => 10
    });


    $predicate = new Cassandra::SlicePredicate();
    $predicate->{'column_names'} = [$WANTED_COLUMN_NAME];

    eval
    {
        $socket = new Thrift::Socket($CASSANDRA_HOST, $CASSANDRA_PORT);
        $transport = new Thrift::BufferedTransport($socket, 1024, 1024);
        $protocol = new Thrift::BinaryProtocol($transport);
        $client = new Cassandra::CassandraClient($protocol);
        $transport->open();


        my($next_start_key, $one_res, $iteration, $have_more, $value,
$local_count, $previous_start_key);

        $iteration = 0;
        $have_more = 1;
        while ($have_more == 1)
        {
            $iteration++;
            $result = undef;

            $result = $client->get_range_slices($keyspace,
$column_parent,
$predicate, $keyrange, $consistency_level);

            # on success, results is an array of objects.

            if (scalar(@$result) == 1)
            {
                # we only got 1 result... check to see if it's the
                # same key as the start key... if so, we're done.
                if ($result->[0]->{'key'} eq
$keyrange->{'start_key'})
                {
                    $have_more = 0;
                    last;
                }
            }

            # check to see if we are starting with some value
            # if so, we throw away the first result.
            if ($keyrange->{'start_key'})
            {
                shift(@$result);
            }
            if (scalar(@$result) == 0)
            {
                $have_more = 0;
                last;
            }

            $previous_start_key = $keyrange->{'start_key'};
            $local_count = 0;

            for (my $r = 0; $r < scalar(@$result); $r++)
            {
                $one_res = $result->[$r];
                $next_start_key = $one_res->{'key'};

                $keyrange->{'start_key'} = $next_start_key;

                if (!exists($returned_keys->{$next_start_key}))
                {
                    $have_more = 1;
                    $local_count++;
                }


                next if (scalar(@{ $one_res->{'columns'} }) == 0);

                $value = undef;

                for (my $i = 0; $i < scalar(@{ $one_res->{'columns'}
});
$i++)
                {
                    if
($one_res->{'columns'}->[$i]->{'column'}->{'name'} eq
$WANTED_COLUMN_NAME)
                    {
                        $value =
$one_res->{'columns'}->[$i]->{'column'}->{'value'};
                        if (!exists($returned_keys->{$next_start_key}))
                        {
                            $returned_keys->{$next_start_key} = $value;
                        }
                        else
                        {
                            # NOTE: prior to Cassandra 0.6.4, the
get_range_slices returns duplicates sometimes.
                            #warn "Found second value for key
[$next_start_key]  was [" . $returned_keys->{$next_start_key} . "] now
[$value]!";
                        }
                    }
                }
                $have_more = 1;
            } # end results loop

            if ($keyrange->{'start_key'} eq $previous_start_key)
            {
                $have_more = 0;
            }

        } # end while() loop

        $transport->close();
    };
    if ($@)
    {
        warn "Problem with Cassandra: " . Dumper($@);
    }

    # cleanup
    undef $client;
    undef $protocol;
    undef $transport;
    undef $socket;
}


HTH
Dave Viner

On Fri, Aug 6, 2010 at 7:45 AM, Adam Crain
<adam.c### @greenenergycorp.com>wrote:

 Thomas,

 That was indeed the source of the problem. I naively assumed that the
token
 range would help me avoid retrieving duplicate rows.

 If you iterate over the keys, how do you avoid retrieving duplicate
keys? I
 tried this morning and I seem to get odd results. Maybe this is just
a
 consequence of the random partitioner. I really don't care about the
order
 of the iteration, but only each key once and that I see all keys is
 important.

 -Adam


 -----Original Message-----
 From: th.he### @gmail.com on behalf of Thomas Heller
 Sent: Fri 8/6/2010 7:27 AM
 To: us### @cassandra.apache.org
 Subject: Re: error using get_range_slice with random partitioner

 Wild guess here, but are you using start_token/end_token here when
you
 should be using start_key? Looks to me like you are trying end_token
 = ''.

 HTH,
 /thomas

 On Thursday, August 5, 2010, Adam Crain
<adam.### @greenenergycorp.com>
 wrote:
 > Hi,
 >
 > I'm on 0.6.4. Previous tickets in the JIRA in searching the web
indicated
 that iterating over the keys in keyspace is possible, even with the
random
 partitioner. This is mostly desirable in my case for testing purposes
only.
 >
 > I get the following error:
 >
 > [junit] Internal error processing get_range_slices
 > [junit] org.apache.thrift.TApplicationException: Internal error
 processing get_range_slices
 >
 > and the following server traceback:
 >
 > java.lang.NumberFormatException: Zero length BigInteger
 >         at
java.math.BigInteger.<init>(BigInteger.java:295)
 >         at
java.math.BigInteger.<init>(BigInteger.java:467)
 >         at

org.apache.cassandra.dht.RandomPartitioner$1.fromString(RandomPartitioner.java:100)
 >         at

org.apache.cassandra.thrift.CassandraServer.getRangeSlicesInternal(CassandraServer.java:575)
 >
 > I am using the scala cascal client, but am sure that
get_range_slice is
 being called with start and stop set to "".
 >
 > 1) Is batch iteration possible with random partioner?
 >
 > This isn't clear from the FAQ entry on the subject:
 >
 > http://wiki.apache.org/cassandra/FAQ#iter_world
 >
 > 2) The FAQ states that start argument should be "". What should
the end
 argument be?
 >
 > thanks!
 > Adam
 >
 >
 >
 >
 >
 >

I can. I'm using the debian distro.  I assume that all that is required is
wiping the data/commitlog directories.

If I do that, I still get the same result.

Here's my CF:

<ColumnFamily Name ="Meas" CompareWith="LongType" />

I'm using this to time series measurement data where the keys are
measurement names and the columns are Long unix epoch timestamps in
millisecs. My use case is then to do a range_slice that asks for the first
X number of rows, but only the most recent measurement by using a
descending order column predicate with a limit of 1.

I have no trouble using this predicate to retrieve columns within a
specified row, but the get_range_slice fails.

-Adam

-----Original Message-----
From: Jonathan Ellis [mailto:jbe### @gmail.com] 
Sent: Thursday, August 05, 2010 12:22 PM
To: us### @cassandra.apache.org
Subject: Re: error using get_range_slice with random partitioner

can you reproduce starting with a fresh install, no existing data?

On Thu, Aug 5, 2010 at 12:09 PM, Adam Crain
<adam.### @greenenergycorp.com> wrote:
 I've never changed the partitioner from the default random. Other
ideas?

 I can insert and do column queries using a single key but not range
on CF.

 -Adam

 -----Original Message-----
 From: Jonathan Ellis [mailto:jbel### @gmail.com]
 Sent: Thursday, August 05, 2010 11:33 AM
 To: us### @cassandra.apache.org
 Subject: Re: error using get_range_slice with random partitioner

 Yes, you should be able to use get_range_slices with RP.

 This stack trace looks like you changed your partitioner after the
 node already had data in it.

 On Thu, Aug 5, 2010 at 10:06 AM, Adam Crain
 <adam.### @greenenergycorp.com> wrote:
> Hi,
>
> I'm on 0.6.4. Previous tickets in the JIRA in searching the web
indicated
> that iterating over the keys in keyspace is possible, even with
the random
> partitioner. This is mostly desirable in my case for testing
purposes only.
>
> I get the following error:
>
> [junit] Internal error processing get_range_slices
> [junit] org.apache.thrift.TApplicationException: Internal error
processing
> get_range_slices
>
> and the following server traceback:
>
> java.lang.NumberFormatException: Zero length BigInteger
>         at
java.math.BigInteger.<init>(BigInteger.java:295)
>         at
java.math.BigInteger.<init>(BigInteger.java:467)
>         at
>
org.apache.cassandra.dht.RandomPartitioner$1.fromString(RandomPartitioner.java:100)
>         at
>
org.apache.cassandra.thrift.CassandraServer.getRangeSlicesInternal(CassandraServer.java:575)
>
> I am using the scala cascal client, but am sure that
get_range_slice is
> being called with start and stop set to "".
>
> 1) Is batch iteration possible with random partioner?
>
> This isn't clear from the FAQ entry on the subject:
>
> http://wiki.apache.org/cassandra/FAQ#iter_world
>
> 2) The FAQ states that start argument should be "". What should
the end
> argument be?
>
> thanks!
> Adam
>
>



 --
 Jonathan Ellis
 Project Chair, Apache Cassandra
 co-founder of Riptano, the source for professional Cassandra support
 http://riptano.com

This is a multi-part message in MIME format.
Hi Jeremy,

So, I fixed my client so it preserves the ordering and I get results that
may be related to the bug.

If I insert 30 keys into the random partitioner with names [key1, key2,
... key30] and then start the iteration (with a batch size of 10) I get the
following debug output during the iteration:

[junit] Query w/ Range(,,10) result size: 10
[junit] key18
[junit] key23
[junit] key26
[junit] key27
[junit] key12
[junit] key28
[junit] key4
[junit] key3
[junit] key1
[junit] key24
[junit] Query w/ Range(key24,,10) result size: 10
[junit] key24
[junit] key5
[junit] key17
[junit] key29
[junit] key19
[junit] key8
[junit] key15
[junit] key22
[junit] key6
[junit] key25
[junit] Query w/ Range(key25,,10) result size: 3
[junit] key25
[junit] key14
[junit] key2
[junit] Query w/ Range(key2,,10), result size: 1
[junit] key2

I never make it back around to key 18 as expected, and I never see all of
the keys.

-Adam

-----Original Message-----
From: Jeremy Hanna [mailto:jeremy.h### @gmail.com]
Sent: Fri 8/6/2010 11:45 AM
To: us### @cassandra.apache.org
Subject: Re: error using get_range_slice with random partitioner
 
Sounds like what you're seeing is in the client, but there was another
duplicate bug with get_range_slice that was recently fixed on cassandra-0.6
branch.  It's slated for 0.6.5 which will probably be out sometime this
month, based on previous minor releases.

https://issues.apache.org/jira/browse/CASSANDRA-1145

On Aug 6, 2010, at 10:29 AM, Adam Crain wrote:

 Thanks Dave. I'm using 0.6.4 since I say this issue in the JIRA, but
I just discovered that the client I'm using mutates the order of keys after
retrieving the result with the thrift API... pretty much making key
iteration impossible. So time to fork and see if they'll fix it :(.
 
 I'll review yours as soon as I get the client fixed that I'm using.
 
 Adam
 
 
 -----Original Message-----
 From: davev### @gmail.com on behalf of Dave Viner
 Sent: Fri 8/6/2010 11:28 AM
 To: us### @cassandra.apache.org
 Subject: Re: error using get_range_slice with random partitioner
 
 Funny you should ask... I just went through the same exercise.
 
 You must use Cassandra 0.6.4.  Otherwise you will get duplicate keys.
 However, here is a snippet of perl that you can use.
 
 our $WANTED_COLUMN_NAME = 'mycol';
 get_key_to_one_column_map('myKeySpace', 'myColFamily', 'mySuperCol',
QUORUM,
 \%map);
 
 sub get_key_to_one_column_map
 {
    my ($keyspace, $column_family_name, $super_column_name,
 $consistency_level, $returned_keys) = @_;
 
 
    my($socket, $transport, $protocol, $client, $result, $predicate,
 $column_parent, $keyrange);
 
    $column_parent = new Cassandra::ColumnParent();
    $column_parent->{'column_family'} = $column_family_name;
    $column_parent->{'super_column'} = $super_column_name;
 
    $keyrange = new Cassandra::KeyRange({
            'start_key' => '', 'end_key' => '', 'count' => 10
    });
 
 
    $predicate = new Cassandra::SlicePredicate();
    $predicate->{'column_names'} = [$WANTED_COLUMN_NAME];
 
    eval
    {
        $socket = new Thrift::Socket($CASSANDRA_HOST,
$CASSANDRA_PORT);
        $transport = new Thrift::BufferedTransport($socket, 1024,
1024);
        $protocol = new Thrift::BinaryProtocol($transport);
        $client = new Cassandra::CassandraClient($protocol);
        $transport->open();
 
 
        my($next_start_key, $one_res, $iteration, $have_more, $value,
 $local_count, $previous_start_key);
 
        $iteration = 0;
        $have_more = 1;
        while ($have_more == 1)
        {
            $iteration++;
            $result = undef;
 
            $result = $client->get_range_slices($keyspace,
$column_parent,
 $predicate, $keyrange, $consistency_level);
 
            # on success, results is an array of objects.
 
            if (scalar(@$result) == 1)
            {
                # we only got 1 result... check to see if it's the
                # same key as the start key... if so, we're done.
                if ($result->[0]->{'key'} eq
$keyrange->{'start_key'})
                {
                    $have_more = 0;
                    last;
                }
            }
 
            # check to see if we are starting with some value
            # if so, we throw away the first result.
            if ($keyrange->{'start_key'})
            {
                shift(@$result);
            }
            if (scalar(@$result) == 0)
            {
                $have_more = 0;
                last;
            }
 
            $previous_start_key = $keyrange->{'start_key'};
            $local_count = 0;
 
            for (my $r = 0; $r < scalar(@$result); $r++)
            {
                $one_res = $result->[$r];
                $next_start_key = $one_res->{'key'};
 
                $keyrange->{'start_key'} = $next_start_key;
 
                if (!exists($returned_keys->{$next_start_key}))
                {
                    $have_more = 1;
                    $local_count++;
                }
 
 
                next if (scalar(@{ $one_res->{'columns'} }) == 0);
 
                $value = undef;
 
                for (my $i = 0; $i < scalar(@{
$one_res->{'columns'} });
 $i++)
                {
                    if
($one_res->{'columns'}->[$i]->{'column'}->{'name'} eq
 $WANTED_COLUMN_NAME)
                    {
                        $value =
 $one_res->{'columns'}->[$i]->{'column'}->{'value'};
                        if
(!exists($returned_keys->{$next_start_key}))
                        {
                            $returned_keys->{$next_start_key} =
$value;
                        }
                        else
                        {
                            # NOTE: prior to Cassandra 0.6.4, the
 get_range_slices returns duplicates sometimes.
                            #warn "Found second value for key
 [$next_start_key]  was [" . $returned_keys->{$next_start_key} . "]
now
 [$value]!";
                        }
                    }
                }
                $have_more = 1;
            } # end results loop
 
            if ($keyrange->{'start_key'} eq $previous_start_key)
            {
                $have_more = 0;
            }
 
        } # end while() loop
 
        $transport->close();
    };
    if ($@)
    {
        warn "Problem with Cassandra: " . Dumper($@);
    }
 
    # cleanup
    undef $client;
    undef $protocol;
    undef $transport;
    undef $socket;
 }
 
 
 HTH
 Dave Viner
 
 On Fri, Aug 6, 2010 at 7:45 AM, Adam Crain
 <adam.c### @greenenergycorp.com>wrote:
 
> Thomas,
> 
> That was indeed the source of the problem. I naively assumed that
the token
> range would help me avoid retrieving duplicate rows.
> 
> If you iterate over the keys, how do you avoid retrieving
duplicate keys? I
> tried this morning and I seem to get odd results. Maybe this is
just a
> consequence of the random partitioner. I really don't care about
the order
> of the iteration, but only each key once and that I see all keys
is
> important.
> 
> -Adam
> 
> 
> -----Original Message-----
> From: th.h### @gmail.com on behalf of Thomas Heller
> Sent: Fri 8/6/2010 7:27 AM
> To: use### @cassandra.apache.org
> Subject: Re: error using get_range_slice with random partitioner
> 
> Wild guess here, but are you using start_token/end_token here
when you
> should be using start_key? Looks to me like you are trying
end_token
> = ''.
> 
> HTH,
> /thomas
> 
> On Thursday, August 5, 2010, Adam Crain
<adam.c### @greenenergycorp.com>
> wrote:
>> Hi,
>> 
>> I'm on 0.6.4. Previous tickets in the JIRA in searching the
web indicated
> that iterating over the keys in keyspace is possible, even with
the random
> partitioner. This is mostly desirable in my case for testing
purposes only.
>> 
>> I get the following error:
>> 
>> [junit] Internal error processing get_range_slices
>> [junit] org.apache.thrift.TApplicationException: Internal
error
> processing get_range_slices
>> 
>> and the following server traceback:
>> 
>> java.lang.NumberFormatException: Zero length BigInteger
>>        at
java.math.BigInteger.<init>(BigInteger.java:295)
>>        at
java.math.BigInteger.<init>(BigInteger.java:467)
>>        at
>
org.apache.cassandra.dht.RandomPartitioner$1.fromString(RandomPartitioner.java:100)
>>        at
>
org.apache.cassandra.thrift.CassandraServer.getRangeSlicesInternal(CassandraServer.java:575)
>> 
>> I am using the scala cascal client, but am sure that
get_range_slice is
> being called with start and stop set to "".
>> 
>> 1) Is batch iteration possible with random partioner?
>> 
>> This isn't clear from the FAQ entry on the subject:
>> 
>> http://wiki.apache.org/cassandra/FAQ#iter_world
>> 
>> 2) The FAQ states that start argument should be "". What
should the end
> argument be?
>> 
>> thanks!
>> Adam
>> 
>> 
>> 
>> 
>> 
>> 
> 
> 
> 
> 
> 
 
 <winmail.dat>