Issue 296 in redis: redis-cli does not handle EOF in monitor mode correctly

Comment #1 on issue 276 by mvolmaro: [Feature Request] redis as embedded  
database
http://code.google.com/p/redis/issues/detail?id=276

Indeed. Having Redis available as embedded DB opens up a lot of  
possibilities that are not possible with the server only version.

Comment #2 on issue 266 by ben.nevile: Sorted set has not a same rank even
 
though it has a same score
http://code.google.com/p/redis/issues/detail?id=266

I understand the strict, academic definition of rank, but I don't believe 

it is a useful number to return in this context. I would be happy if you  
could provide me with a counter-example: why would someone using Redis
want  
to know the exact index of an element in the sorted set? Aren't we using  
Redis so that we can allow it to deal with that kind of data-structure  
detail?

I sumbit that it would be more helpful to have a command like kikiki.blue 

is requesting. Something like ZSCORERANK X, which when given element X
with  
score Y would return the number of elements with score greater than Y plus
 
one. Currently I am hacking this as follows with ZRANGEBYSCORE (node.js  
code):

redis.zrangebyscore(group_key,score+1,'+inf',function(err,reply){
   var rank = 1;
   if (reply) {
     //redis returns null for no matching elements
     rank = reply.toString().split(',').length + 1;
   }
   ...
})

This is fine, but I'm not looking forward to running this code on sorted  
sets with 100,000+ elements (a totally real use case, I'm using Redis for 

Facebook apps with hundreds of thousands of participants.)

Comment #4 on issue 288 by J4yferd: Guru Meditation crash every 1-2 days
http://code.google.com/p/redis/issues/detail?id=288

so for the meantime we have a cron job making sure it's up.

MONITOR has the strangest behavior when it crashes - it seems to output a 

bajillion newlines, which makes it really hard to find the last command.  

I've been trying to get that last line by redirecting the output to a log 

file, but after the crash, the log file was about 22M of gibberish.  It  
nearly crashed `less`.

I'll keep trying on my end, but let me know if there's any other info you 

guys need.

Comment #5 on issue 231 by elreydetodo: Feature request: support for unix 

domain sockets
http://code.google.com/p/redis/issues/detail?id=231

I use Redis extensively in a very performance-critical application. I
would  
love to see Unix sockets available as well!

Comment #1 on issue 181 by jzawodn: master connect timeout neeed for  
replication SLAVEOF command to prevent DoS
http://code.google.com/p/redis/issues/detail?id=181

Adding to this, we were bitten again by this issue recently.

Ideally, I'd like two config file vars to help.  One would set the connect
 
timeout between a slave and its master (letting the OS decide waits far
too  
long!).  Secondly, being able to set the number of retries the slave will 

do will help to mitigate this as well.  In our environment,  I want the  
slave to have a 3 second connect timeout to the master and to try at most
3  
times before giving up.  Then our monitoring system can catch it and get a
 
human involved.

Otherwise, the slave is mostly unresponsive (long pauses in responses to  
other commands) while it's waiting for the timeout to fire during socket  
connection.

Comment #2 on issue 286 by ratman: set mykey "my binary safe value"    ...
 
will not work
http://code.google.com/p/redis/issues/detail?id=286

I have the same issue. On OSX 10.6.4, Redis 1.2.6 installed via 'brew'  
which pulled the archive:  
http://redis.googlecode.com/files/redis-1.2.6.tar.gz

I'm using double quotes but still get the error. Perhaps a bug catching
the  
quotes on OSX?

My exact output:

$ redis-cli
redis> set mykey "my binary safe value"
Wrong number of arguments for 'set'
redis>

Thanks to your replies, I finally installed a specific NGINX HTTP
instance with DAV module to take care of uploads.

As my first need was a sort of message queue with these blobs, I
exchange in a redis queue only previously uploaded NGINX URL.

Light, robust and fast.

Regards

Hey Fabien!

Bigdis it's just a toy currently, something that needs a lot of work
(and volunteers, as I need to focus on Redis itself ;) in order to be
a trusted/good solution.

So I suggest using Redis or another mature FS-based solution as
alternative.

Cheers,
Salvatore

On Mon, Jul 26, 2010 at 10:17 PM, Fabien <fabien.### @gmail.com>
wrote:
 Hi,

 I just discovered Bigdis :

 http://github.com/antirez/Bigdis/

 In the README, antirez wrote :

 

Hi Fabien,

Redis can be used for large values, but this causes Redis to spend a lot
of
time receiving SETs and sending replies back to the client, what can be
seen
as overhead. If this is the case, you should take a step back and see what
the problem is you're trying to solve. With your problem that involves
blobs
with an average size of 1MB, I would recommend not using Redis for serving
them for the simple reason that Redis is not designed to serve large
blobs.
There is no best-practice for a maximum record size, but if your problem
only involves working with the blobs as a whole (not parts, using commands
like APPEND), I would just stick to using a filesystem.

On another note, you not need to use /dev/shm because the OS will take
care
of caching on the fs-level. If your RAM is large enough to hold everything
you access frequently, it will be as speedy as /dev/shm.

Cheers,
Pieter

On Tue, Jul 27, 2010 at 8:11 PM, Jason J. W. Williams <
jasonjw### @gmail.com> wrote:

 Not related to this exactly, but I bet Bigdis performs well on
 Copy-on-Write filesystems.

 -J

 On Mon, Jul 26, 2010 at 2:17 PM, Fabien
<fabien### @gmail.com> wrote:
 > Hi,
 >
 > I just discovered Bigdis :
 >
 > http://github.com/antirez/Bigdis/
 >
 > In the README, antirez wrote :
 >
 >
 

If you use memcache in PHP you can store a whole object just using $mc-
set($key, $object).
I am currently using phpredis and it does not support this feature, If
you try, it says that the function was expecting a string). Do you
know a PHP client that does support object serialization?

Thank you in advance

Redis data is only one dimensional(STRING,LIST,SET) or two
dimensional(ZSET,HASH), so transforming redis commands into Mysql
tables is actually fairly easy.

The idea of having a script that listens to the append-only file (from
Marion) DOES work, it requires some transformations, but is fairly
quick, the key is your data is organized in a way that can easily
NORMALISE.

Building transformations is not that hard: (quick & dirty writeup)
A.) STRING (created by "SET" cmd) can simple be INSERTed into a table
w/ a pk that auto-increments and two varchar(256)s named key and value
  1.) "SET x 10000"-> "INSERT into STRINGS values ("x","10000"); {pk
will be 1}
  2.) "SET y 20000"-> "INSERT into STRINGS values ("y","20000"); {pk
will be 2}
B.) LIST is pretty much the same except the table needs a timestamp
and LPOPs sort to last timestamp and delete.
  1.) "LADD L 10" -> INSERT into LISTS ("L","10") {timestamp would be
1280349447}
  2.) "LADD L 20" -> INSERT into LISTS ("L","20") {timestamp would be
1280349448}
  3.) "LPOP L"    -> DELETE from LISTS where name = "L" and timestamp
= (SELECT max(timestamp) from LISTS where name ="L");
C.) HASH needs two fields (key varchar(32) and value FLOAT), HDEL
needs to create a SQL DELETE with pk=x and key=k.
  1.) "HADD H key1 val1" -> INSERT INTO HASHES values
("H","key1",val1");
  2.) "HDEL H keys1"     -> DELETE FROM HASHES where pk="H" and
key="key1"

Further complicated denormalised key-value sets can also be normalised
back into SQL
Here is an example bash script:
CLI="./redis-cli"
$CLI set user:1:name bill
$CLI set user:1:age 33
$CLI set user:1:status member
$CLI set user:2:name jane
$CLI set user:2:age 22
$CLI set user:2:status premium

Its pretty easy to dump this data, use the unix "sort" command and
insert the following rows into the following table:
#CREATE table users (id int primary key, name varchar(32), age int,
status varchar(32));
#insert into users values(1,"bill",33,"member");
#insert into users values(2,"jane",22,"premium");

So if your data is organised properly, having a script do a "tail -f
appendonly-file | ./script | mysql" will work.

The Key is to make sure your data can be NORMALISED trivially and
systematically.

Hope my explanation was not so quick and dirty that it makes sense :)

On Jul 2, 11:52 pm, Mason Jones <masonj### @charnel.com> wrote:
 As Jeremy asked, I'm also curious why a cron wouldn't work and why
you
 say it would time out...

 One idea, though I'm not sure how difficult it might be (partially
 depending on your specific data) would be to set up Redis to use an
 append-only file, and write a script to read that file, parse it, and
 use it to write to MySQL. In theory I think that should work, though
 in practice...who knows.

 On Friday, July 2, 2010, Mr_Google <kah### @gmail.com> wrote:
 > Hi here's my problem. I need to transfer a large amount of data
(page
 > hits) from Redis to Mysql everyday. Reason I'm doing this is
because
 > redis sort (hits descending) can be slow, so I'm going to have
to use
 > mysql to store and sort hits. Data in in the range of 50 million
keys.

 > I can't use daily cron, because it'll timeout due to the large
amount
 > of data transferred at one go. Anyone got any idea / solution
they
 > implemented to this problem?

 > --
 > 

The situation is interesting to me too.
We have an analyse system to query/analyze the operation data.
Say, we want to know how many registered accounts between the
specified ages.
The only way I could considered is to scan all the accounts,  it's a
low-performance way.

So, I need migrate the redis data to mysql, and make the analyse
system access mysql.
Back to this thread, is there any easy or high-performance way to
migrate the redis data to mysql?


On Jul 3, 12:12 pm, Terry Martin <terryzm### @gmail.com> wrote:
 Your situation is interesting to me.  I'm not sure, but my initial
thought
 is to leverage the pubsub messaging paradigm to push data to MySQL,
etc.
  Perhaps a daemon that brokers transactional requests.  Again, just
my
 initial thought.  I haven't tested this, but it seems like an
interesting
 angle to explore.

 Looking forward to watching this thread.

 - Terry
  @tzmartin <http://twitter.com/tzmartin>



 On Fri, Jul 2, 2010 at 11:52 PM, Mr_Google <kah### @gmail.com>
wrote:
 > Hi here's my problem. I need to transfer a large amount of data
(page
 > hits) from Redis to Mysql everyday. Reason I'm doing this is
because
 > redis sort (hits descending) can be slow, so I'm going to have
to use
 > mysql to store and sort hits. Data in in the range of 50 million
keys.

 > I can't use daily cron, because it'll timeout due to the large
amount
 > of data transferred at one go. Anyone got any idea / solution
they
 > implemented to this problem?

 > --
 > 

Hi Roman,

I am facing the same situation with http client. For some reason JBoss
4.2.2
with Tomcat 5.5 doesn't send proper chunked response.

Can you please let me know whether your issue has been resolved? Is it a
bug
in Tomcat server or httpclient?

Any help would be quite useful.

Thanks
Praveen



RomanY wrote:
 
 We have an XML API service which splits reply XML data into chunks if
it
 is larger then certain amount of bytes. Here is the sample piece of
the
 reply:
 
 HTTP/1.1 200 OK
 Server: nginx/0.6.35
 Date: Wed, 20 Jan 2010 14:53:27 GMT
 Content-Type: text/xml;charset=UTF-8
 Transfer-Encoding: chunked
 Connection: keep-alive
 X-Powered-By: Servlet 2.4; JBoss-4.2.2.GA (build:
SVNTag=JBoss_4_2_2_GA
 date=200710221139)/Tomcat-5.5
 Connection: close
 
 1f0d
 <?xml version='1.0' encoding='UTF-8'?>
 <root>
 [...]
 <Label><![CDATA[Some character data br
 2000
 oken in the middle of the string]]></Label>
 [...]
 <root>
 
 0
 
 
 The problem is when this XML is retrieved via
 httpResponse.getEntity().getContent() I expect all chunks to be
 transformed into single XML with no service information (I'm talking
about
 some strange 2000 number appearing in the middle of the string)
 
 In fact returned content is not always correctly parsed and contains
such
 service information, which in turn makes my XML parser throw an
exception
 
 httpResponse.getEntity().isChunked() always retrieves "true"
 
 Can anyone advice on what am I doing wrong or otherwise provide
 information how to workaround such issue?
 
 Thanks,
 Roman 
 

     [
https://issues.apache.org/jira/browse...nels:all-tabpanel
]

Patrick Wendell updated AVRO-284:

     [
https://issues.apache.org/jira/browse...nels:all-tabpanel
]

Patrick Wendell updated AVRO-284:

     [
https://issues.apache.org/jira/browse...nels:all-tabpanel
]

Jeff Hammerbacher updated AVRO-284:

     [
https://issues.apache.org/jira/browse...nels:all-tabpanel
]

Jeff Hammerbacher resolved AVRO-284.

     [
https://issues.apache.org/jira/browse...nels:all-tabpanel
]

Jeff Hammerbacher reassigned AVRO-284:

    [
https://issues.apache.org/jira/browse...0#action_12879250
] 

Jeff Hammerbacher commented on AVRO-284: