GithubHelp home page GithubHelp logo

rredis's Introduction

IMPORTANT NOTICE

The rredis is defunct and I will not continue developing it. I urge all rredis users to switch to the redux package, https://github.com/richfitz/redux (also on CRAN). The redux package provides a more complete interface to Redis, with a much better (consistent, flexible, simpler) internal design.

It's easy to convert projects that rely on rredis to use redux. See the examples below.

I'll keep rredis around for a long time until I'm sure most folks that depend on it have moved over to redux, so there is no great urgency to switch. But redux is better and you should use it!

This package will be removed from CRAN soon, so switch to the redux package please.

rredis: An R client for Redis

Example

library(rredis)
redisConnect()
redisSet('foo', runif(10))
bar <- redisGet('foo')
bar
# [1] 0.93499818 0.47159536 0.30597259 0.58325228 0.41589498 0.63914212 0.34658694 0.08633471 0.18111369 0.15763507

redisMSet(list(x=pi,y='Cazart',z=runif(2)))
redisMGet(list('z','y'))
#$z
#[1] 0.1155711 0.7166137
#
#$y
#[1] "Cazart"

redisClose()

Use redisCmd to run any Redis command

This is more or less how the redux package works.

Some Redis commands have corresponding convenience wrappers with online help in the R package. Use the generic redisCmd function to run any Redis command, even ones not specifically implemented by the package. For example:

redisCmd("set","key1","foobar")
redisCmd("set","key2","abcdef")
redisCmd("bitop", "and", "dest", "key1", "key2")
#  [1] "6"
redisCmd("get", "dest")
#  [1] "`bc`ab"

# in redux:
library(redux)
con <- hiredis()
con$set("key1", "foobar")
con$get("key1")
# [1] "foobar"

# setting/getting a serialized R value in redux
con$set("key2", serialize(pi, NULL))
unserialize(con$get("key2"))
# [1] 3.14159

New in version 1.7.0

Better value exchange between R and Redis

We implemented a great suggestion by Simon Urbanek. Values obtained from Redis that are not serialized R objects are now decorated with an attribute named "redis string value." The package uses this to automatically maintain fidelity of the original Redis value through repeated download/upload cycles. Previous versions of the rredis package uploaded everything as serialized R values unless explictly told otherwise.

Consider the following interplay between the redis-cli client and R:

redis-cli set key "string value"

And now in R:

> library(rredis)
> redisConnect()
> redisGet("key")
[1] "string value"
attr(,"redis string value")     # <- note the new attribute
[1] TRUE

> redisSet("new key", redisGet("key"))

Recovering the "new key" value from the redis-cli client returns a string value now:

redis-cli get "new key"
"string value"

Before this change, users needed to be careful about converting strings to raw values in R. Now things work much more intuitively.

API change, and option to revert behavior

Set options('redis:num'=TRUE) to return Redis ":" messages as numeric values. This was the default behavior of the rredis package for all versions prior to 1.6.9. For versions of the R package later than that, redis ":" messages are returned as raw Redis string values to correspond to the data types stored in Redis. Set this option to revert to the old behavior.

Redis commands affected by this option importantly include the increment and decrement operations. This change is outlined in the following example:

> library(rredis)
> redisConnect()
> redisSet('x',charToRaw('1'))
[1] "OK"

> redisIncr('x')
[1] "2"
attr(,"redis string value")
[1] TRUE

> options('redis:num'=TRUE)
> redisIncr('x')
[1] 3

> options('redis:num'=c())
> redisIncr('x')
[1] "4"
attr(,"redis string value")
[1] TRUE

Performance

Consider using the redisSetPipeline function to enable pipelining, and also read help about options available to the redisConnect function. Also see the available options in the redisConnect function.

Status

Travis CI status [![codecov.io](https://codecov.io/github/bwlewis/rredis/coverage.svg?branch=master)](https://codecov.io/github/bwlewis/rredis?branch=master) [![CRAN version](http://www.r-pkg.org/badges/version/rredis)](http://cran.rstudio.com/web/packages/rredis/index.html)

rredis's People

Contributors

bwlewis avatar eddelbuettel avatar isomorphisms avatar kennyhelsens avatar paradigm4labs avatar pashields avatar qwzybug avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

rredis's Issues

"Error in readBin(con, raw(), 1000000L) : negative length vectors are not allowed" issued incorrectly

I got this response from redisKeys

$ redisKeys("*")
Error in readBin(con, raw(), 1000000L) :
negative length vectors are not allowed

According to bwlewis/doRedis#4, this error message should be issued when exceeding the data size limit. But this is definitely not the case of redisKeys. There was probably some error around the connection, because after I connected again, I got the correct response with no error:

$ redisConnect('myserver')
$ redisKeys("*")
[1] "jobs:1.alive.1" "x" "jobs:counter" "jobs:1.start.3" "jobs:1.alive.3"
[6] "jobs" "jobs:1" "jobs:1.start.1" "jobs:workers" "jobs:1.env"

So there should probably be a different error message issued in this case.

Question: Make scalar character values standard Redis strings by default instead of serialized R objects?

See the internal .raw function: https://github.com/bwlewis/rredis/blob/master/R/redis-internal.R#L110

The approach defined there is used almost everywhere now already with the notable exception of redis "String" operations like redisSet, redisMSet, etc.

The original intent of this package is to make R objects always the default, letting users convert them to raw values if they desire to inter-operate with other programs.

However, I don't really see the harm in relaxing that slightly for scalar character values in R. The only real chance for problems I see might be in various character encoding data being messed up.

Ideas? Comments?

`redisConnect` errors out when integer port is used.

redisConnect refuses to use integer ports (e.g. redisConnect("localhost", 6379L) will throw an error). The culprit is

stopifnot(class(port) == "numeric")

which should read

stopifnot(is.numeric(port))

(integers are numeric, but they have the class "integer").

Update package on CRAN

Hi Bryan,

Thank you for your work on rredis! I have a suggestion, can you update the CRAN package? Since that rredis has new features like the authenticated connection, it would be great that it serves the community. Thanks!

tag 1.7.1

turns out I had an issue with one of our systems that seems to have been resolved by updating to master, perhaps you should tag it as 1.7.1.

database bloat using rredis

consider the following, set a key from 3 different clients:
SET from: "abcd"

  1. from redis-cli
    127.0.0.1:6379> debug object from:cli
    Value at:00007FE24DB730A0 refcount:1 encoding:embstr serializedlength:5 lru:6561621 lru_seconds_idle:39
  2. from redis-other (almost any other client Predis.php, Redis.jl, Ioredis.js...)
    127.0.0.1:6379> debug object from:other
    Value at:00007FE24DB73120 refcount:1 encoding:embstr serializedlength:5 lru:6561636 lru_seconds_idle:31
  3. from redis-rredis
    127.0.0.1:6379> debug object from:rredis
    Value at:00007FE24780D4C0 refcount:1 encoding:embstr serializedlength:35 lru:6561646 lru_seconds_idle:26

Is this 7x bloat necessary? And is there a workaround in the form of an option/setting?

Blocking operations can hang

Blocking operations like redisBRPop, redisBLPop can hang when communication with the Redis server is interrupted. This is a bug in the internal burn function.

redisHMSet not working

Using rredis 1.6.3, redisHMSet doesn't seem to work:

redisHMSet('A', list(x=1,y=2,z=3))
[1] TRUE
redisHKeys('A')
NULL
redisHSet('A', 'x', 1)
[1] TRUE
redisHKeys('A')
[[1]]
[1] "x"

Starting from an empty Redis:

redisFlushAll()
[1] "OK"
redisKeys()
NULL
redisHMSet('A', list(x=1,y=2,z=3))
[1] TRUE
redisKeys()
[1] 120 1

It also gives errors when the list of fields and values is not exactly 3 elements that make me suspect something is going wrong internally:

redisHMSet('A', list(x=1))
Error in doTryCatch(return(expr), name, parentenv, handler) :
ERR wrong number of arguments for 'hmset' command
redisHMSet('A', list(x=1,y=2))
Error in doTryCatch(return(expr), name, parentenv, handler) :
ERR wrong number of arguments for 'hmset' command
redisHMSet('A', list(x=1,y=2,z=3))
[1] TRUE
redisHMSet('A', list(x=1,y=2,z=3,w=4))
Error in doTryCatch(return(expr), name, parentenv, handler) :
ERR wrong number of arguments for HMSET

Insert multiple list items

It seems like inserting multiple list elements is not possible with the current redisLPush/redisRPush implementation (despite this being supported in redis).

Would it make sense to implement this (e.g. by breaking a vector of values into individual list items and inserting them)? A flag could be used to enable this option (to provide backwards compatibility).

Question: R object size limitation of 512 MB

Hi

Just a quick question. In the vignette, the size limitation of a R object is indicated below.

  • Redis values are limited to 512 MB. R objects that exceed this size once serialized can’t be stored
    in Redis.

When I checked its data types, however, the maximum number of elements of lists, sets and hashes is 2^32 - 1 (http://redis.io/topics/data-types) while the maximum size of its key and value is 512 MB.

So I guess, for example, if a R vector is created, as long as each value (and the key) doesn't exceed 512 MB, it'd be possible to add.

Please inform if the guess is incorrect.

Regards
Jaehyeon

Redis Serialization Protocol support

Hi Bryan,

With the help of your package is there any way to perform RESP serialization of an R dataframe?

If not do you think it's good idea to include such functionality.

Thank you,
Sergey.

Race condition in bulk replies?

I think I am running into some sort of race condition in the hash get functions that receive bulk replies. When I make a request for a lot of data using either HMGET or HGETALL, I can get the following error:

Error in if (nchar(l) < 2) { : argument is of length zero
Calls: redisCmd ... replicate -> sapply -> lapply -> FUN -> .getResponse

I am running rredis 1.6.6 against Redis server version 2.4.10. The code I am running to test this is:

redisConnect('redis host');
a = redisCmd('HGETALL', 'test_redis_bug', raw=TRUE)
redisClose(); 

test_redis_bug is a hash containing 400 vectors of 51,698 doubles and a single vector of 51,698 character strings, all of which have been manually run through serialize() before storage (this is also why they are being retrieved in raw mode).

For small enough requests, I will never see this problem, while for large enough requests it happens every time. There is a middle ground where it happens inconsistently. “Large” seems to vary depending on the speed of the connection to the Redis server, but over our metro Ethernet (which runs at tens of megabits per second), I see consistent problems on this 400 x 50,000 example.

I did a little bit of poking around and when I see this error, l is character(0). If I read the next line, it becomes an empty string, and if I read the line after that I get "$<some length>", as expected.

I suspect a race condition because putting a Sys.sleep(1) at the top of .getResponse() fixes everything.

I was able to improve the situation by looping the l <- readLines(con=con, n=1) until it got something real, but I then less consistently see an error where readLines() ran out of material to read before reading an end-of-line, which will be followed by a garbled message on my next pull (which I theorize is data from the first request that didn’t get there in time).

It's been frustrating to try to figure out more because trying to look at anything that happens before the error seems to introduce enough delay that the race isn't triggered. Let me know if I can do anything to help in tracking this down.

strange characters set by R

Using redis to exchange data between multiple clients I found problems setting a value in R (using rredis) and reading it back from the redis client or from node: the value I get back is not the one I put in, but a string of "strange characters".
It works perfectly only if I get the value back in R or if I read a value stored by the redis client.

  1. In R
> library(rredis)
> redisConnect(timeout=300)
> redisSet('x',10)
[1] "OK"
> redisGet('x')
[1] 10
  1. In redis-cli
redis 127.0.0.1:6379> get x
"X\n\x00\x00\x00\x02\x00\x03\x00\x00\x00\x02\x03\x00\x00\x00\x00\x0e\x00\x00\x00\x01@$\x00\x00\x00\x00\x00\x00"
redis 127.0.0.1:6379> set x 5
OK
  1. In R
> redisGet('x')
[1] "5"

I get the same results using rredis 1.6.7 either with R 3.0.0-64bit (on OSX 10.8) and R 2.15.2-64bit (on Windows 7). Redis version is 2.6.12 (on OSX 10.8), but I tried also with a free redis on-line service.

Multiple databases?

Hello!
Wanted to use this, but i need to be able to specify which database to connect to.
Seems like this is currently not possible.
P

Error setting TCP_NODELAY

On startup, I am consistently getting a fail on the first attempt to connect, but it works on the subsequent attempt(s):

library(rredis)
ref <- redisConnect(returnRef=TRUE)
# Error in .openConnection(host = host, port = port, nodelay = nodelay,  : 
#   Error setting TCP_NODELAY
ref <- redisConnect(returnRef=TRUE)

I get the error with or without returnRef, with non-default host/port, and with timeout specified or not. I've started R with --vanilla --no-site-file --no-init-file to ensure I'm not poisoning it somehow.

It does not fail with nodelay=FALSE.

> sessionInfo()
R version 3.2.3 (2015-12-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 15.10

locale:
 [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8       
 [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
 [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C          
[10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] rredis_1.7.0

loaded via a namespace (and not attached):
[1] compiler_3.2.3 tools_3.2.3   

Very slow performance with a local server

Hey,

I'm having a performance issue problem with running Redis (2.6) on Ubuntu 12.04.

After filling a database by executing the following command in a redis-cli

eval "for i=1,100 do redis.call('sadd', i, 1); end; return 1" 0

Which fills a 100 sets with one element. Then in R running the following commands:

library(rredis)
redisConnect()

system.time (lapply(1:100, function(i) { redisSMembers(as.character(i)) }))
user system elapsed
0.104 0.044 4.000

system.time (lapply(1:100, function(i) { runif(1) }))
user system elapsed
0.000 0.000 0.001

The operation takes about 4 second elapsed time, which seems a bit long especially with regard to the user time ;-). Happens consistently as I just iterated over 400k sets with two Redis operations which lasted about 7,5 hours (26060.933 seconds to be precise).

redis-benchmark results in acceptable performance, so I'm kinda at a loss what's going here! It works fine on my Mac, happens with both 32 and 64 bit R versions (latest major release).

redisTTL returning -2

redisTTL() function is returning undocumented value of -2 for a non-existent key. It should probably return -1 or the documentation should contain the description of another return value of -2.

redisIncrBy broken

Hi,

using rredis 1.6.9 on R 3.1.2 I get the following with the example of redisIncrBy

> redisSet('x','5')
[1] "OK"
> redisIncrBy('x','3')
Error in doTryCatch(return(expr), name, parentenv, handler) : 
  ERR value is not an integer or out of range

The problem might be an encoding issue (?). Because that is what I get in redis:

redis 127.0.0.1:6379> get x
"X\n\x00\x00\x00\x02\x00\x03\x01\x02\x00\x02\x03\x00\x00\x00\x00\x10\x00\x00\x00
\x01\x00\x04\x00\t\x00\x00\x00\x015"
redis 127.0.0.1:6379> set x "5"
OK
redis 127.0.0.1:6379> get x
"5"
redis 127.0.0.1:6379> incrby x 1
(integer) 6

Ciao,
Stefan

'raw' argument across functions

Some get commands have a 'raw' argument:

redisGet(key, raw = FALSE)
redisMGet(keys, raw = FALSE)
redisHGet(key, field, raw = FALSE)

Some do not:

redisHGetAll(key)
redisHMGet(key, fields)

We've taken to using redisCmd() to call a base Redis command when we need a raw version that isn't provided, but it would be nice if this wasn't necessary. Is there some rationale behind which commands have raw versions and which do not?

redisHMSet applied charToRaw on character list

instead of serialization, redisHMSet applied charToRaw on characters and lost information. for example

> letters
 [1] "a" "b" "c" "d" "e" "f" "g" "h" "i" "j" "k" "l" "m" "n" "o" "p" "q" "r" "s" "t" "u" "v" "w" "x" "y" "z"
> redisHMSet("test", list(letters = letters))
[1] "OK"
Warning message:
In charToRaw(values[[j]]) :
  argument should be a character vector of length 1
all but the first element will be ignored
> redisHGet("test", "letters")
[1] "a"
> 

the problem is below line

    > redisHMSet
    function (key, values) 
    {
        a <- c(alist(), list(.raw("HMSET")), list(.raw(key)))
        fieldnames <- lapply(names(values), charToRaw)
        for (j in 1:length(values)) {
            a <- c(a, fieldnames[j])
<<<<   if (is.character(values[[j]])) <<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                a <- c(a, list(charToRaw(values[[j]])))
            else (a <- c(a, list(.cerealize(values[[j]]))))
        }
        do.call(".redisCmd", a)
    }

Is this work as design or an issue?

timeout in redisConnect()

1.6.7 introduces a timeout parameter for redisConnect() that is giving me trouble with R 2.15.1 under OS X 10.8 (Mountain Lion). The issue appears to be that timeouts that large just don't work on Mountain Lion, as the call consistently fails with timeout > 100,000,000, while my colleague's 10.7 machine and our Ubuntu 10.04 servers are fine. I can clearly just set the timeout in the call, but it seems like setting the default to 100,000,000 (~3 years) might be nice. I'm not sure if socketConnection might also allow 0 to mean "no timeout".

[R.app GUI 1.52 (6188) x86_64-apple-darwin9.8.0]

> library(rredis)

> redisConnect("redis-equinix")
Error in socketConnection(host, port, open = "a+b", blocking = TRUE, timeout = timeout) : 
  cannot open the connection
In addition: Warning message:
In socketConnection(host, port, open = "a+b", blocking = TRUE, timeout = timeout) :
  redis-equinix:6379 cannot be opened

> redisConnect("redis-equinix", timeout=100000000L)

> redisConnect("redis-equinix", timeout=100000001L)
Error in socketConnection(host, port, open = "a+b", blocking = TRUE, timeout = timeout) : 
  cannot open the connection
In addition: Warning message:
In socketConnection(host, port, open = "a+b", blocking = TRUE, timeout = timeout) :
  redis-equinix:6379 cannot be opened
>

writeBin() causes "can only write to a binary connection" error

I encountered this error caused by this line in redis-internal.R

writeBin(.raw(hdr), con)

which is supposedly thrown if con is not binary, and not NULL either. I don't see any how it can happen, looking through the code of rredis.

And in fact, when I call rredis::redisGet locally there was no issue. Yet the error popped out for all the calls via OpenCPU.

It took me quite a while and still have no clue how or where con can become a non-binary connection. I don't think it is the problem of either rredis or OpenCPU. I must have missed some very basic thing...

redisIncr problem

In the manual it says that calling redisIncr after redisSet could be easily done. The example from the manual does not seems to work though:

> redisSet('foo', 1)
[1] "OK"
> redisIncr('foo')
Error in .getResponse() : ERR value is not an integer or out of range

If I set the integer value from command line (with redis-cli) the redisIncr works like a charm so it seems that the redisSet does not save the value as integer but as R object? From command line the saved foo (set to: 1) looks like:

"X\\x00\x00\x00\x02\x00\x02\\x00\x00\x02\x03\x00\x00\x00\x00\x0e\x00\x00\x00\x01?\xf0\x00\x00\x00\x00\x00\x00"

If I set the value with charToRaw, it seems to work:

> redisSet('foo', charToRaw('1'))
[1] "OK"
> redisIncr('foo')
[1] 2

The documentation might be updated if my findings look right.

storing large objects

rredis seems to be ok for small sized, data, but when I try to store a large object the connection seems to die:

> redisSet("test", rnorm(1e8))
Error in doTryCatch(return(expr), name, parentenv, handler) : 
  ERR Protocol error: invalid bulk length
In addition: Warning messages:
1: In writeBin(v, con) : problem writing to connection
2: In writeBin(.raw("\r\n"), con) : problem writing to connection

Any idea what is going on?

PLANNED CRAN PACKAGE UPDATE

The next planned CRAN package update is September 15th, 2013.

If there is something you want in the package let me know before then!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.