whatyouhide / redix

Fast, pipelined, resilient Redis driver for Elixir. 🛍

Redix

Fast, pipelined, resilient Redis client for Elixir.

Redix is a Redis and Valkey client written in pure Elixir with focus on speed, correctness, and resiliency (that is, being able to automatically reconnect to Redis in case of network errors).

This README refers to the main branch of Redix, not the latest released version on Hex. Make sure to check the documentation for the version you're using.

Features

• Idiomatic interface for sending commands to Redis
• Pipelining
• Resiliency (automatic reconnections)
• Pub/Sub
• SSL
• Redis Sentinel

Installation

Add the :redix dependency to your mix.exs file. If you plan on connecting to a Redis server over SSL you may want to add the optional :castore dependency as well:

defp deps do
  [
    {:redix, "~> 1.1"},
    {:castore, ">= 0.0.0"}
  ]
end

Then, run mix deps.get in your shell to fetch the new dependencies.

Usage

Redix is simple: it doesn't wrap Redis commands with Elixir functions. It only provides functions to send any Redis command to the Redis server. A Redis command is expressed as a list of strings making up the command and its arguments.

Connections are started via start_link/0,1,2:

{:ok, conn} = Redix.start_link(host: "example.com", port: 5000)
{:ok, conn} = Redix.start_link("redis://localhost:6379/3", name: :redix)

Commands can be sent using Redix.command/2,3:

Redix.command(conn, ["SET", "mykey", "foo"])
#=> {:ok, "OK"}
Redix.command(conn, ["GET", "mykey"])
#=> {:ok, "foo"}

Pipelines are just lists of commands sent all at once to Redis for which Redis replies with a list of responses. They can be used in Redix via Redix.pipeline/2,3:

Redix.pipeline(conn, [["INCR", "foo"], ["INCR", "foo"], ["INCRBY", "foo", "2"]])
#=> {:ok, [1, 2, 4]}

Redix.command/2,3 and Redix.pipeline/2,3 always return {:ok, result} or {:error, reason}. If you want to access the result directly and raise in case there's an error, bang! variants are provided:

Redix.command!(conn, ["PING"])
#=> "PONG"

Redix.pipeline!(conn, [["SET", "mykey", "foo"], ["GET", "mykey"]])
#=> ["OK", "foo"]

Resiliency

Redix is resilient against network errors. For example, if the connection to Redis drops, Redix will automatically try to reconnect periodically at a given "backoff" interval. Look at the documentation for the Redix module and at the "Reconnections" page in the documentation for more information on the available options and on the exact reconnection behaviour.

Redis Sentinel

Redix supports Redis Sentinel out of the box. You can specify a list of sentinels to connect to when starting a Redix (or Redix.PubSub) connection. Every time that connection will need to connect to a Redis server (the first time or after a disconnection), it will try to connect to one of the sentinels in order to ask that sentinel for the current primary or a replica.

sentinels = ["redis://sent1.example.com:26379", "redis://sent2.example.com:26379"]
{:ok, primary} = Redix.start_link(sentinel: [sentinels: sentinels, group: "main"])

Terminology

Redix doesn't support the use of the terms "master" and "slave" that are usually used with Redis Sentinel. I don't think those are good terms to use, period. Instead, Redix uses the terms "primary" and "replica". If you're interested in the discussions around this, this issue in the Redis repository might be interesting to you.

Pub/Sub

A Redix.PubSub process can be started via Redix.PubSub.start_link/2:

{:ok, pubsub} = Redix.PubSub.start_link()

Most communication with the Redix.PubSub process happens via Elixir messages (that simulate a Pub/Sub interaction with the pub/sub server).

{:ok, pubsub} = Redix.PubSub.start_link()

Redix.PubSub.subscribe(pubsub, "my_channel", self())
#=> {:ok, ref}

Confirmation of subscriptions is delivered as an Elixir message:

receive do
  {:redix_pubsub, ^pubsub, ^ref, :subscribed, %{channel: "my_channel"}} -> :ok
end

If someone publishes a message on a channel we're subscribed to:

receive do
  {:redix_pubsub, ^pubsub, ^ref, :message, %{channel: "my_channel", payload: "hello"}} ->
    IO.puts("Received a message!")
end

Using Redix in the Real World™

Redix is low-level, but it's still built to handle most things thrown at it. For many applications, you can avoid pooling with little to no impact on performance. Read the "Real world usage" page in the documentation for more information on this and pooling strategies that work better with Redix.

Contributing

To run the Redix test suite you will have to have Redis running locally. Redix requires a somewhat complex setup for running tests (because it needs a few instances running, for pub/sub and sentinel). For this reason, in this repository you'll find a docker-compose.yml file so that you can use Docker and docker-compose to spin up all the necessary Redis instances with just one command. Make sure you have Docker installed and then just run:

docker-compose up

Now, you're ready to run tests with the $ mix test command.

License

Redix is released under the MIT license. See the license file.

redix's Issues

No function clause match on closed connection

I'm using Redix 0.6.1:

 "redix": {:hex, :redix, "0.6.1", "20986b0e02f02b13e6f53c79a1ae70aa83147488c408f40275ec261f5bb0a6d0", [:mix], [{:connection, "~> 1.0", [hex: :connection, repo: "hexpm", optional: false]}], "hexpm"},

I received the following error (with some info elided):

erlang error: {{:function_clause, [{Redix.Connection, :handle_info, [{:receiver, #PID<0.4376.211>, {:tcp_closed, #Port<0.10631>}}, %Redix.Connection{backoff_current: 500, opts: [socket_opts: [], sync_connect: false, backoff_initial: 500, backoff_max: 30000, log: [disconnection: :error, failed_connection: :error, reconnection: :info], exit_on_disconnection: false, host: "...", port: ..., password: "..."], receiver: nil, shared_state: nil, socket: nil}], [file: 'lib/redix/connection.ex', line: 223]}, {Connection, :handle_async, 3, [file: 'lib/connection.ex', line: 810]}, {:gen_server, :try_dispatch, 4, [file: 'gen_server.erl', line: 601]}, {:gen_server, :handle_msg, 5, [file: 'gen_server.erl', line: 667]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 247]}]}, {:gen_server, :call, [#PID<0.550.0>, {:commands, [[...]], #Reference<0.0.44564487.222310>}, 5000]}}

However, it does appear that handle_info would match. So I'm not sure I'm reading it right.

def handle_info({:receiver, pid, {:tcp_closed, socket}}, %{receiver: pid, socket: socket} = state) do

It seems like it's something that Redix is trying to handle.

Redix.command raises errors

Since 791c078, Redix.command raises errors instead of returning them. This change is not reflected in the README, which still says that command! raises and command returns`. I'd actually prefer that the change be rolled back, but if not, the documentation should be brought up to date.

Upgrade from 0.5.1 to 0.5.2 broke our app

Hello,

looks like from 0.5.2 theres some error which ignores selecting correct database. We use simple:

worker(Redix, [[database: 1], [name: :redix]]),

And with 0.5.2 it ignores database: 1 configuration.

Downgrading to 0.5.1 fixes the issue.

connection dependency problem

We have a problem making all dependencies work well togheter after installing redix, we came up with a solution, maybe the problem is with the "connection" dependency.

defp deps do
    [{:phoenix, "~> 1.2.1"},
     {:phoenix_pubsub, "~> 1.0"},
     {:phoenix_ecto, "~> 3.0"},
     {:postgrex, ">= 0.0.0"},
     {:mariaex, ">= 0.0.0"},
     {:phoenix_html, "~> 2.6"},
     {:phoenix_live_reload, "~> 1.0", only: :dev},
     {:gettext, "~> 0.11"},
     {:cowboy, "~> 1.0"},
     {:httpoison, "~> 0.9"},
     {:ratio, "~> 1.0"},
     {:dialyxir, "~> 0.3", only: [:dev]},
     {:credo, "~> 0.4", only: [:dev, :test]},
     {:timex, "~> 3.0"},
     {:cachex, "~> 2.0"},
     {:redix, "~> 0.4"}]
  end

with this setup many dependencies are kept back: for example phoenix_ecto, postgrex and mariaex. We found that the problem could be solved by relaxing the connection dependency like this:

primait@5aeef83

the test suite passes.

make disconnect error log more clear

connetion.ex line54

def disconnect({:error, reason} = _error, state) do
    Logger.error ["Disconnected from Redis (#{Utils.format_host(state)}): ",
                  :inet.format_error(reason)]

sometime :inet.format_error/1 will return unknown POSIX error for example :inet.format_error(:tcp_closed) will return unknown POSIX error
Maybe it's better to inspect the original reason :)

def disconnect({:error, reason} = _error, state) do
    Logger.error ["Disconnected from Redis (#{Utils.format_host(state)}): #{inspect reason}",
                  :inet.format_error(reason)]

Connection leak

Hey there!

So on a project at work, I pulled in Redix (v0.4.0) as our Redis client library. We call it the dispatcher, and in essence it is what it sounds like. It receives incoming events from various sources (HTTP, SQS, etc.), glues together configuration related to those events and the customer they apply to, and then executes Lua scripts via a scripting engine (a separate application built in C and Go). Lua scripts in turn can make calls to various "blessed" services via a Service.function({ args }) convention, which are executed as HTTP requests via the dispatcher. We use Redis for storing a ring buffer of script logs for each customer, as well as the node registry for the cluster (our environment is built on Kubernetes, so cluster membership is dynamic). Events come in at a rate of 30-100 per second. The script logging goes through a single GenServer, which maintains a Redix connection process, and relies on Redix to deal with reconnecting if the network goes down, etc. If some kind of hard exception occurs, the process goes down, and will be restarted. The cluster membership is monitored by a dedicated process on each node, which grabs the node list, and the nodes from the registry, does a diff, attempts to connect to registered processes it is not yet connected to, and removes any nodes in the registry that are unreachable. A connection is also established at startup and shutdown (to register and unregister the node). We're using sync_connect: true everywhere, and exit_on_disconnection: true in the cluster registry module (since we're only spawning a connection every 30 seconds or so and closing it when the command has been run, we don't care about retries, we just wait until the next interval passes.

Sorry for the brain dump, but I wanted to make sure you had the context, since it may be important for troubleshooting this.

Somehow, connections are leaking. We know this because we eventually hit the max connection limit. From a code perspective we're always calling Redix.stop/1 on the Redix process after a command has been executed (or in the case of the script logging process, in terminate/2). As far as I know, there should be no way that our application is leaking connections. To further cement this, switching over to :eredis works fine, no leaks.

When I was poking around on one of the nodes, I didn't have access to :observer, so I was pulling the process list and mapping over it with Process.info/2 on the last 30-50 processes created. I noticed that all of them were Redix.Connection.Receiver processes, which I see hold a TCP socket. The interesting thing was that when I checked it's parent process, which I believe is Redix.Connection, that process was no longer alive. Now this was with the most recent version prior to 0.4.0, I haven't checked with it yet, but since the behaviour hasn't changed, I assume that the same problem is occurring. My assumption is that the Receiver process isn't linked to it's parent Connection process (and I assume this because when I was poking around, there was no link to the parent's PID in the Receiver process links list), so when the parent crashes for some reason, the Receiver is left orphaned.

We've had to switch to using :eredis for now, but I'd like to switch back to Redix once this is figured out. If I can be of any help, just let me know!

Release new version with latest updates

Hi! I see there's some additions on master that have not been released since May last year (such as Redix.child_spec/1). Could you release a new version please? :)

Remove start_link/1 and only provide start_link/2

As @lexmag suggested, it's confusing to accept options for both Redis and Connection in the same argument for start_link/1. Also, it's confusing to be able to pass a Redis URI and then override parts of that URI with options in start_link/2 (e.g., Redix.start_link("redis://localhost:6380", port: 6379)).

We have to only provide start_link/2:

def start_link(uri_or_redis_opts \\ [], connection_opts \\ [])

The only problem I see with this is that while it works flawlessly with the :host, :port, :username, and :password Redis options (that can all be specified via the URI), it doesn't work so well with the other options that Redix accepts (:max_reconnection_attempts, :backoff). Should they go along the connection opts, so that we can keep just two arguments to start_link/2 and we can allow those options to be tweaked even if the first argument is a URI? Wdyt @lexmag?

Initializing startup options in app start

Hello,

I am experiencing a trouble with the following code. I cant start the child. Any suggestions what I am doing wrong? Thank you.

`def start(_type, _args) do
import Supervisor.Spec, warn: false

#config :redis_client, conn: "redis://localhost:32768", name: [name: "redix"]

conn = Application.get_env(:redis_client, :conn)
name = Application.get_env(:redis_client, :name)

children = [
  worker(Redix, [conn, name])
]

opts = [strategy: :one_for_one, name: RedisClient.Supervisor]
Supervisor.start_link(children, opts)

end`

Cannot get timeout options to work

Similar to this issue: #28

I am having some difficulty passing the :timeout value to Redix.command.

Here is how I am calling Redix.command:

 case Redis.command(save_confirm_uuid, timeout: 1800000) do
  {:ok, message} -> Logger.info message
  {:error, error} -> Logger.error error
end

... and in another module, called Redis, I'm forwarding the call to my Redix worker. You'll see I'm taking an approach right out of your "Real World" docs.

def command(command, options) do
  Redix.command(:"redix_#{random_index()}", command, options)
end

If I inspect options within my Redis.command method, I see what I expect to see. However, when I issue the redis-cli commands to see that my values are TTL'd correctly:

127.0.0.1:6379> keys *
1) "a3334454-144d-4f8d-9e59-3af7291f49ea"
127.0.0.1:6379> ttl a3334454-144d-4f8d-9e59-3af7291f49ea
(integer) -1
127.0.0.1:6379>

... I discover that they are still infinite TTLs. Am I passing anything wrong here?

I've also tried directly calling Redix.command(<conn_name_here>, ~w(SET foo bar), timeout: 1800000) and I have the same issue.

Any ideas? Thanks! :)

Close connection

Hello guys

I'm new in Elixir and I cannot close the connection in my request.

{:ok, conn} = Redix.start_link()
Redix.command(conn, ~w(INCR mykey))

# Want to close the connection here. How can I do this?

Version 0.5.2 causes issues with Redix.PubSub

Hello,

I've updated redix from 0.5.1 to 0.5.2 in a library I'm working on. Here you can find the branch.

The library uses both redix and redix_pubsub: it uses Redis as a data store and also publishes and subscribes to pub-sub messages. I was actually planning to update to redix 0.6.0, but redix_pubsub 0.2.0 is locked to redix ~> 0.5.0.

Everything works fine with redix 0.5.1, but it looks like I can't receive pub-sub messages when using 0.5.2.
An example of the test failures is on Travis, and I've verified the same issue manually.

Is this already on your radar? i.e. are you already working on an update to redix_pubsub that addresses the issue?
If not, do you need help and would you accept a pull request?

Thank you

(Redix.Protocol.ParseError) not a valid integer: "-"

I execute MGET command with the following code:
{:ok, result} = Like4uElixir.RedixPool.command(["MGET"] ++ redis_keys(tasks))
And that's what I see in log. As I understand, the error is not always reproducable :(

Request: GET /client/instagram/follows.json
** (exit) exited in: :gen_server.call(#PID<0.26019.5>, {:commands, [["MGET", "igtasks-732855-60", "igtasks-732855-300", "igtasks-732855-900", "igtasks-732855-3600", "igtasks-732855-14400", "igtasks-732855-86400", "IgFollowTask-lock-732855", "igtasks-711017-60", "igtasks-711017-300", "igtasks-711017-900", "igtasks-711017-3600", "igtasks-711017-14400", "igtasks-711017-86400", "IgFollowTask-lock-711017", "igtasks-88983-60", "igtasks-88983-300", "igtasks-88983-900", "igtasks-88983-3600", "igtasks-88983-14400", "igtasks-88983-86400", "IgFollowTask-lock-88983", "igtasks-701622-60", "igtasks-701622-300", "igtasks-701622-900", "igtasks-701622-3600", "igtasks-701622-14400", "igtasks-701622-86400", "IgFollowTask-lock-701622", "igtasks-739219-60", "igtasks-739219-300", "igtasks-739219-900", "igtasks-739219-3600", "igtasks-739219-14400", "igtasks-739219-86400", "IgFollowTask-lock-739219", "igtasks-737275-60", "igtasks-737275-300", "igtasks-737275-900", "igtasks-737275-3600", "igtasks-737275-14400", "igtasks-737275-86400", "IgFollowTask-lock-737275", "igtasks-726044-60", "igtasks-726044-300", "igtasks-726044-900", "igtasks-726044-3600", "igtasks-726044-14400", "igtasks-726044-86400", ...]]}, 5000)
** (EXIT) an exception was raised:
** (Redix.Protocol.ParseError) not a valid integer: "-"
lib/redix/protocol.ex:144: Redix.Protocol.parse_integer/1
lib/redix/protocol.ex:149: Redix.Protocol.parse_bulk_string/1
lib/redix/protocol.ex:190: Redix.Protocol.take_n_elems/3
lib/redix/protocol.ex:105: Redix.Protocol.parse_multi/2
lib/redix/connection/receiver.ex:76: Redix.Connection.Receiver.new_data/2
lib/redix/connection/receiver.ex:55: Redix.Connection.Receiver.handle_info/2
(stdlib) gen_server.erl:615: :gen_server.try_dispatch/4
(stdlib) gen_server.erl:681: :gen_server.handle_msg/5

Any interest in my implementing a `no_wait_for_reply` opt?

Would anyone be interested in me implementing a no_wait_for_reply boolean opt to command/pipeline, that if enabled, prefixes the command with CLIENT REPLY SKIP, doesn't bother enqueueing it to the SharedState, and returns immediately that the :gen_tcp.send succeeds?

I was thinking of implementing it for an application that will have a very high rate of of key writes (where the application doesn't care enough to do anything if any of them fail) -- doesn't seem much point spending resources having redis respond to each one and redix diligently matching the reply to the request when I'll be ignoring the response anyway.

Edit: didn't get any feedback on this, so went ahead and implemented it in #87

Clustering support

How would I go about handling clustering with redix? My initial thought would be to just maintain a pool of Redix GenServers, each with a different connection string. Is that a valid approach or am I missing something?

Redix crash by Redis connection closed by unknow reason

19:34:36.241 [error] Disconnected from Redis (11.1.1.11:6379): unknown POSIX error
19:34:36.243 [error] GenServer #PID<0.1963.0> terminating
** (FunctionClauseError) no function clause matching in Redix.Connection.Auth.auth/2
    (redix) lib/redix/connection/auth.ex:32: Redix.Connection.Auth.auth(%{opts: [host: "11.1.1.11", port: 6379, password: :redacted, socket_opts: [], backoff: 2000], receiver: #PID<0.1964.0>, reconnection_attempts: 0, socket: #Port<0.22764>, tail: ""}, :redacted)
    (redix) lib/redix/connection/auth.ex:18: Redix.Connection.Auth.auth_and_select_db/1
    (redix) lib/redix/connection.ex:38: Redix.Connection.connect/2
    (connection) lib/connection.ex:742: Connection.connect/3
    (stdlib) gen_server.erl:615: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:681: :gen_server.handle_msg/5
    (stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
Last message: {:timeout, #Reference<0.0.524289.138217>, Connection}
State: %{opts: [host: "11.1.1.11", port: 6379, password: :redacted, socket_opts: [], backoff: 2000], receiver: #PID<0.1964.0>, reconnection_attempts: 0, socket: nil, tail: ""}

Redix.command/3 shouldn't throw if EVAL returns an error

Redix.command/3 shouldn't throw if eval fails with various issues like key missing, script invalid or value not expected.

It should try to stick to {:ok | :error, value} tuples since it's sans !.

Real World Page gives 404

https://github.com/whatyouhide/redix#using-redix-in-the-real-world

It points to https://hexdocs.pm/redix/real-world-usage.html

potential nosql injection with recommended command execution elixir syntax "~w(SET mykey foo)"

Elixir syntax for creating lists of words could become potential injection point if used in production with no user input escaping.

Example:
Redix library user follows how-to examples and using syntax like:
~w(SET mykey foo)

At some point he's using string interpolation like:
~w(SET mykey #{foo_var})
where foo_var is user input.

Given there is no proper escaping of user input, attacker could successfully execute the following:
assuming foo_var = "1 DEL mykey"
~w(SET mykey #{foo_var})
becomes:
["SET", "mykey", "1", "DEL", "mykey"]

Since Redis usually treated as "safe to NoSQL injections" by engineers (http://redis.io/topics/security), that beautiful syntax could have unexpected outcome.

So that's not a security hole in the library itself. But I'd suggest to edit how-to section accordingly.

ERR unknown command 'SCRIPT LOAD'

Is the SCRIPT LOAD command not supported?

{:ok, hash} = Redix.command(conn, ["SCRIPT LOAD", my_script])

** (Redix.Error) ERR unknown command 'SCRIPT LOAD'
                 (redix) lib/redix.ex:334: Redix.command/3

Add SSL support

Thoughts on supporting ssl? We're looking to add SSL support to redix. Even if you guys don't have a solution, any help would be appreciated.

Register :redix application not required with elixir 1.4

defp application() do
  [applications: [:logger, :redix]]
end

is no longer necessary as of elixir 1.4 (see https://sergiotapia.me/application-inference-in-elixir-1-4-ae9e43e90301)

Transactions do not seem to work

Hi there,
it is me again.

So I have trying to get transactions to work but every time I try to execute it, it says "ERR EXEC without MULTI".

The commands I send to Redis look like this:

MULTI
SET foo "bar"
HSET ...
...
EXEC

Of course I pass them to Redix seperately. So every command is a single Redix.command(...).

How does Redix handle transactions? Is it possible to do some?

Thanks in advance!

uri documentation

Hi there,

could you please add an example to the README.md how to configure the REDIS_URL?

I plan to use redis as a background worker.

worker(Redix, [[], [name: :redix]])

Would be great if you could provide a sample configuration.

config :redix,
  uri: System.get_env("REDIS_URL"),

If I'd need connection-pooling, I assume the configuration name needs to be setup accordingly:

config: :redix_1,...

Thanks for your help!

space in a string

Here is my code

    {:ok, conn} = Redix.start_link()
    t = "hello there"
    Redix.command(conn, ~w(RPUSH emails #{t}))

I was expecting "hello there" but I got

2269) "hello"
2270) "there"`

Forbid *SUBSCRIBE commands

Should we forbid these commands explicitly since we don't support them? I'm leaning towards 👍 \cc @lexmag

Multi-word hash setting not working

At the Redis CLI, if you try to pass a multi-word value, you have to quote it

This fails:

127.0.0.1:6379> hset abc123 status my status
(error) ERR wrong number of arguments for 'hset' command

But this works:

127.0.0.1:6379> hset abc123 status "my status"
(integer) 0
127.0.0.1:6379> hgetall abc123
1) "status"
2) "my status"

This works too:

127.0.0.1:6379> hset abc123 status 'my status'
(integer) 0
127.0.0.1:6379> hgetall abc123
1) "status"
2) "my status"

However, I can't seem to make this work with Redix.

iex(1)> {:ok, conn} = Redix.start_link("redis://localhost:6379/1", name: :redix)
{:ok, #PID<0.337.0>}
iex(2)> Redix.command(conn, ~w(HSET abc123 status my status))
{:error,
 %Redix.Error{message: "ERR wrong number of arguments for 'hset' command"}}
iex(3)> Redix.command(conn, ~w(HSET abc123 status "my status"))
{:error,
 %Redix.Error{message: "ERR wrong number of arguments for 'hset' command"}}
iex(4)> Redix.command(conn, ~w(HSET abc123 status 'my status'))
{:error,
 %Redix.Error{message: "ERR wrong number of arguments for 'hset' command"}}
iex(5)> Redix.command(conn, ~w(HSET abc123 status \"my status\"))
{:error,
 %Redix.Error{message: "ERR wrong number of arguments for 'hset' command"}}
iex(6)> Redix.command(conn, ~w(HSET abc123 status \'my status\'))
{:error,
 %Redix.Error{message: "ERR wrong number of arguments for 'hset' command"}}

What can I do?

Is there a way to get numbers instead of strings?

iex(1)> {:ok, conn} = Redix.start_link()
{:ok, #PID<0.261.0>}
iex(2)> Redix.command(conn, ~w(SET testing 1))
{:ok, "OK"}
iex(3)> Redix.command(conn, ~w(GET testing))  
{:ok, "1"}

Is there a way to get 1 when I run the GET testing? Some operations do the parsing, like INCR.

Timeout value is not respected

In my application, Redix always timeout irrespective of the timeout value I set. I get the following exception.

** (stop) exited in: :gen_server.call(#PID<0.14248.2>, {:commands, [["smembers", "shooting _"]]}, 600000)
    ** (EXIT) time out
    (stdlib) gen_server.erl:212: :gen_server.call/3
    lib/redix.ex:221: Redix.command/3
    (poolboy) src/poolboy.erl:76: :poolboy.transaction/3
    (dedup) lib/iindex.ex:9: InvertedIndex.get_attrs/1
    (elixir) lib/enum.ex:1088: Enum."-map/2-lists^map/1-0-"/2
    (elixir) lib/enum.ex:1088: Enum."-map/2-lists^map/1-0-"/2
    (dedup) lib/tagger.ex:10: Tagger.tag/2
    (dedup) lib/tagger.ex:4: Tagger.tag/1
Function: #Function<1.18912432/0 in MatchAggregator.handle_cast/2>
    Args: []

As you can see, I am using poolboy to pool the connections. Timeout passed to command is 600000ms. But the timeout happens even before this. May be in 5-10 seconds.

Any help to debug this would be great!

edge cases in reconnection handling

     Reason:     {{badmatch,{error,closed}},
                  [{'Elixir.Redix.Connection',start_receiver_and_hand_socket,
                       2,
                       [{file,"lib/redix/connection.ex"},{line,244}]},
                   {'Elixir.Redix.Connection',connect,2,
                       [{file,"lib/redix/connection.ex"},{line,95}]},
                   {'Elixir.Connection',enter_connect,5,
                       [{file,"lib/connection.ex"},{line,622}]},
                   {proc_lib,init_p_do_apply,3,
                       [{file,"proc_lib.erl"},{line,247}]}]}
     Offender:   [{pid,<0.2147.0>},
                  {id,'Elixir.Redix'},
                  {mfargs,
                      {'Elixir.Redix',start_link,
                          [<<"redis://redis:7379">>,
                           [{backoff,100},
                            {timeout,5000},
                            {name,'Elixir.Exq.Redis.Client'}]]}},
                  {restart_type,permanent},
                  {shutdown,5000},
                  {child_type,worker}]

    ** (EXIT) an exception was raised:
        ** (MatchError) no match of right hand side value: {:error, :closed}
            (redix) lib/redix/connection.ex:244: Redix.Connection.start_receiver_and_hand_socket/2
            (redix) lib/redix/connection.ex:95: Redix.Connection.connect/2
            (connection) lib/connection.ex:622: Connection.enter_connect/5
            (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
    (stdlib) gen_server.erl:212: :gen_server.call/3
    (redix) lib/redix/connection.ex:53: Redix.Connection.pipeline/3
    (exq) lib/exq/redis/job_queue.ex:133: Exq.Redis.JobQueue.scheduler_dequeue/3
    (exq) lib/exq/scheduler/server.ex:68: Exq.Scheduler.Server.dequeue/1
    (exq) lib/exq/scheduler/server.ex:56: Exq.Scheduler.Server.handle_info/2
    (stdlib) gen_server.erl:601: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:667: :gen_server.handle_msg/5
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3

To give some context, this happens when we try to do the integration test. We run redis inside a docker container and expose it in a port. Then we stop and start the redis server and see if it is handled properly. If no service is listening on a port, :gen_tcp.connect would normally fail. But with docker this doesn't happen. :gen_tcp.connect would succeed and then the connection would get closed immediately. So when redis tries to actually uses the socket for anything like transfer of ownership :gen_tcp.controlling_process(socket, receiver) or authentication, an error gets thrown. Currently, redix doesn't backoff on these failures and leads to either crash/shutdown, which then triggers the cascading failures which lead to the shutdown of the entire node.

Redix with Elixir 1.5.1

I want a global connection to the redis db in my app Foo, which is part of an umbrella App. So I added a line to the children list:

defmodule Foo.Application do

use Application

  def start(_type, _args) do
    children = [
      {Redix, [[], [name: :redix]]}
    ]

    opts = [strategy: :one_for_one, name: Foo.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

When I start the App, I get the error message:

Erlang/OTP 20 [erts-9.0.4] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]


=INFO REPORT==== 3-Sep-2017::10:20:43 ===
    application: logger
    exited: stopped
    type: temporary
** (Mix) Could not start application rooms: Foo.Application.start(:normal, []) returned an error: an exception was raised:
    ** (ArgumentError) The module Redix was given as a child to a supervisor
but it does not implement child_spec/1.

If you own the given module, please define a child_spec/1 function
that receives an argument and returns a child specification as a map.
For example:

    def child_spec(opts) do
      %{
        id: __MODULE__,
        start: {__MODULE__, :start_link, [opts]},
        type: :worker,
        restart: :permanent,
        shutdown: 500
      }
    end

Note that "use Agent", "use GenServer" and so on automatically define
this function for you.

However, if you don't own the given module and it doesn't implement
child_spec/1, instead of passing the module name directly as a supervisor
child, you will have to pass a child specification as a map:

    %{
      id: Redix,
      start: {Redix, :start_link, [arg1, arg2]}
    }

See the Supervisor documentation for more information.

        (elixir) lib/supervisor.ex:594: Supervisor.init_child/1
        (elixir) lib/enum.ex:1255: Enum."-map/2-lists^map/1-0-"/2
        (elixir) lib/supervisor.ex:581: Supervisor.init/2
        (stdlib) supervisor.erl:294: :supervisor.init/1
        (stdlib) gen_server.erl:365: :gen_server.init_it/2
        (stdlib) gen_server.erl:333: :gen_server.init_it/6
        (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3

I use elixir 1.5.1

What is my mistake? Should I prefer poolboy?

Redis Cluster support

I have not found anything about redis cluster support.
Does it exist?

Utils.format_error needs to return a binary

If we don't return a binary we will get problems when Redix.command! throws an exception:

got 'connection timed out' while retrieving Exception.message/1 for %Redix.ConnectionError{message: 'connection timed out'} (expected a string)

Exception.message expects a binary: https://github.com/elixir-lang/elixir/blob/v1.1.1/lib/elixir/lib/exception.ex#L55

We need to convert :inet.format_error(reason) to a binary as it returns a charlist.

I can send a PR later today if this is the correct solution

Update documentation on HexDocs

Hi there,
I really appreciate the work you have done. It truely is a well developed application.

However I got quite confused with the documentation I found on hexdocs. Especially when it comes to Real-world usage. I would be grateful if you could update these Docs with the ones I found in your pages > Real-world usage.md.

More specifically I am talking about the code snippet where you show how to spawn the child processes.

Thank you in advance!

documentation request

Hello.

As a newbie it is pretty hard to know how to actually use this in an application.

I got some guidance on the slack to set up a supervisor which I am now going to attempt.

It would be very nice if you could provide either (or both) boilerplate for setting up a supervisor/pool for the genserver or some instructions on what to set up where.

Client looks good though, cheers :)

Disconnect / Reconnect every 5 minutes in Heroku

Hello, we're using Redix in Heroku, it works great, but we see in the logs that it gets disconnected and reconnected every 5 minutes:

2017-11-21T23:04:52.095960+00:00 app web.1 - - 23:04:52.095 [error] Disconnected from Redis (...): TCP connection closed
2017-11-21T23:04:52.599245+00:00 app web.1 - - 23:04:52.599 [info] Reconnected to Redis (...)

2017-11-21T23:09:53.052947+00:00 app web.1 - - 23:09:53.052 [error] Disconnected from Redis (...): TCP connection closed
2017-11-21T23:09:53.556557+00:00 app web.1 - - 23:09:53.556 [info] Reconnected to Redis (...)

2017-11-21T23:14:54.011798+00:00 app web.1 - - 23:14:54.011 [error] Disconnected from Redis (...): TCP connection closed
2017-11-21T23:14:54.516201+00:00 app web.1 - - 23:14:54.516 [info] Reconnected to Redis (...)

2017-11-21T23:19:55.071229+00:00 app web.1 - - 23:19:55.071 [error] Disconnected from Redis (...): TCP connection closed
2017-11-21T23:19:55.574640+00:00 app web.1 - - 23:19:55.574 [info] Reconnected to Redis (...)

2017-11-21T23:24:56.015412+00:00 app web.1 - - 23:24:56.015 [error] Disconnected from Redis (...): TCP connection closed
2017-11-21T23:24:56.518771+00:00 app web.1 - - 23:24:56.518 [info] Reconnected to Redis (...)

Is this a normal behaviour?

Atomic ZRANGEBYSCORE and ZREM

How can I atomically read a value and delete it using multi?
I tried so much

Redix.command(conn, ["multi"])
{:ok, value} = Redix.command(conn, ["zrangebyscore", name, 0, 10, "limit", 0, 1])
Redix.command(conn, ["zrem", name, value])
Redix.command(conn, ["exec"])

But this does not work, because value = "QUEUED"

Random parse errors

Hey,

I am seeing these sometimes:

GenServer #PID<0.10459.0> terminating
  ** (exit) an exception was raised:
  ** (Redix.Protocol.ParseError) not a valid integer: "343\r"
  lib/redix/protocol.ex:135: Redix.Protocol.parse_integer/1
  lib/redix/protocol.ex:142: Redix.Protocol.parse_bulk_string/1
  lib/redix/protocol.ex:183: Redix.Protocol.take_n_elems/3
  lib/redix/protocol.ex:105: Redix.Protocol.parse_multi/2

It seems to happen especially often when fetching a MGET like query but not all of the time. :(

Let me know if I can help further...

Proper strategies to use Redix under heavy load

I am using Redix. During my application's run, I will spawn close to 10000 processes (Task.async) and all of these will hit Redis multiple times.

My current setup is pooled Redix connections with poolboy. Pool size is set as 100 and max overlow is also set as 100. When I run, I keep seeing the following errors.

12:54:46.787 [error] GenServer #PID<0.12311.1> terminating
** (stop) :eaddrnotavail
Last message: nil
State: %{continuation: nil, opts: [socket_opts: [], backoff: 2000, port: 6379, host: "192.168.99.100", database: 0], receiver: nil, reconnection_attempts: 0, socket: nil, tail: ""}

It looks like for some reason, the address became unavailable. I can confirm that when this happened, I could connect to Redis through the Redis CLI just fine. That says, Redis was not down, but for some reason the genserver was unable to connect to it.

Now after reading the Real world usage page, I was wondering is this the right way of doing it.

What do you suggest to do in such high load? Will manual pooling works?

Timeout trying to handle timeout

This error occurs in 0.6.1 on the following line:

redix/lib/redix/connection.ex

Line 58 in 2efdf4d

Connection.call(conn, {:timed_out, request_id})

This appears to be trying to handle a timed out connection, but times out calling the connection process. My assumption is either that the process is no longer there or there is some deadlock issue. Because of where this crashes, it seems the connection never recovers.

Error:
Process #PID<0.3403.1> terminating: {:timeout, {:gen_server, :call, [#PID<0.389.0>, {:timed_out, #Reference<0.2961049768.3466330114.137120>}]}}

Stacktrace:

(stdlib) gen_server.erl:206: :gen_server.call/2
(redix) lib/redix/connection.ex:58: Redix.Connection.pipeline/3
(redix) lib/redix.ex:327: Redix.command/3
(phoenix_pubsub_redis) lib/phoenix_pubsub_redis/redis_server.ex:34: anonymous fn/3 in Phoenix.PubSub.RedisServer.do_broadcast/8
(poolboy) src/poolboy.erl:76: :poolboy.transaction/3
(middle_out) web/channels/api/v1/profile_channel.ex:43: anonymous fn/3 in MiddleOut.API.V1.ProfileChannel.handle_in/3
(appsignal) lib/appsignal/phoenix/channel.ex:97: Appsignal.Phoenix.Channel.channel_action/5
(phoenix) lib/phoenix/channel/server.ex:244: anonymous fn/4 in Phoenix.Channel.Server.handle_info/2

Allow Redix to handle backoff on initial connection.

When porting from Redo to Redix we ran into the snag where Redix will handle back off and retries but it won't handle it on the first try. It's fairly trivial to handle retries on your own but it would be one less thing a user would need to deal with.

I am not privy to all the reasons why it won't retry on initial connection but I figured I'd post a discussion.

See here for an example case of needing to handle retries on our own. phoenixframework/phoenix_pubsub_redis#14

`Redix.command/2` fails when the command is an empty list

Right now, the failure is just a timeout which is awful. We have to raise a proper error message when Redix.command(conn, []) is called.

Update Hex docs the source link is broken for v 0.6.1

Intermittent issue when fetching multiple keys

Hi, we're seeing the following failure when trying to fetch using an MGET with many keys:

** (exit) exited in: :gen_server.call(#PID<0.960.0>, {:commands, [["MGET", "edge:cache:distance-JP986309-37.7838,-122.401", "edge:cache:distance-JP141681-37.7838,-122.401", "edge:cache:distance-JP543185-37.7838,-122.401", ...]]}, 5000)
** (EXIT) an exception was raised:
** (FunctionClauseError) no function clause matching in anonymous fn/2 in Redix.Protocol.parse_integer/1
(redix) lib/redix/protocol.ex:140: anonymous fn("3", <<131, 104, 2, 100, 0, 2, 111, 107, 98, 0, 0, 135, 196, 13, 10, 36, 49, 51, 13, 10, 131, 104, 2, 100, 0, 2, 111, 107, 98, 0, 0, 168, 38, 13, 10, 36, 49, 51, 13, 10, 131, 104, 2, 100, 0, 2, 111, 107, 98, 0, ...>>) in Redix.Protocol.parse_integer/1
(redix) lib/redix/protocol.ex:196: anonymous fn/3 in Redix.Protocol.resolve_cont/2
(redix) lib/redix/protocol.ex:196: anonymous fn/3 in Redix.Protocol.resolve_cont/2
(redix) lib/redix/connection/receiver.ex:86: Redix.Connection.Receiver.new_data/2
(redix) lib/redix/connection/receiver.ex:62: Redix.Connection.Receiver.handle_info/2
(stdlib) gen_server.erl:615: :gen_server.try_dispatch/4
(stdlib) gen_server.erl:681: :gen_server.handle_msg/5
(stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3

Fetching the same batch of keys results in the above error intermittently, with a high rate of occurrence when we have about 100 keys. Reducing the batch size to around 50 seems to eliminate the errors. As I understand it, Redis is capable of retrieving several thousands of keys at the same time. I've tried reproducing the issue in isolation in a separate repository, but was unable to do so, so I thought I would post the error here to try and get more insight.

Doc improvements

Hello,

(disclaimer, I am just starting with elixir)

I lost a fair bit of time trying to set values containing white spaces.
I think it might be worth it to add to the doc that we can also use something like

cmd = ["HSET", "username:1", "Name", "My Name"]

to prepare commands since

cmd = ~w(HSET username:1 name "My Name")

will not work with values containing white spaces...

Thanks!

PubSub remote redis failing to connect.

I am running into this error when using PubSub with a remote redis. It looks like you've already fixed this issue on master. Would you mind pushing a new release? Thank you.

=INFO REPORT==== 9-May-2016::17:29:23 ===
    application: logger
    exited: stopped
    type: temporary
** (Mix) Could not start application my_application: MyModule.start(:normal, []) returned an error: shutdown: failed to start child: MyModule.Listener
    ** (EXIT) an exception was raised:
        ** (KeyError) key :continuation not found in: %{clients_to_notify_of_reconnection: [], monitors: #HashDict<[]>, opts: [socket_opts: [], backoff: 2000, host: "my.secret.host", port: 6379, password: "my-secret-password"], queue: {[], []}, recipients: #HashDict<[]>, reconnection_attempts: 0, socket: #Port<0.19979>, tail: ""}
            (redix) lib/redix/connection/auth.ex:76: Redix.Connection.Auth.wait_for_response/1
            (redix) lib/redix/connection/auth.ex:39: Redix.Connection.Auth.auth/2
            (redix) lib/redix/connection/auth.ex:18: Redix.Connection.Auth.auth_and_select_db/1
            (redix) lib/redix/pub_sub/connection.ex:31: Redix.PubSub.Connection.connect/2
            (connection) lib/connection.ex:623: Connection.enter_connect/5
            (stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3

Pipelined commands and MULTI/EXEC does not always return a list with answers

Redix.pipeline and MULTI/EXEC currently returns a list when executing multiple commands but only a single answer when executing a single command. I found this quite surprising, ex:

commands = [
  ["SET", "pipe", "10"],
  ["INCR", "pipe"],
  ["GET", "pipe"]
]
Redix.pipeline(conn, commands)
=> {:ok, ["OK", 11, "11"]}

Redix.pipeline(conn, [["SET", "pipe_single", "10"]])
=> {:ok, "OK"}

Expected {:ok, ["OK"]} for last command.

Same for MULTI/EXEC:

Redix.command(conn, ["MULTI"])
=> {:ok, "OK"}
Redix.command(conn, ["INCR", "multifoo"])
=> {:ok, "QUEUED"}
Redix.command(conn, ["INCR", "multibar"])
=> {:ok, "QUEUED"}
Redix.command(conn, ["INCRBY", "multifoo", 4])
=> {:ok, "QUEUED"}
Redix.command(c, ["EXEC"])
=> {:ok, [1, 1, 5]}

Redix.command(conn, ["MULTI"])
=> {:ok, "OK"}
Redix.command(conn, ["SET", "multifoo", "10"])
=> {:ok, "QUEUED"}
Redix.command(c, ["EXEC"])
=> {:ok, "OK"}

Expected {:ok, ["OK"]} for last command.

What's the reasoning behind not including the RedixPool module as part of the library?

Just wondering, thanks!

Support for Redis Sentinel

Hi,

It does not look like that Redix is currently supporting Redis Sentinel and it would be great to do so.
Guidelines for Redis clients can be found here: http://redis.io/topics/sentinel-clients

Support for Redis Cluster

I guess It could be a good thing to have support for cluster - https://redis.io/topics/cluster-tutorial

There is a module, but I guess it should be part of the library - https://github.com/zhongwencool/redix-cluster