openresty / lua-resty-redis Goto Github PK
View Code? Open in Web Editor NEWLua redis client driver for the ngx_lua based on the cosocket API
Lua redis client driver for the ngx_lua based on the cosocket API
I'm not sure why this is returned to the client of lua-resty-redis library:
https://github.com/agentzh/lua-resty-redis/blob/master/lib/resty/redis.lua#L157
It seems wrong to return "userdata" type of things to users of the library?
For example:
local res, err = red:get('non-existing-key')
This returns ngx.null instead of nil as it should be(?).
Hello,
After reading the issue #23 I have updated my configuration to the following:
https://gist.github.com/frozenminds/5a1f0b9047c7af69ec61
Each SSI request will try to get the content from Redis, if key is not saved in cache yet, it will fallback to a PHP route which will generate the content, save it to Redis and return the output. The second time, Nginx will retrieve the content from Redis because it was cached.
On a page I have 4 SSI calls. The last one hangs on the very first request before the page is also stored in Nginx's FastCgi cache.
<!-- include virtual="/ssi?action=a -->
<!-- include virtual="/ssi?action=b -->
<!-- include virtual="/ssi?action=c&product_id=123 -->
<!-- include virtual="/ssi?action=c&category_id=999 -->
I am aware of this:
http://wiki.nginx.org/HttpLuaModule#Mixing_with_SSI_Not_Supported
But what I do not understand, do I have to completely turn off the SSI module and write my own module that will parse the SSI tags () and then use ngx.location.capture or ngx.location.capture_multi the get the content?
If I do have to parse the SSI tags, do you have any idea where do I start from?
Could you please guide me further?
Thank you in advance!
Boby
Hi everyone,
When set the ip to create a connect with my redis server, it work right. But if i set the hostname of my redis server, i get follow message:
2015/05/18 19:33:13 [error] 12297#0: *2592 attempt to send data on a closed socket: u:000000000243ABA8, c:0000000000000000, ft:8 eof:0, client: 179.189.133.35, server
The code that work right:
_REDIS_HOST = "10.254.222.23"
-- _REDIS_HOST = "my-server.8cxqzw.0001.sae1.cache.amazonaws.com"
_REDIS_PORT = "6379"
_REDIS_TIMEOUT = 2000 -- set 2 sec
_REDIS_DATABASE = 8
local redis = require "resty.redis"
local red = redis:new()
local ok, err = red:connect(_REDIS_HOST, _REDIS_PORT)
red:set_timeout(_REDIS_TIMEOUT)
red:select(_REDIS_DATABASE)
The code that break down:
-- _REDIS_HOST = "10.254.222.23"
_REDIS_HOST = "my-server.8cxqzw.0001.sae1.cache.amazonaws.com"
_REDIS_PORT = "6379"
_REDIS_TIMEOUT = 2000 -- set 2 sec
_REDIS_DATABASE = 8
local redis = require "resty.redis"
local red = redis:new()
local ok, err = red:connect(_REDIS_HOST, _REDIS_PORT)
red:set_timeout(_REDIS_TIMEOUT)
red:select(_REDIS_DATABASE)
当我使用redis的smembers的时候会出现下面的error:
2012/06/27 18:08:57 [error] 10960#0: *2564615 lua handler aborted: runtime error: /data/www/m/script/vendor/resty/redis.lua:132: bad argument #1 to 'receive' (bad pattern argument), client: 172.23.11.103, server: m.openapi.cn, request: "GET /app/index?session=222222250f52c5874fc441a8b2596206db@desktop5c03225e3d1a83f875f173a594d25280 HTTP/1.1", host: "..."
是在调用recieve的时候出错....不知道是什么问题导致...寻帮助...
It seems to me (unless I'm missing something) that lua-resty-redis only allows you to connect to database 0 of redis. You cannot specify another number.
ie this uri connects to database 7
redis://[email protected]:6379/7
Useful in contexts where the cosocket api is not available - such as init_by_lua.
I used this to load a script into lua once, set a shared memory variable, and then use evalsha
instead of sending the much larger script over the wire constantly.
Implementation here: openresty/lua-nginx-module#206 (comment)
Hi Yichun,
Thanks a lot for all those modules !
I tried to run your example and can't figure how to actually returns a 404 if the key is not found (hence res == null
in local res, err = red:get(ngx.var.key_that_does_not_exist)
.
Snippet:
if res == ngx.null then
ngx.status = ngx.HTTP_NOT_FOUND
ngx.say("key not found.")
return ngx.status
end
ngx.say("result: ", res)
Also tried:
if res == ngx.null then
ngx.status = ngx.HTTP_NOT_FOUND
ngx.say("key not found.")
ngx.exit(ngx.status)
end
ngx.say("result: ", res)
Output:
$> curl -I redis/key_exists
HTTP/1.1 200 OK
Server: ngx_openresty/1.2.7.1
result: "the key exists!"
$> curl -I redis/key_does_not_exist
HTTP/1.1 200 OK
Server: ngx_openresty/1.2.7.1
key not found.
Any idea by any chance ?
Thanks so much for your great contribution. @agentzh
And there is a problem with redis:subscribe
.
Can I subscribe multiple channels by such code?
redis:subscribe('channel:first', 'channel:second')
or
redis:subscribe({'channel:first', 'channel:second'})
hi,agentzh
just as #22 says, it's a little expensive to use the #
operator in the loop iterations, i'm curious about why you keep the #
operator in the https://github.com/agentzh/lua-resty-redis/blob/master/lib/resty/redis.lua#L418 instead of caching the value in a local variable something like ncmds?
The environment is ngx_openresty/1.4.3.6 on centos 6.5. Version of redis is 2.6.14.
The test code is as follow:
function Testblpop()
local count=1;
local msg=nil;
local err='';
local redis_handle, err=InitializeRedis(redis_host, redis_port);
while true
do
ngx.log(ngx.NOTICE, string.format("count: %d", count));
msg, err=redis_handle:blpop("list1", '0');
ngx.log(ngx.NOTICE, string.format("msg: %s", cjson.encode(msg)));
count=count+1;
ngx.sleep(1);
end
KeepAliveRedis(redis_handle);
end
I put this test code in content_by_lua_file, and call it using "curl".
If the "list1" is empty, the "blpop" won't block the current programme. The loop will go on, and the msg is "nil".
Is this a bug or by design?
Thanks
Hi there,
Is there any "legal" way to connect to redis from a worker created via the new init_worker_by_lua* directives
?
Anything I try gives me the "API disabled in the context of init_worker_by_lua*"
error.
I'm trying to cache errors on connect to a redis instance both by using
ok, err = ....
and wrapping up the code with a pcall
In both cases still an error line goes to nginx error log. I have a high load application and if a redis instance goes down for a few secs my error log gets flooded with messages. Is there a way around it?
Hi! Is it possible to implement cluster functionality to core of this library? I'm not sure if I am able to do it myself, I tried, but code looks bad and it fails sometimes.
Hi
I'm getting this error intermittently on a busy server:
2013/04/08 10:23:33 [error] 52774#0: *752776 connect() to unix://tmp/redis.sock failed (61: Connection refused), client: 46.14.44.217, server: , request: "GET /css/popup.css?v=3 HTTP/1.1"
I have maxclients 100000 in my redis conf and ulimit is 200000 on the server itself. Requests per second is around 1000 and redis connections is peaking around 3000..
What are the possible causes for this?
Many thanks
Richard
How do I skip/ignore errors generated under "content by lua"?
1.i'm using
local zlib = require "zlib"
sometimes the zlib variable returned will be empty or non compatible gzip format.
how do i let it continue to run?
location ~ /cached/ {
content_by_lua '
local zlib = require "zlib"
...
...
if (string.len(inflated) >= 10) then
ngx.say(inflated)
return
end';
}
how do i make it continue running the rest of the other status if it does not match string.len(inflated) ???
Is the following a proper implementation to serve clients the last published value from a Redis PubSub? Or is it possible that it misses some values?
init_worker_by_lua ' local delay = 0; local handler; handler = function() local redis = require "resty.redis" local red = redis:new() local ok, err = red:connect("127.0.0.1", 6379) local ok, err = red:subscribe("streampic") local res, err = red:read_reply() if not res then ngx.log(ngx.ERR, "No response", err) else local item = res[3] ngx.log(ngx.ERR, "Setting value : ", item) local data = ngx.shared.data data:set("last", item) end local ok, err = ngx.timer.at(delay, handler) end local ok, err = ngx.timer.at(delay, handler) ';
then just output the global value:
location /async { keepalive_timeout 0; content_by_lua ' local data = ngx.shared.data ngx.say(data:get("last")); '; }
EDIT:
Looks to me that I'm dropping a lot of values. When publishing 100 numbers (1-100) in a loop
I only catch about 5-6.
Given the following code:
local redis = require "resty.redis"
REDIS_HOST = "127.0.0.1"
REDIS_PORT = 6379
redis_client = redis:new()
local ok, err = redis_client:connect(REDIS_HOST, REDIS_PORT)
If REDIS_HOST
is "localhost"
instead of 127.0.0.1
, then it failed to connect.
Although, in the scope of localhost
vs. 127.0.0.1
it makes no difference, in a real world dynamic production environment, an IP is not likely to be used, rather, a nameserver instead.
Hi,
I have to track a change in a state of user and trigger a http post to another service based on http request from client.
In my request method I am using nginx timer to create a subrequest and using redis pubsub to track state change.
ngx.timer.at(0, subscribe, params.tracker_user_uuid, target_user_uuid)
local function subscribe(premature, user_uuid, channel_id)
local redis_x = require("redis.redis").new()
local res, err = redis_x:subscribe("channel" .. channel_id)
if not res then
ngx.say("1: failed to subscribe: ", err)
return
end
while true do
local erver_url = "http://example.com"
local res, err = redis_x:read_reply()
if res then
local req_body = res[3]
local result, respcode, respheaders, respstatus = http.request({
})
end
redis_x:set_keepalive(250, 200)
end
end
I see following error every minute in my nginx log.
[error] 9938#0: lua tcp socket read timed out, context: ngx.timer, client: 127.0.0.1, server: 0.0.0.0:8080
Is this the best way to do this ?
主要是变量的写法无法确定是否准确。
location /toredis {
default_type text/html;
set $key $url;
set $body $echo_request_body;
set $exptime 300;
content_by_lua '
local redis = require "resty.redis"
local red = redis:new()
red:set_timeout(1000)
local ok, err = red:connect("127.0.0.1", 6379)
if not ok then
return nil
end
local ok, err = red:set(ngx.var.key,ngx.var.body,ngx.var.exptime)
if not ok then
return nil
end
red:close()
';
}
I'm trying to understand what is the best practice for handling Redis connections.
The doc suggests to avoid having a global connection to the Redis server in order to avoid any concurrency problems. This means opening a new connection to Redis for every incoming connection to Nginx, isn't this very inefficient?
Also, I'm trying to reuse the same connection in both access_by_lua
and log_by_lua
, but I get a attempt to send data on a closed socket
error in the log_by_lua
timer callback.
init_by_lua:
redis = require "resty.redis"
access_by_lua:
ngx.ctx.redis = redis:new()
local ok, err = ngx.ctx.redis:connect("127.0.0.1", 6379)
if not ok then
ngx.log(ngx.ERR, "Cant connect to Redis")
end
ngx.ctx.redis:incr("incoming_requests")
log_by_lua:
local function log(premature, redis)
redis:incr("requests_completed")
end
local ok, err = ngx.timer.at(0, log, ngx.ctx.redis)
if not ok then
ngx.log(ngx.ERR, "failed to create timer: ", err)
return
end
And I'm getting the following error in log_by_lua
:
[error] 16644#0: attempt to send data on a closed socket: u:00007FCA22600B78, c:0000000000000000, ft:0 eof:0, context: ngx.timer
I have page with SSI includes,
<!--# include /widget/user/1 -->
<!--# include /widget/advert/37884 -->
Problem is nginx randomly mixes ssi content, so in user-block you can see advert and user info in advertisement block =)
ssi catches by location
location /widget/ {
internal;
lua_code_cache off;
content_by_lua_file '/etc/nginx/lua/widget_redis.lua';
error_page 404 502 500 503 = @generate-widget;
}
redis = require "resty.redis"
red = redis:new()
red:set_timeout(500)
ok, err = red:connect("127.0.0.1", 6379)
if not ok then
ngx.log(ngx.ERR, "Connection error")
ngx.exit(ngx.HTTP_SERVICE_UNAVAILABLE)
return
end
content, err = red:get(ngx.var.uri .. "/")
if not content then
ngx.exit(ngx.HTTP_INTERNAL_SERVER_ERROR)
return
end
if content == ngx.null then
ngx.exit(ngx.HTTP_NOT_FOUND)
return
end
ngx.header["Content-Type"] = "text/html; charset=utf-8"
ngx.print(content)
If Redis-key /widget/user/1/
exists, lua prints it contents.
Else @generate-widget (php-fpm) render and store it in redis.
All is ok when all widgets rendered by @generate-widget, but when some widgets already stored in redis (and other slowly generated by php) they appears in first ssi blocks.
How can I fix it?
Intro
Now we are using http_redis2_module
and load-balancing read-only connections to the redis cluster with http_upstream_module
. But It's annoying to write tons of locations for each kind of request, so we are thinking about migration to lua-resty-redis
or another Lua-based redis client, but I am sure that It is not good idea to re-implement load-balancing (including work with failed servers in upstream) in pure Lua.
The question
So, Is it possible to use the library to connect to Redis using upstreams?
lua+redis
i use zrangebysocre,how to use limit?
--local aSetId = instance:zrangebyscore('jsonjjmatch_ware_dict_usually',iTimeLine, iTime)
local aSetId = instance:zrangebyscore('jsonjjmatch_ware_dict_usually', iTimeLine, iTime, {limit={0,10}})
2013/07/04 15:44:20 [error] 19003#0: *2004 lua handler aborted: runtime error: /usr/local/openresty/lualib/resty/redis.lua:196: bad argument #1 to 'len' (string expected, got table), client: 192.168.20.115, server: localhost, request: "GET /get_ware_dict?waretype=1&id=33&wareid= HTTP/1.1", host: "localhost:85"
Could you provide some method about the lua get data from redis?
I'm using 2.8.17 redis server.
Send requests throw unix.sock by nginx+lua. Each request is one SMEMBERS command. Now I have about 1k req per sec. In the nginx access log I see mostly next
[09/Oct/2014:18:04:49 +0300] [ 1412867089.670 : 0.002 : 190 ] 212.1.66.241 - - "POST /API/0.1/url/getCategory/ HTTP/1.1" 200 29
[09/Oct/2014:18:04:49 +0300] [ 1412867089.670 : 0.002 : 190 ] 212.1.66.241 - - "POST /API/0.1/url/getCategory/ HTTP/1.1" 200 29
[09/Oct/2014:18:04:49 +0300] [ 1412867089.670 : 0.002 : 190 ] 212.1.66.241 - - "POST /API/0.1/url/getCategory/ HTTP/1.1" 200 29
[09/Oct/2014:18:04:49 +0300] [ 1412867089.670 : 0.002 : 190 ] 212.1.66.241 - - "POST /API/0.1/url/getCategory/ HTTP/1.1" 200 29
[09/Oct/2014:18:04:49 +0300] [ 1412867089.671 : 0.001 : 190 ] 212.1.66.241 - - "POST /API/0.1/url/getCategory/ HTTP/1.1" 200 29
[09/Oct/2014:18:04:49 +0300] [ 1412867089.672 : 0.000 : 190 ] 212.1.66.241 - - "POST /API/0.1/url/getCategory/ HTTP/1.1" 200 29
[09/Oct/2014:18:04:49 +0300] [ 1412867089.673 : 0.001 : 190 ] 212.1.66.241 - - "POST /API/0.1/url/getCategory/ HTTP/1.1" 200 29
[09/Oct/2014:18:04:49 +0300] [ 1412867089.673 : 0.001 : 190 ] 212.1.66.241 - - "POST /API/0.1/url/getCategory/ HTTP/1.1" 200 29
[09/Oct/2014:18:04:49 +0300] [ 1412867089.674 : 0.000 : 198 ] 212.1.66.241 - - "POST /API/0.1/url/getCategory/ HTTP/1.1" 200 37
[09/Oct/2014:18:04:49 +0300] [ 1412867089.681 : 0.000 : 190 ] 212.1.66.241 - - "POST /API/0.1/url/getCategory/ HTTP/1.1" 200 29
You can see that request time is not more than 2ms. But almost every second I have a few requests with request time about 100ms
[09/Oct/2014:18:05:22 +0300] [ 1412867122.843 : 0.131 : 190 ] 212.1.66.241 - - "POST /API/0.1/url/getCategory/ HTTP/1.1" 200 29
[09/Oct/2014:18:05:23 +0300] [ 1412867123.212 : 0.100 : 190 ] 212.1.66.241 - - "POST /API/0.1/url/getCategory/ HTTP/1.1" 200 29
[09/Oct/2014:18:05:23 +0300] [ 1412867123.613 : 0.101 : 190 ] 212.1.66.241 - - "POST /API/0.1/url/getCategory/ HTTP/1.1" 200 29
I can't find the problem place.
I checked everything from the latency guide
Slowlog has no entries
System is not swapping
I don't use AOF
I read a lot of posts on stackoverflow like this
http://stackoverflow.com/questions/16841469/redis-performance-tuning/23719773#23719773
and I can't find the solution.
redis-benchmark gives me next
redis-benchmark -s /tmp/redis.sock -q -n 10000 -c 1 -d 50
PING_INLINE: 51282.05 requests per second
PING_BULK: 64935.07 requests per second
SET: 39840.64 requests per second
GET: 28901.73 requests per second
INCR: 26525.20 requests per second
LPUSH: 29411.76 requests per second
LPOP: 23255.81 requests per second
SADD: 21551.72 requests per second
SPOP: 23041.47 requests per second
LPUSH (needed to benchmark LRANGE): 45045.04 requests per second
LRANGE_100 (first 100 elements): 10101.01 requests per second
LRANGE_300 (first 300 elements): 4196.39 requests per second
LRANGE_500 (first 450 elements): 3337.78 requests per second
LRANGE_600 (first 600 elements): 7320.64 requests per second
MSET (10 keys): 62111.80 requests per second
hi, maybe it's a simple question, but it's a real question of mine and i get nothing from readme about this question. The question is how to know if is a redis key exists.
example:
local key = "not_exists"
local val, err = red:get(key)
if not val then
ngx.say("the value is not exists.")
else
ngx.say("val: "..val)
end
when the key is not exits, I will get a error of [error] 26052#0: *1678 lua entry thread aborted: runtime error: /opt/code/lua/redis.lua:22: attempt to concatenate local 'val' (a userdata value)
I know that it's probably that the value is not exists, so i can not use ngx.say()
to print it.
but I do not know how to figure out it with lua code, that I used if not val
, if val ~= nil
, which neither can not work correctly.
Thank you!
楼主,你好。菜鸟学生一枚,想请教下该驱动下,如何select不同的库,官方驱动如下:
package.path = "../src/?.lua;src/?.lua;" .. package.path
pcall(require, "luarocks.require")
local redis = require 'redis'
local params = {
host = '10.20.17.123', --Host
port = 6379, --Port
}
local client = redis.connect(params)
client:select(1) --例如这里
client:set('foo', 'bar')
local value = client:get('foo')
local test1=client:get('pwd');
print('get from redis :',value,test1)
如果使用lua-resty-redis,该如何实现?
The lua-resty-redis module has a problem with connection, when connect() is called in ngx.timer.at callback's function
connect() to unix:/var/lib/redis/redis.sock failed (11: Resource temporarily unavailable), context: ngx.timer
Search logs with grep shows that the problem only occurs in ngx.timer context
grep -o 'context: .*' /var/log/nginx/error.log|sort |uniq -c
5701 context: ngx.timer
I've tested it via http_load tool
for i in `seq 0 3`; do ./http_load -proxy 0:80 -parallel 2000 -seconds 20 url 2>/dev/null & done
I'm also using mongo module (https://github.com/bigplum/lua-resty-mongol) the same way I'm using redis module and it works fine.
Simple config example (one i've tested on):
nginx.conf
user nginx nginx;
worker_processes 12;
events {
worker_connections 2048;
}
pid /var/run/nginx.pid;
http {
include mime.types;
default_type text/html;
keepalive_timeout 0;
init_by_lua '
package.path = package.path .. ";/usr/local/nginx/conf/lua/?.lua"
package.path = package.path .. ";/usr/local/nginx/conf/lua/resty/?.lua"
redis = require "resty.redis.redis"
r = require "r"
';
server {
listen 80 deferred default_server;
server_name .*;
resolver 127.0.0.1;
location / {
access_by_lua_file '/usr/local/nginx/conf/lua/access.lua';
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://$host$request_uri;
log_by_lua_file '/usr/local/nginx/conf/lua/log.lua';
}
}
}
access.lua
local c = r.open()
if c then
c:incr('n')
end
r.close(c)
log.lua
function foo(premature)
if premature then
return
end
local c = r.open()
if c then
c:decr('n')
end
r.close(c)
end
ngx.timer.at(0, foo)
r.lua
local M = {}
function M.open()
local c = redis:new()
c:set_timeout(1000)
local ok, err = c:connect('unix:/var/lib/redis/redis.sock')
if not ok then
ngx.log(ngx.ERR, err)
return nil
end
return c
end
function M.close(c)
if not c then
return
end
local ok, err = c:set_keepalive(1000*10, 32)
if not ok then
c:close()
ngx.log(ngx.ERR, err)
end
end
return M
Hi,
I'm really intrigued in your work of using nginx as app server. I've tried simple benchmark of nginx and lua duo, and it gave me about 100k requests per second, but when I added redis call to code, it dropped to about 18k , is it normal or there is something wrong with my code?
Many thanks,
Tadek
worker_processes 8;
events {
worker_connections 16384;
}
http {
server {
access_log off;
listen 8080;
location / {
default_type text/html;
content_by_lua '
local redis = require "resty.redis"
local red = redis:new()
local ok, err = red:connect("127.0.0.1", 6379)
if not ok then
ngx.say("failed to connect: ", err)
return
end
res, err = red:set("dog", "an aniaml")
if not ok then
ngx.say("failed to set dog: ", err)
return
ngx.say("dog: ", res)
end
';
}
}
}
现在 red:hmget 传入的是可变参数,能否想hmset一样传入数组啊,或者安装redis
原生的命令 hmget hashname field1 field2?
Should hmset return a hash table rather than an array table? Or, we could provided a utility function that converts it. I know I have that code in several of my projects.
I'm trying to share the same Redis client between workers:
--- script/red.lua
local redis = require "resty.redis"
local red = redis:new()
local host = "127.0.0.1"
local port = 6379
red:set_timeout(1000) -- 1 sec
local ok, err = red:connect(host, port)
if not ok then
ngx.exit(ngx.HTTP_FORBIDDEN)
return
end
return red
--- script/auth.lua
local red = require "script.red"
However it throws the following error. Am I doing something wrong? Is this impossible to achieve?
2014/01/17 00:05:34 [error] 7716#0: *1 lua entry thread aborted: runtime error: attempt to yield across C-call boundary
stack traceback:
coroutine 0:
[C]: in function 'require'
/home/radu/stor/script/auth.lua:2: in function </home/radu/stor/script/auth.lua:1>, client: 127.0.0.1, server: , request: "GET /nolua? AUTH_TOKEN=AAAAAAAAAAAAAAAA_BBBBBBBBBBBBBBBB_MGMGMGMGMGMGMGMG HTTP/1.1", host: "localhost:8888"
hello, @agentzh! first of all thank you for this great lib!
have tihs problem:
zrangebyscore ip 92724334 +inf limit 0 1
works fine in redis-cli
can't run this from lua:
res, err = redis:zrangebyscore('ip', 92724334, '+inf', 'limit 0 1')
it returns ERR syntax error
in err
and false
in res
tried many variations but no luck at all, can you suggest something?
thank you in advance!
I am trying to use the resty.redis module same as the example of the synopsis.
lua_package_path "/path/to/lua-resty-redis/lib/?.lua;;";
server {
location /test {
content_by_lua '
local redis = require "resty.redis"
But I am getting the error:
[string "content_by_lua"]:2: module 'resty.redis' not found
the lua_package_path is set correctly to the lua-resty-redis lib
Do you have an idea what might be the problem?
Thanks!
local res, err hmget(mykey, "one", "two", "three")
then
res.one
(not worked)
when mykey not exists,the below code not worked
if (res[2] ~= nil) then
ngx.say('exists') -- when mykey not exists, this line would output
end
and the type of mykey is userdata, how can i convert it to table?
Hi,
I'm doing some redis lookups in my lua script and I'm getting some weird connection problems which I don't know how to properly debug to actually get to the source of the problem.
Here is snip from config
- Connect to Redis
local redis = require "resty.redis"
local red = redis:new()
red:set_timeout(1000)
local ok, err = red:connect("127.0.0.1", 6379)
if not ok then
ngx.log(ngx.ERR, "Redis connection failure: " .. err)
return
end
- rest of the code ...
On a random basis I get a lot of Redis connection failure: timeout in nginx error log. While debug loging redis server I don't see any connections at that same moment.
I'm using latest lua-resty-redis and 0.8.6 ngx_lua module. I also try to upgrade to 0.9.2 but then I get even more strange errors
2013/11/10 23:24:53 [error] 52514#0: *15 attempt to send data on a closed socket: u:0000000000000000, c:0000000000000000, ft:0 eof:0,
2013/11/10 23:24:53 [error] 52514#0: *15 attempt to send data on a closed socket: u:0000000000000000, c:0000000000000000, ft:0 eof:0,
2013/11/10 23:24:53 [error] 52514#0: *15 attempt to send data on a closed socket: u:0000000000000000, c:0000000000000000, ft:0 eof:0,
2013/11/10 23:24:53 [error] 52514#0: *15 attempt to send data on a closed socket: u:0000000000000000, c:0000000000000000, ft:0 eof:0,
I don't know if this one is actually related. Is there anything I can try to get more detail info where things actually broke. Server is FreeBSD with pf enabled, nginx 1.5.x.
Regards,
Uros
The function "set_keepalive" call almost next to the last, the code just as below:
function set_redis_keepalive(client)
local kp_ok, kp_err = client:set_keepalive(0, 100)
if not kp_ok then
ngx.log(ngx.ERR, "Limit ", " Failed to set keepalive: ", kp_err)
end
end
local client = redis:new()
client:set_timeout(1000) -- 1 sec
......
set_redis_keepalive(client)
.....
set_redis_keepalive(client)
Hello. At first, thanks for your work:)
After this commit 00e493e lua-resty starts yelling
2014/10/29 19:40:21 [error] 25299#0: *2 lua entry thread aborted: runtime error: /usr/local/lib/lua-resty-redis/lib/resty/redis.lua:22: attempt to call local 'new_tab' (a table value)
stack traceback:
coroutine 0:
[C]: in function 'require'
/etc/nginx/lua/hosts.wlua:1: in function </etc/nginx/lua/hosts.wlua:1>,
hosts.wlua contains only 'local redis = require "resty.redis"'
Inspired by https://github.com/FGRibreau/node-redis-lua
Abstract the eval/evalsha so that user does not have to check if script has been loaded or not. Also, allow the scripts to be loaded into the module. Something like:
redis.import_script('my_special_function', string_with_lua_script)
-- later with red is a redis connection
red.scripts['my_special_function'](1, key, arg, arg, arg)
Not sure about the public interface, the above is given as a horrible example.
Thoughts? If it looks like something that would be generally useful, I can write some code to get it started.
This file is necessary to install lua-resty-redis
into a custom nginx compilation. Is it available somewhere, or is this meant to be part of another module somehow?
I package redis functions into a module testdb.lua, and include it in another module testm.lua.
testm.lua is required in test.lua, which is running by ngx_lua:
content_by_lua_file luas/test.lua;
Finally, the error occur:
2012/04/07 22:10:32 [error] 24794#0: *185113 1, client: 127.0.0.1, server: localhost, request: "GET /test HTTP/1.1"
2012/04/07 22:10:32 [error] 24794#0: *185113 lua handler aborted: runtime error: attempt to yield across C-call boundary, client: 127.0.0.1, server: localhost, request: "GET /test HTTP/1.1"
The first ngx.log() is excuted, but second not. So is there anything wrong in redis:connect()?
All the files could be found in: https://gist.github.com/2329313
thanks.
I encountered a problem when I set keepalive to redis while using ngx.redirect. My intent is to keep the connection between nginx and redis always established so that I can reduce the cost on TCP connection while I can do query in redis and judge which sites would be chosen to redirect to. Here is my configuration:
server {
listen 80;
set $lua_country us;
rewrite_by_lua '
local redis = require "resty.redis"
local testdemo = redis:new()
local luacountry = ngx.var.lua_country
local luauri = ngx.var.request_uri
testdemo:set_timeout(1000)
local ok,err = testdemo:connect("127.0.0.1",6379)
if not ok then
ngx.say("faild to connect",err)
return
end
local exist = testdemo:exists(luauri)
if exist==1 then
local remote_host=testdemo:hget(luauri,luacountry)
if remote_host then
-- ngx.say("hello")
ngx.redirect("http://"..remote_host..luauri,ngx.HTTP_MOVED_TEMPORARILY)
else
ngx.redirect("http://".."www.google.com.hk",302)
end
end
testdemo:set_keepalive(0,100)
';
}
您好:
我用set_unescape_uri解出来的变量(中文)放入nginx日志,日志名字为变量名,结果在centos 6中日志名中文显示正常,但是日志中的中文显示是“\xE5\xBA\x94\xE7\x94\xA8\xE7\xB1\xBB\xE5\x9E\x8B”这种,请问该如何解决呢?
备注:我的系统和nginx编码都是utf-8
Greetings, I'm absolutely positive this is my fault but I can't figure out what I'm doing wrong.
Seems to be randomly spitting back a "bad request"
I went uber paranoid with error handling and made it not call keepalive unless there were no errors...
hostmap.lua
local module = {}
local REDIS_TIMEOUT = 250
local REDIS_HOST = '127.0.0.1'
local REDIS_PORT = 6379
local REDIS_POOLSIZE = 200
local REDIS_IDLETIME = 60
local MAP_EXPIRES = 3660
function module.init()
local redis = require "resty.redis"
module.red = redis:new()
module.err = false
module.red:set_timeout(REDIS_TIMEOUT)
local ok, err = module.red:connect(REDIS_HOST, REDIS_PORT)
if not ok then
ngx.log(ngx.ERR, "Absolute failure of redis: " .. err)
return nil
end
return true
end
function module.fetch()
local res, err = module.red:get(ngx.var.uid_got)
if not res then
ngx.log(ngx.ERR, "Failed to get value for key (" .. ngx.var.uid_got .. "): " .. err)
module.err = true
return
end
if type(res) == "string" then
ngx.var.httphost = res
ngx.log(ngx.ERR, "Why so bad request? " .. ngx.var.uid_got .. " : " .. MAP_EXPIRES)
local res, err = module.red:expire(ngx.var.uid_got, MAP_EXPIRES)
if not res then
ngx.log(ngx.ERR, "Failed to reset expiry for (" .. ngx.var.uid_got ..") to " .. MAP_EXPIRES .. ": " .. err)
module.err = true
return
end
end
return true
end
function module.store()
local res, err = module.red:setex(ngx.var.uid_got, MAP_EXPIRES, ngx.var.httphost)
if not res then
ngx.log(ngx.ERR, "Failed to store value (" .. ngx.var.httphost .. ") for key (" .. ngx.var.uid_got .. "): " .. err)
module.err = true
return
end
if ngx.var.args then
ngx.log(ngx.ERR, "Not redirecting...")
return true
end
return "/" .. ngx.var.redirecturl
end
function module.clear()
local res, err = module.red:del(ngx.var.uid_got)
if not res then
ngx.log(ngx.ERR, "Failed to clear value for key (" .. ngx.var.uid_got .. "): " .. err)
module.err = true
return
end
end
function module.pool()
if module.err then
ngx.log(ngx.ERR, "Not sending closed connection back to pool")
else
module.red:set_keepalive(REDIS_IDLETIME, REDIS_POOLSIZE)
end
end
return module
select_host.lua
if ngx.var.uid_got then
local map = require "hostmap"
if map then
if map.init() and map.fetch() then
map.pool()
end
end
end
Exception - the "Why so bad request?" lines are me debugging to make sure that values are actually being passed.
2014/05/12 01:39:37 [error] 1002#0: *4 [lua] hostmap.lua:36: fetch(): Why so bad request? vid=020011AC320D70539B01419A02030303 : 3660, client: 172.17.42.1, server: www.example.com, request: "GET / HTTP/1.1", host: "www.yoda.lan"
2014/05/12 01:39:38 [error] 1002#0: *4 [lua] hostmap.lua:36: fetch(): Why so bad request? vid=020011AC320D70539B01419A02030303 : 3660, client: 172.17.42.1, server: www.example.com, request: "GET /js/css_menu.js HTTP/1.1", host: "www.yoda.lan", referrer: "https://www.yoda.lan/"
2014/05/12 01:39:38 [error] 1002#0: *4 [lua] hostmap.lua:36: fetch(): Why so bad request? vid=020011AC320D70539B01419A02030303 : 3660, client: 172.17.42.1, server: www.example.com, request: "GET /css/footer.css HTTP/1.1", host: "www.yoda.lan", referrer: "https://www.yoda.lan/"
2014/05/12 01:39:38 [error] 1002#0: *24 lua entry thread aborted: runtime error: /usr/share/lua/5.1/resty/redis.lua:292: bad request
stack traceback:
coroutine 0:
[C]: in function 'send'
/usr/share/lua/5.1/resty/redis.lua:292: in function 'get'
/etc/nginx/conf.d/lua/hostmap.lua:27: in function 'fetch'
/etc/nginx/conf.d/lua/select_host.lua:4: in function </etc/nginx/conf.d/lua/select_host.lua:1>, client: 172.17.42.1, server: www.example.com, request: "GET /css/jquery-ui-1.7.3.custom.css HTTP/1.1", host: "www.yoda.lan", referrer: "https://www.yoda.lan/"
2014/05/12 01:39:38 [error] 1002#0: *23 lua entry thread aborted: runtime error: /usr/share/lua/5.1/resty/redis.lua:292: bad request
stack traceback:
coroutine 0:
[C]: in function 'send'
/usr/share/lua/5.1/resty/redis.lua:292: in function 'get'
/etc/nginx/conf.d/lua/hostmap.lua:27: in function 'fetch'
/etc/nginx/conf.d/lua/select_host.lua:4: in function </etc/nginx/conf.d/lua/select_host.lua:1>, client: 172.17.42.1, server: www.example.com, request: "GET /css/jquery.simpledialog.0.1.css HTTP/1.1", host: "www.yoda.lan", referrer: "https://www.yoda.lan/"
2014/05/12 01:39:38 [error] 1002#0: *25 [lua] hostmap.lua:36: fetch(): Why so bad request? vid=020011AC320D70539B01419A02030303 : 3660, client: 172.17.42.1, server: www.example.com, request: "GET /css/common-account.css HTTP/1.1", host: "www.yoda.lan", referrer: "https://www.yoda.lan/"
2014/05/12 01:39:38 [error] 1002#0: *29 [lua] hostmap.lua:36: fetch(): Why so bad request? vid=020011AC320D70539B01419A02030303 : 3660, client: 172.17.42.1, server: www.example.com, request: "GET /css/content.css HTTP/1.1", host: "www.yoda.lan", referrer: "https://www.yoda.lan/"
2014/05/12 01:39:38 [error] 1002#0: *29 lua entry thread aborted: runtime error: /usr/share/lua/5.1/resty/redis.lua:292: bad request
stack traceback:
coroutine 0:
[C]: in function 'send'
/usr/share/lua/5.1/resty/redis.lua:292: in function 'expire'
/etc/nginx/conf.d/lua/hostmap.lua:37: in function 'fetch'
/etc/nginx/conf.d/lua/select_host.lua:4: in function </etc/nginx/conf.d/lua/select_host.lua:1>, client: 172.17.42.1, server: www.example.com, request: "GET /css/content.css HTTP/1.1", host: "www.yoda.lan", referrer: "https://www.yoda.lan/"
2014/05/12 01:39:38 [error] 1002#0: *27 [lua] hostmap.lua:36: fetch(): Why so bad request? vid=020011AC320D70539B01419A02030303 : 3660, client: 172.17.42.1, server: www.example.com, request: "GET /css/odds_and_rules.css HTTP/1.1", host: "www.yoda.lan", referrer: "https://www.yoda.lan/"
2014/05/12 01:39:38 [error] 1002#0: *30 [lua] hostmap.lua:36: fetch(): Why so bad request? vid=020011AC320D70539B01419A02030303 : 3660, client: 172.17.42.1, server: www.example.com, request: "GET /css/index.css HTTP/1.1", host: "www.yoda.lan", referrer: "https://www.yoda.lan/"
2014/05/12 01:39:38 [error] 1002#0: *38 lua entry thread aborted: runtime error: /usr/share/lua/5.1/resty/redis.lua:292: bad request
stack traceback:
coroutine 0:
[C]: in function 'send'
/usr/share/lua/5.1/resty/redis.lua:292: in function 'get'
/etc/nginx/conf.d/lua/hostmap.lua:27: in function 'fetch'
/etc/nginx/conf.d/lua/select_host.lua:4: in function </etc/nginx/conf.d/lua/select_host.lua:1>, client: 172.17.42.1, server: www.example.com, request: "GET /css/dev_server.css HTTP/1.1", host: "www.yoda.lan", referrer: "https://www.yoda.lan/"
2014/05/12 01:39:38 [error] 1002#0: *39 [lua] hostmap.lua:36: fetch(): Why so bad request? vid=020011AC320D70539B01419A02030303 : 3660, client: 172.17.42.1, server: www.example.com, request: "GET /css/print.css HTTP/1.1", host: "www.yoda.lan", referrer: "https://www.yoda.lan/"
2014/05/12 01:39:38 [error] 1002#0: *40 [lua] hostmap.lua:36: fetch(): Why so bad request? vid=020011AC320D70539B01419A02030303 : 3660, client: 172.17.42.1, server: www.example.com, request: "GET /css/notification.css HTTP/1.1", host: "www.yoda.lan", referrer: "https://www.yoda.lan/"
2014/05/12 01:39:38 [error] 1002#0: *41 lua entry thread aborted: runtime error: /usr/share/lua/5.1/resty/redis.lua:292: bad request
stack traceback:
coroutine 0:
[C]: in function 'send'
/usr/share/lua/5.1/resty/redis.lua:292: in function 'get'
/etc/nginx/conf.d/lua/hostmap.lua:27: in function 'fetch'
/etc/nginx/conf.d/lua/select_host.lua:4: in function </etc/nginx/conf.d/lua/select_host.lua:1>, client: 172.17.42.1, server: www.example.com, request: "GET /css/navigation-primary.css HTTP/1.1", host: "www.yoda.lan", referrer: "https://www.yoda.lan/"
2014/05/12 01:39:38 [error] 1002#0: *42 [lua] hostmap.lua:36: fetch(): Why so bad request? vid=020011AC320D70539B01419A02030303 : 3660, client: 172.17.42.1, server: www.example.com, request: "GET /css/navigation-secondary.css HTTP/1.1", host: "www.yoda.lan", referrer: "https://www.yoda.lan/"
2014/05/12 01:39:38 [error] 1002#0: *30 [lua] hostmap.lua:36: fetch(): Why so bad request? vid=020011AC320D70539B01419A02030303 : 3660, client: 172.17.42.1, server: www.example.com, request: "GET /js/jquery-1.3.2.min.js HTTP/1.1", host: "www.yoda.lan", referrer: "https://www.yoda.lan/"
2014/05/12 01:39:38 [error] 1002#0: *39 [lua] hostmap.lua:36: fetch(): Why so bad request? vid=020011AC320D70539B01419A02030303 : 3660, client: 172.17.42.1, server: www.example.com, request: "GET /js/jquery-ui-1.7.3.custom.min.js HTTP/1.1", host: "www.yoda.lan", referrer: "https://www.yoda.lan/"
2014/05/12 01:39:38 [error] 1002#0: *66 [lua] hostmap.lua:36: fetch(): Why so bad request? vid=020011AC320D70539B01419A02030303 : 3660, client: 172.17.42.1, server: www.example.com, request: "GET /js/jquery.corner.js HTTP/1.1", host: "www.yoda.lan", referrer: "https://www.yoda.lan/"
2014/05/12 01:39:38 [error] 1002#0: *71 lua entry thread aborted: runtime error: /usr/share/lua/5.1/resty/redis.lua:292: bad request
stack traceback:
coroutine 0:
[C]: in function 'send'
/usr/share/lua/5.1/resty/redis.lua:292: in function 'get'
/etc/nginx/conf.d/lua/hostmap.lua:27: in function 'fetch'
/etc/nginx/conf.d/lua/select_host.lua:4: in function </etc/nginx/conf.d/lua/select_host.lua:1>, client: 172.17.42.1, server: www.example.com, request: "GET /js/jquery-placeholder-0.1.min.js HTTP/1.1", host: "www.yoda.lan", referrer: "https://www.yoda.lan/"
2014/05/12 01:39:38 [error] 1002#0: *72 lua entry thread aborted: runtime error: /usr/share/lua/5.1/resty/redis.lua:292: bad request
stack traceback:
coroutine 0:
[C]: in function 'send'
/usr/share/lua/5.1/resty/redis.lua:292: in function 'get'
/etc/nginx/conf.d/lua/hostmap.lua:27: in function 'fetch'
/etc/nginx/conf.d/lua/select_host.lua:4: in function </etc/nginx/conf.d/lua/select_host.lua:1>, client: 172.17.42.1, server: www.example.com, request: "GET /js/global.js HTTP/1.1", host: "www.yoda.lan", referrer: "https://www.yoda.lan/"
2014/05/12 01:39:38 [error] 1002#0: *73 [lua] hostmap.lua:36: fetch(): Why so bad request? vid=020011AC320D70539B01419A02030303 : 3660, client: 172.17.42.1, server: www.example.com, request: "GET /js/notifications.js HTTP/1.1", host: "www.yoda.lan", referrer: "https://www.yoda.lan/"
2014/05/12 01:39:38 [error] 1002#0: *74 [lua] hostmap.lua:36: fetch(): Why so bad request? vid=020011AC320D70539B01419A02030303 : 3660, client: 172.17.42.1, server: www.example.com, request: "GET /js/dev_server.js HTTP/1.1", host: "www.yoda.lan", referrer: "https://www.yoda.lan/"
2014/05/12 01:39:38 [error] 1002#0: *77 lua entry thread aborted: runtime error: /usr/share/lua/5.1/resty/redis.lua:292: bad request
stack traceback:
coroutine 0:
[C]: in function 'send'
/usr/share/lua/5.1/resty/redis.lua:292: in function 'get'
/etc/nginx/conf.d/lua/hostmap.lua:27: in function 'fetch'
/etc/nginx/conf.d/lua/select_host.lua:4: in function </etc/nginx/conf.d/lua/select_host.lua:1>, client: 172.17.42.1, server: www.example.com, request: "GET /js/jquery.bgiframe.min.js HTTP/1.1", host: "www.yoda.lan", referrer: "https://www.yoda.lan/"
2014/05/12 01:39:38 [error] 1002#0: *66 [lua] hostmap.lua:36: fetch(): Why so bad request? vid=020011AC320D70539B01419A02030303 : 3660, client: 172.17.42.1, server: www.example.com, request: "GET /js/jquery.simpledialog.0.1.js HTTP/1.1", host: "www.yoda.lan", referrer: "https://www.yoda.lan/"
2014/05/12 01:39:38 [error] 1002#0: *84 [lua] hostmap.lua:36: fetch(): Why so bad request? vid=020011AC320D70539B01419A02030303 : 3660, client: 172.17.42.1, server: www.example.com, request: "GET /js/jquery.qtip-1.0.0-rc3.min.js HTTP/1.1", host: "www.yoda.lan", referrer: "https://www.yoda.lan/"
http://example.com/everythingelsegoes?this&that=all
how do i get
http://example.com/
and
everythingelsegoes?this&that=all
inside lua content_by_lua?
great software. thanks.
I'm tring install this lib via luarocks. It desn't work. But i see lots of lib named 'lua-resty-*'. Why not include this? I'm sorry my english is poor.
I'm trying to stream data from a Redis Pub/Sub channel to the websockets. Problem is I don't understand how to wrap the read_reply method. How can I "listen" to the Redis channel in a loop?
The following snippet works, but there's a big delay between server pushes, I think I'm blocking something. Here is my code:
server { listen 8881; server_name localhost; location /stream { lua_socket_log_errors on; lua_check_client_abort on; content_by_lua ' local redis = require "resty.redis" local cjson = require "cjson" local red = redis:new() red:set_timeout(1000) -- 1 sec local ok, err = red:connect("127.0.0.1", 6379) if not ok then ngx.say("failed to connect: ", err) return end local res, err = red:subscribe("streampic") if not res then ngx.say("1: failed to subscribe: ", err) return end local clients = {} local server = require "resty.websocket.server" local wb, err = server:new{ timeout = 5000, max_payload_len = 65535 } if not wb then ngx.log(ngx.ERR, "failed to new websocket: ", err) return ngx.exit(444) end while true do local data, typ, err = wb:recv_frame() -- Websocket related code block (ping/close etc.) --read the pubsub and wire it res, err = red:read_reply() if res then local f = io.open(res[3], "rb") if f then local data = f:read("*all") wb:send_binary(data) end end end wb:send_close()'; }
please add evalsha to the local commands table.
i am trying to use lua-resty-redis in my web server which for a game.
it works very nice when there are few requests per second.
but the cost of close the sokect is too expensive when request frequently.
so i want to implement a redis connection pool for each nginx worker.
we cannot use upstream in the lua code to access redis server ,am i right?
how to do it?
So that SCRIPT family of commands [1] can be used.
Thanks!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.