cloudflare / lua-upstream-cache-nginx-module Goto Github PK
View Code? Open in Web Editor NEWNginx module for ngx_lua to provide Lua API to inspect upstream http cache meta-data
License: Other
Nginx module for ngx_lua to provide Lua API to inspect upstream http cache meta-data
License: Other
Name http_cache - Expose & modify the internal nginx cache metadata. *This module is not distributed with the Nginx source or in the lua-nginx module* Installation is similar to other nginx modules. Status This module is under development and is used in production. Version This document describes http_cache v0.1.1 Synopsis # set search paths for pure Lua external libraries (';;' is the default path): lua_package_path '/foo/bar/?.lua;/blah/?.lua;;'; # set search paths for Lua external libraries written in C (can also use ';;'): lua_package_cpath '/bar/baz/?.so;/blah/blah/?.so;;'; server { listen 8000 default_server; server_name localhost; location / { proxy_cache pcache; proxy_pass http://127.0.0.1:8001/; # Force entries in cache to only be cached for 5 seconds # This is equivalent to: # proxy_ignore_headers "Expires" "Cache-Control; # proxy_cache_valid any 5s; header_filter_by_lua ' local http_cache = require("http_cache") local cache_status = (ngx.var.upstream_cache_status or "") if cache_status == "MISS" or cache_status == "EXPIRED" then local cache_data = http_cache.get_metadata() local new_expire = ngx.time() + 5 if cache_data and cache_data["valid_sec"] then http_cache.set_metadata({ valid_sec = new_expire, fcn = { valid_sec = new_expire, expire = new_expire } }) end end '; } Description This module extends the lua-nginx Lua module. It adds the ability to read and set values from nginx's internal cache metadata. API get_metadata() - returns a table with all cache metadata. The fields are: Resource specific fields key: Hexadecimal cache key for this resource crc32: Numeric crc32 calculated by nginx to check cache collisions valid_sec: Absolute time, in seconds since Epoch, when this entry expires. last_modified: Absolute time of last-modified header in seconds, if present. date: Absolute time of date header in seconds, if present. length: Length of contents fs_size: Size of resource, in disk blocks. NOTE: This can be converted to bytes by multiplying by bsize below. min_uses: Minimum number of uses required to cache this resource. valid_msec: millisecond component of valid_secs (?) Shared Cache information, under "sh" size Cache Data, under "fcn" max_size: maximum allowable cache size in 512B blocks bsize: size of disk block in bytes. inactive: inactive value for cache. files: currently open files used by loader loader_files: maximum number of files for the loader to open at once loader_sleep: time between file opens loader_threshold: aggressive_purge: Flag for whether cache manager will use aggressive purge policy set_metadat(tbl) - sets cache metadata from a table similar to the one returned by get_metadata. Fields that will be written are: Resource specific fields valid_sec, last_modified, date, min_uses, valid_msec Cache Data, under "fcn" uses, valid_msec, expire, valid_sec
This might be an issue not specific to this module but this is where i encountered it.
if i set lua_code_cache off
in http
block, i get the error the module is not found, when i comment that config line, the module loads and everything works.
Any idea why this might be happening?
I needed this in order to work on the scripts without restarting nginx all the time.
Also, not sure if it's relevant, but openresty is running in docker (based on alpine).
Thank you
Hello,
I was just trying to experiment this module and I noticed that every second request goes to backend and every second request's response header says cache-status "EXPIRED" regardless of first request is MISS or EXPIRED. After that all subsequent requests get HIT until it actually expired according to time defined in header_filter_by_lua .
I have used the same code which is given in readme file. And proxy to same nginx server running at higher port.
I have also noticed that the cache file is being created on first request only which is expected behavior and if I remove this header_filter_by_lua code, each second request gets cache HIT.
Is this same code(which is given in readme) working in production servers or is there any change in readme?
Thanks,
Is this module still relevant for nginx 1.9.12 ?
CentOS 7 with nginx 1.9.12 getting errors
nginx configure options (alternate ordering):
./configure --with-ld-opt="-ljemalloc -Wl,-z,relro -Wl,-rpath,/usr/local/lib" --with-cc-opt="-m64 -mtune=native -mfpmath=sse -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2" --sbin-path=/usr/local/sbin/nginx --conf-path=/usr/local/nginx/conf/nginx.conf --with-http_stub_status_module --with-http_secure_link_module --with-openssl-opt="enable-tlsext" --add-module=../nginx-module-vts --with-libatomic --with-threads --with-stream=dynamic --with-stream_ssl_module --with-http_gzip_static_module --add-dynamic-module=../ngx_brotli --add-dynamic-module=../ngx_pagespeed-release-1.10.33.6-beta --with-http_sub_module --with-http_addition_module --with-http_image_filter_module=dynamic --with-http_geoip_module=dynamic --with-http_realip_module --add-module=../ngx-fancyindex-0.3.6 --add-module=../ngx_cache_purge-2.3 --add-module=../ngx_devel_kit-0.3.0rc1 --add-dynamic-module=../set-misc-nginx-module-0.30 --add-dynamic-module=../echo-nginx-module-0.58 --add-module=../redis2-nginx-module-0.12 --add-module=../ngx_http_redis-0.3.7 --add-dynamic-module=../lua-nginx-module-0.10.2 --add-module=../lua-upstream-nginx-module-0.04 --add-module=../lua-upstream-cache-nginx-module-0.1.1 --add-module=../nginx_upstream_check_module-0.3.0 --add-module=../openresty-memc-nginx-module-4f6f78f --add-module=../srcache-nginx-module-0.30 --add-dynamic-module=../headers-more-nginx-module-0.29 --with-pcre=../pcre-8.38 --with-pcre-jit --with-http_ssl_module --with-http_v2_module --with-openssl=../openssl-1.0.2g
checking for OS
- Linux 2.6.32-042stab112.15 x86_64
checking for C compiler ... found- using GNU C compiler
- gcc version: 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC)
configuring additional modules
adding module in ../nginx-module-vts
+ ngx_http_vhost_traffic_status_module was configured
adding module in ../ngx-fancyindex-0.3.6
+ ngx_http_fancyindex_module was configured
adding module in ../ngx_cache_purge-2.3
+ ngx_http_cache_purge_module was configured
adding module in ../ngx_devel_kit-0.3.0rc1
+ ngx_devel_kit was configured
adding module in ../redis2-nginx-module-0.12
+ ngx_http_redis2_module was configured
adding module in ../ngx_http_redis-0.3.7
+ ngx_http_redis_module was configured
adding module in ../lua-upstream-nginx-module-0.04
+ ngx_http_lua_upstream_module was configured
adding module in ../lua-upstream-cache-nginx-module-0.1.1
+ ngx_http_lua_cache_module was configured
adding module in ../nginx_upstream_check_module-0.3.0
checking for ngx_http_upstream_check_module ... found
+ ngx_http_upstream_check_module was configured
adding module in ../openresty-memc-nginx-module-4f6f78f
+ ngx_http_memc_module was configured
adding module in ../srcache-nginx-module-0.30
+ ngx_http_srcache_filter_module was configured
configuring additional dynamic modules
adding module in ../ngx_brotli
checking for Brotli library ... found
+ ngx_brotli was configured
adding module in ../ngx_pagespeed-release-1.10.33.6-beta
mod_pagespeed_dir=../ngx_pagespeed-release-1.10.33.6-beta/psol/include
build_from_source=false
checking for psol ... found
List of modules (in reverse order of applicability): ngx_http_write_filter_module ngx_http_header_filter_module ngx_http_chunked_filter_module ngx_http_v2_filter_module ngx_http_range_header_filter_module ngx_http_gzip_filter_module ngx_http_postpone_filter_module ngx_http_ssi_filter_module ngx_http_charset_filter_module ngx_http_sub_filter_module ngx_http_addition_filter_module ngx_http_userid_filter_module ngx_http_headers_filter_module
checking for psol-compiler-compat ... found
+ ngx_pagespeed was configured
adding module in ../set-misc-nginx-module-0.30
found ngx_devel_kit for ngx_set_misc; looks good.
+ ngx_http_set_misc_module was configured
adding module in ../echo-nginx-module-0.58
+ ngx_http_echo_module was configured
adding module in ../lua-nginx-module-0.10.2
checking for LuaJIT library in /usr/local/lib and /usr/local/include/luajit-2.1 (specified by the LUAJIT_LIB and LUAJIT_INC env, with -ldl) ... found
checking for export symbols by default (-E) ... found
checking for export symbols by default (--export-all-symbols) ... not found
checking for SO_PASSCRED ... found
+ ngx_http_lua_module was configured
adding module in ../headers-more-nginx-module-0.29
+ ngx_http_headers_more_filter_module was configured
checking for zlib library ... found
checking for GD library ... found
checking for GeoIP library ... found
checking for atomic_ops library ... found
creating objs/Makefile
objs/addon/src/ngx_http_lua_cache_control.o: In function `ngx_http_lua_ngx_set_cache_data':
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:276: undefined reference to `luaL_error'
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:318: undefined reference to `luaL_error'
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:283: undefined reference to `luaL_error'
objs/addon/src/ngx_http_lua_cache_control.o: In function `ngx_http_lua_ngx_get_cache_data':
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:112: undefined reference to `luaL_error'
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:66: undefined reference to `luaL_error'
objs/addon/src/ngx_http_lua_cache_control.o:/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:369: more undefined references to `luaL_error' follow
collect2: error: ld returned 1 exit status
make[1]: *** [objs/nginx] Error 1
make[1]: Leaving directory `/svr-setup/nginx-1.9.12'
make: *** [build] Error 2
objs/addon/src/ngx_http_lua_cache_control.o: In function `ngx_http_lua_ngx_cache_purge':
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:366: undefined reference to `lua_gettop'
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:372: undefined reference to `ngx_http_lua_get_request'
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:386: undefined reference to `lua_pushboolean'
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:417: undefined reference to `lua_pushboolean'
objs/addon/src/ngx_http_lua_cache_control.o: In function `ngx_http_lua_inject_cache_control_api':
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:33: undefined reference to `lua_createtable'
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:36: undefined reference to `lua_pushcclosure'
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:37: undefined reference to `lua_setfield'
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:40: undefined reference to `lua_pushcclosure'
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:41: undefined reference to `lua_setfield'
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:44: undefined reference to `lua_pushcclosure'
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:45: undefined reference to `lua_setfield'
objs/addon/src/ngx_http_lua_cache_control.o: In function `ngx_http_lua_ngx_set_cache_data':
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:276: undefined reference to `luaL_error'
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:318: undefined reference to `luaL_error'
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:283: undefined reference to `luaL_error'
objs/addon/src/ngx_http_lua_cache_control.o: In function `ngx_http_lua_ngx_get_cache_data':
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:112: undefined reference to `luaL_error'
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:66: undefined reference to `luaL_error'
objs/addon/src/ngx_http_lua_cache_control.o:/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:369: more undefined references to `luaL_error' follow
collect2: error: ld returned 1 exit status
make[1]: *** [objs/nginx] Error 1
``
hi:
I use the module to set meta info when upstream_cache_status is HIT, but I found this is unvalid.
code:
header_filter_by_lua '
local http_cache = require("http_cache")
local cache_status = (ngx.var.upstream_cache_status or "")
if cache_status == "HIT" and ngx.ctx.flag == "true" then
local cache_data = http_cache.get_metadata()
local new_expire = ngx.time() - 1
if cache_data and cache_data["valid_sec"] then
http_cache.set_metadata({ valid_sec = new_expire,
fcn = { valid_sec = new_expire,
expire = new_expire } })
end
end
';
Can I use it like this? thx!
when using lua-upstream-cache-nginx-module, it reports like this.
2015/03/04 16:11:34 F[ngx_http_lua_headerfilterby.c] L[134] f[ngx_http_lua_header_filter_by_chunk] [error] 12980#0: 3 failed to run header_filter_by_lua: /home/sshd/nginx/lua/hdrfilter.lua:51: module 'http_cache' not found:
it seems that http_cache is needed but can not found.
Does anybody meet same question?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.