cloudflare / lua-upstream-cache-nginx-module Goto Github PK

Nginx module for ngx_lua to provide Lua API to inspect upstream http cache meta-data

License: Other

C 97.93% Perl 6 2.07%

lua-upstream-cache-nginx-module's Introduction

Name
    http_cache - Expose & modify the internal nginx cache metadata.

    *This module is not distributed with the Nginx source or in the lua-nginx
    module*
    Installation is similar to other nginx modules.

Status
    This module is under development and is used in production.

Version
    This document describes http_cache v0.1.1

Synopsis
        # set search paths for pure Lua external libraries (';;' is the default path):
        lua_package_path '/foo/bar/?.lua;/blah/?.lua;;';

        # set search paths for Lua external libraries written in C (can also use ';;'):
        lua_package_cpath '/bar/baz/?.so;/blah/blah/?.so;;';

        server {
            listen       8000 default_server;
            server_name  localhost;

        location / {
            proxy_cache pcache;
            proxy_pass http://127.0.0.1:8001/;

            # Force entries in cache to only be cached for 5 seconds
            # This is equivalent to:
            # proxy_ignore_headers "Expires" "Cache-Control;
            # proxy_cache_valid any 5s;
            header_filter_by_lua '
               local http_cache = require("http_cache")
               local cache_status = (ngx.var.upstream_cache_status or "")

               if cache_status == "MISS" or cache_status == "EXPIRED" then
                   local cache_data = http_cache.get_metadata()
                   local new_expire = ngx.time() + 5

                   if cache_data and cache_data["valid_sec"] then
                       http_cache.set_metadata({ valid_sec = new_expire,
                                                 fcn = { valid_sec = new_expire,
                                                     expire = new_expire } })
                   end
               end
            ';
        }

Description
    This module extends the lua-nginx Lua module. It adds the ability to read
    and set values from nginx's internal cache metadata.

API
    get_metadata() - returns a table with all cache metadata.
    The fields are:
    Resource specific fields
        key: Hexadecimal cache key for this resource
        crc32: Numeric crc32 calculated by nginx to check cache collisions
        valid_sec: Absolute time, in seconds since Epoch, when this entry expires.
        last_modified: Absolute time of last-modified header in seconds, if present.
        date: Absolute time of date header in seconds, if present.
	length: Length of contents
	fs_size: Size of resource, in disk blocks.
           NOTE: This can be converted to bytes by multiplying by bsize below.
	min_uses: Minimum number of uses required to cache this resource.
        valid_msec: millisecond component of valid_secs (?)

    Shared Cache information, under "sh"
        size

    Cache Data, under "fcn"
        max_size: maximum allowable cache size in 512B blocks
        bsize: size of disk block in bytes.
        inactive: inactive value for cache.
        files: currently open files used by loader
        loader_files: maximum number of files for the loader to open at once
        loader_sleep: time between file opens
        loader_threshold:
        aggressive_purge: Flag for whether cache manager will use aggressive
            purge policy

    set_metadat(tbl) - sets cache metadata from a table similar to the one 
    returned by get_metadata. Fields that will be written are:
    Resource specific fields
    valid_sec, last_modified, date, min_uses, valid_msec

    Cache Data, under "fcn"
    uses, valid_msec, expire, valid_sec

lua-upstream-cache-nginx-module's People

Contributors

Stargazers

Watchers

lua-upstream-cache-nginx-module's Issues

http_cache module not found when lua_code_cache is off

This might be an issue not specific to this module but this is where i encountered it.

if i set lua_code_cache off in http block, i get the error the module is not found, when i comment that config line, the module loads and everything works.

Any idea why this might be happening?
I needed this in order to work on the scripts without restarting nginx all the time.
Also, not sure if it's relevant, but openresty is running in docker (based on alpine).

Thank you

Every second request goes to backend.

Hello,

I was just trying to experiment this module and I noticed that every second request goes to backend and every second request's response header says cache-status "EXPIRED" regardless of first request is MISS or EXPIRED. After that all subsequent requests get HIT until it actually expired according to time defined in header_filter_by_lua .

I have used the same code which is given in readme file. And proxy to same nginx server running at higher port.

I have also noticed that the cache file is being created on first request only which is expected behavior and if I remove this header_filter_by_lua code, each second request gets cache HIT.

Is this same code(which is given in readme) working in production servers or is there any change in readme?

Thanks,

nginx 1.9.12 support ?

Is this module still relevant for nginx 1.9.12 ?

CentOS 7 with nginx 1.9.12 getting errors

nginx configure options (alternate ordering):
./configure --with-ld-opt="-ljemalloc -Wl,-z,relro -Wl,-rpath,/usr/local/lib" --with-cc-opt="-m64 -mtune=native -mfpmath=sse -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2" --sbin-path=/usr/local/sbin/nginx --conf-path=/usr/local/nginx/conf/nginx.conf --with-http_stub_status_module --with-http_secure_link_module --with-openssl-opt="enable-tlsext" --add-module=../nginx-module-vts --with-libatomic --with-threads --with-stream=dynamic --with-stream_ssl_module --with-http_gzip_static_module --add-dynamic-module=../ngx_brotli --add-dynamic-module=../ngx_pagespeed-release-1.10.33.6-beta --with-http_sub_module --with-http_addition_module --with-http_image_filter_module=dynamic --with-http_geoip_module=dynamic --with-http_realip_module --add-module=../ngx-fancyindex-0.3.6 --add-module=../ngx_cache_purge-2.3 --add-module=../ngx_devel_kit-0.3.0rc1 --add-dynamic-module=../set-misc-nginx-module-0.30 --add-dynamic-module=../echo-nginx-module-0.58 --add-module=../redis2-nginx-module-0.12 --add-module=../ngx_http_redis-0.3.7 --add-dynamic-module=../lua-nginx-module-0.10.2 --add-module=../lua-upstream-nginx-module-0.04 --add-module=../lua-upstream-cache-nginx-module-0.1.1 --add-module=../nginx_upstream_check_module-0.3.0 --add-module=../openresty-memc-nginx-module-4f6f78f --add-module=../srcache-nginx-module-0.30 --add-dynamic-module=../headers-more-nginx-module-0.29 --with-pcre=../pcre-8.38 --with-pcre-jit --with-http_ssl_module --with-http_v2_module --with-openssl=../openssl-1.0.2g
checking for OS

Linux 2.6.32-042stab112.15 x86_64
checking for C compiler ... found

using GNU C compiler

gcc version: 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC)

configuring additional modules
adding module in ../nginx-module-vts
 + ngx_http_vhost_traffic_status_module was configured
adding module in ../ngx-fancyindex-0.3.6
 + ngx_http_fancyindex_module was configured
adding module in ../ngx_cache_purge-2.3
 + ngx_http_cache_purge_module was configured
adding module in ../ngx_devel_kit-0.3.0rc1
 + ngx_devel_kit was configured
adding module in ../redis2-nginx-module-0.12
 + ngx_http_redis2_module was configured
adding module in ../ngx_http_redis-0.3.7
 + ngx_http_redis_module was configured
adding module in ../lua-upstream-nginx-module-0.04
 + ngx_http_lua_upstream_module was configured
adding module in ../lua-upstream-cache-nginx-module-0.1.1
 + ngx_http_lua_cache_module was configured
adding module in ../nginx_upstream_check_module-0.3.0
checking for ngx_http_upstream_check_module ... found
 + ngx_http_upstream_check_module was configured
adding module in ../openresty-memc-nginx-module-4f6f78f
 + ngx_http_memc_module was configured
adding module in ../srcache-nginx-module-0.30
 + ngx_http_srcache_filter_module was configured
configuring additional dynamic modules
adding module in ../ngx_brotli
checking for Brotli library ... found
 + ngx_brotli was configured
adding module in ../ngx_pagespeed-release-1.10.33.6-beta
mod_pagespeed_dir=../ngx_pagespeed-release-1.10.33.6-beta/psol/include
build_from_source=false
checking for psol ... found
List of modules (in reverse order of applicability):  ngx_http_write_filter_module ngx_http_header_filter_module ngx_http_chunked_filter_module ngx_http_v2_filter_module ngx_http_range_header_filter_module ngx_http_gzip_filter_module ngx_http_postpone_filter_module ngx_http_ssi_filter_module ngx_http_charset_filter_module ngx_http_sub_filter_module ngx_http_addition_filter_module ngx_http_userid_filter_module ngx_http_headers_filter_module
checking for psol-compiler-compat ... found
 + ngx_pagespeed was configured
adding module in ../set-misc-nginx-module-0.30
found ngx_devel_kit for ngx_set_misc; looks good.
 + ngx_http_set_misc_module was configured
adding module in ../echo-nginx-module-0.58
 + ngx_http_echo_module was configured
adding module in ../lua-nginx-module-0.10.2
checking for LuaJIT library in /usr/local/lib and /usr/local/include/luajit-2.1 (specified by the LUAJIT_LIB and LUAJIT_INC env, with -ldl) ... found
checking for export symbols by default (-E) ... found
checking for export symbols by default (--export-all-symbols) ... not found
checking for SO_PASSCRED ... found
 + ngx_http_lua_module was configured
adding module in ../headers-more-nginx-module-0.29
 + ngx_http_headers_more_filter_module was configured
checking for zlib library ... found
checking for GD library ... found
checking for GeoIP library ... found
checking for atomic_ops library ... found
creating objs/Makefile

objs/addon/src/ngx_http_lua_cache_control.o: In function `ngx_http_lua_ngx_set_cache_data':
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:276: undefined reference to `luaL_error'
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:318: undefined reference to `luaL_error'
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:283: undefined reference to `luaL_error'
objs/addon/src/ngx_http_lua_cache_control.o: In function `ngx_http_lua_ngx_get_cache_data':
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:112: undefined reference to `luaL_error'
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:66: undefined reference to `luaL_error'
objs/addon/src/ngx_http_lua_cache_control.o:/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:369: more undefined references to `luaL_error' follow
collect2: error: ld returned 1 exit status
make[1]: *** [objs/nginx] Error 1
make[1]: Leaving directory `/svr-setup/nginx-1.9.12'
make: *** [build] Error 2

objs/addon/src/ngx_http_lua_cache_control.o: In function `ngx_http_lua_ngx_cache_purge':
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:366: undefined reference to `lua_gettop'
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:372: undefined reference to `ngx_http_lua_get_request'
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:386: undefined reference to `lua_pushboolean'
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:417: undefined reference to `lua_pushboolean'
objs/addon/src/ngx_http_lua_cache_control.o: In function `ngx_http_lua_inject_cache_control_api':
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:33: undefined reference to `lua_createtable'
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:36: undefined reference to `lua_pushcclosure'
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:37: undefined reference to `lua_setfield'
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:40: undefined reference to `lua_pushcclosure'
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:41: undefined reference to `lua_setfield'
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:44: undefined reference to `lua_pushcclosure'
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:45: undefined reference to `lua_setfield'
objs/addon/src/ngx_http_lua_cache_control.o: In function `ngx_http_lua_ngx_set_cache_data':
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:276: undefined reference to `luaL_error'
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:318: undefined reference to `luaL_error'
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:283: undefined reference to `luaL_error'
objs/addon/src/ngx_http_lua_cache_control.o: In function `ngx_http_lua_ngx_get_cache_data':
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:112: undefined reference to `luaL_error'
/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:66: undefined reference to `luaL_error'
objs/addon/src/ngx_http_lua_cache_control.o:/svr-setup/nginx-1.9.12/../lua-upstream-cache-nginx-module-0.1.1/src/ngx_http_lua_cache_control.c:369: more undefined references to `luaL_error' follow
collect2: error: ld returned 1 exit status
make[1]: *** [objs/nginx] Error 1
``

the module is unvalid when upstream_cache_status is HIT

hi:
I use the module to set meta info when upstream_cache_status is HIT, but I found this is unvalid.

code:
header_filter_by_lua '
local http_cache = require("http_cache")
local cache_status = (ngx.var.upstream_cache_status or "")

           if cache_status == "HIT" and ngx.ctx.flag == "true" then
               local cache_data = http_cache.get_metadata()
               local new_expire = ngx.time() - 1

               if cache_data and cache_data["valid_sec"] then
                   http_cache.set_metadata({ valid_sec = new_expire,
                                             fcn = { valid_sec = new_expire,
                                                 expire = new_expire } })
               end
           end
        ';

Can I use it like this? thx！

where to find http_cache module

when using lua-upstream-cache-nginx-module, it reports like this.

2015/03/04 16:11:34 F[ngx_http_lua_headerfilterby.c] L[134] f[ngx_http_lua_header_filter_by_chunk] [error] 12980#0: 3 failed to run header_filter_by_lua: /home/sshd/nginx/lua/hdrfilter.lua:51: module 'http_cache' not found:

it seems that http_cache is needed but can not found.

Does anybody meet same question?

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.