GithubHelp home page GithubHelp logo

haproxy-lua-http's Introduction

Lua HTTP client library for HAProxy

This is a pure Lua HTTP 1.1 library for HAProxy. It should work on any modern HAProxy version (1.7+)

The library is loosely modeled after Python's Requests Library using the same attribute names and similar calling conventions for "HTTP verb" methods (where we use Lua specific named parameter support)

In addition to client side, the library also supports server side request parsing, where we utilize HAProxy Lua API for all heavy lifting (i.e. HAProxy handles client side connections, parses the headers and gives us access to request body).

Usage

After downloading this library, you will need to move it into your Lua package package path, to be able to use it from your script. In later HAProxy versions, there is a lua-prepend-path directive which can make your life easier.

Basic usage for parsing client requests, constructing responses, or sending custom requests to external servers is demonstrated bellow. You can use this in your own Lua actions or services:

local http = require('http')

local function main(applet)

    -- 1) Parse client side request and print received headers

    local req = http.request.parse(applet)
    for k, v in req:get_headers() do
        core.Debug(k .. ": " .. v)
    end

    -- You can also parse submitted form data
    local form, err = req:parse_multipart()

    -- 2) Send request to external server (please note there is no DNS
    --    support for Lua on HAProxy

    local res, err = http.get{url="http://1.2.3.4",
        headers={host="example.net", ["x-test"]={"a", "b"}}
    }

    if res then
        for k, v in res:get_headers() do
            core.Debug(k .. ": " .. v)
        end
        -- We can access the response body in content attribute
        core.Debug(res.content)
    else
      core.Debug(err)
    end


    -- 3) Send response to client side
    http.response.create{status_code=200, content="Hello World"}:send(applet)

end

core.register_service("test", "http", main)

Naturally, you need to add your script to main haproxy configuration:

global
  ...
  lua-load test.lua

frontend test
  ...
  http-request use-service lua.test

haproxy-lua-http's People

Contributors

anezirovic avatar epicfilemcnulty avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

haproxy-lua-http's Issues

Incorrect handling of HEAD requests with Content-Length > 0

Example configuration:

global
	lua-prepend-path /redacted/?.lua
	lua-load test.lua

frontend test_fe
    mode http
    bind *:8080
    http-request lua.test
    use_backend test_be

backend test_be
    mode http
    server example example.com:80
local http = require('http')

core.register_action("test", {"http-req"},function ()
	local r, err = http.head {
		url = "http://93.184.216.34/"
	}

	if err ~= nil then
		core.Info("err: " .. err)
	else
		core.Info(r.content)
	end
end)

Now making a request to localhost:8080 will log the following:

[info] 053/235929 (28384) : err: http.head: Receive error (content): connection closed.

RFC 7230#3.3.2 says:

A server MAY send a Content-Length header field in a response to a
HEAD request (Section 4.3.2 of [RFC7231]); a server MUST NOT send
Content-Length in such a response unless its field-value equals the
decimal number of octets that would have been sent in the payload
body of a response if the same request had used the GET method.

and that's what the example.com server is doing: It's sending a non-zero Content-Length in response to the HEAD request, leading to haproxy-lua-http falling over.

How to use lua-prepend-path?

I'd really appreciate an example on the page. Instead of just mentioning this in passing, could you give a quick example of the line that should be added to haproxy.conf?

And for those running an older version - what is the alternative?

Unable to import json library

Hi,

I saw a similar issue earlier this year about using this script to integrate authelia with haproxy 2.2. I have manually imported your lua script into /usr/local/share/lua/5.3/haproxy-lua-http.lua along with https://github.com/TimWolla/haproxy-auth-request/blob/main/auth-request.lua and I have installed lua-json library with apt as I'm running on debian buster.

After this, I tried adding the required loading lines in the global section

lua-prepend-path /usr/local/share/lua/5.3/haproxy-lua-http.lua
lua-load /usr/local/share/lua/5.3/auth-request.lua
When adding the second line, then HAProxy starts to complain about your file and I have no idea why:

Errors found while starting haproxy:

[NOTICE] 106/215753 (2014) : haproxy version is 2.2.22-1~bpo10+1 [NOTICE] 106/215753 (2014) : path to executable is /usr/sbin/haproxy [ALERT] 106/215753 (2014) : parsing [haproxy.cfg:5] : Lua runtime error: error loading module 'json' from file '/usr/share/lua/5.3/http.lua': /usr/share/lua/5.3/http.lua:673: too many C levels (limit is 200) in function at line 619 near 'v'

Can you help me figure this out?

Cheers! And thank you in advance for your time :)

Dechunking of response not working correctly

It appears that the length of a received chunk is completely ignored and instead it is assumed that a chunk will consist of a single line, which clearly is incorrect for any kind of binary data (e.g. a gzip stream).

Unable to handle a single POST parameter

Realized when making a request like this:

curl -v -X POST 127.0.0.1/test -H "Content-Type: application/x-www-form-urlencoded" -d "param1=value1"

That the params was empty. Turned out that the logic required an & in the body (which wouldn't exist if there was a single parameter).

Fixed by minor duplication of the part parsing: if the body had no ampersand the break caused it to exit before parsing any arguments, as well as caused it to ignore the last argument.

Feature Request: Support UNIX sockets

The Socket class used for making connections supports connecting to UNIX sockets, by passing an address of e.g. unix@/run/foo. Right now, it's not possible to use this feature through the HTTP library because the library mandates a prefix of http[s], and also always passes a port.

It would be really useful for tighter control of calls to sensitive endpoints to be able to use UNIX sockets. For example in the Let's Encrypt example, it would be preferable for the proxied ACME endpoints to be exposed on UNIX sockets (where they could be restricted by file system permissions), rather than 127.0.0.1 (where any user with access to the host could hit them).

request.create() function is not instanciating a complete new request object

Hello,

I have found a small issue with the request.create() function.

Let's start with an example:

-- the servers that lua http client will call
servers = {}
table.insert(servers, { name = 'server42', baseUrl = 'http://dummy42' } )
table.insert(servers, { name = 'server44', baseUrl = 'http://dummy44' } )


function replaceBaseUrl(url, newBaseUrl)
	return url:gsub("^(https?://.-)/", newBaseUrl .. '/')
end


function routeBroadcast(applet)
	-- get the incoming request
	local originalRequest = http.request.parse(applet)

	
	-- now create and send modified requests to all servers
	for k, server in pairs(servers) do
		-- copy the orignal request
		local newRequest = http.request.create(originalRequest)
		-- adjust url
		newRequest.url = replaceBaseUrl(originalRequest.url, server.baseUrl)
		newRequest.headers['host'] = nil
		
		-- do some headers customization for each server
		-- in reality, much more complex than this
		if server.name == 'server42' then
			newRequest.headers['my-custom-header'] = nil
		end

		http.send(originalRequest.method, newRequest)
	end
	
	
	-- write final answer to hap client
	local body = 'all done'
	applet:add_header("Content-Length", #tostring(body))
	applet:set_status(200)
	applet:start_response()
	applet:send(tostring(body))
end


core.register_service("broadcast", "http", function(applet)
	return routeBroadcast(applet)
end)

Basically, just a broadcasting backend, sending an incoming request to several servers.
If the incoming request contains a 'my-custom-header' header, the customization in my loop will remove it for the call to the 1st backend. But the header should still be there for the call to the 2nd server.

Seems that the faulty lines are these ones in the http module:

    self.url = t.url or nil
    self.headers = t.headers or {}
    self.data = t.data or nil
    self.params = t.params or {}
    self.auth = t.auth or {}

url and data are properly copied, as they're strings. But for the 3 tables (headers, params and auth), only the reference of the first request is copied.

Each further edit is therefore targeting the original request, as well as its copy.

Using a proper object copy function should do the trick:
Haven't tried it myself for the moment : http://lua-users.org/wiki/CopyTable

A bogus APPCTX [0x7f1a30024b20] is spinning at 100000 calls per second and refuses to die, aborting now!

Reproduction steps:

crasher.cfg

    global
        lua-prepend-path ./?.lua
        lua-load ./crasher.lua

    listen fe1
        mode http
        bind *:8080

        http-request lua.crasher

crasher.lua

local http = require('http')

core.register_action("crasher", { "http-req" }, function(txn)
	local response, err = http.head {
		url = "http://127.0.0.1:9000",
		headers = {},
	}
end)

haproxy -vv

HA-Proxy version 2.3-dev0-950954-25 2020/07/10 - https://haproxy.org/
Status: development branch - not safe for use in production.
Known bugs: https://github.com/haproxy/haproxy/issues?q=is:issue+is:open
Running on: Linux 4.4.0-185-generic #215-Ubuntu SMP Mon Jun 8 21:53:19 UTC 2020 x86_64
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -Wall -Wextra -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits
  OPTIONS = USE_LUA=1 HLUA_PREPEND_PATH=/usr/share/lua/5.2/?.lua

Feature list : +EPOLL -KQUEUE +NETFILTER -PCRE -PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED +BACKTRACE -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H +GETADDRINFO -OPENSSL +LUA +FUTEX +ACCEPT4 -ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=4).
Built with Lua version : Lua 5.3.1
Built with network namespace support.
Built without compression support (neither USE_ZLIB nor USE_SLZ are set).
Compression algorithms supported : identity("identity")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built without PCRE or PCRE2 support (using libc's regex instead)
Encrypted password support via crypt(3): yes
Built with gcc compiler version 5.4.0 20160609

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
            fcgi : mode=HTTP       side=BE        mux=FCGI
       <default> : mode=HTTP       side=FE|BE     mux=H1
              h2 : mode=HTTP       side=FE|BE     mux=H2
       <default> : mode=TCP        side=FE|BE     mux=PASS

Available services : none

Available filters :
	[SPOE] spoe
	[COMP] compression
	[TRACE] trace
	[CACHE] cache
	[FCGI] fcgi-app

List of steps:

  1. haproxy -d -f ./crasher.cfg
  2. printf "foo" |nc -l 127.0.0.1 9000
  3. curl localhost:8080
[NOTICE] 192/134456 (17264) : haproxy version is 2.3-dev0-950954-25
[NOTICE] 192/134456 (17264) : path to executable is *snip*/haproxy
[ALERT] 192/134456 (17264) : A bogus APPCTX [0xa6d090] is spinning at 175268 calls per second and refuses to die, aborting now! Please report this error to developers [strm=0xa6d320 src=<LUA_TCP> fe=LUA-SOCKET be=LUA-SOCKET dst=LUA-TCP-CONN rqf=848000 rqa=0 rpf=8000c020 rpa=0 sif=EST,204048 sib=EST,280058 af=0xa6d090,0 csf=(nil),0 ab=(nil),0 csb=0xa6de60,9000 cof=(nil),0:NONE((nil))/NONE((nil))/NONE(0) cob=0xa6a620,42300:PASS(0xa6deb0)/RAW((nil))/tcpv4(16) filters={}]

Thread 1 "haproxy" received signal SIGABRT, Aborted.
0x00007ffff6faa438 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
54	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) t a a bt

Thread 4 (Thread 0x7fffef445700 (LWP 17271)):
#0  0x00007ffff707cad3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
#1  0x000000000041da40 in _do_poll (p=<optimized out>, exp=0, wake=0) at src/ev_epoll.c:195
#2  0x00000000005242d7 in run_poll_loop () at src/haproxy.c:3003
#3  0x00000000005246ef in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3121
#4  0x00007ffff757d6ba in start_thread (arg=0x7fffef445700) at pthread_create.c:333
#5  0x00007ffff707c4dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 3 (Thread 0x7ffff0047700 (LWP 17270)):
#0  0x00007ffff707cad3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
#1  0x000000000041da40 in _do_poll (p=<optimized out>, exp=0, wake=0) at src/ev_epoll.c:195
#2  0x00000000005242d7 in run_poll_loop () at src/haproxy.c:3003
#3  0x00000000005246ef in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3121
#4  0x00007ffff757d6ba in start_thread (arg=0x7ffff0047700) at pthread_create.c:333
#5  0x00007ffff707c4dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 2 (Thread 0x7ffff0848700 (LWP 17269)):
#0  0x00007ffff707cad3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
#1  0x000000000041da40 in _do_poll (p=<optimized out>, exp=0, wake=0) at src/ev_epoll.c:195
#2  0x00000000005242d7 in run_poll_loop () at src/haproxy.c:3003
#3  0x00000000005246ef in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3121
#4  0x00007ffff757d6ba in start_thread (arg=0x7ffff0848700) at pthread_create.c:333
#5  0x00007ffff707c4dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 1 (Thread 0x7ffff7fcc1c0 (LWP 17264)):
#0  0x00007ffff6faa438 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1  0x00007ffff6fac03a in __GI_abort () at abort.c:89
#2  0x00000000004ab979 in stream_dump_and_crash (obj=obj@entry=0xa6d090, rate=175268) at src/stream.c:2691
#3  0x000000000056442e in task_run_applet (t=0xa6d1a0, context=0xa6d090, state=<optimized out>) at src/applet.c:98
#4  0x0000000000567a22 in run_tasks_from_lists (budgets=budgets@entry=0x7fffffffd9f0) at src/task.c:477
#5  0x0000000000568549 in process_runnable_tasks () at src/task.c:673
#6  0x000000000052431c in run_poll_loop () at src/haproxy.c:2956
#7  0x00000000005246ef in run_thread_poll_loop (data=data@entry=0x0) at src/haproxy.c:3121
#8  0x000000000041b5ed in main (argc=<optimized out>, argv=0x7fffffffdef8) at src/haproxy.c:3819

see also: https://github.com/TimWolla/haproxy-auth-request/runs/860809162?check_suite_focus=true

http.lua still mentions GNU GPL instead of Apache

-- This program is free software: you can redistribute it and/or modify
-- it under the terms of the GNU General Public License as published by
-- the Free Software Foundation, version 2 of the License
should be replaced with:

   Licensed under the Apache License, Version 2.0 (the "License");
   you may not use this file except in compliance with the License.
   You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.

The “All Rights Reserved” within line 19 might also be misleading:

local _copyright = "Copyright 2017-2018. HAProxy Technologies, LLC. All Rights Reserved."

too many C levels

Hi,

I have manually imported your lua script into /usr/local/share/lua/5.3/haproxy-lua-http.lua along with https://github.com/TimWolla/haproxy-auth-request/blob/main/auth-request.lua and https://github.com/rxi/json.lua (I want to use Authelia).

After this, I tried adding the required loading lines in the global section

lua-prepend-path /usr/local/share/lua/5.3/haproxy-lua-http.lua
lua-load /usr/local/share/lua/5.3/auth-request.lua

When adding the second line, then HAProxy starts to complain about your file and I have no idea why:

Errors found while starting haproxy
[NOTICE] 013/044744 (18131) : haproxy version is 2.2.14-a07ac36
[ALERT] 013/044744 (18131) : parsing [/var/etc/haproxy_test/haproxy.cfg:25] : Lua runtime error: error loading module 'json' from file '/usr/local/share/lua/5.3/haproxy-lua-http.lua':
/usr/local/share/lua/5.3/haproxy-lua-http.lua:673: too many C levels (limit is 200) in function at line 619 near 'v' 

Can you help me figure this out?

Many thanks :)

Require json

Hi all,
My environment : Centos 8 + haproxy-2.2.10 + lua-5.3.5 install from source following instruction : https://kifarunix.com/install-haproxy-on-rocky-linux-8/
I followed instruction at : https://www.authelia.com/docs/deployment/supported-proxies/haproxy.html

global
# Path to haproxy-lua-http, below example assumes /usr/local/etc/haproxy/haproxy-lua-http/http.lua
lua-prepend-path /usr/local/etc/haproxy/http.lua
# Path to haproxy-auth-request
lua-load /usr/local/etc/haproxy/auth-request.lua
log stdout format raw local0 debug

When I check haproxy cfg I get errors:

haproxy -c -f haproxy.cfg
[NOTICE] (26946) : haproxy version is 2.4.2-553dee3
[NOTICE] (26946) : path to executable is /usr/local/sbin/haproxy
[ALERT] (26946) : parsing [haproxy.cfg:5] : Lua runtime error: error loading module 'json' from file '/usr/local/etc/haproxy/http.lua':
/usr/local/etc/haproxy/http.lua:673: too many C levels (limit is 200) in function at line 619 near 'v'

[ALERT] (26946) : parsing [haproxy.cfg:51]: 'http-request' expects 'wait-for-handshake', 'use-service', 'send-spoe-group', 'sc-inc-gpc0()', 'sc-inc-gpc1()', 'sc-set-gpt0()', 'do-resolve()', 'cache-use', 'add-acl()', 'add-header', 'allow', 'auth', 'capture', 'del-acl()', 'del-header', 'del-map()', 'deny', 'disable-l7-retry', 'early-hint', 'normalize-uri', 'redirect', 'reject', 'replace-header', 'replace-path', 'replace-pathq', 'replace-uri', 'replace-value', 'return', 'set-header', 'set-log-level', 'set-map()', 'set-method', 'set-mark', 'set-nice', 'set-path', 'set-pathq', 'set-query', 'set-tos', 'set-uri', 'strict-mode', 'tarpit', 'track-sc()', 'set-timeout', 'wait-for-body', 'set-var()', 'unset-var(*)', 'set-priority-class', 'set-priority-offset', 'silent-drop', 'set-src', 'set-src-port', 'set-dst', 'set-dst-port', but got 'lua.auth-request'.
[ALERT] (26946) : Error(s) found in configuration file : haproxy.cfg

How can I fix it ? Please give me some advice , thank you very much.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.