openresty / lua-resty-upload Goto Github PK
View Code? Open in Web Editor NEWStreaming reader and parser for http file uploading based on ngx_lua cosocket
Streaming reader and parser for http file uploading based on ngx_lua cosocket
I'm trying to upload a slightly big file of size ~3MB. And I'm getting a 413 error. The nginx error is:
2016/08/24 21:35:05 [error] 50785#0: *2 client intended to send too large body: 3520276 bytes, client: 127.0.0.1, server: localhost, request: "POST /test HTTP/1.1", host: "localhost:8070".
And my client side error:
<title>413 Request Entity Too Large</title>Is there any configuration I need to do on the Nginx side?
line 8 in upload.lua
local get_headers = ngx.req.get_headers
is now unused.
Thanks
when i use the mouble,i occured a probole.it will be more than one \r in each line.my lua script:
package.path = 'E:/nginx-windows/lua/?.lua;E:/nginx-windows/resty/?.lua;'
package.cpath = 'E:/nginx-windows/?.dll;'
local restyUpload = require "upload"
local chunk_size = 4096
local file
local filelen=0
local filename
local args = ngx.req.get_query_args()
--ngx.say(args)
function get_filename(res)
local filename = ngx.re.match(res,'(.+)filename="(.+)"(.*)')
if filename then
return filename[2]
end
end
local rootDir = "e:/temp/"
local filepath = rootDir .. "/" --args.path ..
local temp = io.open(filepath)
local form = restyUpload:new(chunk_size)
if form == nil then
ngx.log(ngx.ALERT, "parse body failed.")
ngx.exit(500)
end
form:set_timeout(0) -- 1 sec
local i=0
ngx.header.content_type = "text/plain"
while true do
local typ, res, err = form:read()
if not typ then
ngx.say("failed to read body: ", err)
return
end
if typ == "header" then
if res[1] ~= "Content-Type" then
filename = get_filename(res[2])
if filename then
i=i+1
filepath = filepath .. filename
file = io.open(filepath,"w+")
if not file then
ngx.exit(ngx.HTTP_INTERNAL_SERVER_ERROR)
ngx.log(ngx.ALERT, "can not open file: "..filepath)
return
end
end
end
elseif typ == "body" then
ngx.say("type", res);
if file then
--filelen= filelen + tonumber(string.len(res))
file:write(res)
end
elseif typ == "part_end" then
if file then
file:close()
file = nil
ngx.say("success")
end
elseif typ == "eof" then
break
else
end
end
if i==0 then
ngx.exit(ngx.HTTP_NOT_FOUND)
ngx.log(ngx.ALERT, "please upload at least one file!")
return
end
the response is:
type<html>
<body>
<form enctype="multipart/form-data" method="post" action="http://localhost:8090/upload">
<input type="file" name="file"/>
<input type="submit" value="upload"/>
</form>
</body>
</html>
success
but,the file is:
<html>
<body>
<form enctype="multipart/form-data" method="post" action="http://localhost:8090/upload">
<input type="file" name="file"/>
<input type="submit" value="upload"/>
</form>
</body>
</html>
so,when i try to use this script to upload image、zip、doc and other files ,all files are damage.and i found that not all files will damage,same file is right.
I wrote yesterday PHP class to parse multipart request body. For example Chrome sent boundary before headers not in Content-Type header.
Is it possible to make update to handle those changes?
Hi, my version is openresty/1.9.15.1, When I upload 2MB+ size file I got "nginx: lua atpanic: Lua VM crashed, reason: unknown reason" Error,
I could upload 100KB file successfully. Seems some memory configure need changed?
I'm trying to make an anti-virus middle ware which first check the uploaded files with help of lua-resty-upload and if they are not infected proxy_pass them to some dynamic proxy target.
I was able to handle the first part easily thanks to lua-resty-upload but the issue is when I proxy_pass the request after anti-virus check the body is empty. which seems pretty normal because lua-resty-upload has consumed the body already.
So I should reconstruct the body again after virus check, But I don't have the raw binary data of request at that point to easily reconstruct the body, all I have is parsed body which is hard to be converted to raw binary again.
Is there any shortcut method which can give me the raw binary data after process of lua-resty-upload is finished?
One Korean character is 3-byte in UTF-8. And MS Windows limits a single path to 260 characters.
Therefore maximum Korean caharacter path or filename length is about 780 bytes in UTF-8.
local MAX_LINE_SIZE = 512
MAX_LINE_SIZE, 512 is too small to contain max Korean character filename.
How about increase the size MAX_LINE_SIZE to 1024?
Thank you.
Thank you for your work, i use this module without any problems for a year.
But now I tried to handle the upload with the same script, but from a subrequest, and i got this:
attempt to read the request body in a subrequest
stack traceback:
coroutine 0: > [C]: in function 'req_socket' > /usr/lib/lua/luarocks/share/lua/5.1/resty/upload.lua:61: in function 'new'
i called the script with this:
local res = ngx.location.capture("/upl_worker", {share_all_vars = true, always_forward_body=true})
Is it impossible, or i miss something?
文件上传报错:
2020/12/22 18:15:39 [error] 2122#0: *12049 lua entry thread aborted: runtime error: /www/server/openresty/lualib/resty/upload.lua:60: http v2 not supported yet
支持断点续传和分块传输吗?
Does this support the upload of bulk files? If this does not support, Can you give me a reference plz?
lua entry thread aborted: runtime error: /usr/local/openresty/lualib/resty/upload.lua:61: http v2 not supported yet
Hi, ChunGe!
I use the example code:
local upload = require "resty.upload"
local cjson = require "cjson"
local chunk_size = 8192 -- should be set to 4096 or 8192
-- for real-world settings
local form, err = upload:new(chunk_size)
if not form then
ngx.log(ngx.ERR, "failed to new upload: ", err)
ngx.exit(500)
end
form:set_timeout(1000) -- 1 sec
while true do
local typ, res, err = form:read()
if not typ then
ngx.say("failed to read: ", err)
return
end
ngx.say("read: ", cjson.encode({typ, res}))
if typ == "eof" then
break
end
end
local typ, res, err = form:read()
ngx.say("read: ", cjson.encode({typ, res}))
the request raw is below:
POST http://api.31lancai.com/upload_lua HTTP/1.1
Host: api.31lancai.com
Connection: keep-alive
Content-Length: 301
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Origin: http://www.31lancai.com
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryTOm33VnyDJkkXk8V
DNT: 1
Referer: http://www.31lancai.com/user_bank_card_cancel.html?index=97961
Accept-Encoding: gzip, deflate
Accept-Language: zh-CN,zh;q=0.8,en;q=0.6,ja;q=0.4,zh-TW;q=0.2
Cookie: LC_UUID=79b77b68-ee66-4bdf-91ad-9766e81f1199; PHPSESSID=c5cdf77f848185b9d104ec3fe96ee7f8; LC_USER_LOGIN_TYPE=s; LC_LOGGED=1; LC_USER_ID=250402; LC_DISPLAY_NAME=%E5%93%88%E5%A5%B3%E5%A3%AB
------WebKitFormBoundaryTOm33VnyDJkkXk8V
Content-Disposition: form-data; name="data"
{"type":"photo_idcard"}
------WebKitFormBoundaryTOm33VnyDJkkXk8V
Content-Disposition: form-data; name="pic"; filename="test.jpg"
Content-Type: image/jpeg
321321
------WebKitFormBoundaryTOm33VnyDJkkXk8V--
the response is:
HTTP/1.1 200 OK
Server: nginx
Date: Tue, 29 Mar 2016 06:54:33 GMT
Content-Type: text/plain
Transfer-Encoding: chunked
Connection: keep-alive
74
read: ["header",["Content-Disposition","form-data; name=\"data\"","Content-Disposition: form-data; name=\"data\""]]
2d
read: ["body","{\"type\":\"photo_idcard\"}"]
13
read: ["part_end"]
a0
read: ["header",["Content-Disposition","form-data; name=\"pic\"; filename=\"test.jpg\"","Content-Disposition: form-data; name=\"pic\"; filename=\"test.jpg\""]]
4c
read: ["header",["Content-Type","image\/jpeg","Content-Type: image\/jpeg"]]
18
read: ["body","321321"]
13
read: ["part_end"]
e
read: ["eof"]
e
read: ["eof"]
0
you can seen that the header is dumplicated, like read: ["header",["Content-Disposition","form-data; name=\"pic\"; filename=\"test.jpg\"","Content-Disposition: form-data; name=\"pic\"; filename=\"test.jpg\""]]
, there is double "Content-Disposition"
, double "form-data; name=\"pic\"; filename=\"test.jpg\""
why? this is a bug? or I misunderstand?
When I upload a file via lua-resty-upload ,I find the latest uploaded file always lose some data,but when I upload another file, the previous file seems ok,it seems something has been buffered and not flush to the disk immediately。
local upload = require "resty.upload"
local chunk_size = 8192
local form = upload:new(chunk_size)
local uuid = require("resty.uuid")
local file
local currenttime = os.time()
local filename = uuid().."-"..pid
local file_name = uploadpath..filename
if form then
while true do
local typ, res, err = form:read()
if not typ then
ngx.say(cjson.encode({code=1,message="nofile upload"}))
return
end
if file_name then
file = io.open(file_name, "a+")
if not file then
ngx.say(cjson.encode({code=2,message="file save failed"}))
return
end
end
if typ == "body" then
if file then
file:write(res)
end
elseif typ == "part_end" then
file:close()
file = nil
file_name = nil
ngx.say(cjson.encode({code=0,url=ngx.var.scheme.."://"..ngx.var.host.."/"..filename}))
filename = nil
return
elseif typ == "eof" then
break
else
-- do nothing
end
end
else
ngx.say(cjson.encode({code=1,message="nofile upload"}))
return
end
I call file:flush() before file:colse() ,there is no effect
My nginx version is 1.7.9 and use the latest lua-resty-upload lib
I am trying to build a Docker image, unfortunately it is failing:
./configure: error: no ../lua-resty-upload-0.10/config was found
The command '/bin/sh -c ./configure --prefix=/usr/local/nginx --add-module=../headers-more-nginx-module --add-module=../ngx_http_auth_pam_module --add-module=../ngx_cache_purge --add-module=../dav-ext-module --add-module=../ngx_devel_kit --add-module=../echo-nginx-module --add-module=../push-stream-module --add-module=../lua-nginx-module --add-module=../upload-progress-module --add-module=../ngx-fancyindex --add-module=../ngx_http_substitutions_filter_module --add-module=../lua-resty-upload-0.10 --add-module=../ngx_aws_auth --add-module=../set-misc-nginx-module --add-module=../upload-module --with-http_ssl_module --with-http_addition_module --with-http_dav_module --with-http_geoip_module --with-http_gzip_static_module --with-http_image_filter_module --with-http_perl_module --with-http_random_index_module --with-http_secure_link_module --with-http_v2_module --with-http_sub_module --with-http_xslt_module --with-mail --with-mail_ssl_module --with-file-aio --with-threads --with-cc-opt="-O3"' returned a non-zero code: 1
here is the code
local upload = require "resty.upload"
local json = require "cjson"
local s3_upload = require "resty.s3_upload"
local form, err = upload:new(chunk_size)
if not form then
ngx.log(ngx.ERR, "failed to new upload: ", err)
ngx.exit(500)
end
local timeOut = 10000
form:set_timeout(timeOut)
while true do
local typ, res, err = form:read()
if not typ then
ngx.say(ngx.ERR,"failed to read: ", err)
return
end
if typ == "header" then
content_type = res[2]
if content_type == 'application/hta' then
start_upload = true
end
ngx.log(ngx.INFO,"content_type: ", content_type)
ngx.log(ngx.INFO,"header: ", json.encode(res))
elseif typ == "body" then
if start_upload==true and res then
appender = appender..res
end
elseif typ == "part_end" then
if appender ~= '' then
local data_len = string_len(appender)
ngx.log(ngx.NOTICE,"start to upload ",content_type," len=" ,data_len)
## -- ERROR HERE handle string appender
## -- data_len is bigger than I expected when the chunk_size is 50*1024
## -- data_len will bi right when I change the chunk_size as 10*1024
## -- if I post a file in the form
## -- the data_len appender will be 15 byte bigger than the file I posted
appender = ''
start_upload = false
end
elseif typ == "eof" then
break
end
end
I'm using the following version of Nginx.
nginx version: nginx/1.14.0
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC)
built with OpenSSL 1.0.2k-fips 26 Jan 2017
TLS SNI support enabled
configure arguments: --prefix=/opt/nginx --with-http_ssl_module --with-http_v2_module --with-http_realip_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_auth_request_module --with-http_stub_status_module --with-threads --with-stream --with-stream_ssl_module --add-module=/tmp/ngx_devel_kit-0.3.0 --add-module=/tmp/lua-nginx-module-0.10.9rc7
I have defined four virtual servers in http config block. One virtual server is defined with listen 443 ssl;
, while three other virtual servers are defined with listen 443 ssl http2;
. The lua-nginx-module
is used in the server without http2
configured to handle file upload. The code is as follows.
local req_socket = ngx.req.socket
local sock, err = req_socket()
if not sock then
return nil, err
end
Strange thing is the error.log reports that http v2 is not supported yet. But the http2
is not even configured in this virtual server. The error log is as below.
2019/04/29 15:26:19 [error] 22281#22281: *6512 lua entry thread aborted: runtime error: /opt/nginx/co
nf/lua/resty/upload.lua:61: http v2 not supported yet
stack traceback:
coroutine 0:
[C]: in function 'req_socket'
/opt/nginx/conf/lua/resty/upload.lua:61: in function 'new'
It looks like the http2
configuration exists beyond the virtual server in which it's configured, like somehow it becomes a global thing.
After I delete the http2
in all three other virtual servers, the error.log stops reporting the error mentioned above.
I don't know what make this happen. Is it a bug or something wrong with my configuration?
I have a client code in python:
data = {"report_type": self.type}
files = {"logfile": (self.name, open(self.data_path, 'rb'))}
try:
r = requests.post(self.url, files=files, headers = headers, data = data, timeout = 180)
except Exception as e:
error("Failed to upload file %s: %s" % (self.data_path, str(e)))
return False
My lua code is following:
while true do
local typ, res, err = form:read()
if not typ then
ngx.say("failed to read: ", err)
return
end
if typ == "header" then
local file_name = gen_file_name(location, report_type, get_report_host())
if file_name then
file = io.open(file_name, "w+")
if not file then
ngx.say("failed to open file ", file_name)
return
end
end
elseif typ == "body" then
if file then
file:write(res)
end
elseif typ == "part_end" then
file:close()
file = nil
elseif typ == "eof" then
break
else
-- do nothing
end
end
It creates 2 files instead of 1:
How can avoid creating of the second file with request parameter?
Hi, I am really new to use nginx and lua. I am trying to upload a file from my local location to nginx server using lua-resty-upload . I have a docker build with base image openresty/openresty:1.15.8.3-buster
I tried the lua script provided in the readme section and i got the success output saying the file uploaded to location /tmp/[filename]
But on checking the docker location I couldnt find the file any where in the mentioned location. Am I missing anything here . Below is the the lua script
local upload = require "resty.upload"
local function my_get_file_name(header)
local file_name
for i, ele in ipairs(header) do
file_name = string.match(ele, 'filename="(.*)"')
if file_name and file_name ~= '' then
return file_name
end
end
return nil
end
local chunk_size = 4096
local form = upload:new(chunk_size)
local file
local file_path
while true do
local typ, res, err = form:read()
if not typ then
ngx.say("failed to read: ", err)
return
end
ngx.print("the variable value: ", typ)
ngx.print("the res value: ", res)
if typ == "header" then
local file_name = my_get_file_name(res)
if file_name then
file_path = ngx.var.store_path..file_name
file = io.open(file_path, "w")
if not file then
ngx.say("failed to open file ", file_path)
return
end
end
elseif typ == "body" then
if file then
file:write(res)
end
elseif typ == "part_end" then
if file then
file:close()
file = nil
ngx.say("upload to "..file_path.." successfully!")
end
elseif typ == "eof" then
break
else
-- do nothing
end
end
hi,I use lua-resty-upload to handle big file upload, I use an extension of this version which have the part-split upload function, repos link, when the last part upload complete, the nginx error log sometimes show this error:
lua tcp socket read timed out
and did not merge all the part to one file,please give me a help!thx!
this is upload.lua script below:
local upload = require "resty.upload"
local cjson = require "cjson"
local get = ngx.req.get_uri_args()
local chunk_size = 4096
local file_upload_count = 0
local speed = 0
local post = {}
local index
local file
local upload_path = "/tmp/" -- 上传路径
--merge chunks
local function mergeChunks(chunks, name)
local file = io.open(upload_path .. name, "w+")
for i = 0, chunks - 1 do
local chunk_path = upload_path .. name .. "." .. i
local chunk_file = io.open(chunk_path)
while true do
local bytes = chunk_file:read(4096)
if not bytes then
break
end
file:write(bytes)
end
chunk_file:close()
os.remove(chunk_path)
end
file:close()
end
-- rate limit
if get.speed then
speed = math.ceil(get.speed * 1024 / chunk_size)
end
local form, err = upload:new(chunk_size)
if not form then
ngx.say("no zuo no die.")
ngx.exit(200)
end
form:set_timeout(1000) -- one second timeout
while true do
local typ, res, err = form:read()
if not typ then
ngx.say("failed to read: ", err)
return
end
if typ == "header" then
if res[1] == "Content-Disposition" then
local filename = ngx.re.match(res[2], '(.+)filename="(.+)"(.*)')
if filename then
filename = filename[2]
-- part upload
if post["chunks"] then
filename = filename .. "." .. post["chunk"]
end
file = io.open(upload_path .. filename, "w+")
if not file then
ngx.say("failed to open file ")
return
end
else
local name = ngx.re.match(res[2], '(.+)name="(.+)"(.*)')
if name[2] then
index = name[2]
end
end
end
elseif typ == "body" then
if file then
-- record upload times
file_upload_count = file_upload_count + 1
-- rate limit
if speed ~= 0 and file_upload_count % speed == 0 then
ngx.sleep(1)
end
-- write file
file:write(res)
else
if index then
if post[index] == nil then
post[index] = res
else
post[index] = post[index] .. res
end
end
end
elseif typ == "part_end" then
if file then
file:close()
file = nil
if post["chunks"] and post["chunk"] then
if tonumber(post["chunks"]) - tonumber(post["chunk"]) == 1 then
mergeChunks(tonumber(post["chunks"]), post["name"])
end
end
ngx.header["Content-Type"] = 'application/json'
ngx.say(cjson.encode({code=1}))
end
if index then
index = nil
end
elseif typ == "eof" then
break
else
end
end
I get headers like this:
["Content-Disposition","form-data; name=\"nick\"","Content-Disposition: form-data; name=\"nick\""]
While original request only had this:
`Content-Disposition: form-data; name="nick"``
It seems all the headers this returns are doubled? Is that what is expected?
I have some code like this
ngx.ctx.uploads={} --m.getupload()
ngx.req.read_body()
ngx.ctx.posts={}
args, err = ngx.req.get_post_args()
for key, val in pairs(args) do
ngx.ctx.posts[key]=val
ngx.ctx.gets[key]=val
and m.getupload is using your library, to get all the uploads into table hashed by the input_name. it also save the non file data input to table posts
getupload:->
form = upload\new(100000)
file_name=""
input_name=""
resx=""
result={}
while form~=nil do
typ, res, err = form\read()
if typ == "header" then
if res[1]=="Content-Disposition"
tt=m.getfield(res)
file_name = tt.filename
input_name = tt.name
resx=""
elseif typ == "eof"
break
elseif typ == "body"
if input_name
resx=resx..res
elseif typ == "part_ "
if file_name
result[input_name]={file_name,resx}
ngx.ctx.gets[input_name]=file_name
else
ngx.ctx.gets[input_name]=resx
resx=""
if i enable uploads processing then the ngx.req.get_post_args() not work. perhaps i need to detect the enc-type multipart ?
while true do
local typ, res, err = form:read()
if not typ then
ngx.say("failed to read: ", err)
return
end
if typ == "header" then
local key = res[1]
local value = res[2]
if key == "Content-Disposition" then
local kvlist = string.split(value, ';')
for _, kv in ipairs(kvlist) do
local seg = string.trim(kv)
if seg:find("filename") then
local kvfile = string.split(seg, "=")
local filename = string.sub(kvfile[2], 2, -2)
if filename then
ext = getExtension(filename)
local now = os.date("%Y%m%d%H%M%S", os.time())
local filename = now.."-"..string.sub(ngx.md5(now..user_cid), 0, 8)..ext
local image_id = ""
if string.find(uri, "/upload/photo") then
uri = string.gsub(ngx.var.uri, '/upload/photo', "/bfs/g/"..filename)
image_id = "/g/"..filename
elseif string.find(uri, "/upload/voice") then
uri = string.gsub(ngx.var.uri, '/upload/voice', "/bfs/v/"..filename)
else
ngx.print(cjson.encode({code=404, message="Not Found"}))
ngx.exit(404)
end
break
end
end
end
end
--append body
elseif typ == "body" then
if res then
appender = appender..res
end
--one file has been read
elseif typ == "part_end" then
if appender ~= '' then
-- If the uploaded file is a picture,I want it to be a 2048 by 2048 image with maximum height and width
code = upload2BFS(uri, data)
Hi:
When Content-Type is "xxxxxx;boundary=xxxxxxxx", lua-resty-upload cannot parse boundary. I find out """Content-Type := type "/" subtype *[";" parameter]""" in http://www.ietf.org/rfc/rfc1341.txt. Is necessary space between semicolon and parameter?
Best regards,
chenshu
Pretty much as the title says. It'd be great to get v0.10 on OPM!
cjson not found, falling back to stdlib json
Traceback (most recent call last):
File "c:\INTEGRACAO-QI-TECH\5_2_3_CONTRATO_DE_CESSAO\5_2_3_1.py", line 3, in
from jose import jwt
ImportError: cannot import name 'jwt' from 'jose'
Este é o erro que estou enfrentando. Um problema do código da biblioteca. Como consertar?
local upname = nil
while true do
local typ, res, err = form:read()
-- bla bla do something
if typ == "header" then
for k,v in pairs(res) do
local ext = string.match(v,'.+(%.7z.*)"') -- say we only want to permit files with this extension
if ext then
upname = ext
end
end
end
-- crude but we found our extension, if we didn't then
if not upname then
-- cleanup()
ngx.status = 415
ngx.exit(415) -- makes nginx dump binary fuzz into the error.log
end
-- sample bad output in error.log after connnection RST issued
09:17:16 [info] 6461#6461: *511 client sent invalid method while readin g client request line, client: 127.0.0.1, server: xxxxx, request: ▒▒"▒▒ ▒▒f
^^^ that kind of rubbish dumped in the error.log
HOWEVER - the Correct behaviour occurs if the file upload is small (<300-400k) - presumably all read into a single buffer.. by correct i mean the upload is terminated with a redirect to the configured error_page 415 /415.html and the content of the error page is duly served.
If the file is larger than that (500k+) ngx.exit(415) totally fails and spews bad output into error.log
What's going on here? Pipes not emptying properly? Request line disappearing? This is easily reproducible on the latest openresty+debian buster
Hi,
I am following below code on fedora Linux with nginx:
local resty_sha256 = require "resty.sha256"
local upload = require "resty.upload"
local chunk_size = 4096
local form = upload:new(chunk_size)
local sha256 = resty_sha256:new()
local file
local upload_path = '/tmp/uploads/'
while true do
local typ, res, err = form:read()
if not typ then
ngx.say("failed to read: ", err)
return
end
if typ == "header" then
local file_name = upload_path .. ngx.time()
if file_name then
file = io.open(file_name, 'w+')
if not file then
ngx.say("failed to open file ", file_name)
return
end
end
elseif typ == "body" then
if file then
file:write(res)
sha256:update(res)
end
elseif typ == "part_end" then
file:close()
file = nil
elseif typ == "eof" then
break
else
-- nothing
end
end
local digest = sha256:final()
sha256:reset()
local str = require "resty.string"
ngx.say("sha256: ", str.to_hex(digest))```
When I run this as
curl -F "[email protected]" http://127.0.0.1:8089/uploadtest
It gives me below errors:
$ tail -f /usr/local/openresty/nginx/logs/error.log 2016/06/29 15:26:23 [error] 31449#0: *12 lua entry thread aborted: runtime error: /usr/local/openresty/nginx/uploadtest.lua:13: attempt to index local 'form' (a nil value) stack traceback: coroutine 0: /usr/local/openresty/nginx/uploadtest.lua: in function </usr/local/openresty/nginx/uploadtest.lua:1>, client: 127.0.0.1, server: localhost, request: "POST /uploadtest HTTP/1.1", host: "127.0.0.1:8089"
Can you please help me what is wrong here?
Hi,I use the upload component,and success to upload the image to special dir,but the image is damaged,I have no idear.
I use lua restry upload, and ngx.req.read_body() , I put those in /opt/xx.lua;
nginx conf as below ~
access_by_lua_file /opt/xx.lua
proxy_pass http://localhosy:9000
the proxy pass is my django project ,it works ,and I try to change another url~
I can process .lua in get and post(without file,use formdata is ok~) method,but when I post file ,the nginx alway raise error~
"
[error] 25390#0: *442 upstream prematurely closed connection while reading response header from upstream, client: 192.168.10.26, server: localhost, request: "POST /app/get_all_args/ HTTP/1.1", upstream: "http://127.0.0.1:8090/app/get_all_args/", host: "192.168.10.27"
"
IP 192.168.10.26 is my client
IP 192.168.10.27 is my nginx
URL 127.0.0.1:8090/app/get_all_args/ is my project,which is proxied to after lua accept~
client always block,and I try use ngx.say ~ It seems everything is ok~
So I don't know what's the problem is ~
In .lua when auth is ok , I use ngx.exit(ngx.OK) ngx.HTTP_OK is bad ~
when auth filed, I used ngx.say("xxx") then return
Now I am sure I use ngx.exit(ngx.OK)
So please help me~ Thank you ~
Sorry for my poor English~ I can desc it in Chinese~ = =!
@agentzh Does lua-resty upload module support curl's post request ?
There is one problem I got , I tried to use this command : curl -d 'payload' -X POST 'http://localhost/uploadfile?path=/data/test.txt to post data to server,but failed. The error log is posted below
2016/08/31 18:38:56 [error] 48977#0: 77 [lua] savefile.lua:5: the raw header is POST /uploadfile?path=/data/ds HTTP/1.1^M
User-Agent: curl/7.35.0^M
Host: localhost^M
Accept: */^M
Content-Length: 8^M
Content-Type: application/x-www-form-urlencoded^M
^M
, client: 127.0.0.1, server: localhost, request: "POST /uploadfile?path=/data/test.txt HTTP/1.1", host: "localhost"
2016/08/31 18:38:56 [error] 48977#0: *77 [lua] savefile.lua:11: failed to new upload: no boundary defined in Content-Type, client: 127.0.0.1, server: localhost, request: "POST /uploadfile?path=/data/test.txt HTTP/1.1", host: "localhost"
The Content-Type don't have boundary so the upload is failed. Is there any solution to solve this problem ?
When I upload a file via lua-resty-upload ,I find the lastest uploaded file always lose some data,but when I upload another file, the previous file seems ok,it seems something has been buffered and not flush to the disk immediately。
local upload = require "resty.upload"
local chunk_size = 8192
local form = upload:new(chunk_size)
local uuid = require("resty.uuid")
local file
local currenttime = os.time()
local filename = uuid().."-"..pid
local file_name = uploadpath..filename
if form then
while true do
local typ, res, err = form:read()
if not typ then
ngx.say(cjson.encode({code=1,message="nofile upload"}))
return
end
if file_name then
file = io.open(file_name, "a+")
if not file then
ngx.say(cjson.encode({code=2,message="file save failed"}))
return
end
end
if typ == "body" then
if file then
file:write(res)
end
elseif typ == "part_end" then
file:close()
file = nil
file_name = nil
ngx.say(cjson.encode({code=0,url=ngx.var.scheme.."://"..ngx.var.host.."/"..filename}))
filename = nil
return
elseif typ == "eof" then
break
else
-- do nothing
end
end
else
ngx.say(cjson.encode({code=1,message="nofile upload"}))
return
end
I call file:flush() before file:colse() ,there is no effect
Many (if not all) mainstream web servers, Apache(mod_upload
) and Nginx(upload module
) for instance are compatible to request use LF as line break:
https://github.com/Austinb/nginx-upload-module/blob/2.2/ngx_http_upload_module.c#L4123
https://github.com/Austinb/nginx-upload-module/blob/2.2/ngx_http_upload_module.c#L4051
http://apache.webthing.com/mod_upload/mod_upload.c L304
Would it be necessary for the resty-upload module to support this too? Maybe an optional feature and closed by default?
To illustrate when it matters, if a WAF may want to filter multipart forms and use lua-resty-upload
, where a malicious request can bypass its check for headers:
POST /submit.php HTTP/1.1
Host: target.com
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary6GpaBUxso7BnIrnA
Headers-Dont-Matter: it doesnt matter
------WebKitFormBoundary6GpaBUxso7BnIrnA
Content-Type: text/plain
Content-Disposition: form-data; name="something"; filename="doesnt_matter"
[LF rather than CRLF this line]
------WebKitFormBoundary6GpaBUxso7BnIrnA[LF rather than CRLF this line]
Content-Disposition: form-data; name="webshell.php"
malicious script here
------WebKitFormBoundary6GpaBUxso7BnIrnA--
from WAF's point of view, below is a single line:
[LF rather than CRLF this line]
------WebKitFormBoundary6GpaBUxso7BnIrnA[LF rather than CRLF this line]
Content-Disposition: form-data; name="webshell.php"
therefore the boundary is missing and the second uploaded file(including its headers) is seen as part of the first file's body, and the protected server still sees it as two files.
We can make up more methods to bypass WAF based on this.
I'm developing a file server with openresty, the client is a java program which upload the file with stream so that not too much memory used , but I get the error "chunked request bodies not supported yet", so I want to know does lua_resty_upload support chunked request bodies? if not, is there any suggest to handle this, thanks.
``
read: ["header",["Content-Disposition","form-data; name="file1"; filename="a.txt"","Content-Disposition: form-data; name="file1"; filename="a.txt""]]
read: ["header",["Content-Type","text/plain","Content-Type: text/plain"]]
read: ["body","Hello"]
read: ["body",", wor"]
read: ["body","ld"]
read: ["part_end"]
read: ["header",["Content-Disposition","form-data; name="test"","Content-Disposition: form-data; name="test""]]
read: ["body","value"]
read: ["body","\r\n"]
read: ["part_end"]
read: ["eof"]
read: ["eof"]
Each request header content in pairs,
This module works fine when I have upload via http. But when I used httpS it crashes on call upload:new(chunk_size) phase. Does exist method for working with httpS?
I'm new to LUA, I tried to upload file with some meta data with curl command curl -i -X POST -F "uuid=123" -F "[email protected] http://somewhere"
But the value of uuid which is "123" seems be truncated in the the value, and even the line.
["Content-Disposition","form-data; name=\"uuid\"","Content-Disposition: form-data; name=\"uuid\""]
I noticed there're double \r\n pairs located between uuid and its value.
020: 39 36 63 64 63 33 39 31 35 33 0d 0a 43 6f 6e 74 96cdc39153..Cont 0030: 65 6e 74 2d 44 69 73 70 6f 73 69 74 69 6f 6e 3a ent-Disposition: 0040: 20 66 6f 72 6d 2d 64 61 74 61 3b 20 6e 61 6d 65 form-data; name 0050: 3d 22 75 75 69 64 22 0d 0a 0d 0a 31 32 33 0d 0a ="uuid"....123.. 0060: 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d ---------------- 0070: 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 30 62 --------------0b 0080: 39 36 63 64 63 33 39 31 35 33 0d 0a 43 6f 6e 74 96cdc39153..Cont 0090: 65 6e 74 2d 44 69 73 70 6f 73 69 74 69 6f 6e 3a ent-Disposition: 00a0: 20 66 6f 72 6d 2d 64 61 74 61 3b 20 6e 61 6d 65 form-data; name 00b0: 3d 22 66 69 6c 65 6f 62 6a 22 3b 20 66 69 6c 65 ="fileobj"; file 00c0: 6e 61 6d 65 3d 22 31 2e 74 78 74 22 0d 0a 43 6f name="1.txt"..Co 00d0: 6e 74 65 6e 74 2d 54 79 70 65 3a 20 74 65 78 74 ntent-Type: text 00e0: 2f 70 6c 61 69 6e 0d 0a 0d 0a /plain....
BTW. any better solution to obtain the extra information(you defined as header
) from multipart uploading request?
Hi, I encountered an occasional problem in a http request, the following is the sample code
local upload = require "resty.upload"
local form, err = upload.new(4 * 1024)
****
form:set_timeout(5000)
local typ, res, err = form:read()
The value of err is "closed", then server return a message(by using ngx.print
and ngx.exit(ngx.HTTP_OK)
) and client can receive it. It seems that the socket was closed but client can still get the return of server
Hi, I have hosted an nginx server using resty docker base image. Have couple of endpoints expose
one to upload, download, list directory content. What ever files uploaded using upload link to directory need to be download all at once using download option.
With ngx.location.capture of the directory i get the list of files with in directory. How can I pass the file information and content over as http response?
Any help would really be appreciated. :)
Hi,
i am uploading files and writing them to a filesystem using the code from the readme. However, when the upload is aborted, an incomplete file remains on the filesystem. Is there a way to automatically delete it, when the upload is aborted somewhere in the middle?
Best
我使用"curl -i -F '[email protected]' http://172.16.107.137/upload" ,后台能够正确的接收并处理binary。
但,使用rest client,比如java的httpclient或者postman等第三方工具上传的binary,后台无法正确获取。
求指导。
I noticed that the only way to receive the data seems to be to manually iterate until 'eof'
is returned.
I think it would make sense to add a more user-friendly method that works with Luas generic for loop to iterate over the data.
Here's an example implementation:
function _M.each(self)
return function(self)
local type, data, err = self:read()
if not type then
error(err, 2)
elseif type == "eof" then
else
return type, data
end
end, self
end
Then the example of the readme could also be written as
for type, data in form:each() do
ngx.say("read: ", cjson.encode{type, data}
end
local key, value = string.match(line, "([^: \t]+)%s*:%s*(.+)")
if not key then
return 'header', line
end
return 'header', {key, value}
是否可以将string.match(line, "([^: \t]+)%s*:%s*(.+)")解析的部分包装成一个函数。
而原始的read_header函数直接返回 'header', line,因为有些场景情况下需要用到line,而现在我没办法知道line的原始数据是什么
Hello, I'm trying to install two modules
https://github.com/openresty/lua-resty-upload
https://github.com/openresty/lua-resty-redis
on my test machine
nginx -V
nginx version: system-ns.com/LoadBalancer/0.0.1
built by gcc 7.3.1 20180130 (Red Hat 7.3.1-2) (GCC)
built with OpenSSL 1.1.0g-fips 2 Nov 2017 (running with OpenSSL 1.1.0h-fips 27 Mar 2018)
TLS SNI support enabled
configure arguments: --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/run/nginx.pid --lock-path=/run/lock/subsys/nginx --user=nginx --group=nginx --with-file-aio --with-threads --with-ipv6 --with-http_ssl_module --with-http_v2_module --with-http_realip_module --with-http_addition_module --with-http_xslt_module --with-http_image_filter_module --with-http_geoip_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_degradation_module --with-http_stub_status_module --with-http_perl_module --with-mail --with-mail_ssl_module --with-pcre --with-stream --with-pcre-jit --add-module=../lua-nginx-module-0.10.13 --add-module=../nginx_upstream_check_module --add-module=../ngx_pagespeed-1.13.35.1-beta --add-module=../nginx-gridfs --add-module=../nginx-upstream-dynamic-servers --add-module=../nginx-module-vts --add-module=../nginx-push-stream-module --with-google_perftools_module --with-cc-opt='-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -D_GLIBCXX_USE_CXX11_ABI=0' --with-ld-opt='-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Wl,-E'
my config file nginx
lua_package_path "/opt/lua-resty-redis/lib/?.lua;/opt/lua-resty-upload/lib/?.lua;;";
server {
server_name _;
location /test {
content_by_lua '
local upload = require "resty.upload"
local cjson = require "cjson"
local chunk_size = 5 -- should be set to 4096 or 8192
-- for real-world settings
local form, err = upload:new(chunk_size)
if not form then
ngx.log(ngx.ERR, "failed to new upload: ", err)
ngx.exit(500)
end
form:set_timeout(1000) -- 1 sec
while true do
local typ, res, err = form:read()
if not typ then
ngx.say("failed to read: ", err)
return
end
ngx.say("read: ", cjson.encode({typ, res}))
if typ == "eof" then
break
end
end
local typ, res, err = form:read()
ngx.say("read: ", cjson.encode({typ, res}))
';
}
location /redis {
content_by_lua_block {
local redis = require "resty.redis"
local red = redis:new()
red:set_timeout(1000) -- 1 sec
-- or connect to a unix domain socket file listened
-- by a redis server:
-- local ok, err = red:connect("unix:/path/to/redis.sock")
local ok, err = red:connect("127.0.0.1", 6379)
if not ok then
ngx.say("failed to connect: ", err)
return
end
ok, err = red:set("dog", "an animal")
if not ok then
ngx.say("failed to set dog: ", err)
return
end
ngx.say("set result: ", ok)
local res, err = red:get("dog")
if not res then
ngx.say("failed to get dog: ", err)
return
end
if res == ngx.null then
ngx.say("dog not found.")
return
end
ngx.say("dog: ", res)
red:init_pipeline()
red:set("cat", "Marry")
red:set("horse", "Bob")
red:get("cat")
red:get("horse")
local results, err = red:commit_pipeline()
if not results then
ngx.say("failed to commit the pipelined requests: ", err)
return
end
for i, res in ipairs(results) do
if type(res) == "table" then
if res[1] == false then
ngx.say("failed to run command ", i, ": ", res[2])
else
-- process the table value
end
else
-- process the scalar value
end
end
-- put it into the connection pool of size 100,
-- with 10 seconds max idle time
local ok, err = red:set_keepalive(10000, 100)
if not ok then
ngx.say("failed to set keepalive: ", err)
return
end
-- or just close the connection right away:
-- local ok, err = red:close()
-- if not ok then
-- ngx.say("failed to close: ", err)
-- return
-- end
}
}
}
for a simple test I do
curl -I http://test.name/redis
HTTP/1.1 200 OK
Server: system-ns.com/LoadBalancer
Date: Thu, 13 Sep 2018 12:27:37 GMT
Content-Type: application/octet-stream
Connection: keep-alive
curl -I http://test.name/test
HTTP/1.1 500 Internal Server Error
Server: system-ns.com/LoadBalancer
Date: Thu, 13 Sep 2018 12:29:49 GMT
Content-Type: text/html
Content-Length: 186
Connection: close
and in the logs I get an error
*1 lua entry thread aborted: runtime error: content_by_lua(upload.conf:39):3: module 'cjson' not found:
no field package.preload['cjson']
no file '/opt/lua-resty-redis/lib/cjson.lua'
no file '/opt/lua-resty-upload/lib/cjson.lua'
no file './cjson.lua'
no file '/usr/share/luajit-2.1.0-beta3/cjson.lua'
no file '/usr/local/share/lua/5.1/cjson.lua'
no file '/usr/local/share/lua/5.1/cjson/init.lua'
no file '/usr/share/lua/5.1/cjson.lua'
no file '/usr/share/lua/5.1/cjson/init.lua'
no file './cjson.so'
no file '/usr/local/lib/lua/5.1/cjson.so'
no file '/usr/lib64/lua/5.1/cjson.so'
no file '/usr/local/lib/lua/5.1/loadall.so'
stack traceback:
coroutine 0:
[C]: in function 'require'
content_by_lua(upload.conf:39):3: in function <content_by_lua(upload.conf:39):1>, client: my.ip, server: test.server.ip, request: "HEAD /test HTTP/1.1", host: "test.server.ip"
can not find the module cjson
what could be the problem?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.