libmpack / libmpack-lua

libmpack lua binding

libmpack lua binding

Building

LUA_TARGET=$(PLATFORM) make

Where PLATFORM is a supported Lua 5.1 platform: linux (default), freebsd, macosx, ...

For the complete list of targets run:

make; cd .deps/5.1.5/src/lua && make

libmpack-lua's Issues

License

The limpack-lua and libmpack-python repos lack license files. Also seems like it'd be a good time to tag a new version of libmpack-lua.

Fails to unpack with luarocks, "trailing garbage ignored"

The rock fails to unpack with luarocks on OS X 10.11.6

Steps to reproduce

$ cd $(mktemp -d)
$ wget https://raw.githubusercontent.com/libmpack/libmpack-lua/1.0.8/mpack-1.0.8-0.rockspec
$ luarocks unpack --verbose mpack-1.0.8-0.rockspec

io.popen: 	pwd 2> /dev/null

os.execute: 	cd '/var/folders/jv/n5rf3drx1td1ljkt6xk2_60w0000gn/T/tmp.KbIpq2MSor' && mkdir -p 'mpack-1.0.8-0'
Results: 3
  1 (boolean): true
  2 (string): exit
  3 (number): 0

os.execute: 	cd '/var/folders/jv/n5rf3drx1td1ljkt6xk2_60w0000gn/T/tmp.KbIpq2MSor/mpack-1.0.8-0/.' && curl '--version'
curl 7.67.0 (x86_64-apple-darwin15.6.0) libcurl/7.67.0 SecureTransport (OpenSSL/0.9.8zd) zlib/1.2.5 brotli/1.0.7 libidn2/2.2.0 librtmp/2.3
Release-Date: 2019-11-06
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS brotli IDN IPv6 Largefile libz MultiSSL NTLM NTLM_WB SSL UnixSockets
Results: 3
  1 (boolean): true
  2 (string): exit
  3 (number): 0

os.execute: 	cd '/var/folders/jv/n5rf3drx1td1ljkt6xk2_60w0000gn/T/tmp.KbIpq2MSor/mpack-1.0.8-0/.' && curl -k -f -L --user-agent "LuaRocks/3.2.1 macosx-x86_64 via curl" --connect-timeout 30 'https://github.com/libmpack/libmpack-lua/releases/download/1.0.8/libmpack-lua-1.0.8.tar.gz' > '/var/folders/jv/n5rf3drx1td1ljkt6xk2_60w0000gn/T/tmp.KbIpq2MSor/mpack-1.0.8-0/./libmpack-lua-1.0.8.tar.gz' 2> /dev/null
Results: 3
  1 (boolean): true
  2 (string): exit
  3 (number): 0

os.execute: 	cd '/var/folders/jv/n5rf3drx1td1ljkt6xk2_60w0000gn/T/tmp.KbIpq2MSor/mpack-1.0.8-0/.' && 'gunzip' -c '/var/folders/jv/n5rf3drx1td1ljkt6xk2_60w0000gn/T/tmp.KbIpq2MSor/mpack-1.0.8-0/./libmpack-lua-1.0.8.tar.gz' > '/var/folders/jv/n5rf3drx1td1ljkt6xk2_60w0000gn/T/tmp.KbIpq2MSor/mpack-1.0.8-0/./libmpack-lua-1.0.8.tar'

gzip: /var/folders/jv/n5rf3drx1td1ljkt6xk2_60w0000gn/T/tmp.KbIpq2MSor/mpack-1.0.8-0/./libmpack-lua-1.0.8.tar.gz: decompression OK, trailing garbage ignored
Results: 3
  1 (nil): nil
  2 (string): exit
  3 (number): 2

Error: Failed extracting /var/folders/jv/n5rf3drx1td1ljkt6xk2_60w0000gn/T/tmp.KbIpq2MSor/mpack-1.0.8-0/./libmpack-lua-1.0.8.tar.gz: failed extracting /var/folders/jv/n5rf3drx1td1ljkt6xk2_60w0000gn/T/tmp.KbIpq2MSor/mpack-1.0.8-0/./libmpack-lua-1.0.8.tar.gz

os.execute: 	cd '/' && rm '-rf' '/var/folders/jv/n5rf3drx1td1ljkt6xk2_60w0000gn/T/tmp.KbIpq2MSor/mpack-1.0.8-0'
Results: 3
  1 (boolean): true
  2 (string): exit
  3 (number): 0

Build error when using USE_SYSTEM_LUA=1

Hi!

When I try to build using USE_SYSTEM_LUA = 1 I get the following error:

$ make USE_SYSTEM_LUA=1
dir="mpack-src"; \
mkdir -p $dir && cd $dir && \
curl -L -o - https://github.com/tarruda/libmpack/archive/1.0.5.tar.gz | tar xfz - --strip-components=1
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   123    0   123    0     0    159      0 --:--:-- --:--:-- --:--:--   159
100   124    0   124    0     0    122      0 --:--:--  0:00:01 --:--:--   122
100 32966  100 32966    0     0  16163      0  0:00:02  0:00:02 --:--:-- 74415
cc -shared -ansi -O0 -g3 -Wall -Wextra -Werror -Wconversion -Wstrict-prototypes -Wno-unused-parameter -pedantic -fPIC -DMPACK_DEBUG_REGISTRY_LEAK   mpack-src lmpack.c -o mpack.so -llua -lm -ldl
mpack-src: file not recognized: É um diretório
collect2: error: ld returned 1 exit status
make: *** [Makefile:102: mpack.so] Error 1

Thanks,
Ronan

Build error when using GCC7

Hi!

If you try to build libmpack-lua with GCC7, then you will get the following error:

[    3s] cc -shared -ansi -O0 -g3 -Wall -Wextra -Werror -Wconversion -Wstrict-prototypes -Wno-unused-parameter -pedantic -fPIC -DMPACK_DEBUG_REGISTRY_LEAK   lmpack.c -o mpack.so -llua -lm -ldl
[    3s] lmpack.c: In function 'lmpack_unparse_enter':
[    3s] lmpack.c:688:10: error: this statement may fall through [-Werror=implicit-fallthrough=]
[    3s]        if (lmpack_isnil(L, -1)) {
[    3s]           ^
[    3s] lmpack.c:692:5: note: here
[    3s]      default:
[    3s]      ^~~~~~~
[    3s] cc1: all warnings being treated as errors
[    3s] make: *** [Makefile:102: mpack.so] Error 1
[    3s] error: Bad exit status from /var/tmp/rpm-tmp.ARXj9S (%build)

This can be fixed by the following patch:

1   Index: libmpack-lua-1.0.6/lmpack.c                                                      
Index: libmpack-lua-1.0.6/lmpack.c
===================================================================
--- libmpack-lua-1.0.6.orig/lmpack.c
+++ libmpack-lua-1.0.6/lmpack.c
@@ -689,6 +689,7 @@ static void lmpack_unparse_enter(mpack_p
         node->tok = mpack_pack_nil();
         break;
       }
+    /* Fallthrough */
     default:
       luaL_error(L, "can't serialize object");
   }

Is it the right approach? If so, then I can submit a PR.

Thanks,
Ronan

Makefile: Lua C modules shouldn't be linked against liblua

As you can notice here, when you compiling the module, you're trying to link the module against liblua, which is wrong behaviour (and adds problems here and there).

Please, read here (scroll to "Do Not Link Modules to the Lua Core Libraries" paragraph for the reasons).

I was about creating a PR with fix for that issue, but I'd like to do that after PR #7 would be merged (to avoid rebasing after that).

Allow linking against libmpack.so instead of including libmpack amalgamation

Building packages for distributions is difficult now that this has been split out of libmpack but requires libmpack's source to build. At least for Debian, I was hoping I could just do the typical install libmpack-dev/libmpack0 and then build libmpack-lua.

lmpack.c: compatibility with lua5.1-no-legacy API

Although, you have some compatibility logic, unfortunatelly, it has some little flow:
Actually, luaL_reg was a part of 5.0 API (and not 5.1) which was deprecated since 5.1 (although, 5.1 had a kludge-#define for that, so it was hard to notice).
And as a side effect of that, current behaviour brakes the compatibility with modern releases of LuaJIT 2.1 (which dropped 5.0 compatibility defines some time ago).

So, I see the fix in two variants:
[not as good as it can be] 1) detect that it is LuaJIT (even if 5.1 API is in use), and add that typedef in that case too;
[pretty good, imho] 2) fix code logic in a bit reverse way: use "modern" API (such as luaL_Reg instead of luaL_reg), and add #ifndef blocks with that API functions to define them (probably, even without taking LUA_VERSION_NUM into account).

How do you think?

lua-mpack segfaults on i686

Hello,

I've tried to build neovim 0.2.1 for Fedora and it failed to build on i686. I've tracked it down to lua-mpack.

I've just put together some code from the lua test which is part of the package but doesn't require anything fancy to verify it works during the RPM build.

The code is the following:

local mpack = require('mpack')

local t = mpack.unpack('\100')

local obj = {
    k1 = {
        {
            k2 = {{{{{{{{{{{{{{{{{{{{{{{{{{{{{1}}}}}}}}}}}}}}}}}}}}}}}}}}}}}
        }
    }
}
local unpack = mpack.Unpacker()
local pack = mpack.Packer()
local o = unpack(pack(obj))

I've built it using:

CFLAGS='-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m32 -march=i686 -fasynchronous-unwind-tables' make USE_SYSTEM_MPACK=no USE_SYSTEM_LUA=yes LUA_LIB="$(pkg-config --libs lua)" MPACK_LUA_VERSION_NOPATCH=5.3

The resulting segfault is:

Program received signal SIGSEGV, Segmentation fault.
0xb7fa7bd4 in luaH_next (L=L@entry=0x405164, t=0x0, key=0x40a774) at ltable.c:203
203     ltable.c: No such file or directory.
Missing separate debuginfos, use: dnf debuginfo-install ncurses-libs-6.0-13.20170722.fc27.i686 readline-7.0-7.fc27.i686
(gdb) bt
#0  0xb7fa7bd4 in luaH_next (L=L@entry=0x405164, t=0x0, key=0x40a774) at ltable.c:203
#1  0xb7f8884b in lua_next (L=L@entry=0x405164, idx=idx@entry=-2) at lapi.c:1128
#2  0xb7fc9349 in lmpack_unparse_enter (parser=parser@entry=0x40d8d0, node=node@entry=0x40d960) at lmpack.c:574
#3  0xb7fca7a3 in mpack_unparse_tok (enter_cb=0xb7fc8d80 <lmpack_unparse_enter>, exit_cb=0xb7fc6120 <lmpack_unparse_exit>, exit_cb=0xb7fc6120 <lmpack_unparse_exit>, 
    enter_cb=0xb7fc8d80 <lmpack_unparse_enter>, tok=0xbfffe13c, parser=0x40d8d0) at mpack-src/src/object.c:63
#4  mpack_unparse (parser=0x40d8d0, buf=buf@entry=0xbfffe1a4, buflen=buflen@entry=0xbfffe1a8, enter_cb=0xb7fc8d80 <lmpack_unparse_enter>, 
    exit_cb=0xb7fc6120 <lmpack_unparse_exit>, exit_cb=0xb7fc6120 <lmpack_unparse_exit>, enter_cb=0xb7fc8d80 <lmpack_unparse_enter>) at mpack-src/src/object.c:110
#5  0xb7fcaa48 in mpack_unparse (exit_cb=0xb7fc6120 <lmpack_unparse_exit>, enter_cb=0xb7fc8d80 <lmpack_unparse_enter>, buflen=0xbfffe1a8, buf=0xbfffe1a4, 
    parser=<optimized out>) at lmpack.c:783
#6  lmpack_packer_pack (L=0x405164) at lmpack.c:758
#7  0xb7f93c34 in luaD_precall (L=L@entry=0x405164, func=func@entry=0x40a750, nresults=nresults@entry=-1) at ldo.c:434
#8  0xb7fadc5b in luaV_execute (L=<optimized out>, L@entry=0x405164) at lvm.c:1134
#9  0xb7f93f1a in luaD_call (L=L@entry=0x405164, func=func@entry=0x4054cc, nResults=nResults@entry=-1) at ldo.c:499
#10 0xb7f93f84 in luaD_callnoyield (L=L@entry=0x405164, func=0x4054cc, nResults=-1) at ldo.c:509
#11 0xb7f86bf0 in f_call (L=L@entry=0x405164, ud=ud@entry=0xbfffec74) at lapi.c:942
#12 0xb7f93326 in luaD_rawrunprotected (L=L@entry=0x405164, f=f@entry=0xb7f86bd0 <f_call>, ud=ud@entry=0xbfffec74) at ldo.c:142
#13 0xb7f9427c in luaD_pcall (L=L@entry=0x405164, func=func@entry=0xb7f86bd0 <f_call>, u=u@entry=0xbfffec74, old_top=60, ef=48) at ldo.c:729
#14 0xb7f8846c in lua_pcallk (L=0x405164, nargs=0, nresults=-1, errfunc=3, ctx=0, k=0x0) at lapi.c:968
#15 0x0040128f in docall (L=0x405164, narg=0, nres=-1) at lua.c:205
#16 0x00402303 in handle_script (argv=<optimized out>, L=<optimized out>) at lua.c:445
#17 pmain (L=<optimized out>) at lua.c:580
#18 0xb7f93c34 in luaD_precall (L=L@entry=0x405164, func=func@entry=0x40549c, nresults=nresults@entry=1) at ldo.c:434
#19 0xb7f93f0e in luaD_call (L=L@entry=0x405164, func=func@entry=0x40549c, nResults=nResults@entry=1) at ldo.c:498
#20 0xb7f93f84 in luaD_callnoyield (L=L@entry=0x405164, func=0x40549c, nResults=1) at ldo.c:509
#21 0xb7f86bf0 in f_call (L=L@entry=0x405164, ud=ud@entry=0xbfffef34) at lapi.c:942
#22 0xb7f93326 in luaD_rawrunprotected (L=L@entry=0x405164, f=f@entry=0xb7f86bd0 <f_call>, ud=ud@entry=0xbfffef34) at ldo.c:142
#23 0xb7f9427c in luaD_pcall (L=L@entry=0x405164, func=func@entry=0xb7f86bd0 <f_call>, u=u@entry=0xbfffef34, old_top=12, ef=0) at ldo.c:729
#24 0xb7f8846c in lua_pcallk (L=0x405164, nargs=2, nresults=1, errfunc=0, ctx=0, k=0x0) at lapi.c:968
#25 0x00400fbf in main (argc=2, argv=0xbffff044) at lua.c:606

I'm sorry but I have no clue about lua. However I think it is related to:

(gdb) f 2
#2  0xb7fc9349 in lmpack_unparse_enter (parser=parser@entry=0x40d770, node=node@entry=0x40d800) at lmpack.c:574
(gdb) p parent->data[1].i
$1 = -1

Let me know if I can provide additional information and help fixing the issue.

Upload a src.rock too

I am trying to make neovim tests run on nixos and for now I just have the src.rock installable from nix, not the rockspec. Could you please upload a src.rock of mpack to luarocks ? To create it, you can run luarocks pack <mpack.rockspec>. Hopefully that won 't be necessary in the future but for now that would be helpful.

gunzip: libmpack-lua-1.0.8.tar.gz: trailing garbage ignored

Latest HEAD build from MacOS encountered this error:

.deps/usr/bin/luarocks --verbose build mpack 1.0.8-0

os.execute: cd '/var/folders/t2/38gr4l6n5t5d8sfcln4r45c8001ct_/T//luarocks_luarocks-source-mpack-1.0.8-0-2bxV8r' && 'gunzip' -c '/var/folders/t2/38gr4l6n5t5d8sfcln4r45c8001ct_/T//luarocks_luarocks-source-mpack-1.0.8-0-2bxV8r/libmpack-lua-1.0.8.tar.gz' > '/var/folders/t2/38gr4l6n5t5d8sfcln4r45c8001ct_/T//luarocks_luarocks-source-mpack-1.0.8-0-2bxV8r/libmpack-lua-1.0.8.tar'
gunzip: /var/folders/t2/38gr4l6n5t5d8sfcln4r45c8001ct_/T//luarocks_luarocks-source-mpack-1.0.8-0-2bxV8r/libmpack-lua-1.0.8.tar.gz: trailing garbage ignored
Results: 1
1 (number): 512

Error: Failed extracting /var/folders/t2/38gr4l6n5t5d8sfcln4r45c8001ct_/T//luarocks_luarocks-source-mpack-1.0.8-0-2bxV8r/libmpack-lua-1.0.8.tar.gz: failed extracting /var/folders/t2/38gr4l6n5t5d8sfcln4r45c8001ct_/T//luarocks_luarocks-source-mpack-1.0.8-0-2bxV8r/libmpack-lua-1.0.8.tar.gz

I manually download this libmpack-lua-1.0.8.tar.gz file and using the luarock default gunzip to unpack it and it indeed has problem.

However, the package is unpacked successfully with tar xvfz libmpack-lua-1.0.8.tar.gz.

If I do

tar xvfz libmpack-lua-1.0.8.tar.gz
tar cvfz libmpack-lua-1.0.8.new.tar.gz
gunzip -c libmpack-lua-1.0.8.new.tar.gz > libmpack-lua-1.0.8.tar

it works. This does seem to prove that the original .tar.gz file is somehow corrupted OR maybe the Apple gunzip is somehow behaving wrong... but that thing hasn't changed

gunzip --version
Apple gzip 321.100.10.0.1

ls -la /usr/bin/gunzip
.rwxr-xr-x 180,224 root 1 Jan 2020 /usr/bin/gunzip*

Could not install library through luarocks.

I tried sudo luarocks install nvim-client command on my linux system and got next error:

Installing https://luarocks.org/nvim-client-0.2.2-1.src.rock
Missing dependencies for nvim-client 0.2.2-1:
   mpack (not installed)
   luv (not installed)

nvim-client 0.2.2-1 depends on lua >= 5.1 (5.3-1 provided by VM)
nvim-client 0.2.2-1 depends on mpack (not installed)
Installing https://luarocks.org/mpack-1.0.8-0.rockspec

Error: Failed installing dependency: https://luarocks.org/mpack-1.0.8-0.rockspec - Error fetching file: Failed downloading https://github.com/libmpack/libmpack-lua/releases/download/1.0.8/libmpack-lua-1.0.8.tar.gz - libmpack-lua-1.0.8.tar.gz

Such situation with sudo luarocks install mpack

Installing https://luarocks.org/mpack-1.0.8-0.rockspec

Error: Error fetching file: Failed downloading https://github.com/libmpack/libmpack-lua/releases/download/1.0.8/libmpack-lua-1.0.8.tar.gz - libmpack-lua-1.0.8.tar.gz

But I can download source archive manuall by next link https://github.com/libmpack/libmpack-lua/releases/download/1.0.8/libmpack-lua-1.0.8.tar.gz

memory leak in session receive on response?

If I use a closure callback function when creating a request (to keep a reference alive as an upvalue while waiting for the response), I noticed that my program started to leak memory albeit at a very slow pace.

I started to look into this and traced it back to the static int lmpack_session_receive(lua_State *L) function in lmpack.c. It seems that the cookie passed along when creating the request is not being unref'd, so the garbage collector failed to remove the callback function, which in turn kept the upvalue reference alive and so on...

When applying this patch, the leak seems to be fixed for me:

diff --git a/lmpack.c b/lmpack.c
index 4a7e4d1..5ec1a52 100644
--- a/lmpack.c
+++ b/lmpack.c
@@ -904,6 +904,8 @@ static int lmpack_session_receive(lua_State *L)
     case MPACK_RPC_RESPONSE:
       lua_pushstring(L, "response");
       lmpack_geti(L, session->reg, (int)session->unpacked.msg.data.i);
+      lmpack_unref(L, session->reg, (int)session->unpacked.msg.data.i);
+      session->unpacked.msg.data.i = LUA_NOREF;
       break;
     case MPACK_RPC_NOTIFICATION:
       lua_pushstring(L, "notification");

I am not completely convinced this is the right way to fix it, so I didn't submit a patch and post an issue instead.

v1.0.9: Could not install/compile on MacOS (snprintf)

Unfortunately I am not able to compile mpack on MacOS.

luarocks install mpack
Installing https://luarocks.org/mpack-1.0.9-0.src.rock

env MACOSX_DEPLOYMENT_TARGET=10.5 gcc -O2 -fPIC -c lmpack.c -o lmpack.o
lmpack.c:721:3: error: implicitly declaring library function 'snprintf' with type 'int (char *, unsigned long, const char *, ...)' [-Werror,-Wimplicit-function-declaration]
                snprintf(errmsg, 50, "can't serialize object of type %d", type);
                ^
lmpack.c:721:3: note: include the header <stdio.h> or explicitly provide a declaration for 'snprintf'
1 error generated.

Error: Build error: Failed compiling object lmpack.o

Probably a compile switch should be added here ?:
1740a9d

session fails to decode correct mpack rpc string

It seems the session:receive fails continuously to decode a correct msgpack rpc string once it first correctly fails on non rpc mpack data.
I can't seem to find a way to fix it except for creating a new session.

Here's a small script to demonstrate.

local mpack = require('mpack')

-- these data were built on the msgpack website with their "try" page
-- JSON: [0,8,"add",[1,2]]
corr_rpc_hex = [[ 94 00 08 a3 61 64 64 92 01 02 ]]

-- JSON: {"this" : "that", "that" : "this", "numbers" : [1,2,3,4], "sub" : { "a" : 1, "b" : 2} }
corr_mpack_hex = [[ 84 a4 74 68 69 73 a4 74 68 61 74 a4 74 68 61 74
                    a4 74 68 69 73 a7 6e 75 6d 62 65 72 73 94 01 02
                    03 04 a3 73 75 62 82 a1 61 01 a1 62 02 ]]

--------------------------------------------------------------------------------
function string.remove_spaces(str)
  return str:gsub("%s+", "")
end

--------------------------------------------------------------------------------
function string.fromhex(str)
    return (str:gsub('..', function (cc)
        return string.char(tonumber(cc, 16))
    end))
end

--------------------------------------------------------------------------------
-- tests

local rpc_data = corr_rpc_hex:remove_spaces():fromhex()
--print(rpc_data)

local mpack_data = corr_mpack_hex:remove_spaces():fromhex()
--print (mpack_data)

local session = mpack.Session({unpack = mpack.Unpacker()})

-- correctly decodes rpc
ok, rpctype, id, method, args, pos = pcall(session.receive, session, rpc_data)
print(ok, rpctype, id, method, args, pos)

-- correctly detects error
--[-[
ok, rpctype, id, method, args, pos = pcall(session.receive, session, mpack_data)
print(ok, rpctype, id, method, args, pos)
--]]

-- fails to decode rpc
ok, rpctype, id, method, args, pos = pcall(session.receive, session, rpc_data)
print(ok, rpctype, id, method, args, pos)

-- correctly decodes again
local session = mpack.Session({unpack = mpack.Unpacker()})

ok, rpctype, id, method, args, pos = pcall(session.receive, session, rpc_data)
print(ok, rpctype, id, method, args, pos)