susonicth / zli

Portable statically linked lua interpreter

ZLI - Zig Lua Interpreter

ZLI is a portable lua interpreter statically compiled (on non windows with musl) with additional libraries. The binary can be moved from system to system without any dependencies on installed ibraries.

Currently supported platforms

• Windows on x86_64
• linux on x86_64
• linux on aarch64

Other platforms that are supported by zig could work but are untested

• Windows on x86 and aarch64
• linux on x86, armv7a, riscv64 and powerpc64le
• macOS on aarch64 and x86_64

Batteries included

There are many libraries included in the ZIL binary, some 3rd party lua libraries and some developed specifically for ZIL.

included lua libraries

3rd party lua libraries included in zli

library	link	licence	description
argparse	https://github.com/mpeterv/argparse	MIT	Argparse is a feature-rich command line parser for Lua inspired by argparse for Python
LuaSQLite3	http://lua.sqlite.org/	MIT	LuaSQLite 3 is a thin wrapper around the public domain SQLite3 database engine.
LPeg	https://www.inf.puc-rio.br/~roberto/lpeg/	MIT	LPeg is a new pattern-matching library for Lua, based on Parsing Expression Grammars (PEGs)
lua-zlib	https://github.com/brimworks/lua-zlib	MIT	lua binding to the zlib to compress/decompress
luaunit	https://github.com/bluebird75/luaunit	BSD	LuaUnit is a popular unit-testing framework for Lua, with an interface typical of xUnit libraries (Python unittest, Junit, NUnit, ...)
lua-cjson	https://github.com/openresty/lua-cjson	MIT	Fast JSON encoding/parsing
Serpent	https://github.com/pkulchenko/serpent	MIT	Lua serializer and pretty printer.
ftcsv	https://github.com/FourierTransformer/ftcsv	MIT	ftcsv is a fast csv library written in pure Lua
f-string	https://github.com/hishamhm/f-strings	MIT	String interpolation for Lua, inspired by f-strings, a form of string interpolation coming in Python 3.6.

Custom ZLI bindings to 3rd party libraries

there are some custom lua bindings for 3rd party libraries

library	link	licence	description
Crossline	https://github.com/jcwangxp/Crossline	MIT	Crossline is a small, self-contained, zero-config, MIT licensed, cross-platform, readline and libedit replacement.
unzip / zip	https://zlib.net/	MIT	binding to the minizip library included in zlib for zip file handling

Inhouse ZLI libraries

libraries developed specifically for ZLI that don't use any external libraries

library	description
auxiliary	some auxiliary functions to improve the string, table and file handling
collection	a collection library simmilar to java's collection. (Hash)Set, (Hash)map and (Array)list are implemented
filesystem	providing filesystem functions like listing/creating/changing directories, deleting/renaming/moving files & directories.
logger	a very simple logging library
stream	a stream library enspired by the java stream library that brings the functional style programming to lua.
timer	a nanosecond timer for high precision timings available as os.nanotime

Included tools

In the main binary there are also some commandline tools included. More information can be found in Tools

Clone & Build

This works under linux and windows(under gitbash) for x86_64 platforms and on linux arm all it needs git and zig version 0.13.0 on the path

git clone https://github.com/SuSonicTH/zli.git
cd zli
./build.sh

If you want to compile for all the supported platforms use the --all switch, all binaries are saved to the bin directory with the platform suffix.

./build.sh --all

If you want to have a substentially smaller binary you also have to have upx on the path and call:

./build.sh --upx

Tests can be executed by calling

./build.shj --test

Easymode

You can also use the fully automated get-zig.sh script to do everything above and dowload the zig compiler and upx binary for the build if needed. only on x86_64 on windows or linux All you need to have is git, bash and curl. (works on gitbash in windows) Paste the following into your bash terminal. No sudo, no installation, everything happens in the zli directory created in the current directory.

curl -s https://raw.githubusercontent.com/SuSonicTH/zli/master/getzli.sh | bash && cd zli

get-zig.sh can also be used in a cloned repository to download the zig compiler and upx binary if they are not on the path, or with option --force it doloads zig and upx anyway.

zli's Issues

Automatic UPX download in build.zig

automatically download the upx binary in buid.zig so we don't have to rely on the user to have it installed.

collection library: add sorted_set

• add a sorted_set class simmilar to javas

probably by having an _order array that keeps the items in sort order

add memoize function

• add memoize function that cashes results from function calls

something like this:

function memoize (f)
    local mem = {} -- memoizing table
    setmetatable(mem, {__mode = "kv"}) -- make it weak
    return function (x) -- new version of ’f’, with memoizing
        local r = mem[x]
        if r == nil then -- no previous result?
            r = f(x) -- calls original function
            mem[x] = r -- store result for reuse
        end
        return r
    end
end

maybe add a more complex but generic version that can have multiple arguments and multiple return values and may return nils

collection library: add linked list

• add a generic linked list data structure

collection library: add array list

• add an array or array_list class to collection simmilar to javas ArraList

Refactor unit tests

• create a test directory with all tests including binaries for testing for .gz, ,zip,...-
• split tests into libraries
• more verbose output?
• migrate all old tests to new structure
• more coverage for existing libraries/bindings

In filesystem lib add size_tree to get the size of whole directory

add a markdown library

pure lua: https://github.com/mpeterv/markdown
lua binding for cmark: https://github.com/jgm/cmark-lua
very fast c lib need custom binding: https://github.com/mity/md4c

strings with variable/expression substitution

• implement a string replacement/templating system simmilar to groovy g-strings

Excel reader/writer library

add an xlsx reader/writer
maybe https://xlsxwriterlua.readthedocs.io/
and https://github.com/jjensen/lua-xlsx/blob/master/xlsx.lua
both old and unsupported.

Or create own implementation based on java xlsxreader/writer

Make io.open and io.read be aware of the zip payload

• io.open io.read,... can read files from payload files
• should be (optionally?) transparent i.e tries to read a file from fs if not found searches in zip payload

optinally TBD

• add commandline option to use a zipfile on fs instead of (the) one attached to the exe
• add commandline option to use a directory on fs instead of (the) zip

Some libraries for inclusion or inspiration

Some libraries that could be added or taken as inspiration for own libraries

• https://lunarmodules.github.io/Penlight/
• https://github.com/daurnimator/lpeg_patterns
• https://github.com/hoelzro/lua-term
• https://github.com/dankamongmen/notcurses
• https://github.com/nothings/single_file_libs
• https://github.com/nothings/stb
• https://en.cppreference.com/w/c/links/libs

collection library: add sorted_list

• add a sorted_list that keeps its items sorted
• sorting done on add/insert

Support local variables (closures) in lambdas

implement local variable handling in L/lambda like in F

Benchmark performance differences between Fast and Small build

If the differences are small it might be a good idea to only go for the Small build.
Or if they are huge only go for the fast build.

add luacheck, a static analyser and linter

maybe add luacheck as a library https://github.com/lunarmodules/luacheck

add xml library

• add a xml en-/decoder library
• either existing lua lib or custom binding to existing c-lib

collection library: add a ordered_map and ordered_set

• Maybe implement an ordered_map and ordered_set
• they would keep a _order array where items are added in the order they are added

collection library: add tree

• implement a generic tree data structure

add tar library

add a tar/untar library
either a binding to an existing c-lib or one from scratch in zig

create a "compiler" for a zli project which packages scripts and resources into an exe with zip payload

create a "compiler" for a zli project which packages scripts and resources into an exe with zip payload

for refinement:

• multiple ways of creaing/adding to a binary
  • a lua config that has all the directories and files listed
  • and/or a commandline interface to create a exe and add files/directories
• should be able to "extract" the exe from the currently executed binary, even with payload no matter if it's compressed or not
• possibility set the main executable
• posibility to have multiple executables inside the payload (simmilar to zli itself)

Add goals for Release builds

• add build goals for Fast/Small builds
• add build goals for each supported platform
• add goals for cleanup (zig-cache, zig-out, test temp directory)
• get rid of build.sh