meric / l2l Goto Github PK

I'm quite interested in this project; unfortunately I was thinking of using it with a love2d game, which tragically uses LuaJIT compiled without -DLUAJIT_ENABLE_LUA52COMPAT.

Out of curiosity, what kinds of things prevent the codebase from being compatible with 5.1?

I want to add an apply function, but I don't understand enough about the organization of the codebase to know where it should go.

> (do 1 2 3 4)
Compiler error: .\l2l\compiler.lua:290: cannot not statize..1
> (fn () 1 2 3 4)
Compiler error: .\l2l\compiler.lua:290: cannot not statize..1

Works for strings:

> (fn () "a" "b" "c")
function: 0x0063c948
> (do "a" "b" "c")
c

Nice work. I am a bit confused as for bindings. Let's say I want to do Love2d stuff, which is Lua based. Would I write Lisp code and embed Lua code via backslash?

'seti' works fine in REPL, but it does not work in script:

(set x '(0 2 3 4))
(seti x 1 1)

lua: /usr/local/share/lua/5.1/l2l/main.lua:117: attempt to call a nil value
stack traceback:
	/usr/local/share/lua/5.1/l2l/main.lua:117: in function 'run'
	/usr/local/bin/l2l:3: in main chunk
	[C]: at 0x563c6c5731d0

--tested with "l2l version 0.0.2-pre" on Linux - Pepermint 9 [x86_64]

Witness the output of compiling a subtraction:

-- (let (a 1 b 2 c (- b a)) c)
local _var0
do
   local a=1
   local b=2
   local _var4
   _var4=(-b)
   local _var6
   local _var7
   for _var6, _var7 in next, {a} do
      _var4 = _var4 - _var7
   end
   local c=_var4
   _var0=c
end
return _var0

I've found 3 problems preventing running on LuaJIT so far:

• Missing table.pack; easy fix
• vector:insert has an undefined field self.n; not sure why
• function missing parameters or body; stack trace below

ERROR in test_local():
    [string "local lua = require("l2l.lua")..."]:15: function missing parameters or body
stack traceback:
    [string "local lua = require("l2l.lua")..."]:15: in function 'statize'
    ./l2l/compiler.lua:157: in function 'statize'
    ./l2l/compiler.lua:400: in function 'compile'
    ./l2l/compiler.lua:504: in function 'compile_or_cached'
    ./l2l/compiler.lua:327: in function 'build'
    ./l2l/compiler.lua:339: in function 'import'
    ./l2l/reader.lua:324: in function 'import_extension'
    ./l2l/compiler.lua:392: in function 'compile'
    l2l/test.lua:9: in function 'assert_exec_equal'
    l2l/test.lua:17: in function <l2l/test.lua:16>
    [C]: in function 'xpcall'
    /home/phil/.luarocks/share/lua/5.1/lunatest.lua:624: in function 'run_test'
    /home/phil/.luarocks/share/lua/5.1/lunatest.lua:708: in function 'run_suite'
    /home/phil/.luarocks/share/lua/5.1/lunatest.lua:752: in function 'run'
    l2l/test.lua:202: in main chunk
    [C]: at 0x00404d50

Is this due to the fact that Lua 5.1 (and LuaJIT by default) does not support iterators from metatables, or do you think there is a different root cause?

Currently l2l.list is a list data structure that has an inherent memory leak.

Add reference counting to fix.

Have to fix compile_and so if it is called with three or more arguments and the second one is false, it stops evaluating the rest.

local compile_and = variadic(
  function(block, stream, parameters)
    return list.concat(map(bind(compile, block, stream), parameters), " and ")
  end,
  function(reference, value)
    return reference .. " and " .. value
  end, "true",
  function(var) return "if "..var.." then" end,
  function(var) return "end" end)

in the compiler.lua :

local function build(stream)
  local src = {
    "require(\'core\').import(\'core\')"
  }

We should use local variables instead ?

local function build(stream)
  local src = {
[[local repl, compile, compiler, hash, read, reader, exception, raise, eval
do local _ = require('core')
  repl,   compile,   compiler,   hash,   read,   reader,   exception,   raise,   eval =
_.repl, _.compile, _.compiler, _.hash, _.read, _.reader, _.exception, _.raise, _.eval
end]]
  }

Regression errors have been cropping up after bug fixes, perhaps it would be a good idea to add a Makefile that can run samples, add some rudimentary unit tests, run some linting, e.g. (https://github.com/mpeterv/luacheck).

this is actually a query and issue?
I would request that l2l syntax be a strict subset of clojure. That is (when not accessing lua api or jvm api), the every valid l2l program will be a valid clojure program, but reverse may not be true.

in that case, there will be a real practical benefit of porting our small programs to l2l

>> (+ (unpack (vector 1 2 3)))
=   1
>> 

Should print 6, but prints one, it appears (unpack (vector 1 2 3))) is perceived as a single argument, and evaluated to 1 by the compiling function for +.

The issue is, the compiler functions can only manipulate code, and does not evaluate them. It cannot know (unpack (vector 1 2 3))) will return more than 1 value, until that parameter is evaluated, which happens at a later stage.

I'm using the line-numbers-in-comments branch.

Given this input to the l2l compiler:

(local v 1)
(local r nil)

(cond
    (not v) (set r "a")
    (set r "b"))
(print r)

A correct compilation is created:

local v = 1

local r = nil

local _cond_value0

local _cond_found1

if  not v then _cond_found1=true
do end
r="a"
do end
_cond_value0=r
else

end

if  not _cond_found1 then r="b"
do end
_cond_value0=r
end
print(r)

But if we put the cond body inside a high level macro using a quasiquote and some escapes:

-- save as sm_macros.lisp
@import fn
@import quasiquote

(fn test_qq (vn rn)
    `(cond
        (not ,vn) (set ,rn "a")
        (set ,rn "b")))

{
    macro = {
        ["test_qq"] = test_qq,
    }
}

And run the code as:

@import sm_macros

(local v 1)
(local r nil)

(test_qq v r)
(print r)

Then the compilation output is nonsensical:

local v = 1

local r = nil

local _var0 = "b"
print(r)

If "not" is not present in the cond, it appears to create a correct output.
Here I replaced "(not ,vn)" with ",vn" in the macro:

local v = 1

local r = nil

local _cond_value11

local _cond_found12

if v then _cond_found12=true
do end
r="a"
do end
_cond_value11=r
else

end

if  not _cond_found12 then r="b"
do end
_cond_value11=r
end
print(r)

I'm wondering what features of Lua 5.2 are needed by l2l. LuaJIT supports a subset of the 5.2 features, including goto and labels, but I am unsure what would need to be changed in l2l to be backwards compatible with LuaJIT's 5.??? extensions. I will try my hand at this, but it is unlikely I will do too well. Maybe Moonscript (also here on GitHub, https://github.com/leafo/moonscript ) could serve as some example code for how to compile to Lua 5.1 and thus retain compatibility with standard Lua and LuaJIT.
As an aside, this is a pretty cool project and I wish it had gotten some more attention in the community when it was new. A compiler/translator that fits in one, probably-less-than-1000-SLoC file is not something you see every day...

Currently let-binding works like Clojure instead of the traditional Scheme/CL style:

(let ((a b) (f)
      c 3)
      \return a, b, c)

In general I think this is nice; however, the difference between Clojure and l2l is that Clojure doesn't support multiple return values, so (let [[a b] (f)] ...) will destructure the return value of f into two locals. Since we have to support multiple return values, there is some ambiguity about whether binding to two locals would work with multiple values or whether it would destructure.

Personally I am of the opinion that a data structure on the left-hand side should correspond with destructuring a data structure on the right-hand side. So how would it be possible to bind to multiple values? I think it could be done by reintroducing Scheme-style let which has an extra level of parens around each binding group:

(let ((a b (f))
      (c (+ 32 1))
      ((d e) (list 1 2)))
  ...)

It is a little more syntactically noisy, but I believe it is more consistent and flexible if we want to allow the goals of both binding to multiple values and supporting destructuring of data structures.

What do you think?

I'd feel more comfortable hacking on the compiler if the tests would actually fail or pass instead of just printing output that needs to be manually eyeballed. Would you be up for getting it changed over to a proper suite that can report failures? My recommendation is Lunatest: https://github.com/silentbicycle/lunatest but if you prefer a different lib we can try that.

List of variables injected in l2l: Args, BinOp, Block, Chunk, Comment, Exp, ExpList, Field, FieldList, FieldSep, FuncBody, FuncName, FunctionCall, FunctionDef, Label, LiteralString, LongString, Lua, Name, NameList, NameOrLispExp, Numeral, ParList, PrefixExp, RetStat, Stat, TableConstructor, UnOp, Var, VarList, _37, _42, _43, _45, _47, _60, _60_61, _61_61, _62, _62_61, _dot_dot, and_97_110_100, apply, build, call, compile, compiler, compose, contains, copy, destructure, dotmap, each, environ, error_handler, escape, expand, expize, factor, filter, hasmetatable, import, import_extension, inherit, initialize_dependencies, inserts, ipairs, is_space, isalpha, isalphanumeric, keys, len, list, load_extension, loadstring, lua_annotated, lua_args, lua_assign, lua_ast, lua_binop, lua_binop_exp, lua_block, lua_break, lua_chunk, lua_colon_functioncall, lua_comment, lua_do, lua_dot, lua_else, lua_elseif, lua_elseifs, lua_explist, lua_false, lua_field_key, lua_field_name, lua_fieldlist, lua_for, lua_for_in, lua_funcbody, lua_funcname, lua_function, lua_functioncall, lua_goto, lua_if, lua_index, lua_inline_functioncall, lua_label, lua_lambda_function, lua_lazy, lua_local, lua_local_function, lua_long_comment, lua_name, lua_nameize, lua_namelist, lua_nil, lua_none, lua_paren_exp, lua_repeat, lua_retstat, lua_semicolon, lua_step, lua_string, lua_table, lua_true, lua_unop, lua_unop_exp, lua_vararg, lua_varlist, lua_while, mangle, map, not_110_111_116, numeral, or_111_114, prototype, read, read_dict, read_list, read_lua, read_lua_literal, read_quote, read_right_brace, read_right_paren, read_symbol, reader, rep, reverse, set_sourcemap_to, skip_whitespace, spaces, spacing, span, statize, symbol, term, to_stat, torepresentation, trace, unpack, vector

There are many variables injected into scope. A lot of them have to be in there because of the way quoted lua ast nodes become compiled into Lua. IMO good idea to think of a way to reduce possibility of l2l and user names conflicting. At the same time a lot aren't necessary. (e.g. the ones starting with capital letter). List the necessary ones explicitly in compiler.initialize_dependencies instead of wholesale injecting everything available in those modules.

Currently the map function in itertools module, calls the anonymous function with the value, as well as the index of the value.

So, when calling something like (map + '(1 2 3 4)) it doesn't produce '(1 2 3 4) but rather '(2 4 6 8) because map is making + add the value as well as the index of the value together.

Thoughts?

#47 (comment)

The repl supports partial input, but it bases it on the error message string to determine partial input: if(src_or_err:find("no bytes$")) This could be made more explicit somehow.

Printing of regular tables is still Lua's stupid opaque method: table: 0x10c8700 so we should add some pretty-printing at some point.

In the case of syntax error in the lisp source, we want to show pretty exception messages.

When compiling Lisp in debug mode, and there are run time errors, we want to show a traceback in Lisp, in addition to the traceback of the compiled Lua.

In other lisps, (list x y z) is equivalent to (x y z). in l2l it's (list (vector x y z)). Would be good idea to make it more consistent.

Maybe it might be a good idea to rename #import to @import, to free up the # symbol to be used as the length operator, to match Lua.

Right now the length operator in lisp is implemented as "length", which is different to Lua's "#".

I note that escaping into Lua, to use #,(print \#myarray) currently works.

Introduced from 11c1009

The ... method of extracting the current file means, running lua core.lua sample01.lsp or `./l2l sample01.lsp, results in:

lua: ./l2l:5: module 'sample01.lspcompat' not found:
    no field package.preload['sample01.lspcompat']
    no file '/Users/Eric/.luarocks/share/lua/5.2/sample01/lspcompat.lua'
    no file '/Users/Eric/.luarocks/share/lua/5.2/sample01/lspcompat/init.lua'
    no file '/usr/local/share/lua/5.2/sample01/lspcompat.lua'
    no file '/usr/local/share/lua/5.2/sample01/lspcompat/init.lua'
    no file '/usr/local/Cellar/lua/5.2.4_1/libexec/share/lua/5.2/sample01/lspcompat.lua'
    no file '/usr/local/lib/lua/5.2/sample01/lspcompat.lua'
    no file '/usr/local/lib/lua/5.2/sample01/lspcompat/init.lua'
    no file './sample01/lspcompat.lua'
    no file '/Users/Eric/.luarocks/lib/lua/5.2/sample01/lspcompat.so'
    no file '/usr/local/lib/lua/5.2/sample01/lspcompat.so'
    no file '/usr/local/lib/lua/5.2/loadall.so'
    no file './sample01/lspcompat.so'
    no file '/Users/Eric/.luarocks/lib/lua/5.2/sample01.so'
    no file '/usr/local/lib/lua/5.2/sample01.so'
    no file '/usr/local/lib/lua/5.2/loadall.so'
    no file './sample01.so'
stack traceback:
    [C]: in function 'require'
    ./l2l:5: in main chunk
    [C]: in ?

Currently the most-nested macros are evaluated first, when it should be the reverse order. This means "..." varargs are opaque to the nested macros. This is a limitation of macros implemented purely by defcompiler, piecemeal. Use a global macro table (or similar solution) to maintain index of macro transform functions so they can be evaluated in the correct order, from outer most to inner most passing on arguments.

Having the reader responsible for evaluation doesn't make any sense.

What about trying to keep the API for loading code as similar to that of Lua as possible? Adding load, loadstring, and loadfile which return chunk functions would make it more consistent.

Just a idea,
Moving all sample0* inside one samples directory ?

It may pose problems conflicting with the (. a b) operator.

Inside compile_if, we create ref and assign the last value of each branch to it. However, when we have a "possibly-variadic" value, we assign a table to the ref and then unpack it after the if. This is fine as long as we have an else branch, but if we don't then the ref never gets set, but we attempt to unpack it.

I have a feeling this whole thing could be a lot simpler if we could just identify tail positions and insert return calls directly instead of tracking things with _var0 and such, but maybe this is more difficult than it sounds. It would certainly result in dramatically more readable compiler output.

For example, I want to call a two argument function like (2d-point 0.2 -0.2). In the REPL:

> (list 0.2 -0.2)
list(0)

Some way to toggle this behavior by the compiler would be necessary. It can also generate invalid code like:

> (list "a" -2)
ERROR: attempt to perform arithmetic on a string value

Right now I need to write all negative number constants as (- n) to avoid this behavior.

It looks like the entire Lua environment is available to any lisp code to run. Is there any way to limit this just to a specific sandbox? I can see in the compiler.eval function that there's a way to pass in your own env table, but how difficult would it be to limit it to only what's in env?

I could take a swing at implementing this if you point me in the right direction; I've implemented a Scheme before.