kyren / piccolo

An experimental stackless Lua VM implemented in pure Rust

piccolo - An experimental stackless Lua VM implemented in pure Rust

(After four years, now UN-paused!)

Project Goals, in roughly descending priority:

• Be an arguably working, useful Lua interpreter.
• Be an easy way to confidently sandbox untrusted Lua scripts.
• Be resilient against DoS from untrusted scripts (scripts should not be able to cause the interpreter to panic or use an unbounded amount of memory and should be guaranteed to return control to the caller in some bounded amount of time).
• Be an easy way to bind Rust APIs to Lua safely, with a bindings system that is resilient against weirdness and edge cases, and with user types that can safely participate in runtime garbage collection.
• Be pragmatically compatible with some version(s) of PUC-Rio Lua.
• Don't be obnoxiously slow (for example, avoid abstractions that would make the interpreter fundamentally slower than PUC-Rio Lua).

You read more about the design of piccolo (and try it out a live REPL!) in this blog post.

API Instability

Expect frequent pre-1.0 API breakage, this crate is still very experimental. All API incompatible changes will be accompanied by minor version bumps, but these will be very common.

Safety

The goal with piccolo is to have the majority of it written in safe Rust. Currently, there are a few sources of unsafety, but crucially these sources of unsafety are isolated. piccolo will avoid at all costs relying on abstractions which leak unsafety, it should always be possible to interact with even low level details of piccolo without using unsafe.

The current primary sources of unsafety:

• The particularly weird requirements of Lua tables require using hashbrown's low level RawTable API.
• Userdata requires unsafety to allow for downcasting non-'static userdata with a safe interface.
• The implementation of async Sequences require unsafety to "tunnel" the normal Sequence method parameters into the future (this is completely hidden from the user behind a safe interface).
• Unsafe code is required to avoid fat pointers in several Lua types, to keep Value as small as possible and allow potential future smaller Value representations.

(piccolo makes no attempt yet to guard against side channel attacks like spectre, so even if the VM is memory safe, running untrusted scripts may carry additional risk. With no JIT or callback API to accurately measure time, this might be practically impossible anwyay.)

A unique system for Rust <-> GC interaction

The garbage collector system for piccolo is now in its own repo, and also on crates.io. See the README in the linked repo for more detail about the GC design.

piccolo has a real, cycle detecting, incremental garbage collector with zero-cost Gc pointers (they are machine pointer sized and implement Copy) that are usable from safe Rust. It achieves this by combining two things:

1. An unsafe Collect trait which allows tracing through garbage collected types that, despite being unsafe, can be implemented safely using procedural macros.
2. Branding Gc pointers by unique, invariant "generative" lifetimes to ensure that such pointers are isolated to a single root object, and to guarantee that, outside an active call to mutate, all such pointers are either reachable from the root object or are safe to collect.

Stackless VM

The mutate based GC API means that long running calls to mutate can be problematic. No garbage collection can take place during a call to mutate, so we have to make sure to regularly return from the mutate call to allow garbage collection to take place.

The VM in piccolo is thus written in what is sometimes called "stackless" or "trampoline" style. It does not rely on the rust stack for Lua -> Rust and Rust -> Lua nesting, instead callbacks can either have some kind of immediate result (return values, yield values from a coroutine, resume a thread, error), or they can produce a Sequence. A Sequence is a bit like a Future in that it is a multi-step operation that the parent Executor will drive to completion. Executor will repeatedly call Sequence::poll until the sequence is complete, and the Sequence can yield values and call arbitrary Lua functions while it is being polled.

As an example, it is of course possible for Lua to call a Rust callback, which then in turn creates a new Lua coroutine and runs it. In order to do so, a callback would take a Lua function as a parameter, then create a new coroutine Thread from it and return SequencePoll:Resume to run it. The outer main Executor will run the created Thread, and when it is finished it will "return" via Sequence::poll (or Sequence::error). This is exactly how the coroutine.resume Lua stdlib function is implemented.

As another example, pcall is easy to implement here, a callback can call the provided function with a Sequence underneath it, and the sequence can catch the error and return the error status.

Yet another example, imagine Rust code calling a Lua coroutine thread which calls a Rust Sequence which calls yet more Lua code which then yields. Our stack will look something like this:

[Rust] -> [Lua Coroutine] -> [Rust Sequence] -> [Lua code that yields]

This is no problem with this VM style, the inner Rust callback is paused as a Sequence, and the inner yield will return the value all the way to the top level Rust code. When the coroutine thread is resumed and eventually returns, the Rust Sequence will be resumed.

With any number of nested Lua threads and Sequences, control will always continuously return outside the GC arena and to the outer Rust code driving everything. This is the "trampoline" here, when using this interpreter, somewhere there is a loop that is continuously calling Arena::mutate and Executor::step, and it can stop or pause or change tasks at any time, not requiring unwinding the Rust stack.

This "stackless" style has many benefits, it allows for concurrency patterns that are difficult in some other VMs (like tasklets), and makes the VM much more resilient against untrusted script DoS.

Async Sequences

The downside of the "stackless" style is that writing things as a Sequence implementation is much more difficult than writing in normal, straight control flow. This is identical to the problem Rust had before proper async support, where it required implementing Future manually or using difficult to use combinators. Ideally, if we could somehow implement Collect for the generated state machine for a rust async block, then we could use rust async (or more directly, unstable Rust coroutines) to implement our Sequence state machines.

Unfortunately, implementing a trait like this for a Rust async (coroutine) state machine is not currently possible. HOWEVER, piccolo is currently still able to provide a safe way to implement Sequence using async blocks by using a clever trick: a shadow stack.

The async_sequence function can create a Sequence impl from an async block, and the generated Future tells the outer sequence what actions to take on its behalf. Since the Rust future cannot (safely) hold GC pointers (since it cannot possibly implement Collect in today's Rust), we instead allow it to hold proxy "stashed" values, and these "stashed" values point to a "shadow stack" held inside the outer sequence which allows them to be traced and collected properly! We provide a Locals object inside async sequences and this is the future's "shadow stack"; it can be used to stash / fetch any GC value and any values stashed using this object are treated as owned by the outer Sequence. In this way, we end up with a Rust future that can store GC values safely, both in the sense of being sound and not leading to dangling Gc pointers, but also in a way that cannot possibly lead to things like uncollectable cycles. It is slightly more inconvenient than if Rust async blocks could implement Collect directly (it requires entering and exiting the GC context manually and stashing / unstashing GC values), but it is MUCH easier than manually implementing a custom Sequence state machine!

Using this, it is easy to write very complex Rust callbacks that can themselves call into Lua or resume threads or yield values back to Lua (or simply return control to the outermost Rust code), while also maintaining complex internal state. In addition, these running callbacks are themselves proper garbage collected values, and all of the GC values they hold will be collected if they are (for example) forgotten as part of a suspended Lua coroutine. Without async sequences, this would require writing complex state machines by hand, so this is critical for very complex uses of piccolo.

Executor "fuel" and VM memory tracking

The stackless VM style "periodically" returns control to the outer Rust code driving everything, and how often this happens can be controlled using the "fuel" system.

Lua and Lua driven callback code always happens within some call to Executor::step. This method takes a fuel parameter which controls how long the VM should run before pausing, with fuel measured (roughly) in units of VM instructions.

Different amounts of fuel provided to Executor::step bound the amount of Lua execution that can occur, bounding both the CPU time used and also the amount of memory allocation that can occur within a single Executor::step call (assuming certain rules are followed w.r.t. provided callbacks).

The VM also now accurately tracks all memory allocated within its inner gc-arena::Arena using gc-arena memory tracking features. This can extend to userdata and userdata APIs, and assuming the correct rules are follwed in exposed userdata and callbacks, allows for accurate memory reporting and memory limits.

Assuming that both of these mechanisms work correctly, and assuming that all callback / userdata APIs also follow the same rules, this allows for completely sandboxing untrusted scripts not only in memory safety and API access but also in CPU and RAM usage. These are big assumptions though, and piccolo is still very much WIP, so ensuring this is done correctly is an ongoing effort.

What currently works

• An actual cycle detecting, incremental GC similar to the incremental collector in PUC-Rio Lua 5.3 / 5.4
• Lua source code is compiled to a VM bytecode similar to PUC-Rio Lua's, and there are a complete set of VM instructions implemented
• Almost all of the core Lua language works. Some tricky Lua features that currently actually work:
  • Real closures with proper upvalue handling
  • Proper tail calls
  • Variable arguments and returns and generally proper vararg (...) handling
  • Coroutines, including yielding that is transparent to Rust callbacks
  • Gotos with label handling that matches Lua 5.3 / 5.4
  • Proper _ENV handling
  • Metatables and metamethods, including fully recursive metamethods that trigger other metamethods (Not every metamethod is implemented yet, particularly __gc finalizers).
• A robust Rust callback system with sequencing callbacks that don't block the interpreter and allow calling into and returning from Lua without using the Rust stack, and a way to integrate Rust async so that implementing these callbacks is not wildly painful.
• Garbage collected "userdata" with safe downcasting.
• Some of the stdlib (almost all of the core, fundamental parts of the stdlib are implemented, e.g. things like the coroutine library, pcall, error, most everything that exposes some fundamental runtime feature is implemented).
• A simple REPL (try it with cargo run --example interpreter)

What currently doesn't work

• A large amount of the stdlib is not implemented yet. Most "peripheral" parts of the stdlib are this way, the io, file, os, package, string, table, and utf8 libs are either missing or very sparsely implemented.
• There is no support yet for finalization. gc-arena supports finalization in such a way now that it should be possible to implement __gc metamethods with resurrection and tables with weak keys / values and ephemeron tables fully, but it has not been done yet. Currently, the __gc metamethod has no effect.
• The compiled VM code is in a couple of ways worse than what PUC-Rio Lua will generate. Notably, there is a JMP chaining optimization that is not yet implemented that makes most loops much slower than in PUC-Rio Lua.
• Error messages that don't make you want to cry
• Stack traces
• Debugger
• Aggressive optimization and real effort towards matching or beating (or even just being within a respectable distance of) PUC-Rio Lua's performance in all cases.
• Probably much more I've forgotten about

What will probably never be implemented

This is not an exhaustive list, but these are some things which I currently consider almost definite non-goals.

• An API compatible with the PUC-Rio Lua C API. It would be amazingly difficult to implement and would be very slow, and some of it would be basically impossible (longjmp error handling and adjacent behavior).
• Perfect compatibility with certain classes of behavior in PUC-Rio Lua:
  • PUC-Rio Lua behaves differently on systems depending on the OS, environment, compilation settings, system locale, etc. (In certain versions of PUC-Rio Lua, even the behavior of the lexer changes depending on the system locale!) piccolo is more or less aiming to emulate PUC-Rio Lua behavior with the "C" locale set with the default settings in luaconf.h on 64-bit Linux.
  • The specific format of error messages.
  • The specific iteration order of tables, and the specific behavior of the length operator (the length operator currently functions correctly and will always return a table "border", but for tables that are not sequences, the choice of border that is returned may differ).
• The debug library is unimplemented and much of it will probably never be implemented due to fundamental VM differences.
• Compatibility with PUC-Rio Lua bytecode
• os.setlocale and other weirdness inherited from C
• package.loadlib and all functionality which allows loading C libraries.
• Perfectly matching all of the (sometimes quite exotic) garbage collector corner case behavior in PUC-Rio Lua.

Why is it called 'piccolo'?

It's a cute little "pico" Lua, get it?

It's not really all that "pico" anymore, but it's still a cute little instrument you can safely carry with you anywhere!

Wasn't this project called something else? Luster? Deimos?

There was an embarassing naming kerfluffle where I somehow ended up with other people's project names twice. They're all the same project. I promise I'm done renaming it.

License

piccolo is licensed under either of:

• MIT license LICENSE-MIT or http://opensource.org/licenses/MIT
• Creative Commons CC0 1.0 Universal Public Domain Dedication LICENSE-CC0 or https://creativecommons.org/publicdomain/zero/1.0/

at your option.

piccolo's Issues

Suggestion: Nested FromLuaMulti

Hi, I recently created a PR on rlua that adds ability to nest MultiValues which makes it very convenient to allow new kinds of parameter unpacking in registered functions. It hasn't been accepted yet, but I propose this addition to piccolo as well.

The PR contains very verbose explanation of these changes as well as use case examples.

I find it weird that from_lua_multi can take any number of arguments without telling the caller how many were consumed (which can be ignored if not needed). I do see that it's only really intended for function arguments and Variadic is just a nicety that comes from it, but given that something like this greatly improved DX on my project I think it could be a useful to others as well.

RawTable fmt::Debug implementation can overflow stack

When printing a table with {:?}, if it has __index set to itself, it overflows the stack by recursively printing itself

Not entirely sure what the desired behavior should be, especially since the cycles probably won't always be that simple? But right now, debug printing the wrong thing will cause it to crash 😬

Error: ThreadError(BadCall(TypeError { expected: "function", found: "nil" })

[zaoqi@zaoqi-pc lua]$ lua -i lang.lua 
Lua 5.3.5  Copyright (C) 1994-2018 Lua.org, PUC-Rio
> 
[zaoqi@zaoqi-pc lua]$ ............../luster/target/debug/luster -r lang.lua
Error: ThreadError(BadCall(TypeError { expected: "function", found: "nil" }))

https://gitlab.com/the-language/the-language/blob/d13341505075cb8cc4fa9b23212fd9c9932bc556/core/pure/lua/lang.lua

question

Hello, I would like to know what you had envisioned for the rust API <-> lua binding. What's the progress on that front and how one could contribute? Thanks.

`'x' as u8` constants in the lexer can use b'x' literals instead.

I assume the reason these constants exists is pattern-matching: https://github.com/kyren/luster/blob/413b5c675d5747b26c19bd34eb4ed56e9fa956ff/src/lexer.rs#L852-L906

But instead of e.g match c { SPACE => ... }, one can write match c { b' ' => ... }.

TailCail OpCode bug

So I tried testing https://github.com/kikito/middleclass/blob/master/middleclass.lua (which actually works when you patch for this and allow type to accept nil) and was bumping into a mysterious error.

I did some print-debugging and managed to get this minimal example to reproduce the problem.

local b = {}
function b.inner(a, bk)
  print(a, bk, b == bk)
end
setmetatable(b, {
  __call = function(_, ...)
    b.inner(...)
    return b.inner(...)
  end
})

local a = b("a")

PUC-Lua Output

a       nil     false
a       nil     false

Piccolo Output

a       nil     false
a       <table [ADDR]>  true

Piccolo Compiler Dump

===FunctionProto(0x16b156288)===
fixed_params: 0, has_varargs: true, stack_size: 5
---constants---
0: Integer(0)
1: String("inner")
2: String("setmetatable")
3: String("__call")
4: String("a")
---opcodes---
<line 1>
0: OpCode(NewTable { dest: RegisterIndex(0), array_size: 0, map_size: 0 })
1: OpCode(LoadConstant { dest: RegisterIndex(1), constant: ConstantIndex16(0) })
<line 2>
2: OpCode(Closure { dest: RegisterIndex(1), proto: PrototypeIndex(0) })
3: OpCode(SetTableCR { table: RegisterIndex(0), key: ConstantIndex8(1), value: Reg
<line 5>
4: OpCode(GetUpTableC { dest: RegisterIndex(1), table: UpValueIndex(0), key: Const
5: OpCode(Move { dest: RegisterIndex(2), source: RegisterIndex(0) })
6: OpCode(NewTable { dest: RegisterIndex(3), array_size: 0, map_size: 1 })
7: OpCode(Closure { dest: RegisterIndex(4), proto: PrototypeIndex(1) })
8: OpCode(SetTableCR { table: RegisterIndex(3), key: ConstantIndex8(3), value: Reg
9: OpCode(LoadConstant { dest: RegisterIndex(4), constant: ConstantIndex16(0) })
10: OpCode(Call { func: RegisterIndex(1), args: VarCount(Opt254(Some(2))), returns
<line 12>
11: OpCode(Move { dest: RegisterIndex(1), source: RegisterIndex(0) })
12: OpCode(LoadConstant { dest: RegisterIndex(2), constant: ConstantIndex16(4) })
13: OpCode(Call { func: RegisterIndex(1), args: VarCount(Opt254(Some(1))), returns
14: OpCode(Jump { offset: 0, close_upvalues: Opt254(Some(0)) })
<line 13>
15: OpCode(Return { start: RegisterIndex(0), count: VarCount(Opt254(Some(0))) })
---upvalues---
0: Environment
---prototypes---
  ===FunctionProto(0x6000025ac3c0)===
  fixed_params: 2, has_varargs: false, stack_size: 6
  ---constants---
  0: String("print")
  ---opcodes---
  <line 3>
  0: OpCode(GetUpTableC { dest: RegisterIndex(2), table: UpValueIndex(1), key: Con
  1: OpCode(Move { dest: RegisterIndex(3), source: RegisterIndex(0) })
  2: OpCode(Move { dest: RegisterIndex(4), source: RegisterIndex(1) })
  3: OpCode(GetUpValue { dest: RegisterIndex(5), source: UpValueIndex(0) })
  4: OpCode(EqRR { skip_if: false, left: RegisterIndex(5), right: RegisterIndex(1)
  5: OpCode(Jump { offset: 1, close_upvalues: Opt254(None) })
  6: OpCode(LoadBool { dest: RegisterIndex(5), value: false, skip_next: true })
  7: OpCode(LoadBool { dest: RegisterIndex(5), value: true, skip_next: false })
  8: OpCode(Call { func: RegisterIndex(2), args: VarCount(Opt254(Some(3))), return
  <line 4>
  9: OpCode(Return { start: RegisterIndex(0), count: VarCount(Opt254(Some(0))) })
  ---upvalues---
  0: ParentLocal(RegisterIndex(0))
  1: Outer(UpValueIndex(0))
  ===FunctionProto(0x6000025ac460)===
  fixed_params: 1, has_varargs: true, stack_size: 2
  ---constants---
  0: String("inner")
  ---opcodes---
  <line 7>
  0: OpCode(GetUpTableC { dest: RegisterIndex(1), table: UpValueIndex(0), key: ConstantIndex8(0) })
  1: OpCode(VarArgs { dest: RegisterIndex(2), count: VarCount(Opt254(None)) })
  2: OpCode(Call { func: RegisterIndex(1), args: VarCount(Opt254(None)), returns: VarCount(Opt254(Some(0))) })
  <line 8>
  3: OpCode(GetUpTableC { dest: RegisterIndex(1), table: UpValueIndex(0), key: ConstantIndex8(0) })
  4: OpCode(VarArgs { dest: RegisterIndex(2), count: VarCount(Opt254(None)) })
  5: OpCode(TailCall { func: RegisterIndex(1), args: VarCount(Opt254(None)) })
  <line 9>
  6: OpCode(Return { start: RegisterIndex(0), count: VarCount(Opt254(Some(0))) })
  ---upvalues---
  0: ParentLocal(RegisterIndex(0))

The only difference I see between the correct call and the incorrect one is the use of TailCall for Call so something about that implementation is causing this.

Implement `__newindex` Metamethod

Hey there! We're considering using piccolo in Jumpy and as I was just browsing and playing around I realized that __newindex isn't implemented yet. We'd probably need that as a way to create proxy types that let us hook into variable assignments.

I might be able to implement it myself, and there's still a chance we don't end up using piccolo or we don't get to testing it out just yet, but I figured I'd open an issue to put it on the radar and see if there was anything I should know if I tried to implement it.

Split out gc_arena into its own repo

gc_arena and friends seem very useful! It could be a separate crate.

Question: Serialization of entire Lua state

Not proposing this as a feature but just wondering if it would be possible in theory to save and reload the entire Lua state? Given the "stackless vm" design and the "reified stack", it seems like it may be more possible than in other designs so I thought I would ask. I expect that if nothing else, the Rust callbacks that are stored in the state would be unserializable, but curious what other deal breakers there might be?

Passing Lua bytecode to the compiler and getting an AST-Like return value

How would I do this? I've skimmed through the compiler code but as its 23:30 of the time of writing this i can't fully look into the entire source to see how Lua gets compiled to bytecode gets compiled to instructions or if it's Lua straight to instructions and skips bytecode entirely.

I've looked through the compiler example and it seems to be exactly what I'm looking for but i don't know if the prototype is a tree-like value that i can walk through with a simple for loop and a match statement.

Problem with multiple return values and assignments

I've noticed that assignments with functions with multiple return values doesn't work. For example:

function mulret()
    return 1,2,3
end

a,b,c = mulret()
print(a,b,c)

In lua gives:

1       2       3

In Luster:

1       nil     nil

I think the problem is in the handling of assigment statements in the compiler, which seems not to have allowed for multiple returns yet (the function is called with 1 expected return value).

Unexpected panic: `thread finalization was missed`

Unfortunately, this one is way more vague than my previous report. I've yet to figure out exactly how I manged to cause it or to reproduce it reliably. I actually stumbled across #44 while trying to repro this one. The best I can do is provide a description of what I'm attempting in the hopes that some of the details might be relevant. I don't have a stack trace on hand, but the panic is coming from this expect.

I'm working on building a bevy integration. Scripts are loaded module-style and are expected to return a table of functions, which is stashed and stored in a bevy resource which associates it with the AssetId it was loaded from. Since piccolo is !Send, it relies heavily on thread-locals. The Lua environment, a StashedExecutor, and the StashedTables loaded from the compiled scripts are all kept in thread-local data. If the lua system detects that either the StashedTable for a particular asset is missing, or the timestamp for the table is older than that of the most recent asset reload, the module is re-evaluated. On every Update, for each loaded script, it looks up the update method in its table and executes it.

The StashedExecutor is created once alongside the Lua environment and kept around for all operations. Loading a script from the compiled asset (a CompiledPrototype) is done by creating a FunctionPrototype, then a Closure, and finally a Function with which the Executor is restarted, then stepped to completion. The Table result is then taken and stashed. When running the "update" function, the function is looked up from the table, used to restart the executor, and then Lua::execute is used to drive execution.

It seems like the panic was happening shortly after I had edited the script and seen it reload, but not immeditely. My suspicion is that the Executor got left in an odd state after one or more load or execution failures on one thread, but not all threads, so the panic is somewhat nondeterministic as bevy schedules the system to different threads. But it should be reproducible if I can figure out what the conditions actually are, since I'm not using unsafe anywhere and things should be deterministic within each individual thread.

Unexpected panic: `assertion failed: self.stack.is_empty()`

From what I can tell, this occurs when directly returning the result of an attempted call to a nonexistent function. Using an intermediate variable produces a Result::Err as expected.

Full example to reproduce:

use std::string::String as StdString;

use piccolo::*;

const SOURCE: &str = r#"
local function format(who)
    return "Hello, "..who.."!"
end

-- Intentional typo here, causes a failed assertion
return fomat("world")

-- Splitting it across a variable binding causes a runtime error
-- local out = fomat("world")
-- return out
"#;

fn main() {
    let mut lua = Lua::full();

    let exec = lua.enter(|ctx| ctx.stash(Executor::new(ctx)));

    lua.try_enter(|ctx| {
        let closure = Closure::load(ctx, None, SOURCE.as_bytes())?;

        ctx.fetch(&exec).restart(ctx, closure.into(), ());

        Ok(())
    })
    .expect("load closure");

    let res = lua.execute::<StdString>(&exec);

    match res {
        Ok(s) => println!("{s}"),
        Err(e) => println!("execution returned error: {e}"),
    }
}

Full backtrace:
backtrace.txt

Evaluate Profile-Guided Optimization (PGO)

Hi! Thanks for the project!

Recently I did a lot of PGO benchmarks on different kinds of software - the results are here. I think the same optimization option could be useful to Piccolo too.

I understand that the project is in the early stages of development so it's probably not the right time for asking about such things. But maybe one day this day will come and we will be able to perform PGO testing (and documenting its effects on Piccolo). For the users will be helpful to see information about PGO effects (and other performance tuning techniques if any) to improve Piccolo performance.

If you are going to play with PGO, I recommend starting with cargo-pgo.

Async callbacks

Hello,

I have just been pointed to this repository, as I'm currently investigating scripting languages accessible from rust that would support builtins that return -> impl Future<Output = TheActualResult> (and thus have all lua functions when accessed from rust that would implicitly return -> impl Future<Output = TheActualResult>) in a way that would be transparent to the scripted language, for the configuration file of a server I'm writing.

Until now, the only language I had found that matched this property was gluon, but going with a brand new scripting language makes me uneasy, as users would have to learn it in addition to understanding how my server works.

So I wonder, is async builtins / callbacks being transparently “async'd” through the lua call stack until the initial rust caller something you are considering for luster?

Cheers, and good luck with this rewrite!
Leo

Tables need to support "dead keys"

To support concurrent mutation of the map portion of tables during iteration, PUC-Rio Lua uses something called "dead keys".

Removed entries from tables keep their key value, and if the removed key is a GC value it is marked as a "dead key". "Dead keys" are GC keys which are not removed from the table but also not traced, and can be collected as normal. The implementation of next checks for existing keys by including these "dead keys" and checking for key equality based on pointer identity. In this way, it is always safe to remove any entry from a Lua table during normal iteration.

https://www.lua.org/source/5.4/ltable.c.html#equalkey

[BUG] Examples do not compile

Problem

Self descriptive

Versions

Rust: 1.72
Piccolo: Both 0.1.0 and 0.1.1

Reasons

Iterpreter example

Changes in API. No reset method for Thread and more

Compiled example

The trait bounds are not satisfied

Luster Naming

I just published a crate named luster, not realizing that there was a Rust program of that name.
I dont mind trying to change my crate if you might still want the name 'luster'.

Just let me know!

This project is no longer on hold!

Been going through my open source projects which I've been neglecting and figured I should write something here about where I am with this project.

Unfortunately I'm no longer working on a larger project where I think this might be useful
(for my current needs wasm is a better fit), so I'm not going to update this
project at least in the near term. I still think the core part of this project (safe interaction with a garbage collector via a futures-ish API) was a neat idea, and I'd still like to explore
this in the future when the hopefully the compiler is a bit more ready.

Before I pick this up again though I think that there needs to be additional
support in the Rust compiler for doing this, because while it is possible to use a GC safely via combinators, it is not at all pleasant. What I'd like is for it
to be possible to have a safe GC'd API using async / await or generators, but
AFAICT right now it is not possible. I tried for a while recently to see if I could come up with even a very limited version of the "sequence" API that worked with async / await functions and I couldn't find anything that worked.

You can't auto-generate Collect for closures or generators right now which is certainly a
limitiation, but it isn't actually the biggest problem currently. Right now
there's no way I can find to pass a Context<'gc> with a unique, branded 'gc
lifetime through an async function while having that async function not also be
of the 'gc lifetime. We need the async function to strictly outlive 'gc so
that it can't hold Gc pointers across await points, or we need some other
solution. (Incidentally I can't make this work at all right now, but even if
I could I know that you can't make the lifetimes work out so that the async
function lives for longer than 'gc).

Once I find any way of proceeding that enables generators-like functions instead of combinators, I think I'll pick this back up, but until then I'm not going to update this.

I'll keep thinking about the problem though! If anybody else has any ideas about how to make this work, let me know!

Potential wrong operator.

https://github.com/kyren/luster/blob/3e3a6ea12f6f523c105abd7fbe9d0ad226be784c/gc-arena/src/util.rs#L47

This operation looks like it's supposed to be a bitmask, but instead of doing an AND, it does and OR. As such, the inequality will always be true.

next() function

I've started to have a play with luster to see how it might fit into another project.
I was wondering about adding some of the table functions, but notice that the next function isn't implemented yet.

Before I start poking around in Table, do you have thoughts on an approach? Replacing the FxHashMap with something like IndexMap might work (that IndexMap doesn't quite do everything needed as far as I can tell).

Thanks,

Chris

Memory Corruption when running tests

I just checked out this project and it seems like there's a memory corruption happening when running the tests. I'm not sure you are aware of that, so I'm just letting you know in case.

Critical error detected c0000374
Exception thrown at 0x00007FFC01204D1B (ntdll.dll) in basic-66257f80e6d12447.exe: 0xC0000374: Ein Heap wurde beschädigt (parameters: 0x00007FFC012697B0).

Is TSETV a LuaJIT opcode or Lua opcode?

Seeing how Lua 5.1 doesn't talk about it but LuaJIT does, I'm guessing it's LuaJIT?

Future of this project?

Hi! I'm wondering what the future of this project will be now that Chucklefish has decided to not use Rust for their next title.

Could use a git submodule for the Lua testsuite.

I noticed in c78bdf8 all the files from the official tests were added.
But they're also available at https://github.com/lua/tests, so maybe a git submodule would be better?
(feel free to close this issue if it's not worth the trouble)

Small detail about JS Functions in your post.

In your post you do something that looks like (() => { /* code here */ }).bind(this)

This is good code but this is already bound.

JavaScript arrow functions are different to function functions in that they bind this in their closure.

It would help my JS brain if this was fixed. Thanks!

(Question) There's a CLI tool?

Like the official Lua interpreter.

Unexpected values when calling function

So, on latest (9d4906b as of writing)

local function abc(a, b, c)
	print(b, c)
	return a.x
end

print(abc {x = 3})

Piccolo Output

0   nil
3

PUC-Rio Lua Outut

nil     nil
3

Mapping a Lua function over a Rust Iterator

I suspect I'm running up against the downside of the "stackless" style here.

In Rust, I have a reference to some World that I'm wrapping in a piccolo_util::Frozen and sticking in a global Lua variable to be accessed from Lua. This works great for World methods that return static things. When doing more complex queries on the world though, I end up with an iterator which contains a reference to the World, which thanks to Frozen, is only valid within the call to Frozen::with.

My first approach was to pass a Function from Lua to Rust which takes the iterator's Item, and create an Executor to run it inside the callback. Then I found this blurb in the Executor docs:

Executors are not meant to be used from callbacks at all, and Executors should not be nested. Instead, use the normal mechanisms for callbacks to call Lua code so that it is run on the same executor calling the callback.

I'm guessing these "normal mechanisms" are CallbackReturns like Call and Sequence? Since neither the World reference nor the iterator can escape the Frozen::with callback, it doesn't seem like I'd be able to wrap and return the iterator directly, and it can't be recreated and resumed from an arbitrary point. And even if that were possible, it would be handing out references which themselves couldn't be Function::bind'ed and escape.

It seems like the best approach available to me is to copy/collect the iteration results into a Table and return that back to Lua, but I'd like to avoid that overhead if possible.

Is there anything I've missed? Any other approaches to consider?

Multiline strings aren't removing the initial newline

Found a thing:

print "---"
print [[
Random
]]

This prints

---

Random
 

in Piccolo, but

---
Random
 

in Lua.

From the Lua Reference Manual "Literal strings can also be defined using a long format enclosed by long brackets. [...] Literals in this bracketed form can run for several lines, do not interpret any escape sequences, and ignore long brackets of any other level. Any kind of end-of-line sequence (carriage return, newline, carriage return followed by newline, or newline followed by carriage return) is converted to a simple newline. When the opening long bracket is immediately followed by a newline, the newline is not included in the string."

question: are async calls into the vm Send?

I'm using mlua quite heavily in a couple of applications.
One pain point that I have in one of them is that that futures that result from async calls into the VM are !Send, making it rather awkward to integrate in the regular multi threaded tokio runtime.

Is piccolo different?

Also: one thing I love about mlua is how easy it is to define custom embedded modules and types and map them into lua.
How's that look in piccolo? I'm happy to read code examples, but I didn't spot anything obviously like that on a very very quick glance. Can you point me to stuff to look at?

Thanks! This project sounds very interesting to me!