Openresty about raptorjit

[lukego]

I would also like to note that this is a two-sided question:

Is compatibility with Openresty a goal of this project?

The other side is: is the Openresty community interested in getting involved in RaptorJIT?

I would like to align RaptorJIT and OpenResty becuase I feel that the LuaJIT community is much too fragmented and that we need to spend some time on understanding each others' requirements, getting on the same page, and moving the whole ecosystem forward together. I would like to build a community here and that means compromising between what all the participants need.

Having said that, if OpenResty hackers prefer to stick with LuaJIT v2.1 indefinitely for whatever reason then that would be one less reason for RaptorJIT to support features like the standard Lua-C API. Just at the moment it is not clear to what extent RaptorJIT users will be other refugees from the LuaJIT community verses people developing new applications without prior Lua/LuaJIT experience.

[lukego]

@agentzh I'd also like to share the observation that you, me, @fsfod, and surely others have been doing a lot of parallel development when it comes to diagnostics and debugging. You have written Python code to decode the LuaJIT heap as a gdb extension, I have written Smalltalk code to decode fragments of the LuaJIT heap as an interactive object inspector, @0xfsfod has written something similar that runs inside the VM and dumps pre-decoded data structures (IIUC.)

I feel like it would be very positive for us to understand each others' work better. I'd love to merge gdb extensions into RaptorJIT for example but I don't know much about your work and how much of it is open source.

Maybe we could all meet up at the Lua room at FOSDEM next year or something like that? Just for the sake of understanding what we are each working on, cross-community reuse, and potentially some "eventual consistency."

[agentzh]

@splitice To clarify some things on the OpenResty side (as the official OpenResty responses):

1. We're indeed retiring most of the CFunction-based Lua API in the OpenResty core, mostly for code maintenance overhead on our side. Maintaining two parallel Lua API implementations, CFunction-based and FFI-based, is really no fun.
2. We have no intention to retire all our CFunction-based Lua APIs, however, because 1) there will be no performance benefit for certain Lua APIs since they cannot be JIT compiled anyway due to the limitations in the LuaJIT's JIT compiler itself, and 2) deep C-Lua interoperability like calling into Lua code from the C land (we cannot just call C from Lua, but also the other way around, and LuaJIT's FFI C callback support is way too limited).
3. OpenResty maintains its own branch for LuaJIT for quite some time now: https://github.com/openresty/luajit2 We keep sync'ing from the upstream LuaJIT repos for years. It has special optimizations and features for OpenResty so the use of other LuaJIT forks/branches will slow things down unless they also merge our patches. The development work in this LuaJIT branch is also supported by the OpenResty Inc. company. OpenResty Inc. may also provide commercial support for OpenResty's LuaJIT in the near future. The open source OpenResty team will also continue maintaining and supporting OpenResty's LuaJIT as long as OpenResty lives ;)

[agentzh]

@lukego Sorry for the late replay.

Yeah, I open sourced both gdb tools and systemtap tools on github:

https://github.com/openresty/openresty-gdb-utils

https://github.com/openresty/stapxx

https://github.com/openresty/openresty-systemtap-toolkit

I wonder if the smalltalk code and @0xflotus's debugging tools are available somewhere?

You're free to take whatever code from the repositories mentioned above as long as keeping the copyright notice. They are BSD licensed.

I'd surely love to meet you guys though I'm not sure if I can make it to FOSDEM. It looks like a European conference. I'm in SF Bay Area most of the time.

[lukego]

FYI It would be really good if there was a bit more information regarding 3rd party integrations in the README

Can you give an example of what you mean? I'm overdue to revise the README to capture the outcomes of the many and varied wild discussions we have had on Issues lately.

I would like to support OpenResty. I have only recently dived in to understand how that project works. It's quite interesting and demonstrates the value of compatibility with the Lua-C API and LuaRocks ecosystem.

My take on OpenResty is that it is a Lua program that is gradually transitioning to become a LuaJIT program. Today it uses the Lua C-API heavily, and this limits what the JIT is able to do (for better and for worse), and since the JIT is not being pushed hard this means that it's also fine to use standard Lua libraries from LuaRocks too. So it is in a comfortable "medium performance" space.

However, OpenResty is in a very competitive space, and more performance might really help to keep it competitive in the future. So hypothetically if OpenResty could achieve a 2x or better speedup by embracing JIT/FFI then that might be a major win that can really help the project to keep on growing quickly for years to come.

I see OpenResty core hackers already working hard on migrating from the C-API to the FFI over at https://github.com/openresty/lua-resty-core. Once this is complete I think it will naturally follow to want more of the library ecosystem to be based on FFI too.

So in summary I would like to support OpenResty with RaptorJIT, and this means that maintaining C-API compatibility is valuable and important, but it doesn't change my view that the C-API is a backwards-compatibility feature and should be considered deprecated in favour of the FFI for new code.

That answer your question?

[splitice]

@agentzh is the guy to talk to re; Openresty. I just thought you might have done some research into it and might have an opinion regarding if interop with existing projects is a goal, or if you want to do your own thing (as the name somewhat implies). Openresty is certainly one of the bigger LuaJIT users that I am aware of. It's the reason that I do any Lua work at all.

Regarding compatibility with C-API's there are two parts:

1. Ability to call C APIs from within the Lua VM. Realistically FFI does a great job eliminating the need for the Lua C API and I don't see this as a show stopper for the likes of Openresty.
2. Ability to embed RaptorJIT into C programs (for use as interpreter or scripting language), this would include the ability for C programs to manipulate the VM.

Given that the Lua has been a very popular language for scripting purposes in games I can't see why you would want to eliminate the second. All languages need users, and right now there are many people using Lua/LuaJIT for scripting.

I can explain a bit of the ecosystem for you:

• lua-resty-core implements most of the APIs provided by lua-nginx-module (& stream-lua-nginx-module) via FFI. It was discussed years ago but I don't know if @agentzh plans to take the leap and ever eliminate the C API variants and mandate lua-resty-core.
• lua-nginx-module & stream-lua-nginx-module are the integrations implemented as nginx modules. Lua is integrated very tightly into nginx here and keep tight to the non-blocking event driven model used by nginx. There is also a meta module slated to generate the common code between the two projects, as I understand it this is still not yet in full usage.
• OPM: package manager replacing luarocks specifically for Openresty.

In the 3 projects I maintain in that ecosystem there are only a 3 C API modules in use, and 3 of them I am aware of FFI replacements. If the C API was to disappear in exchange for 2x performance I would certainly be happy. I'd estimate a couple man days development work plus testing :)

My main reasons for this issue were:

1. To make sure you were aware of the Openresty project. Perhaps inspiring you to make contact and set something in motion.
2. To get a bit of a feeling for the future direction of LuaJIT (& forks). Recently I got involved in a new project using Openresty and the possible lack of future support for LuaJIT has been a point of concern for myself and potential investors.

[lukego]

Thanks for commenting @agentzh. I have been following your branch of LuaJIT and think you are doing a great job. I really appreciate knowing that there are other kindred spirits wanting to keep LuaJIT alive and moving forward and I hope that some day, some how, we can end up benefiting from each other's work. Meanwhile I am just happy that stuff is happening and people are fighting for the future too :).

I'll find a reason to use RaptorJIT with OpenResty before too long and when I do I'll share what I come up with. Maybe somebody will beat me to it of course, we'll see.

[agentzh]

@lukego BTW, my email address is yichun at openresty dot com. Let's keep in touch :)