lune-org / lune Goto Github PK

PlushNameGui.rbxm
When inserted in Studio, the font is Fredoka One. When read and written to a place file, the font is Arial (Legacy).

Some APIs such as process.spawn tell the user to look at docs for global types:

The third argument, `options`, can be passed as a dictionary of options to give to the child process.
Refer to the documentation for `ProcessSpawnOptions` for specific option keys and their values.

But global types are not currently included in the output for generated docs.

We should probably also show the luau type signature of functions and members somewhere.

We could generate the entire wiki from the type definitions file in the same way we already generate the docs file. Type definitions already contain all globals and their members in a tree structure, similar to how a GitHub wiki repo is structured, with its markdown files and dirs.

Info as a name is ambigious.

This code would print Got Cookie but wouldn't print Complete

-- Main Module
local request = require("CookieRequest")
print("Complete")

-- CookieRequest
local roblox = require("@lune/roblox")
local cookie = roblox.getAuthCookie()
print("Got cookie!")
return function()
end

Created this issue for the sake of record keeping and further discussion, but this was already very briefly discussed in the Roblox OSS Discord.

There are very few RegEx implementations written in Lua, and the ones that do exist are slow and don't have full feature coverage. Rust already has a performant and feature-complete RegEx implementation, and it just needs to be exposed in a Lua API.

Questions:

• Should the RegEx API be exposed as a global (JavaScript style, with RegEx.new("...")) or via a require (local regex = require("@lune/regex"))?

cc @matthargett

We could simplify the setup and running process for users with the help of an extension in their editor.

Setup

Setup and downloading could be prompted using similarly to how the Rojo VSCode extension does things, it also uses aftman which is already our preferred install method and listed under current installation instructions. This should only happen if lune-specific directories are present.

Prompt ideas:

• Generate type definitions if they do not exist but the lune runtime or directories are detected as present, maybe even add them to .gitignore automatically
• Re-generate type definitions if the lune version does not match the tagged version in type definitions

Command ideas:

• Running a script from a list, should be the same scripts as the --list command shows
• Inline "run script" button at the top of a file when a lua file that is definitely a lune script is the opened editor

Any script running in Lune can steal cookies with roblox.getAuthCookie(), read and write arbitrary files with fs, and send network traffic with net.

It's very important to block access to these when running scripts that aren't your own.

An additional argument could be added to the Lune executable that specifies permissions, for example:
lune rwnc arg arg arg
where each letter in the arg rwnc represents a feature that is allowed.

---- - No perms.
rw-- - File read / write perms.
r-n- - File read and net perms.
r-nc - File read, net, and cookie perms.
Etc.

The library Process should be merged with the os library.

Grouping the process library's content inside the os library would be a good idea for a variety of reasons.

Grouping them as seperate libraries does not make much sense as the purpose of each library is the same, yet they are in different global std libs.

It would make it Lune more familliar to vanilla Lua, lessening confusion to people coming from regular Lua. Also some other languages use the os name as well.

Also os is a much more ergonomic name than Process. os is much easier to type and remember than Process.

For example, on this page I assume these methods take in arguments, but they are not named nor typed resulting in a lot of guessing and checking.
https://lune.gitbook.io/lune/api-reference/fs

The os.date / os.time functions included with Luau are very minimal, and properly dealing with ISO standards and timezones is notoriously difficult. The built-in functions also contain unexpected behavior that does not conform to ISO standards, one example of this being that the first day of the week is Sunday instead of Monday.

We could use the chrono crate and reference some prior work and make something like a DateTime library for Lune:

• https://github.com/chronotope/chrono
• https://tc39.es/proposal-temporal/docs/index.html
• https://create.roblox.com/docs/reference/engine/datatypes/DateTime

This would also let us return much more ergonomic values for date/time in future additions such as #51 .

Similar to how Deno has a compile subcommand for generating a standalone executable, we could pack a Lune script into a standalone executable. Some notes on implementation:

• We would need to implement a bundler that also ensures that all our requires are static.
• We would only generate a direct executable and not some kind of package like .app, .dmg, .msi, AppImage, ...
• We should probably not try to compile Lune from scratch or strip out parts of the Lune runtime itself. The Lune binary is very small to begin with and correctly bundling only the Luau VM and Lune builtins necessary is a risky endeavour that's not worth the effort. Appending / combining the main Lune executable with the desired script(s) is good enough.
• Cross-compilation is more involved and something we probably wouldn't do for a first implementation. Downloading different versions of Lune in a GitHub workflow and generating executables using their own specific runners using a matrix is fine, similar to our current release workflow: https://github.com/filiptibell/lune/blob/86a0059af53a0f2a6ce38e66758aa9c26c797947/.github/workflows/release.yaml#L83-L101

Process.cwd prepends "\?" to the path on Windows, which would force you to use workarounds such as stringLiteral:sub(5) to be actually able to work with the given path.

I don't wanna taint the math.random, and I want to have a psuedorandom thing for my code.

Add a REPL so that we can run luau code with the lune runtime without having to create a script, would be useful for modifying Roblox place files interactively from the command line. Suggested this to Remodel before, but since Lune is a successor to Remodel, I'd love for Lune to have it rojo-rbx/remodel#90

When using --generate-selene-types, the resulting lune.yml file has a couple of issues.

• task.spawn, task.defer, and task.delay all require the vararg to be used when it should be optional.
• task.cancel takes an argument of type thread which isn't actually defined anywhere. This actually causes an error within Selene that gives us a confusing error message and prevents it from running. (see below)

It's shorter and vanilla Lua uses the same name.

The ability to create TCP clients and servers.
Currently, I would like to be able to connect to an IRC server through Lune, but cannot due to the inability to open a TCP connection.

A TCP server is probably not required as much since it goes against what this project is trying to achieve, but a TCP client is often required to connect to systems like databases.

Needed for parsing strings as Lua.

Already been discussed in Discord, but creating an issue for formality.

Lune could be an excellent option for unit-testing Roblox scripts outside a Roblox environment. If scripts are decoupled from engine APIs, there's no reason they shouldn't be able to run in an environment like Lune. Lune could support Roblox-like requires as an opt-in feature, using a Rojo source map to resolve file paths.

In the script environment, this would require a few things:

• A new Instance userdata which allows for indexing children and getting the parent. Specific instance types shouldn't be necessary; there just needs to be a way to traverse the datamodel.
• A script global, which is simply the current script's Instance.
• A game global, an Instance which represents the root of the datamodel.
  • How much functionality should be provided here? To maintain compatibility with most Roblox scripts, GetService should be supported so that datamodel services can be indexed in an idiomatic way.
• An alternative require implementation which supports referencing Instances. Much of the existing async require logic can be reused.

In addition, under the existing roblox global, Lune could support scripts implementing functionality for Roblox services. Lune can't reimplement everything (and it shouldn't aim to), but allowing users to reimplement what they need would be very powerful. Something like this:

roblox.implementService("HttpService", function(service)
    service:AddMethod("JSONEncode", ...)
end)

One thing to consider is how this pairs with the existing DataModel implementation for reading place/model files. How much functionality there can be reused?

Lune has ergonomic access for instances and their properties in the roblox builtin, meaning you can create, get and set them the exact same way you would in the Roblox engine.
It could be beneficial to take this a step further and allow users to access metadata about these classes and properties, which is currently contained in the reflection database and only exists internally.

Roblox allows usage of _G but it seems like in Lune it is entirely walled off.

fs.copy(from: string, to: string)
fs.copy(from: string, to: string, overwrite: boolean)
fs.copy(from: string, to: string, options: table)

This should throw an error by default if the path to already exists, and this behavior should be able to be overridden by passing either true for overwrite or some table of options, in case the user is 100% sure about what they are doing and don't want the extra overhead.

We will need to take some extra care to handle symlinks properly here.

The generated files from --generate-luau-types, --generate-selene-types and --generate-docs-file don't include types nor documentation for roblox apis.

Right now we only have aftman and direct download from GitHub releases as viable installation methods.
It would be great to have support for more package managers, some examples could be:

• Homebrew
• Chocolatey
• Scoop
• MacPorts
• Pacman

The docs says that print respects colors, but it does not. stdio.write does, however.

local stdio = require("@lune/stdio")

stdio.write(stdio.color("red"))
print("This text will be red")
stdio.write(stdio.color("reset"))
print("This text will be normal")

local stdio = require("@lune/stdio")

stdio.write(stdio.color("red"))
stdio.write("This text will be red")
stdio.write(stdio.color("reset"))
stdio.write("This text will be normal")

Introduction

In Lune, we currently have no way to dynamically run source/bytecode whatsoever; loadstring() is omitted from the environment afaik.

In standalone Luau, due to multiple sandboxing reasons, loadstring() can only run direct source code. (as opposed to loading raw bytecode in vanilla Lua) The function also deoptimizes the environment of the loaded closure, as it's directly returned and can be modifed/manipulated in any way:

local Closure = loadstring("print('hello world')")

-- Modifying the global env etc
setfenv(Closure, {})

Closure() -- Will err as "print" now doesn't exist in the env!

Proposition

This default behavior isn't even intended for most using loadstring(), and disables safeenv internally in the VM.

I propose some sort of built-in library for Lune like "luau", which could provide some fairly basic functions and utils for compiling/loading source code or bytecode as-is, as we don't really need to sandbox for this seperate behavior.

(Potential) API

^{Also, these should probably all error for basic logical reasons, like incorrect type input etc. They shouldn't error by themselves though, as these functions would be how a script generally knows if compilation is successful.}

• Compile Luau Source Code to Bytecode

function luau.compile(
    source: string,
    options: {
        optimizationLevel: number = 1?,
        debugLevel: number = 1?,
    }?
): (
    bytecode: string | nil,
    compileError: string | nil
)

It would take in input source code, as well as an optional options table for certain basic Luau::Compile compilation options. If successful, it'd return the binary bytecode file/chunk as a string, and no compiler error ofcourse. (string, nil) Otherwise, it'd just return nil, string; string being the Luau compiler error.

Example Usage:

local Bytecode, Error = luau.compile("do end", {
    optimizationLevel = 2,
    debugLevel = 0,
})

if not Bytecode and Error then
    print("Compiler Error: " .. Error)
    stdio.exit(1)
end

...

• Load & Run Source/Bytecode

function luau.load(
    source: string,
    options: {
        debugString: string | nil = nil?,
        safeLoad: boolean = false?,
    }?
): (
    closure: (...any) -> (...any) | nil,
    compileError: string | nil
)

This would be almost identical to loadstring()'s API, but also accepting bytecode for source, just like vanilla Lua. (Luau doesn't have this behavior at all, it wouldn't be default expected behavior)

It would just return the closure as-is, like normal loadstring would. By default, this would mean safeenv would naturally be disabled in the VM internally for this closure env. However, if options.safeLoad is set to true, the closure return will just proxy back the return of a call, similarly to how require() would, and not allow the raw closure environment to be modified from outside of itself. This would solve any safeenv VM optimization issues.

• Set safeenv for VM on Current Environment

function luau.setSafeEnv(newValue: boolean)

(Suggested from @TheGreatSageEqualToHeaven) Sets L->safeenv directly on the current L state.

Conclusion

There should probably be much more added to interact with low level Luau APIs, but these are just basic suggestions for such a library. If you have any disagreements or issues with these ideas, please lmk!

The file this is in is placed in C:/Projects/Roblox/.../lune/libs, so it just appends @lune/fs hoping to find it.

Luckily you don't need to actually require all some of the apis to use them, but it is done in every example.
(the serde and roblox apis do need to be required to be used)

I have attached a repro file to this issue named file. If you rename the file to file.rbxl and open it with Studio, it opens without issue. Though lune's roblox library is unable to parse it.

files_from_cdn.zip

To repro, run the following in lune after unzipping the downloaded file:

local roblox = require("@lune/roblox")

roblox.readPlaceFile("file")

In this file, I get the error Failed to read document from buffer - stream did not contain valid UTF-8 but for other files I'll get Unknown document format

Terrain.SmoothGrid / Terrain.MaterialColors example:

local roblox = require("@lune/roblox")
local Instance = roblox.Instance
local Color3 = roblox.Color3

local game = Instance.new("DataModel")
local workspace = game:GetService("Workspace")

workspace.Terrain.SmoothGrid = roblox.readModelFile("TerrainRegion.rbxm")[1].SmoothGrid
workspace.Terrain.MaterialColors = "![CDATA[AAAAAAAAAIAAP39rf2Y/ilY+j35fi21PZmxvZbDqw8faiVpHOi4kHh4lZlw76JxKc3trhHtagcLgc4RKxr21zq2UlJSM]]"

roblox.writePlaceFile("test.rbxl", game)

Errors: SmoothGrid is not a valid member of TerrainRegion

I'm opening this specifically for emoji url encoding for discord, but url encoding has many usecases. Just a nice feature to have.

Currently there is no way to retrieve file metadata such as creation/modification date, file/folder sizes, and the like. At present Lune does not provide a method to get this information and a workaround is using powershell (or otherwise) and the "Get-Item" command, this does work to a point - retrieving the modification date & file size in bytes, though it's not nearly everything.

My use-case for this is to 'diff' two files (an asset and a place) to ensure that a file that's been modified will be updated and synced so long as it's modification date precedes the last modification date of a target place file, this would then allow me to move away from the not-so-agnostic powershell 'Get-Item' command, and also provide even more useful information.

It would also be helpful if the DateTime library came with this change so parsing of the dates is easier, attached is my solution to the DateTime and file metadata problem (thought do note these files are from when lune was moderately fresh and I was learning the ropes), the main file (for this issue) is canSyncAsset.luau though to understand it's use I've provided the full sync and build scripts for context.
sync scripts.zip

Our Luau type definitions are really small (less than 20kb), and also contain documentation and metadata for all Lune APIs, so we could generate both the selene yaml & docs json files from just the definitions file, instead of downloading from GitHub releases, which has been prone to errors in the past.

idk if its a me issue or whatever but when i use this the script stops directly

when i run this it just stops running directly after

local port = 12941

local handle = net.serve(port, {
	handleWebSocket = function(socket)
		print("Got new web socket connection!")
		while true do
			local message = socket.next()
			if message ~= nil then
				socket.send("Echo - " .. message)
			end
		end
		print("Web socket disconnected.")
	end,
})

print(`Listening on 127.0.0.1:{port} :3`)

fs.rename(from: string, to: string)
fs.rename(from: string, to: string, overwrite: boolean)
fs.rename(from: string, to: string, options: table)

This should throw an error by default if the path to already exists, and this behavior should be able to be overridden by passing either true for overwrite or some table of options, in case the user is 100% sure about what they are doing and don't want the extra overhead.

Remodel is a tool that is widely used in the Roblox community and was one of the motivations for making Lune. It would be great to have some kind of compatibility layer and/or some way to require a module that contains the same APIs that Remodel does. We could use the module support in mlua for this.

Wally is a package manager specifically developed for the Roblox ecosystem - and as a consequence, require paths must use the script global. Requires are however still relative to file, and we could automagically translate paths such as script.Parent.Parent.Foo to a Lune-compatible path such as "../../Foo" if the user enables a configuration for it.

Right now the setup experience for generating type definitions and documentation is not great and relies on users reading the entire wiki docs. We could improve it quite substantially by making it interactive:

1. Streamline the separate subcommands into lune --setup
2. Let the user set up tools using multiselect similar to stdio.prompt("multiselect")
3. If some file already exists but is an old version, display the version change, and default to selecting it for generating
4. Ask the user if they want to add generated files to their .gitignore, defaulting to yes

Title essentially says it all, so here are some simple repro cases.

local stdio = require("@lune/stdio")

stdio.write("First")
stdio.ewrite("Second")

SecondFirst

local stdio = require("@lune/stdio")

stdio.write("First")
print("Second")

Second
First

typeof(), a function exclusive to Luau, seems to behave similarly or identical to type(), which should not be the case.
One example is to feed an instance to typeof, like typeof(Instance.new("DataModel")); this should return Instance but userdata is printed instead.

Some code may be designed to run on both Roblox and on Lune. Right now the cleanest way to determine if the script is running on Lune is to do something like if pcall(require, "@lune/fs") then which is what I'm doing now, but this seems a bit messy.

module.luau

net.request("https://google.com")
return {}

script.luau

require("./module")

There are at least two ways to solve this:

1. Make require async and use async filesystem APIs, but this is really tricky with caching and making sure requires only happen once (users can run require in a task.spawn, ..)
2. Use a bundler to always bundle scripts that run, even in the CLI, to not have to use require at all, this would disallow dynamic requires, and potentially mess up stack traces

Note that this bug does not occur unless it is in the main script of the required module, this works fine:

module2.luau

return function()
    net.request("https://google.com")
end

script2.luau

require("./module2")()

Lune Version: 0.6.7 (Win64)
Step 1: Create a very deeply nested Instance. Around 2,870 nests should be enough, but I'm using 32,768 nests as an example.

local j = {Instance.new("Part")}
for i=1,32768 do
	local part = Instance.new("Part")
	part.Parent = j[i]
	table.insert(j, part)
end

Step 2: Write that instance to a file or model.

roblox.writeModelFile("./myNestedPlace.rbxm", {j[1]})

Actual result: Lune crashes with a fatal unrecoverable error: thread 'blocking-1' has overflowed its stack
Expected result: Lune either writes the file or complains that it's too deeply nested.

We currently leak web socket objects due to lifetime issues here:

https://github.com/filiptibell/lune/blob/36a3bd2113ca12ec9e7c9557277f9466a034b8f6/packages/lib/src/lua/net/websocket.rs#L49-L72

There's no great solution using the mlua crate directly since web socket streams do not implement Clone and cannot be properly stored in the lua registry, and we can't use a LuaUserData since the functions need to be async (without using the native mlua async feature).

Most current usage of web sockets involves clients or servers with a small amount of connections during the life of the program (eg. things like Discord bots) but this still needs to be fixed as soon as possible.

Code used:

local success, result = pcall(function()
    local spawnResult = process.spawn("ffmpeg", '--dry-run', 'cmd', "inherit")

end)

print(success, result)

When trying to spawn a process using any value except for nil for the 3rd shell argument, an error like this will be raised:

Failed to convert Luau type 'string' into Rust type 'table'!
[Stack Begin]
    Script '[C]'
    Script 'async', Line 2 - function process.spawn
    Script '../test', Line 9
    Script '[C]' - function pcall
    Script '../test', Line 8 - function convert
    Script '../test', Line 49
    Script '[C]' - function pcall
    Script '../test', Line 42
[Stack End]

This same issue also happens if the 2nd argument is set to an empty string '':

local spawnResult = process.spawn("ffmpeg", '', nil, "inherit")

Error message:

Failed to convert Luau type 'string' into Rust type 'Vec'!
Details:
        expected table
[Stack Begin]
    Script '[C]'
    Script 'async', Line 2 - function process.spawn
    Script '../test', Line 9
    Script '[C]' - function pcall
    Script '../test', Line 8 - function convert
    Script '../test', Line 49
    Script '[C]' - function pcall
    Script '../test', Line 42
[Stack End]

We currently have a drop-in migration module for Remodel, which we added in Lune 0.7.0 - remodel.luau
There may be other use cases or tools that would suit Lune well and that are no longer maintained, we could also make migration modules for those. If anyone that has migrated to Lune sees this and would like to contribute to that, I would much appreciate it!

It would be great to have migration as part of our documentation site, too.

Hello 👋

I noticed that errors are getting automatically wrapped up into some opaque type. I can stringify the error and see what was originally thrown, but I can't get back the thrown value. For example, it would be great if I could do something like this:

local function throwError()
    error({ message = "error" })
end

local success, err = pcall(throwError)

print("success", success) --> false

print("err", type(err)) --> userdata

print("err.message", err.message) --> throws: attempt to index userdata with 'message'

Please support this, I beg. This would be a life saver to me, and my tooling. Tell me if you need anything more information, and I'll be happier to provide it.

Something like os.listdir() in Python:

import os, sys

# Open a file
path = "/var/www/html/"
dirs = os.listdir(path)

# This would print all the files and directories
for file in dirs:
   print(file)

Which would print:

stamp
faq.htm
_vti_txt
robots.txt
itemlisting
resumelisting
writing_effective_resume.htm
advertisebusiness.htm
papers
resume

process.os = "linux" | "macos" | "windows"
process.arch = "x86_64" | "aarch64"

We shouldn't need to call out to some external process to determine what kind of machine our script(s) are running on. Luau and Lune currently only support the above operating systems and architectures so this should be pretty straightforward to add.