lorettadevs / loretta Goto Github PK

Add validation for language semantics.
Including but not limited to:

• Duplicate goto label detection;
• Missing goto label detection;
• const variable assignment;
• close variable assignment.

Reported by: bmcq#0001
Multiline comments with different amount of equal signs on start and end are treated as single line comments

Code:

--[==[..................................]=]

Current behavior:
Parsed as a single line comment.

Expected behavior:
Error indicating unfinished comment.

Reported by: Slappy826#0001

Error:

FunctionDefinitionStatement.Children enumerator throws an exception when there is a vararg because a vararg is not an IdentifierExpression.

C# Code:

var tree = Parse(@"function a(...) end");
var func = tree.Body.Single();

var children = func.Children.ToArray();
// Throws exception

Feature:
Add Lua 5.3 bitwise operators: & (BAND), | (BOR), ~ (BXOR and BNOT), << (left shift), >> (right shift).

Blocked by: #14

Reported by @Fiusen.

When an IVariable has a read with a write inside it (writes for assignments and local var decls span the entire node instead of only the name), the last use location wasn't being obtained properly leading to the outer node being marked as the last use instead of the read inside the write.

Example:

   local x = 0
   x = x + 1
-- ^^^^^^^^^ write
--     ^     read

Here, the write node was being considered the last use while the read one should've been as it's the one furthest in the code.

Feature:
Add a setting to enable or disable handling of each lua extension Loretta supports along with presets for Lua, Roblox, LuaJIT and GLua.

Reported by: bmcq#0001

Error:

When parsing a NumericForLoopStatement, the for loop scope is spilled.

Lua Code:

local a = 1
for a = 2, 4, 2 do print(a) end
a = 2 -- Refers to the `a` in the `for` loop instead of `a` in `local a`

Reported by: bmcq#0001

Code:

do return; end;

Current behavior:
Exception in ParseExpression.

Expected behavior:
No exceptions.

Suggested by: bmcq#0001

Feature:
An AssumeNoMetatableOverrides option to enable more aggressive constant folding for cases like Roblox where debug.getmetatable and debug.setmetatable aren't available therefore the behavior of operators can't be modified for base types.

Reported by: bmcq#0001
The constant folder can handle the <number> <op> <number>, <nil> <op> <any>, <bool-like> <op> <nil> cases but not <bool> <op> <bool> nor <bool-like> <op> <bool-like> cases.

Current behavior:

-- Transforms:
print(true == true)
-- Into:
print(true == true)

Expected behavior:

-- Transforms:
print(true == true)
-- Into:
print(true)

LuaJIT (with FFI support enabled) has the ULL and LL suffixes for numbers.

ULL results in an unsigned 64-bit integer.
LL results in an signed 64-bit integer.

Requested by @bmcq-0.

Reported by: bmcq#0001
Folding of not expressions is incorrectly handling the case where the operand is not nil nor boolean.

Example:

print(not a)

Results in:

print(true)

Example script:

local str = [[
a]]

str is actually only "a", as the first line break is skipped if its the first character on the string for whatever reason

NormalizeWhitespace does not insert/preserve spaces between function call arguments. Having spaces between arguments is considered "normal" and should be added.

print(1, 2)

After NormalizeWhitespace() is called on the tree, it will be formatted as the following:

print(1,2)

Describe the bug
LuaJIT has 3 number suffixes when FFI is enabled:

• LL for 64-bit signed integers
• ULL for 64-bit unsigned integers
• and i for imaginary numbers

To Reproduce
Steps to reproduce the behavior:

1. Type in 1I or 1ull or 1ll
2. Error is generated

Expected behavior
No error is generated.

Additional context
Loretta Version: master

This was supposed to have been implemented in #26 and #62 but I forgot.

Error:

When a local variable is redeclared, a new Variable isn't created. In lua, variable shadowing is allowed so a new variable should be assigned to variables in the local declaration statement.

Certain statements will not have their scopes in their parent scope's ContainedScopes list.

See the following Lua code:

(function(Variable) end)()

It's expected that the anonymous function's scope should be in the ContainedScopes list of the file's scope, but it is not.

var source = "(function(Variable) end)()";
var syntaxTree = LuaSyntaxTree.ParseText(source, LuaParseOptions.Default);
var script = new Script(ImmutableArray.Create(syntaxTree));
var fileScope = script.GetScope(tree.GetRoot());

Console.WriteLine(fileScope.ContainedScopes.Count()) // expected 1, got 0

The following aren't being parsed correctly:

• Empty type packs ();
• Empty type argument lists: a<>.

I noticed this supports 5.3, but what about 5.1 or 5.2 as well? Cant those be made supported as well for old games that use old lua 5.1 or 5.2?

Also I have a version of unluac that supports 5.1 and 5.2 which I can help provide too you to help you out (would need it to be available to nuget.org so then I could drop my version of an lua decompiler for this sort of project just because it seems far better.

Likewise the compiler I would need to be able to tell it to compile using 5.1 or 5.2 as well and not just 5.3 (for games like Elsword of Grand Chase). For the game versions they used the normal lua 5.1 compiler but for some reason it supported ellipsis which my decompiler either screws it up or the compiler does. Also the files contain Korean words and so must be made to support compiling and decompiling files encoded in UTF-8 WITH BOM. The normal compiler I got forces me to resave without BOM that might be what screws up the korean letters / the '...' characters inside of them when I use the tool to regenerate the KIM file after the raw lua bytecode is generated.

I plan to eventually make an KimLib in c# for my Els_kom program that could then be responsible for regenerating the KIM file / updating it / reading it but it takes time to do.

I probably could theoretically update the lua engine but do not want to chance breaking the entire already fragile setup (even though I personally think it only works due to UB and how dumb their devs were to not care how to properly code to not make it barely operable at all).

The website lags my entire computer and im unable to actually load anything, the colors are rlly shit consider using gitbook thanks 😛 😛 😛 😛

Describe the bug
ObjectDisplay outputs unicode escapes instead of original character, especially large issue for Lua 5.1 because it does not have unicode escapes

To Reproduce
Steps to reproduce the behavior:
Call .ToFullString on a node

Expected behavior
E.g issue

print("Hello World")

Will become

print("Hello\u{0020}World")

Luau has a concept of 'typed Lua' - i.e TypeScript style type annotations for Lua.

This is a custom syntax extension specific to Luau/Roblox.

See syntax here: https://roblox.github.io/luau/syntax

Some examples:

function foo(x: number, y: string): boolean
    local k: string = y:rep(x)
    return k == "a"
end

type Point = { x: number, y: number }
type Array<T> = { [number]: T }
type Something = typeof(string.gmatch("", "\d"))

Reported by: bmcq#0001
In Roblox (but not GLua), continue is parsed as a contextual keyword. Lorreta does not do that.

Code sample:

local continue = 100
for k, v in ipairs(tbl) do
      if v == continue then
        continue
    end
end

Current behavior:
Fatal exception: Expected an Identifier but got a Keyword instead on continue in local continue = 100.

Expected behavior:
continue is correctly parsed as a contextual keyword and no exceptions happen.

The current SyntaxNormalizer was partially based on VB's but the languages are too different for it to work properly and it requires a lot of extra configuration that could be handled in an easier manner.

Reported by: Slappy826#0001

Error:

local variable declaration doesn't create a new variable.

Lua Code:

local a = b[a] -- the `a` in `b[a]` refers to the `a` in `local a`

Not sure if Loretta supports Compound assignment support as I looked over the parser.
Reasons for support:

• Roblox added support for compound assignments.

Suggested by: bmcq#0001

Feature:
Make the constant folder handle math operations between values that can be converted to numbers instead of only numbers.

Example:

-- Transform:
print("0xFF.F" + 1)
-- Into:
print(256.9375)

Blocked by: #7

These are the set of commands contributors should run on your local repositories to migrate from old repo:

git remote set-url origin "https://github.com/LorettaDevs/Loretta.git"
git branch -m master main
git fetch origin
git branch -u origin/main main
git remote set-head origin -a

Describe the bug
Loretta reports incorrectly the bitwise operators are not supported when an intersection or union type is used.

To Reproduce
Parse the following code using LuaSyntaxOptions.Luau:

type A = A & B
type B = A | B

Expected behavior
No syntax diagnostics are emitted as the code is syntactically correct.

Additional context
Loretta Version: v0.2.10-nightly.73

Describe the bug
When creating a string literal node with the SyntaxFactory the inputted stirng

To Reproduce
Run

        Console.WriteLine(SyntaxFactory.LiteralExpression(SyntaxKind.StringLiteralExpression, SyntaxFactory.Literal(@"Hi From
        Loretta
        ")).ToFullString());

Expected behavior
The node should output

Hi From
        Loretta
        

but outputs

"Hi\u{0020}From\r\n\u{0020}\u{0020}\u{0020}\u{0020}\u{0020}\u{0020}\u{0020}\u{0020}Loretta\r\n\u{0020}\u{0020}\u{0020}\u{0020}\u{0020}\u{0020}\u{0020}\u{0020}"

LuaJIT (with FFI support enabled) has the i suffix for numbers.

i = Complex

Requested by @bmcq-0.

Support for unicode escape codes.

Format: \u{XX} where XX is less than (and not equal to) 0x110000.

Describe the bug
When typed-Luau was added all constructors relying on the syntax, which removed a lot of old constructors which was an oversight

To Reproduce
Attempting to use old constructors will throw errors
node.Update(node.FunctionKeyword, node.Name, node.Parameters, node.Body, node.EndKeyword, node.SemicolonToken)

Expected behavior
The code should work alongside its typed-constructors

Support for integers.

Integer examples: 3, 345, 0xff, 0xBEBADA, 0b1010, 0o77
Float examples: 3.0, 3.1416, 314.16e-2, 0.31416E1, 34e1, 0x0.1E, 0xA23p-4, 0X1.921FB54442D18P+1

In Lua 5.1, certain invalid escape sequences will not error the lexer and instead will be saved.

https://www.lua.org/source/5.1/llex.c.html#read_string

if (!isdigit(ls->current)) save_and_next(ls); /* handles \\, \", \', and \? */

The code here improperly handles things like \F, \O, \B, etc. that should not normally be accepted and will return itself (eg: \A -> A)

Luau has interpolated strings

local world = "World"
local foo = `Hello {world}`
print(foo)

Is your feature request related to a problem? Please describe.
Having to provide every single node for .Update is very annoying and could be better as what we have for LuaSyntaxOptions.With

Describe the solution you'd like
The same solution as applied to LuaSyntaxOptions.With.

Describe alternatives you've considered
Providing every single parameter every time or using a combination of multiple .With* methods which would result in a lot of unnecessary allocations.

Additional context
None.

Loretta's Script class initialization of a big script takes way too long:

Time on a 20MB file:

•   Script init: 9s
  

•   Parse time: 0.9s
  

How to replicate:

• Have a random script (like allopcodes-5.1.lua) and copy and paste it until ur file gets to ~20MB

var scriptParsed = new Script(ImmutableArray.Create(LuaSyntaxTree.ParseText(code)));
var root = Parsed[0].GetRoot();

SyntaxNode node = root.ChildNodes().FirstOrDefault(); // whatever

scriptParsed.GetScope(node) // long wait here

Does Loretta support both ways of conversion from:
Lua->AST
AST->Lua
Thanks for reading.

Suggested by: bmcq#0001

Feature:
Make constant folder handle ==, ~= and .. for strings.

Example:

-- Transform:
print(("hello".."there") == "hellothere")
-- Into:
print(true)

Blocked by: #7

Reported by: bmcq#0001
FormattedLuaCodeSerializer serializes not expressions incorrectly by not adding a space after the keyword when serializing.

Current behavior:

-- Transforms:
print(not a)
-- Into:
print(nota)

Expected behavior:

-- Transforms:
print(not a)
-- Into:
print(not a)

The Syntax.xml source generator is generating invalid factory methods for lists where MinCount="1".

It creates an overload where that list is default when it shouldn't.

Not adding a body to most nodes is a mistake and is not something we should support in the minimal factory methods.

We should also look into other things that we might want to make mandatory other than function bodies.

This will result in breaking changes, but that is the point of this.

print(0X1) --> goes nuts
print(0x1) -- works fine

Describe the bug
Lua 5.1 has a warning when you nest long comment or long string delimiters.
Further lua versions do not have this, nor do LuaJIT or Luau.

To Reproduce
Parse the following file with Loretta

local longstr = [[Show me this
yes!]]
--[[
local unused_string = [[Line1
Line2]]
local longstr = [[
Do not show me this
Multiline string]]
--]]

print(longstr)

Expected behavior
An error diagnostic is generated from [[Line1 to Line2]].

Additional context
Loretta Version: v0.2.9

Reported by: bmcq#0001

For with multiple values after in is not parsed properly.

Code:

for i, v in {},{},{},{},{},{},{},{},{},{} do end

Current behavior:
Exception.

Expected behavior:
No exceptions.

Reported by: bmcq#0001
Parser accepts any expression as a statement

Code:

a + b

Current behavior:

StatementList
\_ ExpressionStatement
    \_ BinaryOperationExpression

Expected behavior:
Error.