I'm changing a bunch of things, mainly in order to finalize LESv3. However, the changes to operator precedence/classification affect both LESv2 and LES3, partly because both languages share precedence-choosing code but also for the sake of consistency. These changes are not committed as of today.
!! to suffix operator
To reiterate, !!
changed from an infix to a suffix operator based on Kotlin's !!
operator. I'm duplicating this change now in LESv2. And I noticed that suffix operator printing seems broken in LESv2 (e.g. a++
came out as `'++suf`a
- it parses fine but looks ugly) so I fixed that.
.dotKeywords
will be stored as #hashKeywords
and #
is an id-char
To save us the trouble of changing the whole Loyc codebase, the dot in a .keyword
will be changed into a #
in the Loyc tree. Also, #
will be an identifier character. Thus .return x
is equivalent to #return(x)
. I'm also reverting the definition of LNode.IsSpecialName
to exclude .
.
.keyw0rds
can now contain digits
.keywords
will be allowed to be any valid non-backquoted identifier.
Number parsing change
In order to simplify the lexer by removing a lookahead loop, I'm removing the requirement to have a "p0" clause on hexadecimal floating-point numbers. Consequently, 0x1.Ep0
can be written as 0x1.E
, which will be treated as a floating-point number instead of 0x1 . E
. However, 0x1.Equals(1)
now has the bizarre interpretation quals"0x1.E"(1)
.
Weird Operators
Given a "weird" operator like x *+ y
, the precedence of *+
was previously based on the first character, +
. I think the motivation for this was to act similar to Nim operators. But then I noticed something: MATLAB and Julia have "elementwise" operators like X .* Y
which multiply pairs of scalars in matrices X
and Y
. So I checked several other languages. Apparently, most other languages do not have any "compound" operators, but Swift does. Swift has &*
, &/
, &+
, &-
, and &%
. In Swift, MATLAB and Julia, the last character determines the precedence. Bacause Swift, MATLAB, etc. are more popular than Nim, I'm changing LES to decide precedence based on the last character instead of the first. Sound good @jonathanvdc ?
Earlier I added a !.
operator with precedence of .
so that Kotlin's !!.
operator would have the correct precedence. This change makes the !.
operator redundant. Null-dot ?.
must still be special-cased as its precedence is lower than .
.
Note: Long operators like |%|
will continue to be based on the first and last character. In the case of combo operators, like x s>> y
, the initial identifier is ignored for the purpose of deciding precedence so that this particular operator has the same precedence as >>
.
Operators and fractions
I decided that, to better match other programming languages, x*.2
should be parsed as x * 0.2
rather than as x *. 2
. However, the tricky thing is that 0..2
and 0...3
should still be parsed as ranges, 0 .. 2
and 0 ... 3
respectively. To achieve this I tried splitting Operator
into two rules:
token DotOperator returns [object result] :
'.' OpChar*
{$result = ParseOp(out _type);};
token Operator returns [object result] :
('$'|OpCharExceptDot) (OpCharExceptDot | '.' ((~'0'..'9' | EOF) =>))*
{$result = ParseOp(out _type);};
[inline] extern OpCharExceptDot :
'~'|'!'|'%'|'^'|'&'|'*'|'-'|'+'|'='|'|'|'<'|'>'|'/'|'?'|':';
[inline] extern OpChar :
'~'|'!'|'%'|'^'|'&'|'*'|'-'|'+'|'='|'|'|'<'|'>'|'/'|'?'|':'|'.';
This is an efficient implementation. However, with this grammar, If you write x.+.2
it is parsed as x .+. 2
which is not necessarily the desired result. It's a perfectly reasonable way to parse, but I think it would be better to match Julia/MATLAB. So I thought of an alternative which actually looks simpler in the grammar:
token Operator returns [object result] :
( '$'? (OpCharExceptDot | '.' ((~'0'..'9' | EOF) =>) '.'*)+ / '$' )
{$result = ParseOp(out _type);};
This turns out to generate more, and slower, code. But for the sake of compatibility, I'll accept that.
::
operator
I think the precedence of ::
should be changed to match C++ and EC#. In LESv2 I chose the syntax x::Loyc.Symbol
for variable declarations, which required ::
to have a lower precedence than .
. Why didn't I just use the syntax x: Loyc.Symbol
? I think it was because it conflicted with LES's old "Python mode" where you could write
Meaning if c { print "c is true!" }
. This feature was removed 2016-06-21, I think in order to make the language easier to parse and ensure colon would behave like any other operator.
So I will raise the precedence of ::
up to .
, but will make the change exclusive to LESv3 because I have a fairly large amount of LESv2 code relying on the old precedence.
Minor implementation detail
?.
will now be classified as TokenType.Dot
while .*
will no longer be classified as TokenType.Dot
.
Arrow operators
I have reduced the precedence of arrow operators -> <-
. Technically their precedence is now above &&
but below | ^
. My thinking is that some people would like to use arrows as assignment operators: flag <- x > y || y < 0
gets the structure (flag <- (x > y)) || (y < 0)
.
The previous precedence of -> <-
was sort of a compromise between C (which wants high precedence) and other languages that want lower precedence. But arguably the old precedence served neither case very well. Now I'm thinking that people wanting to use a C++-style arrow operator as in obj->f()
should pick another operator, such as obj*.f()
.
Continuators
The set of possible continuators should be carefully considered because it cannot be compatibly changed later. Since no one has offered an opinion, I am going to suggest that the set be the ten words else, elseif, elsif, catch, except, finally, while, until, plus, using
, plus the set of all identifiers that begin with two underscores (__etc
). Note that instead of double-underscores it would seem more natural to add #hashKeywords
to the set; however, this creates an ambiguity in case of code like the following:
I think it's more likely this was intended to be two separate statements, rather than that #bar(x)
is intended to continue the .foo
statement. I selected __
, a traditional "potential reserved word" marker in C++, because it is not currently used for any purpose in Loyc. Unlike continuators like else
that will be stored with #
in the Loyc tree (#else
), double-underscore identifiers will be stored unchanged. Though it could equally be argued that #
s should not be added even in the former case.
Since continuators are not allowed to be used as binary operators, I removed and, or, but
from the continuator set, thinking that some users would prefer to use them as binary operators.
And introducing token lists / prefix-expressions / whatever you call 'em
I proposed that LESv3 not have token literals (unlike LESv2 and EC#) and instead adopt "token lists" such as ' + x 1
. No one offered an opinion on this, or about whether '(+ x y)
should be represented as `'`(`'()`(`'+`, x, y))
or as `'`(`'+`(x, y))
so I'm somewhat arbitrarily selecting the first representation.
Decisions on other questions previously raised
- Should attributes be allowed in the middle of an expression, as in
foo: @immutable string
? Still undecided; currently, no.
- Should comma be allowed as a separator within braces, as in
{ a, b, c }
? I don't think so. The parser isn't currently complaining about it, but I may add a check, possibly a check dependent on whether {
is followed by a string (to carve out an exception for JSON syntax).
- Should comma be allowed as a separator within tuples as in
(x, y)
? Yes, but (x,)
will be a syntax error and parsed as (x, ``)
.
- What should
#
mean? It will be treated as an identifier character like an underscore.
- Should continuator keywords be bona fide keywords? No, I think not.
- Edit: Should non-ASCII identifiers be supported without backquotes? I suppose that can wait for the next version.
Idea
Numeric literals can have any identifier as a suffix, including backquoted identifiers. This feature could be used to support my favorite feature that few languages allow: compound unit types like 1.2`kB/record`
or 3e8`m/s`
. Currently an expression like size `kB`
is meaningless, but I suppose it could be defined as some sort of suffix operator which could then be used for unit types. However, this idea has the disadvantage that a numeric value with units would have a different syntactic structure than any other expression with units, and 3`px`
would have a different meaning than 3 `px`
.