Comments (4)
Thanks for the info!
I am not very familiar with UTF8/Unicode, but I would suggest we at least fix the lexer to not produce an error when encountering a UTF8 character.
I am not so familiar with the difference between semantic position and visual position, so I vote for whatever is easier to implement, which is probably UTF8 semantic position.
from smlfmt.
By the way, what is the accepted standard practice these days for visually handling "characters" that are encoded as more than one UTF8 character? E.g., the flag emoji "πΊπΈ" is actually two UTF8 characters ("πΊ" followed by "πΈ"). But of course, it is intended to be visually represented as a single character.
My initial thought is that this is important for smlfmt
because we need to know positions to vertically align things correctly. Do we use the UTF8 semantic position, or the intended visual position? I'm inclined to use UTF8 semantic position...
from smlfmt.
Supporting this won't be too bad, but will require changes in a few places.
For the lexer, we'll need to skip over UTF8 characters in the function advance_oneCharOrEscapeSequenceInString
. Note that this function already skips over escape sequences; handling UTF8 should be similar. And then we can selectively enable this functionality by adding an additional flag to the lexer functions Lexer.next
and Lexer.tokens
.
We'll need to update the implementation of Source
, too, such as Source.absoluteStart
which returns the position (line and col) of a source file segment. Currently these are computed via byte offsets, which is no longer correct under UTF8. I believe other functions will need to be updated, too, to ensure that a Source.t
never starts or ends in the middle of a UTF8 sequence.
from smlfmt.
It occurred to me that a simpler way to support this is to allow for UTF-8 bytes but not check for validity of a UTF-8 byte sequence. #74 implements this.
By default, this is disabled. It can be enabled with -allow-extended-text-consts true
at the command-line, or with the "allowExtendedTextConsts true"
annotation within an MLB.
Your example above should now be working. Let me know if you have any trouble!
from smlfmt.
Related Issues (20)
- Grouping of function application HOT 2
- Infix style with large expressions HOT 1
- Style for nested `if` expressions?
- SuccessorML syntax HOT 8
- Long single-line comments can force large indentation increases
- Format of `andalso` and `orelse` HOT 5
- Support top-level expressions HOT 5
- Rightward drift in deep function application
- Revisit `Promise` overhead in document structure
- Non-idempotent formatting
- Parser bug: unexpected functor HOT 1
- Heuristic for placing fun decl body on separate line HOT 1
- Automatically detect if stdin isn't a terminal HOT 2
- Remove `--stdio` flag
- Comments immediately before end of file don't format well HOT 3
- Editor integration HOT 6
- Continuous Integration Support HOT 6
- TODO: new release HOT 1
- Unusual command line interface HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from smlfmt.