JSON:
- http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf
- how are strings encoded? See standard. It's non-trivial.
- Object keys are encoded as regular strings.
sqlite:
- how are strings encoded?
- Looks like it can be UTF-8, UTF-16BE, or UTF-16LE.
https://www.sqlite.org/datatype3.html
This is determined by the open function that's used
(https://www.sqlite.org/c3ref/open.html).
- text is converted on-the-fly (https://www.sqlite.org/version3.html)
- sql processed as utf-8 (implementation detail)
V8/JavaScript:
- Does JavaScript say how strings are encoded?
Looks like UCS-2: https://mathiasbynens.be/notes/javascript-encoding
BUT WITH SURROGATE PAIRS!
- Does V8 describe more precisely how strings are encoded in memory?
- Two-byte string: each character is a two-byte uint16_t, and it's "Unicode"
- ExternalTwoByteStrings appear to be explicitly UTF-16, but I think it's
really UCS-2 because the characters really are only two bytes each.
- All strings can contain null bytes
- Elsewhere in objects.h, it says strings are either ASCII or two-byte "UC16",
which isn't a thing.
Design points:
- what kind of strings do we want the libpmx utilities to consume? UTF-8?
- for labels?
- for strings?
- if so, how do consumers provide those strings?
- what if the string inside the C program isn't valid?
- representation inside the JavaScript program
- representation in JSON/sqlite
- do we want to have to re-encode every string?
Options:
- For strings represented as ASCII strings inside V8, can use a function that
takes an array of ASCII characters (with a length), and just emit them. Maybe
try to deal with non-7-bit-clean characters. Definitely deal with embedded
NULs.
- For strings represented as two-byte strings inside V8, we can either convert
them to UTF-8 for export or we can base64-encode the UCS-2/UTF-16 value. I
lean towards converting to UTF-8. The downside is perf (but base64-encode is
going to be crappy too) and potential loss of information (e.g., memory
usage).
- For both cases, we want to provide metadata indicating:
- the string contained invalid characters (e.g., non-7-bit-clean ASCII or an
unconvertable UCS-2/UTF-16 value), in which case the UTF-8 representation is
the closest approximation but may be truncated or otherwise incomplete.
- the length (however V8/JavaScript represent this, which I think will be in
code units).
- the original string representation, so that the consumer can determine
memory usage?
For sqlite, we could store the UTF-16 value if we wanted, but I don't think
there's any advantage. The underlying strings are probably UCS-2 anyway --
would they need a conversion? I say we just do the same thing as we do for
JSON.
Remaining open questions:
- What's the best way to convert JavaScript-flavor UCS-2 to UTF-8 in C?
- Can I use the mbtowc and wctomb functions? Am I not supposed to use wchar?
Is the encoding (UCS-2 vs. UTF-8) expressed in the locale, and I need to
specify which locale I want?
- Should I be using something else instead?
- See the implementation of: ToCString() in objects.cc, which uses
Utf8::Encode in unicode-inl.h.
Looks like they've got their own non-trivial implementation :-/