grpc-device should convert string encoding to/from UTF-8 when necessary about grpc-device

[bkeryan]

I finally got back to this and it seems to work ok, at least for NI-DCPower.

[dmondrik]

Related, the test QuotedLocalizedDescription_ApiErrorAndDescriptionToStatus_MessageIsEncoded is not behaving in a sufficiently predictable way that meets everyone's expectations. While I'm baffled by the nuances of non-ANSI characters in C string constants, @bkeryan and Ben Craig are not convinced the code is doing anything to encode the characters.

• Brad thought the constant std::string("\"chaîne localisée 😀\"") in MSVC would issue a code page warning, but I can confirm it does not.
• I tried making the constant explicitly specify u8 encoding, but Ben pointed out that code would cease to compile when we move to a newer C++ standard.
• I tried keeping a "normal" C string and using explicit Unicode characters std::string("\"cha\u00EEne localis\u00E9e\U0001F600\"") but that causes 2 problems:
  • Compiling: warning C4566: character represented by universal-ch aracter-name '\U0001F600' cannot be represented in the current code page (1252)
  • Running: C++ exception with description "[json.exception.type_error.316] invalid UTF-8 byte at index 5: 0x6E" thrown in the test body

The last part is the most concerning to me, as I would expect either inserting into the JSON object or stringifying it would take care of the encoding, but it doesn't seem to be the case. This likely needs to be addressed as part of solving this problem.

Also see Brad's comments here and here.

[bkeryan]

Long story short, MSVC has two character sets: source and execution. It uses the source character set to read the .cpp file and it uses the execution character set to write string constants to .obj files. By default, both character sets are ANSI. If you want to specify UTF-8 string constants using UTF-8 source code, you should set both character sets to UTF-8.

Brad thought the constant std::string(""chaîne localisée 😀"") in MSVC would issue a code page warning, but I can confirm it does not.

It depends on the code page and the character values. Some sequences of Windows-1252 (Latin-1) characters or UTF-8 bytes are not valid Windows-932 (simplified Chinese) byte sequences. Depending on the characters used, this may generate warnings when compiling with a Chinese locale.

Compiling: warning C4566: character represented by universal-ch aracter-name '\U0001F600' cannot be represented in the current code page (1252)

This confirms that the execution character set is Windows-1252. This code page cannot represent emoji.

Running: C++ exception with description "[json.exception.type_error.316] invalid UTF-8 byte at index 5: 0x6E" thrown in the test body

MSVC converted \u00EE to the Windows-1252 representation of 'î' and the resulting Windows-1252 byte sequence is not a valid UTF-8 byte sequence.

u8"\u00EE" would work, except for the C++20 changes.

"\xC3\xAE" should work regardless of source/execution encodings or language standard.

[reckenro]

@bkeryan , are you good with marking this Issue closed? I'm good with the changes in the PR + tests that ran and the additional issue David made to capture the additional work to use the better suited DAQmx DLL.

But you're more of the expert with this issue so wanted to check with you to see if you're good with closing it too.

[bkeryan]

I'd like to do a little testing to see if the system as a whole handles i18n properly.