VS2017 build warning C4819 on non-english code page. about abseil-cpp

[JonathanDCohen]

Which character was the compiler complaining about?

[gavxin]

I found two position.

unaligned_access.h the two double quotes are unicode chars.

abseil-cpp/absl/base/internal/unaligned_access.h

Line 145 in 46ed9d9

// “ARMv7 or higher”, so we have to filter away all ARMv5 and ARMv6

absl/strings/str_split_test.cc the test case TEST(Split, UTF8) is testing utf-8 string splitting. It's just test case so maybe no need to change.

abseil-cpp/absl/strings/str_split_test.cc

Lines 622 to 642 in 46ed9d9

	TEST(Split, UTF8) {
	// Tests splitting utf8 strings and utf8 delimiters.
	{
	// A utf8 input std::string with an ascii delimiter.
	std::vector<absl::string_view> v = absl::StrSplit("a,κόσμε", ',');
	EXPECT_THAT(v, ElementsAre("a", "κόσμε"));
	}
	{
	// A utf8 input std::string and a utf8 delimiter.
	std::vector<absl::string_view> v = absl::StrSplit("a,κόσμε,b", ",κόσμε,");
	EXPECT_THAT(v, ElementsAre("a", "b"));
	}
	{
	// A utf8 input std::string and ByAnyChar with ascii chars.
	std::vector<absl::string_view> v =
	absl::StrSplit("Foo hällo th丞re", absl::ByAnyChar(" \t"));
	EXPECT_THAT(v, ElementsAre("Foo", "hällo", "th丞re"));
	}
	}

After modify these two file, the build success.

[JonathanDCohen]

Since Abseil is fairly low-level, instead of forcing utf-8 in msvc builds, I think the correct move is to make our source files all ascii. So I'll write up a patch to change that test to use unicode escape characters, which should make it agnostic to file encoding.

That being said, Abseil files are utf8 internally, so I'm not sure what other monsters there may be lurking if someone tries to use abseil source with other encodings.

[gavxin]

It make sense, it's better make source file as ascii than forcing utf-8 for msvc.

Thanks

[manshreck]

Note that the test is actually testing utf8 characters. I don't see how we can have a test for utf8 in ASCII ...

We should definitely prefer ASCII for source files, IMO, but I'm not sure requiring them to be ASCII is the best option.

[mbxx]

Jon and I chatted about this yesterday, and I came to the conclusion that Abseil files should be ASCII. Abseil strives to be as broadly usable as possible. Usually this takes the form of writing code that works on a wide variety of platforms, dodging bugs in those platforms if necessary. Here it means avoiding UTF-8 because some toolchains we want to support can't handle it.

The alternative isn't as bad as I feared: escaping the unicode characters in the test strings lets them be UTF-8 test strings that are specified using only ASCII characters in the source code. I'm a little sad, because suddenly it's no longer clear at a glance that the escaped characters are well-formed UTF-8, but this feels like the right thing to do to me.

[jorgbrown]

The MSVC error message says "Save the file in Unicode format to prevent data loss"

Just curious: What does it mean to save the file in Unicode format? Does it just add a UTF-8 BOM to the beginning of the file? Or is there additional metadata in the filesystem to tell MSVC that the file is Unicode?

[JonathanDCohen]

"Unicode format" seems ambiguous. Anyways it looks like my internal patch to remove non-ascii characters has been upstreamed, and ASCII fits in all flavors of Unicode. Is this problem still persisting?

[gavxin]

Try to build 0ec11ba , str_split_test.cc still cause following warnings.

ERROR: E:/xx/abseil-cpp/absl/strings/BUILD.bazel:214:1: C++ compilation of rule '//absl/strings:str_split_test' failed (Exit 2)
absl/strings/str_split_test.cc(624): error C2220: warning treated as error - no 'object' file generated
absl/strings/str_split_test.cc(624): warning C4566: character represented by universal-character-name '\u1F79' cannot be represented in the current code page (936)
absl/strings/str_split_test.cc(644): warning C4566: character represented by universal-character-name '\u00E4' cannot be represented in the current code page (936)
absl/strings/str_split_test.cc(645): warning C4566: character represented by universal-character-name '\u00E4' cannot be represented in the current code page (936)

It just test, maybe I should ignore it.

[ahedberg]

I'm looking into fixing the str_split_test.cc failures.

[ahedberg]

This has been fixed internally; just waiting for the change to be propagated to GitHub.

[mbxx]

I just pushed the change to GitHub.