`String#==` does not work the same way as MRI expects about artichoke

[lopopolo]

I'm not sure if String#length is the right conditional here. Look at this for Binary and ASCII strings:

$ irb
[3.1.2] > s = "\u{1F600}"
=> "😀"
[3.1.2] > s.encoding
=> #<Encoding:UTF-8>
[3.1.2] > s.length
=> 1
[3.1.2] > a = s.force_encoding(Encoding::ASCII)
=> "\xF0\x9F\x98\x80"
[3.1.2] > b = s.b
=> "\xF0\x9F\x98\x80"
[3.1.2] > a.encoding
=> #<Encoding:US-ASCII>
[3.1.2] > b.encoding
=> #<Encoding:ASCII-8BIT>
[3.1.2] > b == a
=> false
[3.1.2] > a == b
=> false

I think we'll have to dive though the MRI sources to find out what the extra conditional check is.

[lopopolo]

That length comparison should be self.char_len(). len() is not encoding aware, it just looks at the underlying bytes.

[lopopolo]

Let's add test cases for this in the string_test.rb functional tests.

[b-n]

Suspect:

[3.1.2] > s = "\u{1F600}"
=> "😀"
[3.1.2] > s.force_encoding(Encoding::ASCII)
=> "\xF0\x9F\x98\x80"
[3.1.2] > t = s.b
=> "\xF0\x9F\x98\x80"
[3.1.2] > s == t
=> false
[3.1.2] > s.encoding
=> #<Encoding:US-ASCII>
[3.1.2] > t.encoding
=> #<Encoding:ASCII-8BIT>
[3.1.2] > s.valid_encoding?
=> false
[3.1.2] > t.valid_encoding?
=> true
[3.1.2] > s = "A"
=> "A"
[3.1.2] > s.force_encoding(Encoding::ASCII)
=> "A"
[3.1.2] > t = s.b
=> "A"
[3.1.2] > s == t
=> true
[3.1.2] > s.valid_encoding?
=> true
[3.1.2] > t.valid_encoding?
=> true

e.g. \xF0 is not a valid US-ASCII char. Ruby is still doing it's best to show the underlying bytes of course. And since all bytes are valid in ASCII-8BIT

[AI-Mozi]

Hey!
I will be happy to try to solve this issue. Is there anything else I should know about it?

[lopopolo]

Go for it @AI-Mozi

[b-n]

@AI-Mozi In case it helps, I think this line in the Ruby docs is the crux of the issue:

Returns false if the two strings' encodings are not compatible

☝️ from here: https://ruby-doc.org/core-3.1.2/String.html#method-i-3D-3D

I spent some time with a colleague at work thinking about what this actually means, and I think I now have a grasp. Sorry in advance if my explanation is lacking and/or this becomes a thesis 😬.

In short, it comes to how the characters are represented, instead of their binary values. This hopefully can be explained by these two code pages:

• https://en.wikipedia.org/wiki/ISO/IEC_8859-1#Code_page_layout
• https://en.wikipedia.org/wiki/ISO/IEC_8859-2#Code_page_layout

Here, you can see that \x30 is represented by 0 in both code pages. However later on, \xA1 is represented by a ¡ in 8859-1, where as Ą in 8859-2. So although their binary contents are the same, how they are displayed to the user would be different.

Note, this is confusing in ruby, because it's reliant on whether your shell is set up to view these character sets or not (I believe this is what the limitation is, but not 100% sure - echo $LANG and you'll likely see UTF-8 for example). How this manifests is as follows:

[3.1.2] > s = "\x30".force_encoding(Encoding::ISO_8859_1)
=> "0"
[3.1.2] > t = "\x30".force_encoding(Encoding::ISO_8859_2)
=> "0"
[3.1.2] > s.encoding
=> #<Encoding:ISO-8859-1>
[3.1.2] > t.encoding
=> #<Encoding:ISO-8859-2>
[3.1.2] > s == t
=> true
[3.1.2] > u = "\xA1".force_encoding(Encoding::ISO_8859_1)
=> "\xA1"
[3.1.2] > v = "\xA1".force_encoding(Encoding::ISO_8859_2)
=> "\xA1"
[3.1.2] > u == v
=> false

Some analysis:

• \x30 outputs as a 0 since this symbol is encoded the same as what shell supports (UTF-8 \x30 is also represented by 0)
• \xA1 is output as hex, since that character is not the same in UTF-8 as it is in the encoding specified. e.g. My shell knows this text is not UTF-8, but also it doesn't know how to output it
• When ruby says: "Returns false if the two strings' encodings are not compatible" - I believe it's saying: "If all characters are represented in the same way in both encodings, then it's equal, otherwise it is not"

☝️ In saying all the above, Artichoke currently only supports Binary, ASCII, and UTF-8 strings. The good news, is that the first 128 characters (0 indexed) are represented the same across all of these encodings. e.g. any two byte strings that only include the characters \x00 => \x7F should be equal, regardless of encoding. If the strings contain anything \x80 and above, I'd expect to give equality. Proof:

[3.1.2] > s = "\x80"
=> "\x80"
[3.1.2] > t = s.b
=> "\x80"
[3.1.2] > u = s.dup.force_encoding(Encoding::ASCII)
=> "\x80"
[3.1.2] > s.encoding
=> #<Encoding:UTF-8>
[3.1.2] > t.encoding
=> #<Encoding:ASCII-8BIT>
[3.1.2] > u.encoding
=> #<Encoding:US-ASCII>
[3.1.2] > s == t
=> false
[3.1.2] > t == u
=> false
[3.1.2] > s == u
=> false

Hopefully the above makes some sense. I'm not actually sure how/where I'd implement this, but I thought the info might help

[b-n]

In saying this:

[3.1.2] > u = "\xD6".force_encoding(Encoding::ISO_8859_1)
=> "\xD6"
[3.1.2] > v = "\xD6".force_encoding(Encoding::ISO_8859_2)
=> "\xD6"
[3.1.2] > u == v
=> false

☝️ I'm not sure if this is "correct" from MRI. In both code pages, this would be Ö so "In theory" they should equal since the those two encodings are compatible for that character. I imagine this is because it would be massively hard to manage giant equality tables of "this code point looks like this here" etc. Although I do like the challenge of making a Rust library that can do this sort of equality 😬.

Note, I suspect MRI (and we could also use this same logic in Artichoke) uses these conditions for equality:

• internal bytes are the same
• Encoding is the same OR both strings only contain ascii chars

[AI-Mozi]

Hey! I've had a bit of a break but now I'd like to complete this task :)
Is spinoso-string the only place where I should add changes?
And could you please provide some examples of tests that would hep me check if my changes are correct?

[lopopolo]

hi @AI-Mozi. you'll want to modify the PartialEq implementation on EncodedString to also check for the left and right sides having the same char_len:

artichoke/spinoso-string/src/enc/mod.rs

Lines 60 to 74 in 2db5303

	impl PartialEq for EncodedString {
	fn eq(&self, other: &Self) -> bool {
	// Equality only depends on each `EncodedString`'s byte contents.
	//
	// ```
	// [3.0.2] > s = "abc"
	// => "abc"
	// [3.0.2] > t = s.dup.force_encoding(Encoding::ASCII)
	// => "abc"
	// [3.0.2] > s == t
	// => true
	// ```
	*self.as_slice() == *other.as_slice()
	}
	}

[AI-Mozi]

And thats all? Just check if have same char_len?