Encoding Error about anglesharp

[FlorianRappl]

Thanks, I will look into that!

[FlorianRappl]

With what version of AngleSharp have you tried? The latest one on NuGet, a specific (maybe older) version or the latest one available here?

I used the following code:

const String TestUrl = @"http://www.baidu.com";
Regex TitleRegex = new Regex("<title>(.*?)</title>");

var angleSharpTitle = DocumentBuilder.Html(new Uri(TestUrl, UriKind.Absolute)).Title;
var plainTitle = TitleRegex.Match(new HttpClient().GetStringAsync(TestUrl).Result).Groups[1].Value;

Then I inspected the two titles (as you did, I guess). However, to me both look the same:

I also compared the byte content via (e.g.)

Encoding.Unicode.GetBytes(plainTitle)

Both strings have 18 bytes and the same sequence / values (126,118,166,94,0,78,11,78,12,255,96,79,49,92,229,119,83,144). Can you re-check please? What is the title supposed to be (in bytes)?

I can only compare pictures here. For me the VS and browser generated symbols (Opera 26) look the same:

Thanks!

[h82258652]

the third character is not correct.

[FlorianRappl]

Again: Version? I can't reproduce it...

[h82258652]

I think I may be found why this error occurred.
If the response content encoding is different with the meta charset, it will occurred.
I make an example, here is the link:
https://onedrive.live.com/redir?resid=B406AF403E9168EA!13195&authkey=!AB4w4T2GQOur2XE&ithint=file%2c7z

I run it in IE11, Chrome, the title and body are no problems.
But in AngleSharp, it is wrong.
PS: I use AngleSharp version is 0.7, published at 2014-11-08.

[FlorianRappl]

I also tried the current version on NuGet. Still - I get the same for the HttpClient raw and the AngleSharp. The chosen character set is utf-8. What character set do you have? The only explanation I have is the local encoding start value... (therefore what document encoding has been selected for you might be an indicator in this case).

var document = DocumentBuilder.Html(new Uri(TestUrl, UriKind.Absolute));
var charset = document.CharacterSet;

Maybe this way we can catch the origin of the bug.

[h82258652]

Today, I try another test.

    class Program
    {
        static void Main(string[] args)
        {
            const String TestUrl = @"http://www.baidu.com/";
            Regex TitleRegex = new Regex("<title>(.*?)</title>");
            var angleSharpTitle = DocumentBuilder.Html(new Uri(TestUrl, UriKind.Absolute), new Configuration()
            {
                Culture = new CultureInfo("en-US")
            }).Title;
            var angleSharpTitle2 = DocumentBuilder.Html(new Uri(TestUrl, UriKind.Absolute), new Configuration()
            {
                Culture = new CultureInfo("zh-CN")
            }).Title;
            var plainTitle = TitleRegex.Match(new HttpClient().GetStringAsync(TestUrl).Result).Groups[1].Value;
        }
    }

en-US is right and zh-CN is wrong.
In your computer, en-US is the default culture info, but zh-CN is my computer's default culture info.
If we don't tell the DocumentBuilder which culture info it use, it will use the local culture info same with the computer, that's the reason the same codes will get different result between you and me.

and document.CharacterSet is utf-8 the same with you.

[FlorianRappl]

Alright, that means my guess (and the only remaining possibility) was right that the already consumed stream might be transformed wrong after the encoding changes (Windows-1252 to UTF-8 vs GB18030 to UTF-8). I can reproduce the bug now, thanks. Will be fixed soon.