string_utils should replace windows-1252 encoded special characters about juriscraper

[freelawbot]

Ah, OK. Our algorithm is indeed to check the page's charset and to use that, whether it's in the HTML or in the HTTP header. If that fails, we try to sniff the charset using the chardet library. This is the same algo/library that browsers use, so we're doing pretty well. However, I guess when the charset is commented out, we fail.

I did make one change to our code as a result of this bug. Since iso-8859-1 is a superset of cp1252 , I've hardcoded our crawler so that if iso-8859-1 is detected, we use cp1252 instead. Hard-won lessons, eh?

For future scrapers, if we need to, we can hardcode the encoding in the scraper itself, but this one will be fine when it runs due to the above change.

---

Original Comment By: Mike Lissner

[freelawbot]

I think Python may have been suggesting to me that it was 1252 when I was working on this scraper, but maybe my source was just the HTML page source, which has the following line (at line 30 when I looked today):

{{{
#!HTML

}}}

Can we not look for "charset" in the HTML source?

---

Original Comment By: Brian Carver

[freelawbot]

Oh, and please close this if you don't see any solutions here (as I don't). I'm just keeping this open until we know where you got cp1252 from.

---

Original Comment By: Mike Lissner

[freelawbot]

Well, this took some work, but I figured it out. The problem is that the page referenced doesn't declare an encoding, so we (and our browser) think that it's iso-8859-1. We then decode it to unicode automatically, work with it internally, then encode it as utf-8 for storage and such.

Since we can't detect the proper encoding at the outset, we're pretty much screwed. Internally, we see \x97, which in unicode is U+0097 END OF GUARDED AREA. Not an emdash, alas.

Anyway, you said the page was cp1252. How'd you know that? Firefox, Chrome, and chardet all disagree?

---

Original Comment By: Mike Lissner