Comments (4)
Ah, OK. Our algorithm is indeed to check the page's charset and to use that, whether it's in the HTML or in the HTTP header. If that fails, we try to sniff the charset using the chardet library. This is the same algo/library that browsers use, so we're doing pretty well. However, I guess when the charset is commented out, we fail.
I did make one change to our code as a result of this bug. Since iso-8859-1 is a superset of cp1252 , I've hardcoded our crawler so that if iso-8859-1 is detected, we use cp1252 instead. Hard-won lessons, eh?
For future scrapers, if we need to, we can hardcode the encoding in the scraper itself, but this one will be fine when it runs due to the above change.
Original Comment By: Mike Lissner
from juriscraper.
I think Python may have been suggesting to me that it was 1252 when I was working on this scraper, but maybe my source was just the HTML page source, which has the following line (at line 30 when I looked today):
{{{
#!HTML
}}}
Can we not look for "charset" in the HTML source?
Original Comment By: Brian Carver
from juriscraper.
Oh, and please close this if you don't see any solutions here (as I don't). I'm just keeping this open until we know where you got cp1252 from.
Original Comment By: Mike Lissner
from juriscraper.
Well, this took some work, but I figured it out. The problem is that the page referenced doesn't declare an encoding, so we (and our browser) think that it's iso-8859-1. We then decode it to unicode automatically, work with it internally, then encode it as utf-8 for storage and such.
Since we can't detect the proper encoding at the outset, we're pretty much screwed. Internally, we see \x97, which in unicode is U+0097 END OF GUARDED AREA. Not an emdash, alas.
Anyway, you said the page was cp1252. How'd you know that? Firefox, Chrome, and chardet all disagree?
Original Comment By: Mike Lissner
from juriscraper.
Related Issues (20)
- Map state oral arguments sources to download HOT 3
- `mass` and `massappct` are blocking our scraping server HOT 3
- IndexError: Parsing pacer_doc_id from flmd NEF HOT 1
- `cafc` oral argument scraper broken HOT 1
- IndexError: Querying DocketReport
- Idaho Supreme or Appellate Court appears broken HOT 1
- ASBCA appears broken HOT 1
- Rhode Island failures HOT 3
- Scraper Down (Oklahoma Criminal)
- Scraper down (coloctapp) HOT 6
- Nebraska Ct app (scraper down) HOT 1
- Minnesota is back to blocking or SSL issues
- `njtaxct_u` gap
- `nyag` scraper failing
- IndexError: Parsing Attorney in ACMS docket JSON object HOT 1
- `opinions.united_states_backscrapers` folder excluded from tests HOT 1
- Include de_seq_num when available for downloading PACER documents HOT 1
- `massappct_u` failing with 'Site' object has no attribute 'headers'
- HTML parsing not working for `oklaag`
- Update PACER parsers to include office code, judge initials, and defendant number HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from juriscraper.