Crawler: error trying to parse this page: c02.xhtml about safaribooks

[jrwagz]

@azec-pdx , I'm using an M series MacOS device and I was able to use the code on your branch (commit azec-pdx@a2be61e) and was able to get around this same problem for myself. Thank you!

[azec-pdx]

@lorenzodifuccia && @ivanpagac ,

I believe there is a set of problems introduced with later versions of Python that LXML hasn't addressed yet.
I am watching the following:

1. https://bugs.launchpad.net/lxml/+bug/1949271
2. https://github.com/Donohue/medium-to-jekyll/pull/4/files
3. Donohue/medium-to-jekyll#3

Regardless of this external change in lxml, I found the issue in this project with handling emojis and other special unicode characters when requesting lxml to parse the document, for the versions of Python with which lxml behaves well.

I have addressed the issue in https://github.com/azec-pdx/safaribooks/tree/apiv2 .
I was able to confirm positive results with testing on Book with IDs: 9781098156817 and 9781617297274 which both have some emojis and other offending characters. However, I was able to only get the parsing right with Python 3.9.16 and while using Python 3.9.10, it is still broken (I believe because of the additional issue linked above).

[azec-pdx]

I've had different behaviors of lxml on same Python version between macOS running Apple M1 chip and macOS running Apple Intel chip. On M1 macOS, it basically errors as described above and my branch is handling that now, but on Intel macOS it never errors out.

[trsudarshan]

@azec-pdx thank you, is there a version of lxml (fixing Python at 3.9.x), where this error can be avoided? If so, patching requirements.txt to that version of lxml may allow users to locally work around this problem, until a formal PR resolving it, gets merged.

[dreampuf]

#347 fixed this issue.