only parsing first page of 136 page pdf about fonduer

[clayms]

ok! The following works

first (assuming you have mutool installed):

import subprocess
subprocess.Popen('mutool draw -o "ICUS_Rules_Clean_up.html" "ICUS_Rules_Clean_up.pdf"', shell=True) 

then

corpus_parser = Parser(structural=True, lingual=True, visual=True, pdf_path=pdf_path)
%time corpus_parser.apply(doc_preprocessor, parallelism=PARALLEL)

very slow for one document though

[INFO] fonduer.utils.udf - Clearing existing...
[INFO] fonduer.utils.udf - Running UDF...
CPU times: user 1.16 s, sys: 484 ms, total: 1.64 s
Wall time: 4min 44s

then

from fonduer import Document, Sentence
print("Documents:", session.query(Document).count())
print("Sentences:", session.query(Sentence).count())

Documents: 1
Sentences: 6858

mutool is open source so it seems you should be recommending this as the conversion tool.

[clayms]

related to this HazyResearch/pdftotree#27 (comment) ?

When trying to use pdftotree I get the same error as reported in that issue when converting the pdf linked to above.

[lukehsiao]

@clayms can you try converting the PDF to HTML using Adobe Acrobat to see if that is successful?

[clayms]

I don't have access to Acrobat, only the reader which only appears to convert to [.docx, .doc, .rtf, .xlsx, or .pptx]

As suggested in HazyResearch/pdftotree#38 (comment), I will convert to html through MS Word. I will let you know the results.

[clayms]

Using MS Word to convert leads to this error:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x95 in position 240614: invalid start byte

I replaced the MS Word generated html file with the Poppler pdftohtml file and ran the same code again, and it worked, but only output 19 sentences from the first page again.

[clayms]

possible alternative?

https://github.com/coolwanglu/pdf2htmlEX

[clayms]

this looks promising

https://mupdf.com/docs/manual-mutool-draw.html