Huge memory/cpu utilization for 1 page PDF extraction about pypdf

[alisufian]

Hi Matt,
Thank you for the prompt response. I agree the page is kind of heavy. Takes some time to render on my machine as well.

I greatly appreciate your time and effort on creating and maintaining this library.

[pubpub-zz]

@MartinThoma
Here you are the doc (download was OK for me)
PUC_Quiet crossing_school boundaries_11X17.pdf

I've opened the PDF in Acrobat reader and this file looks very heavy (lot of drawing/images?).

[mstamy2]

More evidence that extractText() needs work. It is a very complex file, however, and opening it in a PDF viewer was somewhat difficult for my system.

[MartinThoma]

@alisufian I'm currently looking into performance topics. The link seems not to load on my machine. Do you still have that document somewhere?

[MartinThoma]

This callgraph was created via:

$ python -m cProfile -o profile.pstats script.py
$ gprof2dot -f pstats profile.pstats | dot -Tsvg -o mine.svg

with

from PyPDF2 import PdfReader

reader = PdfReader('PUC_Quiet.crossing_school.boundaries_11X17.pdf')
text = ""
for page in reader.pages:
    text += page.extract_text()

[MartinThoma]

Some micro-benchmarks on Python 3.6:

peek in (b"\r", b"\n") vs peek in b"\r\n" vs peek == b"\r" or peek == b"\n"

In [8]: %timeit peek not in b"\r\n"
258 ns ± 3.11 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

In [11]: %timeit a != b"\r" and a != b"\n"
65.9 ns ± 0.618 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)

In [9]: %timeit peek not in (b"\r", b"\n")
56 ns ± 0.351 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)

That is surprising to me.

Decimal instanciation

In [13]: %timeit Decimal(0)
168 ns ± 0.3 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)

In [14]: %timeit Decimal("0")
223 ns ± 1.89 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

In [15]: %timeit Decimal(0.0)
430 ns ± 4.32 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

[MartinThoma]

After applying #1014 :

[MartinThoma]

One of the performance killers is creating the FloatObjects. We do it 6 million times in this example and it is 8% of the workload. But remplacing the FloatObject with normal floats (or decimals) would be a quite massive change.

[pubpub-zz]

@MartinThoma
with the new PR can you rerun to check for improvements?

[MartinThoma]

I'm closing this issue now as I don't have any further approach to speed this up (except for writing a C/C++/Rust module, which would be a very different beast)