len/count regression since 0.20.6 - 11x times slower in sample about polars

[ritchie46]

Found that we accidentally introduced quadratic behavior for the new string type. This is resolved by: #14705

[ritchie46]

Can we get a bisect to see which commit introduced the slow down?

I expect this is the new string type. Isn't reader the file just slower?

[ritchie46]

I do see a difference in performance on linux, but not as drastically as on windows. I believe windows is due to the allocator we compile on windows which is very bad. We have to try to switch to default allocator.

[ritchie46]

I think the difference is due to utf8 validation. This is much more expensive because we cannot do it on the whole buffer anymore:

I will see if we can do something smart here.

[yuuuxt]

version 0.20.11 indeed fixes this issue:

# issue exists on random generated data
# 0.20.5  - 253 ms ± 3.68 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# 0.20.6  - 2.28 s ± 7.81 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# 0.20.10 - 3.03 s ± 19.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# 0.20.11 - 747 µs ± 5.34 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit pl.scan_parquet(data_path / "sample_data_gen_4m.parquet").select(pl.len()).collect()

# no issue in this case
# 0.20.5   - 79.9 ms ± 116 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
# 0.20.6   - 114 ms ± 604 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
# 0.20.10 - 113 ms ± 545 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
# 0.20.11 - 687 µs ± 5.12 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit pl.scan_parquet(data_path / "sample_data_4m.parquet").select(pl.len()).collect()

Thanks for the quick fix!