Size tresholds for changing the compression parameters penalizes 256K inputs about zstd

[Cyan4973]

It used to be that the settings for the 256KB tier were clearly better than the settings of the first tier for most data <= 256 KB.
Indeed, since then, a few internals of the library have changed. So this table tier is probably due a performance settings update.

Another problem with these "tiers" is that they decide settings based on training on some diverse sample sets.
But we know that different files react differently to each individual setting.
Therefore, the result on this training is some kind of "rough average" that tends to work fine on the diverse sample set,
but it would be possible to focus on just one sample and show that the new setting is actually detrimental for this one case.

And that's another issue we may face here : we have no idea what's the data used in your test, and it's difficult to claim generality from a single sample set focused on one category. The results are therefore better for RGBA tiles, but who's to say they are better for 256 KB inputs in general, at all levels, which is what the proposed change would enforce ?

This is probably worth testing as part of the diverse sample set.
Therefore. if there are public sample sets representative of your use case that could be used for compression level training, this would be a great contribution.

Finally, the user can always take in charge compression parameters directly. That's why the advanced interface exists (see --zstd=). All parameters from the first tier can be enforced on any data of any size if the user wishes so. Granted, this is more complex than just invoking zstd with default settings, unfortunately complexity is a cost to pay for fuller fine-grained control.

edit : I tested the tier 2 (<= 256KB) parameters vs the tier 1 parameters on 256 KB inputs,
and found nothing obvious.
In general, the tier 2 settings tend to compress more but also slower than the tier 1.
I only found one counter example (level 3), and that was by a tiny margin.
On the other hand, tier 2 settings tend to scale better, and as compression level increases, the difference between these two tiers of settings tend to increase too. For example, Tier 2 level 9 compresses a bit faster and bit better than Tier 1 level 12 on this sample set.
Another way to state that the comparison is not obvious and requires multiple data points.

[lucianpls]

@Cyan4973, thanks for your quick answer.

Using the RGB 8bit test images from http://imagecompression.info/test_images/

I'm using GDAL built with MRF using ZSTD. GDAL is a very common GIS library for reading and writing data. It has lots of dependencies, the only strictly required ones for this test should be PROJ4 and sqlite3. And the ZSTD library, of course. MRF is a format that I wrote and still maintain, it uses ZSTD as one of the possible tile compressions.
The compression code is at https://github.com/OSGeo/gdal/blob/master/frmts/mrf/mrf_band.cpp#L360C21-L403
The filter applied for the single band case ends up just subtracting the previous pixel value, taking advantage of the locality present in images. The default ZSTD level is 9.

In a folder with the unpacked rgb8bit.zip test images. The ppm format is a tiny text header followed by raw data, very little overhead. I timed running this bash script, which converts each ppm in sequence to an MRF using ZSTD compression per tile.

for f in *.ppm
do
  gdal_translate -q -co COMPRESS=ZSTD -co INTERLEAVE=BAND -co BLOCKSIZE=512 $f dest512/${f%.*}.mrf
done

The BLOCKSIZE parameter is the linear tile size. Band interleave means every band is written as a separate 512x512x1 byte, thus 256K exactly. On a laptop, the script ran in 15.828s, on an SSD drive and lots of RAM.
Then I changed the script a bit, to make the page slightly larger than 256K. Using 513x513 tiles, the size is about 257K.

for f in *.ppm
do
  gdal_translate -q -co COMPRESS=ZSTD -co INTERLEAVE=BAND -co BLOCKSIZE=513 $f dest513/${f%.*}.mrf
done

This script ran in 10.57s. There is some constant bash/gdal/MRF overhead, the real ZSTD speed difference is larger than these numbers suggest. The size difference, just looking at the output folder sizes, they were empty to start with:

$ du -ks dest*
249880  dest512
244028  dest513

To really compare apples to apples, I swapped a modified zstd library that uses first tier parameters for 256K, so it can generate exactly the same 512x512 tiles. The compression script was the first one above. It took 10.750s and the output folder reported 243960 KB.

Looking at the individual images, only artificial.ppm image is slightly smaller with the tier two, all the other ones are smaller with the tier one parameters. I found this to be the case in all the other images I've tested. And the compression itself is twice as fast, which is very valuable even if the size would be slightly larger. I agree that it might not be the case for all data, but it does look very consistent for natural images. I didn't select these images in any way, it's just the first available image set that I've found on a quick internet search.

There are petabytes of natural imagery being used in GIS and many other fields, this is likely to impact current and future users, and it's hard to notice. I only found out about it by chasing strange performance figures, yet ZSTD has been used in MRF for a couple of years now. I am likely to switch to using the streaming API in MRF, which side-steps this problem, yet I wanted to raise the awareness of this significant issue.