Question: how does dictionary achieve superior compression for small data? about zstd

[Cyan4973]

If you were to concatenate all the same small files into a single blob, and then compress the resulting blob with zstd, what compression ratio would you obtain ?

[tomerr90]

I havent setup a test case of my own, Im asking based on what I saw in the GitHub readme "The Case For Small Data"

[Cyan4973]

When using different inputs, one logically obtains different compression performance.

If you want to compare small data with large data, use the same data.
Either concatenate the small ones to create large data (preferable), or split the large data to create small data.
Otherwise, it's not comparable.

Compressing large data should always win, though by how much depends on the data (incompressible data remain incompressible). Dictionary will help to close the gap, but typically cannot overtake the large data scenario, especially when the cost of the dictionary (its size) is taken in consideration.
It's likely possible to create a contrarian scenario where above statement is false, by taking advantage of imperfections in the compression process, since it's using fast imperfect heuristics. That would be an exception though, not the general expectation.

[tomerr90]

Thank you for your response!
Thats not quite what Im asking though, you are explaining a general scenario, Im asking about the results that are published in Zstd's GitHub page.
It makes sense to me the the ratio for big files show be the upper limit, however, it seems like for small data, its able to achieve much more, is it just because the input used happens to be such that is ver compressible?
Meaning if I would take all the samples in the scenario described and concatenate them, would I achieve the same ratio (~10) without the dictionary as well?

[Cyan4973]

Yes,
dictionary compression works best on structured data featuring a lot of redundancy across messages, though very little within the message.
This is what the github collection sample achieves: it's just a bunch of json records, with very similar structure.
If they were compressed all concatenated together, the compression ratio would be greater than 10.

Dictionary compression is for scenarios where one cannot concatenate these similar records together, for example because the records must be sent immediately and can't wait inside a batch queue.

[tomerr90]

I see, now I understand, thank you!