[question] Seek for insights on the suitable case for zstd dictionary compression about zstd

[Cyan4973]

Then what's exactly kind of case where zstd dictionary compression can actually stand out? Can I say, it's the case of many small files but don't allow to be concatenated?

Yes. It's a case where random access is required, or there is no time or memory to allow concatenation.

These cases are more common in data centers than on personal computers.

[Cyan4973]

lz4 already supports it.

zlib should also support it, the interface may be less obvious though.

No idea about snappy.

[Cyan4973]

This is highly unlikely.

There might be ways to manufacture a case to explicitly advantage the dictionary compression method, but even in this case, such a difference would end up being unimpressive.

Sort of such cheating technique, I would expect the more natural "stream the whole file" method to win every time, and sometimes by very significant margins.

[Nor7th]

@Cyan4973 Thanks for the very clear conclusion.

Given many files to compress, it's usually understandable that to compress the concatenating file of all can get a better compression effectiveness than compress each file one by one.

https://stackoverflow.com/questions/48145313/how-is-concatenating-files-before-compressing-more-effective-than-just-compressi

Then what's exactly kind of case where zstd dictionary compression can actually stand out? Can I say, it's the case of many small files but don't allow to be concatenated?

[Nor7th]

@Cyan4973 BTW, from your view, how difficult it is to expand the dictionary mechanism into other compression algorithms like snappy, lz4, gzip, zlib?