[Question] Understanding of compression level with external sequence producer about zstd

[Cyan4973]

Is the compressionLevel in ZSTD_sequenceProducer_F function always the same value as ZSTD_c_compressionLevel set by ZSTD_CCtx_setParameter?

Yes.
It's mostly informational btw. The sequence producer is free to do whatever it wants with this information, including ignoring it.

My assumption is that ZSTD_c_compressionLevel somehow influences the 'compress sequences' step.

It's true. It's more a side-effect though.

There are multiple cpu / compression trade-offs in the pipeline. They are roughly influenced by the compression level.
Now, because there are so many of them, I can't tell for sure which ones have impacted your observations.

As a simple example, the "minimum compression" threshold, which is both a guess and a verification deciding to not compress literals if it doesn't produce at least a certain amount of savings compared to "just store uncompressed", this threshold changes depending on compression level.
And there are others too. For example, there is a block splitter which is triggered at some point. This advanced strategy decides to cut a block into 2 or more parts if it can produce more adjusted statistics for each part, resulting in better compression ratio. This is expensive, but compared to the rest of the pipeline, it's negligible at high compression levels. This might be the one you observed in your tests.

So there are multiple heuristics decisions across the pipeline, and their behavior is influenced by the compression level. Sometimes, there are some advanced flags that can take over from the heuristics to unconditionally enable or disable the feature, sometimes not. This is mostly because these features were added later on, one by one, and there was a lack of common framework, resulting in each feature implementing its own slightly different way to be controlled and triggered. A refactoring could be needed at some point, but until then, the general idea is that "higher compression level" means "more cpu available for compression ratio", but the exact technical translation is more complex to detail (and changes with versions).

[b3ng1998]

Thanks for the explanation.