Question around ZSTD_decompressContinue about zstd

[Cyan4973]

Hi Beyers

You are correct.

The current API uses an integrated frame format.
This frame format specify a header in front of each block.
This header is necessary to know the size of the block.

Longer term, another API, at "block level", will be provided.
It will be similar to the current LZ4 one, meaning one can decide its own encapsulation format, or simply decode on its own the block header. But my understanding is that it's only remotely related to your problem.

Your problem is that, because of current design, it's not possible to provide current block to decompress before receiving the header of next one, which basically means, the next block must be already compressed from the sending side.
This is obviously a shitty design. This simple side-effect totally evaded my attention.

Unfortunately, the other idea, of setting srcSize to be [block header + compressed block], doesn't work either, since by definition, you don't know the size of compressed block before reading and decoding block header.

Even longer term, there will be a "buffered" version of the API, similar to current "lz4frame", where all these side-effects will be automatically handled by an internal buffer, allowing any size to be used as input.

But that's not for now. So we need a shorter term solution.

Simplest solution I can come up with :
divide input into 2 phases, one reading block header, the other reading compressed block.

This is supposed to be slightly detrimental to performance, since instead of one function call, there will be 2 (same for fread). But hopefully, this will be totally negligible. And the added flexibility is very well worth it.

Comments welcomed.

[bcronje]

Thank you for the info Yann. I do realize these are early days for ZSTD, so please excuse me for jumping the gun a bit.

OK I now understand your design decision here, the assumption is that typically the compressed data will be read from disk, so there it makes sense to optimize so that you only fread once as per your current implementation.

My use case is that the compressed blocks are already in memory (received via network or IPC/RPC etc.). So I do not have to worry about taking a 2nd fread penalty as I already have the header and block in memory.

So yes, I can certainly use your 2 phase input suggestion.

A ZSTD_setNextcBlockSize public function would be ideal? That way you can still use a single fread if accessing from disk, and would support cases where the header and block is already in memory. The only drawback I can see with this is that the API user would then have to call this manually, since that functionality will have to be removed from ZSTD_decompressContinue. Unless you add a argument to ZSTD_decompressContinue to ask it to read the next header or not, but this would not be so clean.

Any thoughts on this?

[Cyan4973]

Thanks for comments. Difficult to be sure, although I suspect 2 things :

• The penalty from having 2 fread() will likely be small enough as to be negligible. This will have to be properly benchmarked to be confirmed.
• Complexifying an API is never good. Any additional function prototype, and any additional argument, must really pay off. In this case, I'm afraid the relative performance gap is probably too little to grant another dedicated prototype.

One idea I had in mind when starting the design of current API,
was a prototype with a "totally free input size".
That is, srcSize would be provided as an size_t* pointer, and contain the input size.
On return, the function would modify this pointer content, in order to indicate how many bytes were effectively read from the source buffer.
This is basically current lz4frame design.

This way, if not enough input data is provided, it would return "0", basically stating that it has not even started the decompression process because there is not enough data. Conversely, when provided too much data, it would simply return currentBlockSize + nextBlockHeader within this parameter, inviting the user to present the rest of data to next function call.

In the end, I scrapped the idea, on the ground that I felt it was a bit too complex for casual users. The flexible design was indeed required for the buffered version (lz4frame), because the API has to accept any crazy size, and even the internal buffer can be overrun by some crazy memory sizes combinations (large input buffer + small output buffer), so it needs a signal to tell "I haven't been able to deal with all provided input".

But here, since this version is bufferless, it seemed it was better to be strict on input size to provide a synchronous decompression process, which is likely the one most programmers will want, since its behavior is simple to grasp.

It required though to create a second prototype, to know which size to provide to the function, since now it's a strict value. Hence was born ZSTD_getNextcBlockSize().
I believe it's relatively trivial to modify its behavior in order to provide once the header size, and once the next compressed block size. However, I also wonder if the function name is still valid (ZSTD_getNextcBlockSize), since it's no longer "just" about next compressed block size.

In order to allow a "single fread+decompress cycle", I could, for example, authorize the input to provide "currentBlockSize + nextBlockHeader", and deal with it transparently.
It would work this way :
User call ZSTD_getNextcBlockSize(), and receive next compressed block size (say 100).
User nonetheless provides 103 within srcSize argument to ZSTD_decompressContinue().
ZSTD_decompressContinue() makes an exception for this case, and silently accept to handle 103 bytes in a single row. This would simply be documented in the API comment section as a valid input size.

However, there is also a risk that user might be confused, and for example misled header read phase with cBlock read phases, providing 3+3=6 bytes as srcSize. In which case, function would exit with an error code.

In the end, I'm not even sure if that's a good idea to authorize such a behavior, should it risk generating some confusion.

Maybe the answer to this question will depend on benchmark results : if it measures a performance advantage in making a single call instead of 2, it's likely worth it. If not, it's probably better to keep the API behavior simpler.

Regards

[bcronje]

Personally I think what would be least confusing is something like:

1. read block header from file
2. call ZSTD_getNextcBlockSize passing in pointer to header from above, returns cBlock size
3. read cBlock from file
4. call ZSTD_decompressContinue passing in pointer to cBlock

I'll put together a couple of benchmarks with the different options and report back sometime this week.

[Cyan4973]

There's a new API proposed at :
https://github.com/Cyan4973/zstd/tree/streamAPI2

which tries to solve this situation.

The API is very close to the previous one, with the following 2 changes :

• Get the size to load using : ZSTD_nextSrcSizeToDecompress()
• Consequently, ZSTD_decompressContinue() is called alternatively to decode a block header, or to decode a compressed block. When it is a block header, the function result is 0, since it has not generated any data into output buffer.

The rest is the same.
For example, end of frame is still detected by ZSTD_nextSrcSizeToDecompress() == 0;

The new API passes automated tests.
But what matters is if it can be properly understood.
(Note : I still have to add a few comments into zstd_static.h)

[bcronje]

Yann, just to follow up, I'll get a chance this week to test out the streamAPI2 branch and will report back. Sorry that it is taking so long.

[Cyan4973]

You're welcomed

[Cyan4973]

The streamAPI2 feature branch has been merged into dev.
https://github.com/Cyan4973/zstd/tree/dev

If all tests and reports do well, it will land next into master.
Regards

[Cyan4973]

Merged to master