Streaming API for transcoding? about simdutf

[lemire]

The functions that return an error are not meant to process truncated inputs, they are meant to help identify genuine errors. If you have simply a truncated input, the simplest strategy is to roll it back to the previous complete character.

Please see #349

I have added convenience functions.

[kovidgoyal]

Yes, pre-trimming is also an option, however in my use case, truncated final character should be rare, as such I was hoping to avoid the cost of reading the last 3 bytes on every chunk. Are you saying that I cannot rely on the function returning an error in case of truncated last char? My potential algorithm was run the simd transcode function, if there is a TOO_SHORT error trim the last char and re-run.

Thanks for the convenience function.

[lemire]

I was hoping to avoid the cost of reading the last 3 bytes on every chunk.

It is very cheap unless you are processing tiny pieces, in which case, you have other performance constraints.

Are you saying that I cannot rely on the function returning an error in case of truncated last char?

It will return an error. The error should be TOO_SHORT, as you indicated.

My potential algorithm was run the simd transcode function, if there is a TOO_SHORT error trim the last char and re-run.

That's likely slower than what I propose if you expect that the input is valid.

[kovidgoyal]

That's likely slower than what I propose if you expect that the input is valid.

I dont know if the input is valid, it could be invalid and have a trailing truncated char as well. Therefore, I have to use the function that returns an error anyway. Thus, given that a trailing truncated char is rare pre-trimming cannot be faster. But anyway, that's easily checked by measuring.

I was just hoping to convince you to add a field to the return struct indicating how much was output before the error was encountered. But I can work with the existing API, given truncated trailer is rare for me, trascode, check error return, strip trailer, re-transcode should not be too bad. I will benchmark and see.

[lemire]

@kovidgoyal Yes, you want support for decoding invalid inputs. It is not something simdutf supports at this time. The assumption is that the input is valid. We'll extend the library at some point in the future to include support for invalid inputs.