danmasta / csv Goto Github PK

Currently, if you are using streaming mode and using raw buffers for the chunks, if there happens to be a chunk boundary that falls between bytes on a multibyte character, it will fail to properly output the full character. I simulated a chunk boundary by creating a stream that pushes a chunk for each single byte in a buffer, instead of each character.

Example:

 { col0: '��', col1: '��', col2: '���' }

vs

{ col0: 'é', col1: '£', col2: '€'}

This is a known issue and currently being worked on in a separate branch, a fix will be merged soon. Again this only affects streams in buffer mode. If you are parsing whole files, or strings it will work as intended.