The 2nd+ JPG images in a zip stream being created are corrupt for some reason? about fflate

[101arrowz]

I haven't tried cloning your repo yet but based on that error message I'm pretty sure I know why you're having issues. A "detached ArrayBuffer" is essentially an ArrayBuffer (i.e. the underlying block of memory for a Uint8Array) that has either been invalidated, or been moved to a separate thread or process.

In the context of WebAssembly, this usually happens when you construct a Uint8Array that points to a location in the WASM memory (likely as an output pointer for your WASM code to write the converted image to), but then you have to grow the WASM memory to do some temporary heap allocations. When you grow the WASM memory all pointers to the old memory are invalidated, and their ArrayBuffers detached.

In this case I'd guess that the WASM executed by convertImageWithImageMagick is allocating a new output buffer for the image each time it's called. Since you aren't manually freeing the memory it returns (you'd probably have to call some release function on the returned Uint8Array), the allocator is running out of memory space and is therefore growing the entire WASM memory on each call. In other words, every time you call convertImageWithImageMagick, all its previous return values become detached and therefore break.

This issue gets resolved if you immediately copy the data out of the WASM memory into JavaScript-owned memory (e.g. with out.slice(0), as you observed) because then you're no longer keeping a long-term reference to the WASM memory. But when you use ZipPassThrough directly, fflate directly moves your input to the output stream with zero copies, and here, that input is a Uint8Array that points to WASM memory that gets invalidated after future calls to convertImageWithImageMagick. So when you try to concatenate all the output buffers and download them with new Blob(streams), you're still using those invalid WASM pointers.

My recommendation here is to modify your convertImageWithImageMagick function to return a copy of the WASM memory, rather than a view into WASM memory (e.g. if it's currently returning origBuffer, return origBuffer.slice() instead), and deallocate the original view back using your WASM binary's allocator (maybe your bindings have some sort of free(origBuffer) function?). This should fix your current problem and any future unexpected behavior you could encounter due to that function returning a "temporary" buffer.

Let me know if you need any more clarification or if you have any further issues!

[lancejpollard]

I tried the fflate.zipSync and same error. Also, the files which are corrupt are random it seems, it changes on each download. Maybe I'm doing something wrong?

[101arrowz]

By any chance have you tried this with fflate v0.8.2? I released it just a few hours ago and this issue seems very similar to something I fixed in that release.

[lancejpollard]

I'm using 0.8.1, let me try 0.8.2 then :) Thanks.

[lancejpollard]

Ah now with 0.8.2 I'm getting this when I create the zip with 2 files:

[101arrowz]

Can you send me a repo with this code or at least some examples of data you're compressing here? The corrupted ZIP would help too.

[lancejpollard]

@101arrowz okay I whipped together this demo, which seems to be demonstrating the latest error of the unopenable archive.

https://github.com/lancejpollard/zip-test-with-fflate

I use pnpm install && pnpm dev to start it, then upload 2 or 3 PNGs and it will convert them to JPG with imagemagick-wasm.

Here are the PNGs I used if it makes a difference.

Archive.zip

I noticed before I added imagemagick-wasm just now, if I simply selected PNGs and zipped them without conversion, it worked. SO it could also be something with imagemagick, I'm not sure. Oh I also tried UZIP.js and it worked for more files but still resulted in corrupted images, so if that is also failing, it could also be something I'm doing, not sure.

Thanks for taking a look.

[lancejpollard]

I just tried once using this:

zipFile.push(out.slice(0), true);

And it seems to have not corrupted the zipped images? Could it be something to do with the error you sometimes get with array buffers like:

Cannot perform %TypedArray%.prototype.set on a detached ArrayBuffer

[lancejpollard]

Wow thank you for that! It really clears things up, I was just about to ask about that on the web. This is a really great explanation and makes total sense. I will move the .slice() to inside convertImageWithImageMagick, so it's abstracted away I guess. Thanks again!