Website bug: seems to not be compressing anything about fflate

[101arrowz]

That's odd. I've tried this with multiple massive files and haven't had this problem.

Could you try the "streaming GZIP" preset? It's likely that the browser failed to load the 2.7GB of data into memory. Streaming avoids this problem.

[feross]

The streaming example just hangs:

[101arrowz]

Compressing 2.7GB of data can take around a minute or two, longer on slow computers. I'll run the demo on a large filestream and let you know my findings.

[feross]

Actually, I might have misunderstood the UI -- I'll let it run and report back if it finishes.

[feross]

It looks like the streaming example throws an exception:

Uncaught (in promise) TypeError: Failed to fetch
Promise.then (async)
eval @ VM459:12
Br @ sandbox.ts:144
onClick @ index.tsx:510
j @ preact.module.js:1

[101arrowz]

I found the issue, I'm trying to concatenate the output buffer into an ArrayBuffer, which throws because it's over 2GB, the max in browsers. Will fix by only calculating the length rather than concatenating.

[feross]

Makes sense, thanks for the quick debugging!

[101arrowz]

Try pasting this into the code box:

const { AsyncGzip } = fflate;
// Theoretically, you could do this on every file, but I haven't done that here
// for the sake of simplicity.
const file = files[0];
const gzipStream = new AsyncGzip({ level: 6 });
// We can stream the file through GZIP to reduce memory usage
const gz = file.stream().pipeThrough(toNativeStream(gzipStream));
let sz = 0
gz.pipeTo(new WritableStream({
  write(dat) { sz += dat.length; console.log(sz); },
  close() { callback('Length: ' + sz); }
}));

It took a while but worked for me.

[feross]

Perfect, that worked!