Library stop working after webpack bundle it about fflate

[101arrowz]

Check v0.4.8 on NPM, which should have fixed the issue for your Chrome extension. Thanks for the bug report and especially for your continued help in debugging!

[101arrowz]

Sure, that's fine. I know the files are minified, but I'm still able to (and want to) debug the minified builds with JSBeautifier, searching for function names, etc. This is mainly because I can't reproduce the error locally; I'd like to see exactly where the bug is appearing.

[realrecordzLab]

Sure, that's fine. I know the files are minified, but I'm still able to (and want to) debug the minified builds with JSBeautifier, searching for function names, etc. This is mainly because I can't reproduce the error locally; I'd like to see exactly where the bug is appearing.

Mail sent!

[realrecordzLab]

I've looked at the minified source and it seems that you're using zipSync there because I cannot find the asynchronous zipping function. However, your stack trace and previous issue indicate the use of zip. Could you verify that you've sent the same version of the file you created the error log with?

Sorry, I've replaced the zip function with zipSync in non minified production build, I've mailed you the correct files now.

[101arrowz]

I've extracted the minified version of fflate from your build and run my test suite on it, but everything seems to be working fine. Therefore, the failure must be a difference in environment. Could you describe your operating system and browser (name and version)? If only a specific set of files causes the error, could you send those over too?

[101arrowz]

Yeah that would probably be issue, you can send it to me and I'll take a look.

[realrecordzLab]

I managed to force the exact same error to appear by making the data that the Worker receives null. The question remains why exactly the worker receives the null value. I'm afraid I'll need to take a look at your source code just to kill this bug once and for all.

By the way, the file you sent seems to still work with the minified Webpack build, the unminified Node build, and the minified Parcel build on https://101arrowz.github.io/fflate. Argh!

it's strange because it will not work in my production envoirment. Are you used vue 3? I've sent to you an invitation to the repo with the source code. Let me know

[101arrowz]

Well, I finally was able to recreate the bug. I'll try my best to find what exactly your code does differently from mine that causes fflate to fail.

[101arrowz]

The problem was with the library, not your code, although you did have an unusual use case. The Content Security Policy (CSP) is used to prevent arbitrary code injection and execution from untrusted sources, which can somewhat improve security. Most of the time, I test with a very loose Content Security Policy, which is the default for most environments. However, Chrome extensions are not really allowed to execute anything but their own code, so I had to rewrite one function that used dynamic evaluation to make everything work. I hope the library works well for your codebase!

[realrecordzLab]

The problem was with the library, not your code, although you did have an unusual use case. The Content Security Policy (CSP) is used to prevent arbitrary code injection and execution from untrusted sources, which can somewhat improve security. Most of the time, I test with a very loose Content Security Policy, which is the default for most environments. However, Chrome extensions are not really allowed to execute anything but their own code, so I had to rewrite one function that used dynamic evaluation to make everything work. I hope the library works well for your codebase!

Not so unusual to see a strict CSP on certain websites nowdays, the update to support CSP is a great think trust me. I will link to the extension repo in my app where this project is used. You did a great job. Thank you!

[101arrowz]

I'd like to have access to the code in write-zip.896bce5d.js and chunk-vendors.64883a9e.js for debugging purposes; could you send/pastebin them? Otherwise it's hard to tell what exactly went wrong.

[realrecordzLab]

I'd like to have access to the code in write-zip.896bce5d.js and chunk-vendors.64883a9e.js for debugging purposes; could you send/pastebin them? Otherwise it's hard to tell what exactly went wrong.

Unfortunately the files that cause the error are minified so I don't know if it can be useful for debug. I can mail you the files since pastebin will not accept it's content and mark them as potentially offensive

[101arrowz]

I've looked at the minified source and it seems that you're using zipSync there because I cannot find the asynchronous zipping function. However, your stack trace and previous issue indicate the use of zip. Could you verify that you've sent the same version of the file you created the error log with?

[realrecordzLab]

I've extracted the minified version of fflate from your build and run my test suite on it, but everything seems to be working fine. Therefore, the failure must be a difference in environment. Could you describe your operating system, browser (name and version), and the version of fflate you are using?

No, I haven't used any specific file, I've tested the app I'm working on with some random images and files I have on the pc.
I'm on macOS 10.13.6 and the library is used inside a vue 3 powered browser extension. I'm creating the build using vue ui and all the test that give the error are done on chrome 87 and firefox developer edition 85.0b4. The fflate version is 0.4.6

[101arrowz]

That's frustrating; my environment is virtually identical: operating system, chrome version, and fflate version are all the same. I think the only possible error could be in your loading script. Have you tried running a production build with a hardcoded input to zip? Instead of passing in the result from your FileReader, try this:

zip({
  // this is "hello world"
  'test-file.txt': new Uint8Array([
    104, 101, 108, 108,
    111,  32, 119, 111,
    114, 108, 100
  ])
}, (err, dat) => ...)

If the error persists, I have no idea what's going on and will need a day or two to look into it further. Otherwise, it's a bug in the FileReader script (though I'm totally lost as to how switching from development to production could cause that).

[realrecordzLab]

That's frustrating; my environment is virtually identical: operating system, chrome version, and fflate version are all the same. I think the only possible error could be in your loading script. Have you tried running a production build with a hardcoded input to zip? Instead of passing in the result from your FileReader, try this:

zip({
  // this is "hello world"
  'test-file.txt': new Uint8Array([
    104, 101, 108, 108,
    111,  32, 119, 111,
    114, 108, 100
  ])
}, (err, dat) => ...)

If the error persists, I have no idea what's going on and will need a day or two to look into it further. Otherwise, it's a bug in the FileReader script (though I'm totally lost as to how switching from development to production could cause that).

I will try and let you know

[realrecordzLab]

Tested with the Unit8Array suggested from you and no error occur, I'm able to download the zip file with the test-file.txt iniside.

[101arrowz]

OK, the error is coming from the loading script then. Try copy-pasting this code line-for-line, since it works in my environment:

import { zip } from 'fflate';
function readFile(file) {
  return new Promise((resolve, reject) => {
    const reader = new FileReader();

    reader.onloadend = () => {
      resolve(new Uint8Array(reader.result));
    };
    reader.onerror = reject;
    reader.readAsArrayBuffer(file);
  });
}
async function createZip() {
  // You can replace this with what you need, but it needs to be a
  // FileList object or an array of File objects.
  const files = document.querySelectorAll('input[type="file"]')[1].files;
  const toZip = {};
  for (const file of files) {
    toZip[file.name] = await readFile(file);
  }
  return new Promise((res, rej) => {
    zip(toZip, { consume: true, level: 9 }, (err, dat) => err ? rej(err) : res(dat))
  });
}

[realrecordzLab]

OK, the error is coming from the loading script then. Try copy-pasting this code line-for-line, since it works in my environment:

import { zip } from 'fflate';
function readFile(file) {
  return new Promise((resolve, reject) => {
    const reader = new FileReader();

    reader.onloadend = () => {
      resolve(new Uint8Array(reader.result));
    };
    reader.onerror = reject;
    reader.readAsArrayBuffer(file);
  });
}
async function createZip() {
  // You can replace this with what you need, but it needs to be a
  // FileList object or an array of File objects.
  const files = document.querySelectorAll('input[type="file"]')[1].files;
  const toZip = {};
  for (const file of files) {
    toZip[file.name] = await readFile(file);
  }
  return new Promise((res, rej) => {
    zip(toZip, { consume: true, level: 9 }, (err, dat) => err ? rej(err) : res(dat))
  });
}

What do you mean with loading script? I see that you have called the reader.readAsArrayBuffer() after the loadend, I'm calling it before loadendin my readFile function

    readFile(file) {
      return new Promise( (resolve, reject) => {
        const reader = new FileReader();
        reader.readAsArrayBuffer(file);
        reader.onerror = reject;
        reader.onloadend = () => {
          resolve(new Uint8Array(reader.result));
        }
      });
    }

Also the zip function is wrapped inside a promise, can you explain me why this choice? I have a single function to create the zip and save it, if I wrap the zip inside a Promise I will need to create a dedicated method to generate the zip? I'm editing my code now, I will update you after test in production :)

[101arrowz]

You should be able to make whatever changes you think will work, but just to rule out all the possibilities, it's best to see if this works, then start shifting closer to what you wrote originally. I have a feeling the readFile method was at fault, but I can't be sure so I suggest copying as closely as possible to avoid whatever bug caused the failure in the first place.

[realrecordzLab]

I've modified the code in this way, like you've suggested, inside vue methods:

    handleFiles(e) {
      e.dataTransfer.files.forEach( (file) => {
        this.filesList.push(file.name);
        this.files.push(file);
      });
    },
    readFile(file) {
      return new Promise( (resolve, reject) => {
        const reader = new FileReader();
        reader.onloadend = () => {
          resolve(new Uint8Array(reader.result));
        }
        reader.readAsArrayBuffer(file);
        reader.onerror = reject;
      });
    },
    async processFiles() {
      this.isProcessing = true;
      let archiveContent = {};
      for(const file of this.files){
        let fileBuffer = await this.readFile(file);
        archiveContent[file.name] = fileBuffer;
      }
      //debug
      const zipFile = await this.createZip(archiveContent);
      console.log(zipFile);

      
      //production
      // const data = zipSync(archiveContent, {consume: true, level: 9});
      // const chance = new Chance();
      // saveAs(new Blob([data.buffer]), `${chance.hash({length: 6})}.zip`);
      // this.isProcessing = false;
      // this.$router.push({path: '/'});
    },
    //debug 
    createZip(fileBuffer) {
      return new Promise((resolve, reject) => {
        zip(fileBuffer, { consume: true, level: 9 }, (err, dat) => err ? reject(err) : resolve(dat));
      });
    }

The error still remain

chunk-vendors.080bcd71.js:6 Uncaught TypeError: Cannot read property 'length' of null
    at A (chunk-vendors.080bcd71.js:6)
    at chunk-vendors.080bcd71.js:6
    at Worker.s.onerror (chunk-vendors.080bcd71.js:6)
A @ chunk-vendors.080bcd71.js:6
(anonymous) @ chunk-vendors.080bcd71.js:6
s.onerror @ chunk-vendors.080bcd71.js:6
error (async)
r @ chunk-vendors.080bcd71.js:6
X @ chunk-vendors.080bcd71.js:6
re @ chunk-vendors.080bcd71.js:6
ce @ chunk-vendors.080bcd71.js:6
h @ chunk-vendors.080bcd71.js:6
Ae @ chunk-vendors.080bcd71.js:6
(anonymous) @ write-zip.547e918f.js:1
createZip @ write-zip.547e918f.js:1
(anonymous) @ write-zip.547e918f.js:1
u @ chunk-vendors.080bcd71.js:8
(anonymous) @ chunk-vendors.080bcd71.js:8
(anonymous) @ chunk-vendors.080bcd71.js:8
a @ chunk-vendors.080bcd71.js:1
s @ chunk-vendors.080bcd71.js:1
Promise.then (async)
a @ chunk-vendors.080bcd71.js:1
s @ chunk-vendors.080bcd71.js:1
Promise.then (async)
a @ chunk-vendors.080bcd71.js:1
s @ chunk-vendors.080bcd71.js:1
Promise.then (async)
a @ chunk-vendors.080bcd71.js:1
s @ chunk-vendors.080bcd71.js:1
Promise.then (async)
a @ chunk-vendors.080bcd71.js:1
s @ chunk-vendors.080bcd71.js:1
(anonymous) @ chunk-vendors.080bcd71.js:1
(anonymous) @ chunk-vendors.080bcd71.js:1
processFiles @ write-zip.547e918f.js:1
(anonymous) @ write-zip.547e918f.js:1
(anonymous) @ chunk-vendors.080bcd71.js:6
Ee @ chunk-vendors.080bcd71.js:6
Re @ chunk-vendors.080bcd71.js:6
t @ chunk-vendors.080bcd71.js:6

[101arrowz]

This is the strangest bug I've seen in a while, especially since it doesn't appear for me. Try removing consume: true from the ZIP options; if that still fails, just share the source code and I'll try my best to recreate the bug.

[realrecordzLab]

@101arrowz I've done some test with the consume: true and without it, with some single files that vary from pdf and text documents to images and I've noticed that the error will occur when I try to zip a particular file, a screen taken with macOS that is a PNG. All the tested files were read with success and the zip Unit8Array is logged in console correctly. I assume that the file is causing the error? I can send it to you if want to give a try and investigate about ?

[101arrowz]

I managed to force the exact same error to appear by making the data that the Worker receives null. The question remains why exactly the worker receives the null value. I'm afraid I'll need to take a look at your source code just to kill this bug once and for all.

By the way, the file you sent seems to still work with the minified Webpack build, the unminified Node build, and the minified Parcel build on https://101arrowz.github.io/fflate. Argh!

[realrecordzLab]

Well, I finally was able to recreate the bug. I'll try my best to find what exactly your code does differently from mine that causes fflate to fail.

Ok, send me a mail with details if you find something wrong in the code. For curiosity, do you have used the same image that give me the problem or another file? Thank you for the help.

[101arrowz]

Whew, that was quite the debugging experience! I finally found that the bug was the use of new Function() in fflate to dynamically create the web workers in the asynchronous version. Although this works in most situations, Chrome extensions block this kind of call by default due to the content security policy, which bans unsafe-eval. I'm going to update fflate to no longer need new Function.

The reason some files worked and others didn't was that fflate only uses the worker version for larger files. The problem occurred for all files that were sufficiently large, and only appeared in production because the Content Security Policy is loosened in development.

[realrecordzLab]

Check v0.4.8 on NPM, which should have fixed the issue for your Chrome extension. Thanks for the bug report and especially for your continued help in debugging!

You are welcome!If you want to collaborate for some projects it will be nice. What caused the bug, is an error inside my code or other things?