foliojs / brotli.js Goto Github PK

I'm using brotli with react version 16.13. By just import brotli in react component file shows this error.
This is the error stack:

Uncaught TypeError: Cannot read property 'slice' of null
    at encode.js:11
    at Object.<anonymous> (encode.js:11)
    at Object../node_modules/brotli/build/encode.js (encode.js:56)
    at __webpack_require__ (bootstrap:784)
    at fn (bootstrap:150)
    at Object../node_modules/brotli/compress.js (compress.js:1)
    at __webpack_require__ (bootstrap:784)
    at fn (bootstrap:150)
    at Object../node_modules/brotli/index.js (index.js:1)
    at __webpack_require__ (bootstrap:784)
    at fn (bootstrap:150)
    at Module../src/actions/appActions.js (role-resource-constants.js:91)

Brotli version - 1.3.2

Because the Emscripten compile is really huge, which sucks for browsers, and because it uses a ton of memory on Windows. See foliojs/fontkit#11.

I'm trying to build a rollup bundle which includes this library, and getting the following failure:

Circular dependency: 
    node_modules/brotli/dec/decode.js -> 
    node_modules/brotli/dec/dictionary.js -> 
    node_modules/brotli/dec/dictionary-browser.js -> 
    node_modules/brotli/dec/decode.js

I'm passing in a buffer and getting null

standalone or not, it returns null

const compressed = compress(buffer);
const compressed2 = brotli.compress(buffer);

The buffer logs as a bugger before this, then no errors or anything but returns null

Apologies if this is not the proper place to report this issue, but I'm still not completely sure of what the problem is and I don't use bortli directly (it was added to my project as a dependency of a dependency).

I have an uncaughtException handler on my project that is supposed to log errors and all that, and I noticed that it's not being called anymore. Upon debugging, I noticed that an additional exception handler is being added to process, and it seems to come from brotli:

These lines seem to be interfering with my other error handlers, not letting the error reach them. If I clear all the error listers, and re-add mine, issue goes away. Can someone please help?

There's a Browser.T referenced in the emscripten runtime output of the current published build that doesn't refer to anything as far as I can tell, causing the output of either browserify or webpack to produce the error Uncaught ReferenceError: Browser is not defined.

Rebuilding the library fixes this, but adds an error that encode.js.mem can't be found. Adding Module['readAsync']=function(_,f){f(Module['readBinary']())}; seems to fix this, but then I get Uncaught Error: [ReadHuffmanCode] invalid num_codes or space.

You'd lose the benefits of compressing the memfile, but it may be worth considering switching to emscripten's upcoming SINGLE_FILE flag that does base64 encoding if there's no other workaround. (Is there actually a space advantage to compressing it at this stage vs leaving it to be brotli'd/gzip'd at the same time as the rest of the JS in the user's build pipeline? I'd expect the latter to at least be slightly more CPU-efficient since it'd be decompressed by native code.)

Misc issues:

• The bro command in the makefile no longer works with the latest brotli; the equivalent command is now brotli -c. (Alternatively, there's always ./vendor/brotli/bin/bro.)

• NO_BROWSER has been removed from emscripten.

I'm using brotli 1.2.0, with this simple file:

var b = require('brotli');

Executing the file with node yields no errors, but when the same command is executed in the node REPL, brotli fails with an exception:

$ node --version
v4.5.0

$ cat test.js
var b = require('brotli');

$ node test.js

$ node
> .load test.js
> var b = require('brotli');
TypeError: Cannot read property 'replace' of undefined
    at Object.<anonymous> (/Users/dbruhn/repos/brotliTest/node_modules/brotli/build/encode.js:2:461)
    at Module._compile (module.js:409:26)
    at Object.Module._extensions..js (module.js:416:10)
    at Module.load (module.js:343:32)
    at Function.Module._load (module.js:300:12)
    at Module.require (module.js:353:17)
    at require (internal/module.js:12:17)
    at Object.<anonymous> (/Users/dbruhn/repos/brotliTest/node_modules/brotli/compress.js:1:76)
    at Module._compile (module.js:409:26)
    at Object.Module._extensions..js (module.js:416:10)
>

Any idea why brotli would act differently in the REPL?

Would be nice to have a brotli binary that you could pipe to.

https://github.com/devongovett/brotli.js/blob/e974146a5a38c8ff00f7866a05a238a6a334012d/dec/dictionary-browser.js requires base64-js, but the dependency is not declared in package.json and it is not a core node module. I think this might be contributing to problems using PDFkit 0.8.0 in the browser.

Running npm install brotli, and then running browserify index.js -o brotli.browser.js -s Brotli yields a file that cannot be used in the browser, citing:

[warning] TypeError: asm.js type error: Disabled by debugger
[error] ReferenceError: Browser is not defined (brotli.browser.js:703:191)

I found 2 license-related problems:

• missing LICENSE file at top of repo
  • it should contain your copyright info
  • you've indicated MIT: https://opensource.org/licenses/MIT
• missing LICENSE file in the release tarball (under https://registry.npmjs.org/)

$ node
> var brotli = require('brotli');
undefined
> brotli.compress(Buffer.from([255, 255, 255]))
null
> brotli.compress(Buffer.from([255, 255, 255, 255, 255, 255, 255, 255, 255]))
null
> brotli.compress(Buffer.from([255, 255, 255, 255, 255, 255, 255, 255, 255, 255]))
Uint8Array [ 27, 9, 0, 0, 38, 254, 227, 145, 64, 0 ]

I need to have 10 bytes or more in order to compress the buffer. I was unable to find any documentation regarding this as a limitation of the Brotli algorithm, so I assume it's a bug.

As a workaround, I have to pad my data just to get it to compress.

There did not much happen since the last release

v1.3.2...master

But I have seen much slower compression compared to the iltorb package.
I think we should focus on one package (iltorb) and deprecate this one as the other one gets regular updates, has much better performance and is actively maintained.

Any plans to update npmjs w/ the latest?

Is there a way to compress json files in my angular code?
I tried Browserify but some how the compress doesnt work.
Could be related to issue #20

It would be very helpful to include a pre-built standalone decompressor (and compressor?) for use in browsers #2 , as not everyone is familiar with Node.js.

Here is a unbrotli.js and unbrotli.min.js for use in browsers: unbrotli.zip
unbrotli.min.js is a 92Kb download, or 59Kb if Brotli-compressed by your web server. Much of that is the base64-encoded dictionary.

Called with:

unbrotli(buffer, [outSize])

I built it with the following commands (Ubuntu 16.04):

apt-get install nodejs npm
sed -i "s/dictionary-data/dictionary-browser/" dec/dictionary.js
ln -s /usr/bin/nodejs /usr/bin/node
npm install -g browserify
npm install base64-js
browserify decompress.js --standalone unbrotli -o unbrotli.js

(I couldn't get tinyify or uglify to run without errors, so I used a separate Javascript-minifier.)

I'm getting this error Output buffer is not large enough and I can't figure out why.
The code is

const br = require('brotli');

let decodedData = br.decompress(content);
const buf = new Buffer(decodedData);

callback(buf.toString('utf-8'));
return;

This doesn't always happen. After a few retry with the same content it works.
Any ideas why this is happening?

Thank you

Hello!

Aliasify is complaining about const declarations. I suppose you used const for a reason and will keep it?

browserify node_modules/brotli/dec/decode.js -g aliasify -o test.js

SyntaxError: Unexpected token (26:6) (while aliasify was processing node_modules/brotli/dec/decode.js) while parsing file: node_modules/brotli/dec/decode.js

Thank you.

Would there be code size and decompression speed benefits if the decompressor was written in WebAssembly?

I get the error Array buffer allocation failed on machine that used for continious integration.
Machine parameters: Windows10, 1GB, 1 CPU.

I reseached code and found the strange code in build/encode.js:

for (var I, P, J, ta, K, ua, L, N, va = 0, B = 0, wa = 0, z = 0, xa = 0, ya = 0, C = 0, za = b.TOTAL_STACK || 5242880, D = b.TOTAL_MEMORY || 318767104, R = 65536; R < D || R < 2 * za; )
	R = 16777216 > R ? 2 * R : R + 16777216;
R !== D && (D = R);
assert("undefined" !== typeof Int32Array && "undefined" !== typeof Float64Array && !!(new Int32Array(1)).subarray && !!(new Int32Array(1)).set, "JS engine does not provide full typed array support");
var buffer;
buffer = new ArrayBuffer(D); //D equals 318767104 - 304 * 1024 * 1024
...

Why does your module allocate in loop 304 Mb?
It is very strange because V8 has limit 512 Mb for x86.

Please look at bpampuch/pdfmake#935

hello!
i found vulnerability in brotli module.
where can I report this?

Is there a way to use the decompressor part in browser?

tried requiring the decompress function and got an error

ERROR in ./node_modules/brotli/dec/dictionary-browser.js 2:9-22
Module not found: Error: Can't resolve 'fs' in 'C:\redacted\node_modules\brotli\dec'
 @ ./node_modules/brotli/dec/dictionary.js 18:11-39
 @ ./node_modules/brotli/dec/decode.js 19:23-46
 @ ./node_modules/brotli/decompress.js 1:0-63
 @ ./redacted.js
 @ ./redacted.js

i am using webpack 5.

im pretty sure that #22 solved this but its not released.

I've worked up a full solution for testing out Brotil: https://github.com/replete/brotli-experiments

However, the script complains about node's process being missing.

Have you had this working in browserify at all?

Brotli could be used with custom dictioanry instead of the hard-coded static dictionary, but didn't find a good way to plugin to this js decoder. Could we enable this?

Haven't worked how exactly this happens but here is the code:

const brotli = require('brotli');

  try {
    result = brotli.compress(data, {
      mode: 1, // 0 = generic, 1 = text, 2 = font (WOFF2)
      quality, // 0 - 11
      lgwin: 22, // window size
    });
    result = Buffer.from(result);
  } catch (e) {
    console.error('error: ', e);
    console.log('data: ', data);
    console.log('typeof data: ', typeof data);
    console.log('isBuffer?: ', Buffer.isBuffer(data));
    result = data;
  }

I get error: The first argument must be one of type string, Buffer, ArrayBuffer, Array, or Array-like Object. Received type object Error

so I added some more logging to double check what's coming to this compress function. The output:

data:  <Buffer 7b 22 6d 22 3a 22 50 4c 41 4e 5f 43 48 41 4e 47 45 22 2c 22 64 22 3a 7b 22 70 6c 61 6e 4e 65 77 22 3a 22 73 74 61 72 74 22 2c 22 70 6c 61 6e 4f 6c 64 ... >
typeof data:  object
isBuffer?:  true

After the failure, I continue using the result, which becomes uncompressed Buffer and just saves it to the database as binary data. Buffer seems to be perfectly fine. While I also can get it from the database, convert to string, parse JSON out of it and use as intended.

Nodejs version v10.8.0
npm brotli: 1.3.2

This is how i build this data variable:

    let data = {
        m: message || '',
        d: details || '',
        t: moment().unix(),  // when created timestamp
      };

      data = JSON.stringify(data);
      data = Buffer.from(data, 'utf8');

Thank you for the very interesting project!
I want to integrate decoder into web application with large compressed stream >100MB
Right now I am fetching data using chunk-by-chunk.
Can decoder be modified to accept chunks?
More importantly, can decoder produce output data in chunks?

An example of mine: a javascript-file that was 485kb before compression is 299kb after compression with brotli.js. The same file is 94.4kb when compressed 'on-the-fly' with ngx_brotli. I'm using quality 11 compression.

Steps to reproduce would be to install ngx_brotli on an nginx server, and compare file-sizes with the result of the following:

var fs = require('fs')
var brotli = require('brotli')
var result = brotli.compress(someFile, { quality: 11 })
fs.writeFile(someDir, result)

I have tried with Content-Type: br and Content-Type: text/html, aswell as both with and without the Accept-Encoding: br header.

Ever attempt has resulted in the browser only displaying the compressed html as a string without decompressing it at all. Happens in both latest version of chrome and firefox.

const pageFormat = brotli.compress( fs.readFileSync( './html/index.html', { encoding: 'utf8' } ) )

app.get( '/', ( req, res ) => {
	res.setHeader( 'Content-Type', 'text/html' )
	res.setHeader( 'Accept-Encoding', 'br' )
	res.end( pageFormat )
} )

Hi, is the quality option in the compress function OK?

https://github.com/foliojs/brotli.js/blob/master/compress.js#L18=

  } else if (typeof opts === 'object') {
    quality = opts.quality || 11;
    mode = opts.mode || 0;
    lgwin = opts.lgwin || 22;
  }

README says quality could be 0, but then it should turn into 11. My tests confirm that 0 works like 11.

Where is a mistake? I am extremely sorry if I messed up something

We use the decompress.js to decode string like Arabic, Russian, Bulgarian, Greek , the decoded string is invalid.

The encoder, in some scenarios, produce an incompatible output for the brotli 1.0.4 decoder C version. Need a lib upgrade and rebuild?

Same data on the C version works correctly.

I have a code snippet:

var data = new Buffer('This is a string');
var compressed;
var decompressed;

 var brotli = require("brotli");
 compressed = brotli.compress(data);
 decompressed = brotli.decompress(compressed);

 console.log('brotli:');
 console.log(JSON.stringify(decompressed));
 console.log(decompressed.toString());

 var iltorb = require('iltorb');
 compressed = iltorb.compressSync(data);
 decompressed = iltorb.decompressSync(compressed);
 console.log('iltorb:');
 console.log(JSON.stringify(decompressed));
 console.log(decompressed.toString());

Then I run this code the I get output:

brotli:
{"0":84,"1":104,"2":105,"3":115,"4":32,"5":105,"6":115,"7":32,"8":97,"9":32,"10":115,"11":116,"12":114,"13":105,"14":110,"15":103}
84,104,105,115,32,105,115,32,97,32,115,116,114,105,110,103
iltorb:
{"type":"Buffer","data":[84,104,105,115,32,105,115,32,97,32,115,116,114,105,110,103]}
This is a string

If I change code to:

var brotli = require("brotli");
 compressed = brotli.compress(data);
 decompressed = brotli.decompress(compressed);

 decompressed = new Buffer(decompressed);
 console.log('brotli:');
 console.log(JSON.stringify(decompressed));
 console.log(decompressed.toString());

>brotli:
>{"type":"Buffer","data":[84,104,105,115,32,105,115,32,97,32,115,116,114,105,110,103]}
>This is a string

then all works right