tcorral / jsonc

JSON compressor and decompressor

JSONC

Update to version 1.6.1

Changelog

Background

One of the problems you can have developing rich internet applications (RIA) using Javascript is the amount of data being transported to and from the server. When data comes from server, this data could be GZipped, but this is not possible when the big amount of data comes from the browser to the server.

JSONC is born to change the way browser vendors think and become an standard when send information to the server efficiently.

JSONC has two differents approaches to reduce the size of the amount of data to be transported:

• JSONC.compress - Compress JSON objects using a map to reduce the size of the keys in JSON objects.
  • Be careful with this method because it's really impressive if you use it with a JSON with a big amount of data, but it could be awful if you use it to compress JSON objects with small amount of data because it could increase the final size.
  • The rate compression could variate from 7.5% to 32.81% depending of the type and values of data.
• JSONC.pack - Compress JSON objects using GZIP compression algorithm, to make the job JSONC uses a modification to use the gzip library and it encodes the gzipped string with Base64 to avoid url encode.
  • Gzip - @beatgammit - https://github.com/beatgammit/gzip-js
  • Base64 - http://www.webtoolkit.info/
  • You can use pack to compress any JSON objects even if these objects are not been compressed using JSONC See Usage for more details.

Usage

Compress a JSON object:

// Returns a JSON object but compressed.
var compressedJSON = JSONC.compress( json );

Decompress a JSON object:

// Returns the original JSON object.
var json = JSONC.decompress( compressedJSON );

Compress a normal JSON object as a Gzipped string:

// Returns the LZW representation as string of the JSON object.
var lzwString = JSONC.pack( json );

Compress a JSON object as a Gzipped string after compress it using JSONC:

// Returns the LZW representation as string of the JSON object.
var lzwString = JSONC.pack( json, true );

Decompress a normal JSON object from a Gzipped string:

// Returns the original JSON object.
var json = JSONC.unpack( gzippedString );

Decompress a JSON compressed object using JSONC from a Gzipped string:

// Returns the original JSON object.
var json = JSONC.unpack( gzippedString, true );

Examples of compression

Example data.js.

Original - 17331 bytes
Compressed using JSONC - 16025 bytes
Compression rate - 7.5%


Original compressed using gzip.js - 5715 bytes
Compressed using JSONC using gzip.js - 5761 bytes


Compression rate from original to compressed using JSONC and gzip.js - 66.76%

Example data2.js.

Original - 19031 bytes
Compressed using JSONC - 12787 bytes
Compression rate - 32.81%


Original compressed using gzip.js - 4279 bytes
Compressed using JSONC using gzip.js - 4664 bytes


Compression rate from original to compressed using JSONC and gzip.js - 75.49%

Next steps

Implement the gzip class in different languages (Java, Ruby...)

jsonc's Issues

compress in php and decompress in Javascript

Is it possible to compress in php backend and decompress in javascript frontend?

Need to escape the regEx characters on line 239

Line 239
str = str.replace(new RegExp('"' + sKey + '"', 'g'), '"' + oKeys[sKey] + '"');

I think you might forget to escape the regEx characters here it should be:

str = str.replace(new RegExp(escapeRegExp('"' + sKey + '"'), 'g'), '"' + oKeys[sKey] + '"');

If not, when trying to decompress, this case /"("/ will corrupt the process. Resulted with error Uncaught SyntaxError: Invalid regular expression: /"("/: Unterminated group

ReferenceError: zlib is not defined

Maximum call size exceeded

I found when I was decompressing a large piece of data in Chrome stable, I would get a 'RangeError: Maximum call size exceeded'. I tracked this down, the problem is in JSONC.unpack, which calls String.fromCharCode.apply() on the decoded data. When the data is very long, you reach the argument limit of the VM. See https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Function/apply

But beware: in using apply this way, you run the risk of exceeding the JavaScript engine's argument length limit. The consequences of applying a function with too many arguments (think more than tens of thousands of arguments) vary across engines...

I patched this locally by wrapping the call in a try/catch, and if the RangeError happens, I loop through the data and call fromCharCode() on each iteration, instead of calling apply- same effect. I will submit a pull request.

Decompressed result is not the same as the raw content

Found a case where the decompressed file is not the same as the original raw JSON content.

Case: The value of the key-value pair is a single character such as "N", however when it gets decompressed it was replaced with an another existing key in the JSON object.

Serious performance issues

I attempted the compression of objects with more than 5k key->value pairs, and the use of arrays and several iterations slows down the processing terribly.

I would suggest to optimize:

• contains, use object instead of array
• unique, same as above, just stick all the keys into an object and then Object.keys()
• _getKeys, use === undefined instead of hasOwnProperty (line 149)
• _compressOther, do not use json.parse/stringify, replace the key names in place without going over the whole string

I wanted to use this for a huge number of objects, but it keeps my node cli running forever with just over 4k keys in my object.

Need python version

Uncaught ReferenceError: global is not defined at eval (eval at zip (jsonc.js:2940), <anonymous>:1:1) at Object.zip (jsonc.js:2940) at Object.JSONC.pack (jsonc.js:3314)

Uncaught ReferenceError: global is not defined
at eval (eval at zip (jsonc.js:2940), :1:1)
at Object.zip (jsonc.js:2940)
at Object.JSONC.pack (jsonc.js:3314)

gzip has no method zip

I've npm installed jsoncomp

but something is wrong...

I can do JSONC.compress and JSONC.decompress they both worked strait away no problem. all I needed to do was

var JSONC=require('jsoncomp/src/JSONC.js') 

But as soon as tired to do JSON.pack it would say Base64 is undefined

So I went into JSONC.js and added

var Base64=null;

  isNodeEnvironment = typeof exports === 'object' && typeof module === 'object' && typeof module.exports === 'object' && typeof require === 'function';
  if(isNodeEnvironment===true){
    Base64=require('../vendor/base64.js');
    }

then it says it the same about gzip, so i do the same with gzip...

then it says gzip has no method zip!?

console.dir says gzip looks like this:

{ gzip:
   { zip: [Function: zip],
     unzip: [Function: unzip],
     DEFAULT_LEVEL: [Getter] } }

What is wrong? Are there instructions to start using this?

Base64 is not defined in Node.js environment

JSONC.js:275

return Base64.encode(String.fromCharCode.apply(String, gzip.zip(str,{level:9})));

Bower complaining that "Package jsoncomp not found"

I put below version to my bower.json file.
"jsoncomp": "1.6.1"

Getting the below error when I do "bower install"
bower ENOTFOUND Package jsoncomp not found

TypeError: crc32 is not a function at Object.zip (jsonc.js:2940) at Object.JSONC.pack (jsonc.js:3314)

TypeError: crc32 is not a function
at Object.zip (jsonc.js:2940)
at Object.JSONC.pack (jsonc.js:3314)

How to handle the packed string on Ruby server side?

If I create a packedString of a JSON object using JSONC as so:

var packedString = JSONC.pack(JSONObject)

Then I send this packedString to a Ruby on Rails server and try to "unpack" it back to the JSON object, I am running into problems.
I've tried in controller action

packed_string = params["packedString"]
gz = Zlib::GzipReader.new(StringIO.new(packed_string)) # errors out here with Zlib::GzipFile::Error: not in gzip format
uncompressed_string = gz.read
json = JSON.parse(uncompressed_string)

I've also tried to decode it first thinking the issue is that it is in base64 encoded format

packed_string = params["packedString"]
decoded_string = Base64.decode64(packed_string)
gz = Zlib::GzipReader.new(StringIO.new(decoded_string)) # errors out here with Zlib::GzipFile::Error: not in gzip format
uncompressed_string = gz.read
json = JSON.parse(uncompressed_string)

But in both cases Zlib::GzipReader doesn't think it is receiving something in a gzip format with the error message: Zlib::GzipFile::Error: not in gzip format

What should be done to properly unpack this on Ruby on Rails serve side?

Not Work with my JSON

Rename "index.txt" to index.js and execute

index.txt

can i use fixed map

First, thank you for this lib.

I will store the json into database, for the compressed json, the values are import, but the map are fixed, I don't need to save them once and once again. I can I use a fixed map when compress ?

It's impossible to have legit one char key without it having the chance of being changed to another value

With this object as source, "G" will get changed to "config" after unpacking because G will be the key used to replace "config" key .
var tt = { config:{ positions:{ F:200, G:200, }, A:{ test:123, test2:"ABC" } } }

results in (after unpacking)

var tt = { config:{ positions:{ F:200, config:200, }, A:{ test:123, test2:"ABC" } } }

Expose a demo

I see potencial on this to be applied widely, but I think a demo is important for the developers accept it as a choice, at least.

After googling "jsonc online" I found:
http://www.javascriptoo.com/jsonc

This isssue is a request to place it on the README or to encourage a demo creation.

Cant run in web app

I tried to run this inside a javascript function on the browser. I'm running react. during the build process I encountered the following error
ERROR in ./node_modules/graceful-fs/graceful-fs.js
Module not found: Error: Can't resolve 'fs' in '(my path)/node_modules/graceful-fs

Is JSONC only meant to be run on the server and not in a web app on the browser?

JSONC gzip compression is not compatible with gzip on Apache server

I have compressed the JSON array using JSONC.pack(json) and sending this array to Apache Server. But Apache Server is not able to recognize the payload and responding with 400 Bad Request.

Bug in vendor base64

The base64 implementation has a somewhat pointless line of code to normalise line endings. This could(and did for me personally) lead to random data corruption. See this comment on StackOverflow, and the other comments on that answer.

I have committed a fix removing the offending line as part of my pull request #3.

Base64 is not defined when calling pack

Base64 is not defined when calling pack