nvdnkpr / jparser

Parsing binary files made easy

jParser - Parsing binary files made easy.

jParser makes it easy to parse binary files in Javascript.

• You write the structure once, it gets parsed automatically.
• The parsing process can be extended with custom functions. It allows to parse non trivial files with ease.
• It works both in the browser and NodeJS as it is powered by jDataView.

API

Primitive Structures:

• Unsigned Int: uint8, uint16, uint32
• Signed Int: int8, int16, int32
• Float: float32, float64
• String: char, string(len)
• Array: array(type, len)
• BitField: (bitCount)
• Position: tell, skip(len), seek(pos), seek(pos, func)
• Conditionals: if(predicate, type)

jParser Methods:

• parse(value): Run the parsing, can be used recursively.
  • Number: Reads bitfield of given length in left-to-right mode and returns them as unsigned integer (so you can work with them using simple JavaScript binary operators). Please note that you can mix bitfields with primitive and complex types in one structure or even use them in own functions, but ALWAYS make sure that consecutive bitfields are padded to integer byte count (or 8*N bit count) before reading any other data types; most popular data formats already follow this rule but better to check out when writing own structures if you don't want to get unexpected behavior.
  • Function: Calls the function.
  • String: Dereferences the value in the structure.
  • Array: Function call, the function is the first element and arguments are the following.
  • Object: Returns an object with the same keys and parses the values.
• tell(): Return the current position.
• skip(count): Advance in the file by count bytes.
• seek(position): Go to position.
• seek(position, callback): Go to position, execute the callback and return to the previous position.
• current: The current object being parsed. See it as a way to use what has been parsed just before.

jParser Constructor:

• new jParser(data, structure)
  • data is a jDataView. You can give pretty much anything (String, ArrayBuffer, Node Buffer), it will be casted to jDataView automatically.
  • structure is an object with all the defined structures.

Examples

Basic C Structure You have the ability to define C-like structures. It's a Javascript object where keys are labels and values are types.

var parser = new jParser(file, {
  header: {
    fileId: 'int32',
    recordIndex: 'int32',
    hash: ['array', 'uint32', 4],
    fileName: ['string', 256],
    version: 2,
    flags: {
      precisionFlag: 1,
      marker: {
       part1: 2,
       part2: 2
      }
    },
    _reserved: 1 // padding to 8*N bits
  }
});
parser.parse('header');
// {
//   fileId: 42,
//   recordIndex: 6002,
//   hash: [4237894687, 3491173757, 3626834111, 2631772842],
//   fileName: ".\\Resources\\Excel\\Items_Weapons.xls",
//   version: 3,
//   flags: {
//     precisionFlag: 1,
//     marker: {
//       part1: 2,
//       part2: 0
//     }
//   },
//   _reserved: 0
// }

References Structures can reference other structures. Use structure name within a string in order to reference it. The following is an example from World of Warcraft model files.

nofs: {
  count: 'uint32',
  offset: 'uint32'
},
 
animationBlock: {
  interpolationType: 'uint16',
  globalSequenceID: 'int16',
  timestamps: 'nofs',
  keyFrame: 'nofs'
},
 
uvAnimation: {
  translation: 'animationBlock',
  rotation: 'animationBlock',
  scaling: 'animationBlock'
}

Helpers It is really easy to make new primitive types. You can either use existing constructions such as objects (float3) or arrays (float4). In case you want to do something more complicated, you always have the option to define a new function and use this.parse to keep parsing (hex32, string0).

float3: {
  x: 'float32',
  y: 'float32',
  z: 'float32'
},
float4: ['array', 'float32', 4],
hex32: function () {
  return '0x' + this.parse('uint32').toString(16);
},
string0: function (length) {
  return this.parse(['string', length]).replace(/\0+$/g, '');
}

Back Reference Instead of using an integer for the array size, you can put a function that will return an integer. In this function, you can use this.current to reference the englobing object being parsed.

image: {
  width: 'uint8',
  height: 'uint8',
  pixels: [
    'array',
    ['array', 'rgba', function () { return this.current.width; }],
    function () { return this.current.height; }
  ]
}

Advanced Parsing The best part of jParser is that complicated parsing logic can be expressed within the structure. It allows to parse complex files without having to split structure from parsing code.

entryHeader: {
  start: 'int32',
  count: 'int32'
},

entry: function (type) {
  var that = this;
  var header = this.parse('entryHeader');

  var res = [];
  this.seek(header.start, function () {
    for (var i = 0; i < header.count; ++i) {
      res.push(that.parse(type));
    }
  });
  return res;
},

name: {
 language: 'int32',
 text: ['string', 256]
},

file: {
  names: ['entry', 'name']
}

Get Started

NodeJS: Just use npm to install jParser and you are set :)

npm install jParser

var fs = require('fs');
var jParser = require('jParser');

fs.readFile('file.bin', function (err, data) {
  var parser = new jParser(data, {
    magic: ['array', 'uint8', 4]
  });
  console.log(parser.parse('magic'));
});

Browser: I've patched jQuery to allow to download binary files using the best binary format. You include this patched jQuery, jDataView and jParser and you are set :)

<script src="https://raw.github.com/vjeux/jDataView/master/jquery/jquery-1.7.1-binary-ajax.js"></script>
<script src="https://raw.github.com/vjeux/jDataView/master/src/jdataview.js"></script>
<script src="https://raw.github.com/vjeux/jParser/master/src/jparser.js"></script>

<script>
$.get('file.bin', function (data) {
  var parser = new jParser(data, {
    magic: ['array', 'uint8', 4]
  });
  console.log(parser.parse('magic'));
}, 'dataview');
</script>

Caveats

This tool works thanks to a feature that is not in the Javascript specification: When you iterate over an object keys, the keys will be listed in their order of insertion. Note that Chrome and Opera do not respect this implicit rule for keys that are numbers.

If you follow those two rules, the library will work in all the current Javascript implementations.

• Do not start a key name with a digit
• Do not put the same key twice in the same object

Demos

ICO Parser. This is a basic example to parse a binary file in NodeJS. It shows how to solve many common issues with binary file parsing.

• ico.js: jParser structure.
• ico.json: parsed file.

Tar Extractor. This is a basic example to parse a binary file in the browser.

• tar.html: jParser structure.

World of Warcraft Model Viewer. It uses jParser to read the binary model and then WebGL to display it.

• m2.js: jParser structure.
• model.json: parsed file.

Diablo 3 Internal Files.

• convert.coffee: jParser structure. CoffeeScript makes it even easier to write file structure.
• Example of parsed files:
  • Items_Weapons.json
  • Quest/ProtectorOfTristam.json
  • TreasureClass/SkeletonKing