googlechromelabs / json-parse-benchmark

Benchmark comparing JSON.parse vs. equivalent JavaScript literals across JavaScript engines.

JSON.parse benchmark

See the cost of parsing JSON.

Instructions

Clone this repository, and then run these commands:

npm install
export PATH="${HOME}/.jsvu/bin:${PATH}"
npm run bench

Analysis

JetStream2/SeaMonster/inspector-json-payload.js contains a 7.33-MB JavaScript array literal containing 846 object literals. This entire array literal can be serialized to JSON and turned into a 8.2-MB JavaScript string literal, which can be passed to JSON.parse() to produce an equivalent array.

This repository tests each approach in 100 different contexts in the dumbest possible way, i.e. by simply invoking d8 a 100 times per script. That is, we perform 100 cold loads, and measure everything (parsing, compilation, and execution) until the program halts.

On my workstation (HP Z840 with 2 × 14-core Intel Xeon E5-2690 v4 processors @ 2.6GHz, 35MB Cache + 128GB DDR4 2400MHz RAM), I got the following results:

	JS literal	JSON.parse	Speed-up
V8 v7.5	23.765 s	15.766 s	1.5×
V8 v7.6	23.639 s	14.102 s	1.7×
V8 v7.7	23.489 s	13.886 s	1.7×
Chakra v1.11.10.0	24.999 s	16.547 s	1.5×
JavaScriptCore v246878	28.503 s	14.315 s	2.0×
SpiderMonkey v68.0b13	25.554 s	21.174 s	1.2×

For V8 specifically, you can get detailed metrics for a single run by using --runtime-call-stats:

$ v8 --runtime-call-stats out/js.js | grep Parse
                                      ParseProgram    124.33ms  47.00%         1   0.00%

$ v8 --runtime-call-stats out/json.js | grep Parse
                                         JsonParse     47.67ms  28.29%         1   0.07%
                                      ParseProgram     43.17ms  25.62%         1   0.07%

Licensing

The source files in this repository are released under the Apache 2.0 license, as detailed in the LICENSE file.

The scripts in this repository dynamically download JetStream2/SeaMonster/inspector-json-payload.js, which has its own license:

/*
 * Copyright (C) 2018 Apple Inc. All rights reserved.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions
 * are met:
 * 1. Redistributions of source code must retain the above copyright
 *    notice, this list of conditions and the following disclaimer.
 * 2. Redistributions in binary form must reproduce the above copyright
 *    notice, this list of conditions and the following disclaimer in the
 *    documentation and/or other materials provided with the distribution.
 *
 * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
 * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
 * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
 * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
 * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
 * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
 * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
 * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
 * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
 * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
 * THE POSSIBILITY OF SUCH DAMAGE.
*/

This license also applies to the generated *.js files produced by the scripts in this repository.

json-parse-benchmark's Issues

Results can be a bit misleading, as they show the sum of the executions

@mathiasbynens Thank you for the great work, I learned a lot:

1. Inner workings of JSON vs JS literals in JS engines, and yes JSON parsing is faster because of the limited grammar compared to Objects.
2. jsvu, didn't know about that one.

But I have a question regarding methodology. Currently timings are the aggregated results for the hundred runs, e.g.

  printf "Benchmarking JS literal on ${bin}… ";
  time (for i in {1..100}; do $bin out/js.js; done);

So that means, that the stats from the README:

	JS literal	JSON.parse	Speed-up
V8 v7.5	23.765 s	15.766 s	1.5×

could also read:

	JS literal	JSON.parse	Speed-up
V8 v7.5	237.765 ms	157.66 ms	1.5×

So it takes V8 v7.5 either 157 or 237 _milli_seconds for parsing a 8 megabyte file. This is an impressive feature, but one would save "only" 80ms vs the 8 seconds as the README suggests.

README instructions are insufficient

npm run bench fails because none of the applications it is looking for are in the PATH.

sorry, I do not understand this principle of optimality

Test case:

const vm = require('vm')
function run (mark, cb, times = 1000) {
	console.time(mark)
	for (let i = 0; i < times; i++) {
		cb()
	}
	console.timeEnd(mark)
}

run('obj', () => {
	const objScript = new vm.Script(`
		const data = { foo: 42, bar: 1337 }
	`)
	objScript.runInNewContext()
})

run('str', () => {
	const strScript = new vm.Script(`
		const data = JSON.parse('{"foo":42,"bar":1337}')
	`)
	strScript.runInNewContext()
})

After running with nodejs v12.2.0, I got result like:

obj: 1616.828ms
str: 1583.700ms

There seems no obvious difference.