luizperes / simdjson_nodejs

Node.js bindings for the simdjson project: "Parsing gigabytes of JSON per second"

simdjson_nodejs

Node.js bindings for simdjson, a gigabytes-per-second JSON parser, possibly the fastest JSON parser at the moment. simdjson as well as simdjson_nodejs work on popular platforms such as OS X, Linux, and Windows.

Installation

The installation can be done in one step with npm:

npm install simdjson

Usage

API Documentation

See index.d.ts for available simdjson binding functions and their input/output types.

Check if a JSON string is valid:

const simdjson = require('simdjson');

const jsonString = "{ \"answer\": 42 }";
const valid = simdjson.isValid(jsonString); // true

Parsing a JSON string (lazily)

Obs.: Please see that the overhead of converting a C++ object to a JS object might make the parsing time in the NodeJS slower for the simdjson. Therefore, parsing it lazily is preferrable. For more information check issue #5.

const simdjson = require('simdjson');

const jsonString = "{   \
  \"foo\": {            \
    \"bar\": [ 0, 42 ]  \
  }                     \
}";
const JSONbuffer = simdjson.lazyParse(jsonString); // external (C++) parsed JSON object
console.log(JSONbuffer.valueForKeyPath("foo.bar[1]")); // 42

Parsing a JSON string

Obs.: Parsing a JSON lazily is preferrable.

const simdjson = require('simdjson');

const jsonString = "{   \
  \"foo\": {            \
    \"bar\": [ 0, 42 ]  \
  }                     \
}";
const parsedJSON = simdjson.parse(jsonString); // parsed JSON object

Benchmarks

Requirements: git, npm, node, a linux-like system, a recent compiler (GCC,clang)

To benchmark directly from the project:

git clone https://github.com/luizperes/simdjson_nodejs.git
cd simdjson_nodejs
npm install
npm run benchmark

filename	filesize (MB)	JSON.parse(ms)	simdjson.lazyParse (ms)	JSON.parse (GB/s)	simdjson.lazyParse (GB/s)	X faster
apache_builds.json	0.13	0.303	0.158	0.42	0.80	1.91
canada.json	2.25	14.332	7.584	0.16	0.30	1.89
citm_catalog.json	1.73	5.387	5.768	0.32	0.30	0.93
github_events.json	0.07	0.192	0.091	0.34	0.71	2.10
gsoc_2018.json	3.33	5.420	3.839	0.61	0.87	1.41
instruments.json	0.22	0.673	0.619	0.33	0.36	1.09
marine_ik.json	2.98	13.169	6.397	0.23	0.47	2.06
mesh_pretty.json	1.58	5.704	3.043	0.28	0.52	1.87
mesh.json	0.72	2.856	1.404	0.25	0.52	2.03
numbers.json	0.15	0.643	0.280	0.23	0.54	2.30
random.json	0.51	1.914	2.447	0.27	0.21	0.78
sf_citylots.json	189.78	1492.166	709.692	0.13	0.27	2.10
twitter.json	0.63	1.621	2.112	0.39	0.30	0.77
twitterescaped.json	0.56	1.924	0.959	0.29	0.59	2.01
update_center.json	0.53	2.803	2.715	0.19	0.20	1.03

Results from a 2018 MacBook Pro with 2.3GHz Intel Core i9.

Ops/sec

apache_builds.json#simdjson x 6,844 ops/sec ±1.61% (90 runs sampled) => 0.146ms
apache_builds.json#JSON x 3,274 ops/sec ±1.83% (90 runs sampled) => 0.305ms
canada.json#simdjson x 183 ops/sec ±7.69% (72 runs sampled) => 5.475ms
canada.json#JSON x 71.87 ops/sec ±0.92% (74 runs sampled) => 13.914ms
citm_catalog.json#simdjson x 155 ops/sec ±2.95% (72 runs sampled) => 6.470ms
citm_catalog.json#JSON x 207 ops/sec ±59.63% (90 runs sampled) => 4.820ms
github_events.json#simdjson x 11,783 ops/sec ±1.80% (90 runs sampled) => 0.085ms
github_events.json#JSON x 120 ops/sec ±192.65% (92 runs sampled) => 8.310ms
gsoc_2018.json#simdjson x 307 ops/sec ±1.81% (79 runs sampled) => 3.255ms
gsoc_2018.json#JSON x 166 ops/sec ±47.75% (81 runs sampled) => 6.029ms
instruments.json#simdjson x 3,691 ops/sec ±2.06% (87 runs sampled) => 0.271ms
instruments.json#JSON x 2,155 ops/sec ±0.83% (93 runs sampled) => 0.464ms
marine_ik.json#simdjson x 140 ops/sec ±3.08% (73 runs sampled) => 7.133ms
marine_ik.json#JSON x 51.20 ops/sec ±60.81% (76 runs sampled) => 19.533ms
mesh_pretty.json#simdjson x 504 ops/sec ±1.93% (87 runs sampled) => 1.985ms
mesh_pretty.json#JSON x 251 ops/sec ±1.02% (85 runs sampled) => 3.987ms
mesh.json#simdjson x 733 ops/sec ±0.99% (92 runs sampled) => 1.363ms
mesh.json#JSON x 349 ops/sec ±0.64% (89 runs sampled) => 2.869ms
numbers.json#simdjson x 4,099 ops/sec ±1.10% (94 runs sampled) => 0.244ms
numbers.json#JSON x 1,372 ops/sec ±38.74% (95 runs sampled) => 0.729ms
random.json#simdjson x 504 ops/sec ±0.97% (91 runs sampled) => 1.983ms
random.json#JSON x 538 ops/sec ±0.93% (91 runs sampled) => 1.859ms
sf_citylots.json#simdjson x 1.76 ops/sec ±4.02% (9 runs sampled) => 569.580ms
sf_citylots.json#JSON x 0.53 ops/sec ±20.38% (6 runs sampled) => 1880.563ms
twitter.json#simdjson x 478 ops/sec ±0.68% (89 runs sampled) => 2.093ms
twitter.json#JSON x 612 ops/sec ±0.90% (91 runs sampled) => 1.633ms
twitterescaped.json#simdjson x 1,203 ops/sec ±1.80% (88 runs sampled) => 0.831ms
twitterescaped.json#JSON x 488 ops/sec ±2.33% (82 runs sampled) => 2.051ms
update_center.json#simdjson x 469 ops/sec ±3.43% (85 runs sampled) => 2.132ms
update_center.json#JSON x 293 ops/sec ±12.87% (57 runs sampled) => 3.410ms

Observation:

Please refer to the the original repository benchmarks for more information about the performance of simdjson https://github.com/lemire/simdjson.

simdjson_nodejs's Issues

Parsing time is dominated by makeJSONObject (in the wrapper)

At the C++ level, isValid and parse do exactly the same work. This is obvious from the code in this wrapper:

bool simdjson::isValid(std::string_view p) {
  ParsedJson pj = build_parsed_json(p);
  return pj.isValid();
}

Versus

Napi::Object simdjson::parse(Napi::Env env, std::string_view p) {
  ParsedJson pj = build_parsed_json(p);
  if (!pj.isValid()) {
    Napi::Error::New(env, "Invalid JSON Exception").ThrowAsJavaScriptException();
  }
  ParsedJson::iterator pjh(pj);
  return simdjson::makeJSONObject(env, pjh).As<Napi::Object>();
}

However, in simdjson_nodejs, parse is at least 20x slower than isValid. This indicates that the running time is entirely dependent on makeJSONObject.

This function is written in a sensible manner but it is nevertheless quite slow. Evidently, this defeats the purpose of the fast parsing.

I would say it is a priority to fix this performance issue. It seems that there would be different valid ways to go about it.

Update bindings to simdjson 0.3

Once simdjson is updated to its version 0.3, we no longer will need to pass flags such as the -mavx2. That will fix issues #3 and #23

Keep the parser alive between parses

The simdjson parser has some allocation to do when it is initialized, and currently simdjson_nodejs recreates the parser every single time.

We could use napi's napi_set/get_instance_data to store an instance of the parser for each JS worker thread, keeping that internal memory around. We would have to extract the document from the parser at the end of the parse with std::move, but it would get rid of the internal buffer allocation and even keep them hot--which seems likely to more than counterbalance any performance degradation we'd get from said extraction.

If we could tell whether the user kept any instances of the document around when they call parse() again, we could even use a copy-on-write scheme and only move the document away if the user calls parse() again while there are still live document instances in JS (many people just read a doicument extract what they want, and throw away the JSON before doing anything else, so it would be a win for a lot of cases)..

@lemire ^^ relevant to simdjson's ability to work well with bindings.

Debian Stretch 9.11 Docker Build Fails

I try to run npm i sindjson and get the following error. In the main project you advice to use the gcc-8 so i tried to install it (gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516 is currently installed). The issue here is that I could not figure out how to get the experimental version to work. I am still new to linux...

node@1e2ecaefc3ce:/workspaces/node-ms/test/json$ npm i simdjson

> [email protected] install /workspaces/node-ms/test/json/node_modules/simdjson
> node-gyp rebuild

make: Entering directory '/workspaces/node-ms/test/json/node_modules/simdjson/build'
make: Warning: File 'simdjson.target.mk' has modification time 48042 s in the future
  CXX(target) Release/obj.target/simdjson-avx2/simdjson/main.o
In file included from ../simdjson/bindings.h:3:0,
                 from ../simdjson/main.cpp:3:
../simdjson/src/simdjson.h:36731:23: fatal error: string_view: No such file or directory
 #include <string_view>
                       ^
compilation terminated.
simdjson-avx2.target.mk:113: recipe for target 'Release/obj.target/simdjson-avx2/simdjson/main.o' failed
make: *** [Release/obj.target/simdjson-avx2/simdjson/main.o] Error 1
make: Leaving directory '/workspaces/node-ms/test/json/node_modules/simdjson/build'
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:194:23)
gyp ERR! stack     at ChildProcess.emit (events.js:223:5)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:272:12)
gyp ERR! System Linux 4.9.184-linuxkit
gyp ERR! command "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /workspaces/node-ms/test/json/node_modules/simdjson
gyp ERR! node -v v12.14.1
gyp ERR! node-gyp -v v5.0.5
gyp ERR! not ok 
npm WARN [email protected] No description
npm WARN [email protected] No repository field.

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/node/.npm/_logs/2020-02-04T00_38_19_085Z-debug.log

Here some more information about the system

cat /etc/os-release

PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
VERSION_CODENAME=stretch
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

lsb_release -a

No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 9.11 (stretch)
Release:        9.11
Codename:       stretch

uname -r

4.9.184-linuxkit

npm config list

; cli configs
metrics-registry = "https://registry.npmjs.org/"
scope = ""
user-agent = "npm/6.13.4 node/v12.14.1 linux x64"

; globalconfig /usr/local/etc/npmrc
prefix = "/usr/local/share/npm-global"

; node bin location = /usr/local/bin/node
; cwd = /workspaces/node-ms/test/json
; HOME = /home/node
; "npm config ls -l" to show all defaults.

Use the following steps to reproduce the error:

1. install vs code
2. install docker
3. install vs code extension Remote - Containers
4. open empty folder in vs code
5. run vs code action Remote-Containers: Add Development Container Configuration Files...
6. select Node 12 and Typescript
7. add to the Dockerfile USER node and then ARG HOME=/home/node
8. run vs code action Remote-Containers: Open Folder in Container...
9. run in bash npm init
10. run npm i simdjson

Detect dynamically AVX2 within npm

When can detect AVX2 dynamically within npm without user intervention, see for example...

https://github.com/SalvatorePreviti/roaring-node/blob/master/binding.gyp#L24-L29

Is this project still maintained?

Well. Is it?

what(): stoi Aborted (core dumped)

Hi,
I have implemented simdjson in my server to parse the incoming body. But after some requests server breaks with the error. Also, I am not able to replicate the issue at my end but this is affecting our clients.

terminate called after throwing an instance of 'std::invalid_argument'
  what():  stoi
Aborted (core dumped)

@luizperes Can you help me with this?

This package size is 39MB

the package size is 39MB becuase of the examples are publshed:

$ du -h -d 1 .
8.0K	./benchmark
1.4M	./simdjson
4.0K	./tests
 36M	./jsonexamples
4.0K	./.github
8.0K	./lib
 38M	.

Can't build on travis

Node version: v14.15.4
Npm version: 6.14.10
OS: Ubuntu 16.04.6 LTS
simdjson version: 0.9.2

In file included from ../simdjson/bindings.h:2:0, from ../simdjson/main.cpp:2: ../simdjson/src/simdjson.h:22936:58: error: ambiguous template specialization ‘get<>’ for ‘simdjson::simdjson_result<simdjson::fallback::ondemand::array> simdjson::fallback::ondemand::document::get() &’ template<> simdjson_really_inline simdjson_result<array> document::get() & noex

In file included from ../simdjson/bindings.h:2:0, from ../simdjson/main.cpp:2: ../simdjson/src/simdjson.h:22936:58: error: ambiguous template specialization ‘get<>’ for ‘simdjson::simdjson_result<simdjson::fallback::ondemand::array> simdjson::fallback::ondemand::document::get() &’ template<> simdjson_really_inline simdjson_result<array> document::get() & noex

Slower than JSON.parse

Hi @luizperes

I know this library was made to handle large JSON files, but there i occurred to some performance stranges when tried to parse my json and benchmarked, this library is slow, up to 6-x slower.

Here result:

json.parse - large: 985.011ms
simdjson - large: 4.756s

Also, lazyParse does not return expected result for me or i'm doing something wrong, and even using lazyParse, performance still slow. How we can improve this?

Code to test

const simdJson = require("simdjson");

const bench = (name, fn) => {
  console.time(name);
  for (let i = 0; i < 200000; i++) {
    fn();
  }
  console.timeEnd(name);
};

// Create large JSON file
let JSON_BUFF_LARGE = {};
for (let i = 0; i < 20; i++) { // 20 is close to 0.5Kb which very small, but you can increase this value
  JSON_BUFF_LARGE["key_" + i] = Math.round(Math.random() * 1e16).toString(16);
}
JSON_BUFF_LARGE = JSON.stringify(JSON_BUFF_LARGE);

console.log(
  "JSON buffer LARGE size is ",
  parseFloat(JSON_BUFF_LARGE.length / 1024).toFixed(2),
  "Kb"
);

bench("json.parse - large", () => JSON.parse(JSON_BUFF_LARGE));
bench("simdjson - large", () => simdJson.parse(JSON_BUFF_LARGE));

Cannot install with Visual Studio Community 2022 on Windows 10 x64

I followed the instructions at https://github.com/nodejs/node-gyp#on-windows to install node-gyp on Windows. I already have Visual Studio Community 2022 installed with "Desktop development with C++" before so I ran npm config set msvs_version 2022 instead of npm config set msvs_version 2017. The installation was successful and my Node.js is also on the latest LTS version (v16.14.0). I tried to install simdjson but I got this compilation error:

D:\Apps\JishinAlert\Discord>npm install simdjson --save
npm ERR! code 1
npm ERR! path D:\Apps\JishinAlert\Discord\node_modules\simdjson
npm ERR! command failed
npm ERR! command C:\WINDOWS\system32\cmd.exe /d /s /c node-gyp rebuild
npm ERR! Building the projects in this solution one at a time. To enable parallel build, please add the "-m" switch.
npm ERR!   main.cpp
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\node-addon-api\napi-inl.h(2155,9): warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify /EHsc [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR!   simdjson.cpp
npm ERR!   bindings.cpp
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\node-addon-api\napi-inl.h(2155,9): warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify /EHsc [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(21,54): error C2440: '<function-style-cast>': cannot convert from 'simdjson::dom::element' to 'simdjson::dom::array' [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(21,54): message : No constructor could take the source type, or constructor overload resolution was ambiguous [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(21,31): error C2143: syntax error: missing ';' before ':' [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(21,7): error C2440: '<function-style-cast>': cannot convert from 'simdjson::dom::element' to 'simdjson::dom::array' [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(21,7): message : No constructor could take the source type, or constructor overload resolution was ambiguous [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(21,52): error C2143: syntax error: missing ';' before ')' [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(29,47): error C2440: '<function-style-cast>': cannot convert from 'simdjson::dom::element' to 'simdjson::dom::object' [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(29,47): message : No constructor could take the source type, or constructor overload resolution was ambiguous [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(29,7): error C3531: 'field': a symbol whose type contains 'auto' must have an initializer [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(29,23): error C2143: syntax error: missing ';' before ':' [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(29,7): error C2440: '<function-style-cast>': cannot convert from 'simdjson::dom::element' to 'simdjson::dom::object' [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(29,7): message : No constructor could take the source type, or constructor overload resolution was ambiguous [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(29,45): error C2143: syntax error: missing ';' before ')' [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(32,66): error C2660: 'simdjsonnode::makeJSONObject': function does not take 1 arguments [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(15,27): message : see declaration of 'simdjsonnode::makeJSONObject' [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(32,12): error C2672: 'Napi::Object::Set': no matching overloaded function found [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(32,67): error C2780: 'void Napi::Object::Set(uint32_t,const ValueType &)': expects 2 arguments - 1 provided [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\node-addon-api\napi.h(682): message : see declaration of 'Napi::Object::Set' [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(32,67): error C2780: 'void Napi::Object::Set(const std::string &,const ValueType &)': expects 2 arguments - 1 provided [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\node-addon-api\napi.h(645): message : see declaration of 'Napi::Object::Set' [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(32,67): error C2780: 'void Napi::Object::Set(const char *,const ValueType &)': expects 2 arguments - 1 provided [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\node-addon-api\napi.h(638): message : see declaration of 'Napi::Object::Set' [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(32,67): error C2780: 'void Napi::Object::Set(Napi::Value,const ValueType &)': expects 2 arguments - 1 provided [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\node-addon-api\napi.h(631): message : see declaration of 'Napi::Object::Set' [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(32,67): error C2780: 'void Napi::Object::Set(napi_value,const ValueType &)': expects 2 arguments - 1 provided [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\node-addon-api\napi.h(624): message : see declaration of 'Napi::Object::Set' [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(37,37): error C2440: 'initializing': cannot convert from 'simdjson::dom::element' to 'std::basic_string_view<char,std::char_traits<char>>' [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(37,37): message : No constructor could take the source type, or constructor overload resolution was ambiguous [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(41,53): error C2664: 'Napi::Value Napi::Value::From<int64_t>(napi_env,const T &)': cannot convert argument 2 from 'simdjson::dom::element' to 'const T &' [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR!           with
npm ERR!           [
npm ERR!               T=int64_t
npm ERR!           ]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(41,53): message : Reason: cannot convert from 'simdjson::dom::element' to 'const T' [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR!           with
npm ERR!           [
npm ERR!               T=int64_t
npm ERR!           ]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(41,46): message : No user-defined-conversion operator available that can perform this conversion, or the operator cannot be called [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\node-addon-api\napi-inl.h(926,14): message : see declaration of 'Napi::Value::From' [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(43,54): error C2664: 'Napi::Value Napi::Value::From<uint64_t>(napi_env,const T &)': cannot convert argument 2 from 'simdjson::dom::element' to 'const T &' [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR!           with
npm ERR!           [
npm ERR!               T=uint64_t
npm ERR!           ]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(43,54): message : Reason: cannot convert from 'simdjson::dom::element' to 'const T' [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR!           with
npm ERR!           [
npm ERR!               T=uint64_t
npm ERR!           ]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(43,47): message : No user-defined-conversion operator available that can perform this conversion, or the operator cannot be called [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\node-addon-api\napi-inl.h(926,14): message : see declaration of 'Napi::Value::From' [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(45,52): error C2664: 'Napi::Value Napi::Value::From<double>(napi_env,const T &)': cannot convert argument 2 from 'simdjson::dom::element' to 'const T &' [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR!           with
npm ERR!           [
npm ERR!               T=double
npm ERR!           ]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(45,52): message : Reason: cannot convert from 'simdjson::dom::element' to 'const T' [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR!           with
npm ERR!           [
npm ERR!               T=double
npm ERR!           ]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(45,45): message : No user-defined-conversion operator available that can perform this conversion, or the operator cannot be called [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\node-addon-api\napi-inl.h(926,14): message : see declaration of 'Napi::Value::From' [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(47,50): error C2664: 'Napi::Value Napi::Value::From<bool>(napi_env,const T &)': cannot convert argument 2 from 'simdjson::dom::element' to 'const T &' [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR!           with
npm ERR!           [
npm ERR!               T=bool
npm ERR!           ]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(47,50): message : Reason: cannot convert from 'simdjson::dom::element' to 'const T' [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR!           with
npm ERR!           [
npm ERR!               T=bool
npm ERR!           ]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(47,43): message : No user-defined-conversion operator available that can perform this conversion, or the operator cannot be called [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\node-addon-api\napi-inl.h(926,14): message : see declaration of 'Napi::Value::From' [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(85,71): error C2664: 'Napi::Value simdjsonnode::findKeyPath(Napi::Env,std::vector<std::string,std::allocator<std::string>>,simdjson::dom::element)': cannot convert argument 3 from 'simdjson::simdjson_result<simdjson::dom::element>' to 'simdjson::dom::element' [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(85,51): message : No user-defined-conversion operator available that can perform this conversion, or the operator cannot be called [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(75,27): message : see declaration of 'simdjsonnode::findKeyPath' [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(88,64): error C2664: 'Napi::Value simdjsonnode::findKeyPath(Napi::Env,std::vector<std::string,std::allocator<std::string>>,simdjson::dom::element)': cannot convert argument 3 from 'simdjson::simdjson_result<simdjson::dom::element>' to 'simdjson::dom::element' [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(88,55): message : No user-defined-conversion operator available that can perform this conversion, or the operator cannot be called [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(75,27): message : see declaration of 'simdjsonnode::findKeyPath' [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(118,50): error C2664: 'Napi::Value simdjsonnode::makeJSONObject(Napi::Env,simdjson::dom::element)': cannot convert argument 2 from 'simdjson::simdjson_result<simdjson::dom::element>' to 'simdjson::dom::element' [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(118,44): message : No user-defined-conversion operator available that can perform this conversion, or the operator cannot be called [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! D:\Apps\JishinAlert\Discord\node_modules\simdjson\simdjson\bindings.cpp(15,27): message : see declaration of 'simdjsonnode::makeJSONObject' [D:\Apps\JishinAlert\Discord\node_modules\simdjson\build\simdjson.vcxproj]
npm ERR! gyp info it worked if it ends with ok
npm ERR! gyp info using [email protected]
npm ERR! gyp info using [email protected] | win32 | x64
npm ERR! gyp info find Python using Python version 3.10.2 found at "C:\Users\Admin\AppData\Local\Programs\Python\Python310\python.exe"
npm ERR! gyp info find VS using VS2022 (17.0.32002.185) found at:
npm ERR! gyp info find VS "C:\Program Files\Microsoft Visual Studio\2022\Community"
npm ERR! gyp info find VS run with --verbose for detailed information
npm ERR! gyp info spawn C:\Users\Admin\AppData\Local\Programs\Python\Python310\python.exe
npm ERR! gyp info spawn args [
npm ERR! gyp info spawn args   'C:\\Program Files\\nodejs\\node_modules\\npm\\node_modules\\node-gyp\\gyp\\gyp_main.py',
npm ERR! gyp info spawn args   'binding.gyp',
npm ERR! gyp info spawn args   '-f',
npm ERR! gyp info spawn args   'msvs',
npm ERR! gyp info spawn args   '-I',
npm ERR! gyp info spawn args   'D:\\Apps\\JishinAlert\\Discord\\node_modules\\simdjson\\build\\config.gypi',
npm ERR! gyp info spawn args   '-I',
npm ERR! gyp info spawn args   'C:\\Program Files\\nodejs\\node_modules\\npm\\node_modules\\node-gyp\\addon.gypi',
npm ERR! gyp info spawn args   '-I',
npm ERR! gyp info spawn args   'C:\\Users\\Admin\\AppData\\Local\\node-gyp\\Cache\\16.14.0\\include\\node\\common.gypi',
npm ERR! gyp info spawn args   '-Dlibrary=shared_library',
npm ERR! gyp info spawn args   '-Dvisibility=default',
npm ERR! gyp info spawn args   '-Dnode_root_dir=C:\\Users\\Admin\\AppData\\Local\\node-gyp\\Cache\\16.14.0',
npm ERR! gyp info spawn args   '-Dnode_gyp_dir=C:\\Program Files\\nodejs\\node_modules\\npm\\node_modules\\node-gyp',
npm ERR! gyp info spawn args   '-Dnode_lib_file=C:\\\\Users\\\\Admin\\\\AppData\\\\Local\\\\node-gyp\\\\Cache\\\\16.14.0\\\\<(target_arch)\\\\node.lib',
npm ERR! gyp info spawn args   '-Dmodule_root_dir=D:\\Apps\\JishinAlert\\Discord\\node_modules\\simdjson',
npm ERR! gyp info spawn args   '-Dnode_engine=v8',
npm ERR! gyp info spawn args   '--depth=.',
npm ERR! gyp info spawn args   '--no-parallel',
npm ERR! gyp info spawn args   '--generator-output',
npm ERR! gyp info spawn args   'D:\\Apps\\JishinAlert\\Discord\\node_modules\\simdjson\\build',
npm ERR! gyp info spawn args   '-Goutput_dir=.'
npm ERR! gyp info spawn args ]
npm ERR! gyp info spawn C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Current\Bin\MSBuild.exe
npm ERR! gyp info spawn args [
npm ERR! gyp info spawn args   'build/binding.sln',
npm ERR! gyp info spawn args   '/clp:Verbosity=minimal',
npm ERR! gyp info spawn args   '/nologo',
npm ERR! gyp info spawn args   '/p:Configuration=Release;Platform=x64'
npm ERR! gyp info spawn args ]
npm ERR! gyp ERR! build error
npm ERR! gyp ERR! stack Error: `C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Current\Bin\MSBuild.exe` failed with exit code: 1
npm ERR! gyp ERR! stack     at ChildProcess.onExit (C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\lib\build.js:194:23)
npm ERR! gyp ERR! stack     at ChildProcess.emit (node:events:520:28)
npm ERR! gyp ERR! stack     at Process.ChildProcess._handle.onexit (node:internal/child_process:291:12)
npm ERR! gyp ERR! System Windows_NT 10.0.19043
npm ERR! gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild"
npm ERR! gyp ERR! cwd D:\Apps\JishinAlert\Discord\node_modules\simdjson
npm ERR! gyp ERR! node -v v16.14.0
npm ERR! gyp ERR! node-gyp -v v8.4.1
npm ERR! gyp ERR! not ok

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\Admin\AppData\Local\npm-cache\_logs\2022-03-05T02_09_31_907Z-debug-0.log

can anyone help me with this? I don't have lots of C++ knowledge so I can't fix that by myself
I'm using Visual Studio Community 2022 v17.0.3 with Desktop development with C++

Memory Leak

When running the benchmarks locally, the process gradually consumes all memory on my machine until it runs out of memory (~28GB)

Commenting out simdjson.lazyParse in the benchmark removes leak behavior, so I don't think its the test setup. Printing the node heap statistics shows that it is stable ~80 mb, so the leak is likely in the c++ code.

Usage: Get keys without values.

I want to read a very large file and conditionally traverse into it. This means that I want to be able to read property keys without parsing the values. Ideally, I'd like to be able to get the type of the values as well. I can't tell whether this is a limitation of simdjson or whether it's a limitation of this wrapper.

Consider updating to simdjson version 0.4.0

Highlights

• Test coverage has been greatly improved and we have resolved many static-analysis warnings on different systems.

New features:

• We added a fast (8GB/s) minifier that works directly on JSON strings.
• We added fast (10GB/s) UTF-8 validator that works directly on strings (any strings, including non-JSON).
• The array and object elements have a constant-time size() method.

Performance:

• Performance improvements to the API (type(), get<>()).
• The parse_many function (ndjson) has been entirely reworked. It now uses a single secondary thread instead of several new threads.
• We have introduced a faster UTF-8 validation algorithm (lookup3) for all kernels (ARM, x64 SSE, x64 AVX).

System support:

• C++11 support for older compilers and systems.
• FreeBSD support (and tests).
• We support the clang front-end compiler (clangcl) under Visual Studio.
• It is now possible to target ARM platforms under Visual Studio.
• The simdjson library will never abort or print to standard output/error.

Consider updating to simdjson 0.1.1

We have a new version which has precomputed string lengths, and supporting strings with null characters in them.

Consider updating to latest release (0.1.2)

It fixes a few potential buffer overruns for adversarial inputs. It should be safer.

Update mocha to a version > 7.1.1

Problem
Mocha 7.1.0 is using an outdated version of mkdirp and that is causing a warning of low severity vulnerability. Updating mocha to 7.1.1 (the latest at the moment on npm) is no good as well, as it is using [email protected] and we get a deprecated message: Legacy versions of mkdirp are no longer supported. Please update to mkdirp 1.x. (Note that the API surface has changed to use Promises in 1.x.).

Solution
We will have to wait for mocha to update its version so to fix that deprecated warning.

CC @nojvek

Can't build on windows

NODE_VERSION: 15.10.0
TARGET_PLATFORM_VERTION: 10.0.17763.0
VS_VERTION: VS2017 (15.9.28307.1440)
TOOLSET_VERTION: v141
node-addon-api\napi-inl.h(2155): warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify simdjson\simdjson\bindings.cpp(15): error C2653: 'dom': is not a class or namespace name simdjson\simdjson\bindings.cpp(15): error C2061: syntax error: identifier 'element' simdjson\build\simdjson.vcxproj] simdjson\simdjson\bindings.cpp(15): error C2244: 'makeJSONObject': unable to match function definition to an existing simdjson\simdjson\bindings.cpp(75): error C2653: 'dom': is not a class or namespace name simdjson\simdjson\bindings.cpp(75): error C2061: syntax error: identifier 'element' simdjson\simdjson\bindings.cpp(75): error C2244: 'findKeyPath': simdjson\simdjson\bindings.cpp(118): error C2664: 'Napi::Value simdjsonnode::makeJSONObject(Napi::Env,simdjson::dom::element)': cannot convert argument 2 from 'simdjson::simdjson_result<simdjson::dom::element>' to 'simdjson::dom::element' simdjson\simdjson\bindings.cpp(118): note: No user-defined-conversion operator available that can perform this conversion, or the operator cannot be called

Considering upgrading to simdjson 0.3.0

Version 0.3 of simdjson is now available

Highlights

• Multi-Document Parsing: Read a bundle of JSON documents (ndjson) 2-4x faster than doing it individually. API docs / Design Details
• Simplified API: The API has been completely revamped for ease of use, including a new JSON navigation API and fluent support for error code and exception styles of error handling with a single API. Docs
• Exact Float Parsing: Now simdjson parses floats flawlessly without any performance loss (simdjson/simdjson#558).
  Blog Post
• Even Faster: The fastest parser got faster! With a shiny new UTF-8 validator
  and meticulously refactored SIMD core, simdjson 0.3 is 15% faster than before, running at 2.5 GB/s (where 0.2 ran at 2.2 GB/s).

Minor Highlights

• Fallback implementation: simdjson now has a non-SIMD fallback implementation, and can run even on very old 64-bit machines.
• Automatic allocation: as part of API simplification, the parser no longer has to be preallocated-it will adjust automatically when it encounters larger files.
• Runtime selection API: We've exposed simdjson's runtime CPU detection and implementation selection as an API, so you can tell what implementation we detected and test with other implementations.
• Error handling your way: Whether you use exceptions or check error codes, simdjson lets you handle errors in your style. APIs that can fail return simdjson_result, letting you check the error code before using the result. But if you are more comfortable with exceptions, skip the error code and cast straight to T, and exceptions will be thrown automatically if an error happens. Use the same API either way!
• Error chaining: We also worked to keep non-exception error-handling short and sweet. Instead of having to check the error code after every single operation, now you can chain JSON navigation calls like looking up an object field or array element, or casting to a string, so that you only have to check the error code once at the very end.

Very big JSON (up to 2gb)

I'm working on a project which needs to parse some very large stringified JSON (up to 2GB).

It isn't clear to me whether this issue is the same as #5 or not

Windows build failure on outdated node-gyp (msvs_version=2015) @ Windows 7 x64

I'm eager to test this out on my project rurudb, a prototypal db which currently uses JSON.parse against loaded db file JSON string.

Sad thing is I don't know shit about C++ so this might be tricky.

I'm on Win7 x64 with visual studio build tools, and have existing packages in my projects which are also built using node-gyp.

C:\Users\1234>npm config get python
C:\Users\1234\.windows-build-tools\python27\python.exe

C:\Users\1234>npm config get msvs_version
2015

yarn add simdjson
yarn add v1.13.0-20181107.1254
warning ..\package.json: No license field
[1/4] Resolving packages...
[2/4] Fetching packages...
info [email protected]: The platform "win32" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
warning " > [email protected]" has incorrect peer dependency "jest@>=22 <24".
[4/4] Building fresh packages...
error C:\cygwin64\home\1234\rurudb\node_modules\simdjson: Command failed.
Exit code: 1
Command: node-gyp rebuild
Arguments:
Directory: C:\cygwin64\home\1234\rurudb\node_modules\simdjson
Output:
C:\cygwin64\home\1234\rurudb\node_modules\simdjson>if not defined npm_config_node_gyp (node "C:\Program Files\nodejs\node_modules\npm\bin\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp.js" rebuild )  else (node "" rebuild )
gyp info it worked if it ends with ok
gyp info using [email protected]
gyp info using [email protected] | win32 | x64
gyp info spawn C:\Python27\python.EXE
gyp info spawn args [ 'C:\\Program Files\\nodejs\\node_modules\\npm\\node_modules\\node-gyp\\gyp\\gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'msvs',
gyp info spawn args   '-G',
gyp info spawn args   'msvs_version=auto',
gyp info spawn args   '-I',
gyp info spawn args   'C:\\cygwin64\\home\\1234\\rurudb\\node_modules\\simdjson\\build\\config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   'C:\\Program Files\\nodejs\\node_modules\\npm\\node_modules\\node-gyp\\addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   'C:\\Users\\1234\\.node-gyp\\10.10.0\\include\\node\\common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=C:\\Users\\1234\\.node-gyp\\10.10.0',
gyp info spawn args   '-Dnode_gyp_dir=C:\\Program Files\\nodejs\\node_modules\\npm\\node_modules\\node-gyp',
gyp info spawn args   '-Dnode_lib_file=C:\\Users\\1234\\.node-gyp\\10.10.0\\<(target_arch)\\node.lib',
gyp info spawn args   '-Dmodule_root_dir=C:\\cygwin64\\home\\1234\\rurudb\\node_modules\\simdjson',
gyp info spawn args   '-Dnode_engine=v8',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'C:\\cygwin64\\home\\1234\\rurudb\\node_modules\\simdjson\\build',
gyp info spawn args   '-Goutput_dir=.' ]
gyp info spawn C:\Program Files (x86)\MSBuild\14.0\bin\msbuild.exe
gyp info spawn args [ 'build/binding.sln',
gyp info spawn args   '/clp:Verbosity=minimal',
gyp info spawn args   '/nologo',
gyp info spawn args   '/p:Configuration=Release;Platform=x64' ]
Building the projects in this solution one at a time. To enable parallel build, please add the "/m" switch.
  nothing.c
  win_delay_load_hook.cc
  nothing.vcxproj -> C:\cygwin64\home\1234\rurudb\node_modules\simdjson\build\Release\\nothing.lib
  main.cpp
  bindings.cpp
  win_delay_load_hook.cc
c:\cygwin64\home\1234\rurudb\node_modules\simdjson\simdjson\src/simdjson.h(42): warning C4244: 'return': conversion from 'unsigned __int64' to 'int', possible loss of data (compiling source file ..\simdjson\bindings.cpp) [C:\cygwin64\home\1234\rurudb\node_modules\simdjson\build\simdjson.vcxproj]
c:\cygwin64\home\1234\rurudb\node_modules\simdjson\simdjson\src/simdjson.h(46): warning C4244: 'return': conversion from 'unsigned __int64' to 'int', possible loss of data (compiling source file ..\simdjson\bindings.cpp) [C:\cygwin64\home\1234\rurudb\node_modules\simdjson\build\simdjson.vcxproj]
c:\cygwin64\home\1234\rurudb\node_modules\simdjson\simdjson\src/simdjson.h(42): warning C4244: 'return': conversion from 'unsigned __int64' to 'int', possible loss of data (compiling source file ..\simdjson\main.cpp) [C:\cygwin64\home\1234\rurudb\node_modules\simdjson\build\simdjson.vcxproj]
c:\cygwin64\home\1234\rurudb\node_modules\simdjson\simdjson\src/simdjson.h(46): warning C4244: 'return': conversion from 'unsigned __int64' to 'int', possible loss of data (compiling source file ..\simdjson\main.cpp) [C:\cygwin64\home\1234\rurudb\node_modules\simdjson\build\simdjson.vcxproj]
c:\cygwin64\home\1234\rurudb\node_modules\simdjson\simdjson\src/simdjson.h(462): error C2039: 'string_view': is not a member of 'std' (compiling source file ..\simdjson\main.cpp) [C:\cygwin64\home\1234\rurudb\node_modules\simdjson\build\simdjson.vcxproj]
  C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include\sstream(12): note: see declaration of 'std' (compiling source file ..\simdjson\main.cpp)
c:\cygwin64\home\1234\rurudb\node_modules\simdjson\simdjson\src/simdjson.h(462): error C2039: 'string_view': is not a member of 'std' (compiling source file ..\simdjson\bindings.cpp) [C:\cygwin64\home\1234\rurudb\node_modules\simdjson\build\simdjson.vcxproj]
c:\cygwin64\home\1234\rurudb\node_modules\simdjson\simdjson\src/simdjson.h(462): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int
(compiling source file ..\simdjson\main.cpp)C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include\sstream(12): note: see declaration of 'std' (compiling source file ..\simdjson\bindings.cpp) [C:\cygwin64\home\1234\rurudb\node_modules\simdjson\build\simdjson.vcxproj]

c:\cygwin64\home\1234\rurudb\node_modules\simdjson\simdjson\src/simdjson.h(462): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int
(compiling source file ..\simdjson\bindings.cpp) [C:\cygwin64\home\1234\rurudb\node_modules\simdjson\build\simdjson.vcxproj]
c:\cygwin64\home\1234\rurudb\node_modules\simdjson\simdjson\src/simdjson.h(462): error C2146: syntax error: missing ';' before identifier 'get_corpus' (compiling source file ..\simdjson\bindings.cpp) [C:\cygwin64\home\1234\rurudb\node_modules\simdjson\build\simdjson.vcxproj]
c:\cygwin64\home\1234\rurudb\node_modules\simdjson\simdjson\src/simdjson.h(462): error C2146: syntax error: missing ';' before identifier 'get_corpus' (compiling source file ..\simdjson\main.cpp) [C:\cygwin64\home\1234\rurudb\node_modules\simdjson\build\simdjson.vcxproj]
c:\cygwin64\home\1234\rurudb\node_modules\simdjson\simdjson\src/simdjson.h(35739): fatal error C1021: invalid preprocessor command 'warning' (compiling source file ..\simdjson\main.cpp) [C:\cygwin64\home\1234\rurudb\node_modules\simdjson\build\simdjson.vcxproj]
c:\cygwin64\home\1234\rurudb\node_modules\simdjson\simdjson\src/simdjson.h(35739): fatal error C1021: invalid preprocessor command 'warning' (compiling source file ..\simdjson\bindings.cpp) [C:\cygwin64\home\1234\rurudb\node_modules\simdjson\build\simdjson.vcxproj]
gyp ERR! build error
gyp ERR! stack Error: `C:\Program Files (x86)\MSBuild\14.0\bin\msbuild.exe` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onExit (C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\lib\build.js:262:23)
gyp ERR! stack     at ChildProcess.emit (events.js:182:13)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:240:12)
gyp ERR! System Windows_NT 6.1.7601
gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild"
gyp ERR! cwd C:\cygwin64\home\1234\rurudb\node_modules\simdjson
gyp ERR! node -v v10.10.0
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok
info Visit https://yarnpkg.com/en/docs/cli/add for documentation about this command.

Project doesn't seem to be working for main OSs

It seems that simd_nodejs on the master branch is only currently working for Linux (the current tag version 0.0.2 on npm has only been tested with Mac and seems to be fine.

At the moment, we have two reported error in Windows and Mac:

• Windows: #1
• Mac: (see output below)

d142-058-221-020:simdjson_nodejs luizperes$ npm run build

> [email protected] build /Users/luizperes/Projects/simdjson_nodejs
> node-gyp rebuild

  CC(target) Release/obj.target/nothing/node_modules/node-addon-api/src/nothing.o
  LIBTOOL-STATIC Release/nothing.a
  CXX(target) Release/obj.target/simdjson/simdjson/main.o
In file included from ../simdjson/main.cpp:2:
In file included from ../simdjson/bindings.h:2:
../simdjson/src/simdjson.h:74:2: warning: "BMI is missing?" [-W#warnings]
#warning "BMI is missing?"
 ^
../simdjson/src/simdjson.h:35739:2: warning: "We require AVX2 support!" [-W#warnings]
#warning "We require AVX2 support!"
 ^
2 warnings generated.
  CXX(target) Release/obj.target/simdjson/simdjson/bindings.o
In file included from ../simdjson/bindings.cpp:1:
In file included from ../simdjson/bindings.h:2:
../simdjson/src/simdjson.h:74:2: warning: "BMI is missing?" [-W#warnings]
#warning "BMI is missing?"
 ^
../simdjson/src/simdjson.h:35739:2: warning: "We require AVX2 support!" [-W#warnings]
#warning "We require AVX2 support!"
 ^
2 warnings generated.
  CXX(target) Release/obj.target/simdjson/simdjson/src/simdjson.o
In file included from ../simdjson/src/simdjson.cpp:2:
../simdjson/src/simdjson.h:74:2: warning: "BMI is missing?" [-W#warnings]
#warning "BMI is missing?"
 ^
../simdjson/src/simdjson.h:35739:2: warning: "We require AVX2 support!" [-W#warnings]
#warning "We require AVX2 support!"
 ^
../simdjson/src/simdjson.cpp:408:34: error: variable has incomplete type 'struct avx_processed_utf_bytes'
  struct avx_processed_utf_bytes previous{};
                                 ^
../simdjson/src/simdjson.cpp:408:10: note: forward declaration of 'avx_processed_utf_bytes'
  struct avx_processed_utf_bytes previous{};
         ^
2 warnings and 1 error generated.
make: *** [Release/obj.target/simdjson/simdjson/src/simdjson.o] Error 1
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:262:23)
gyp ERR! stack     at ChildProcess.emit (events.js:182:13)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:240:12)
gyp ERR! System Darwin 18.2.0
gyp ERR! command "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users/luizperes/Projects/simdjson_nodejs
gyp ERR! node -v v10.14.1
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok 
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] build: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/luizperes/.npm/_logs/2019-03-03T08_12_36_813Z-debug.log

It seems that the flags __AVX2__ and BMI are not being recognized on Mac OS (it was tested outside npm and runs fine)

a idea to enhance simdjson.parse performance

hello, I find that the simdjson.parse is very slow. i think if your use napi to create string, nodejs would copy the string bytes. the bestway to create string is use v8's String.substring, which would create a sliceString in v8 that no need copy memory. simdjson can generate a new buffer, and use javascript to constructe the json object via the buffer.

Error while starting

Hi @luizperes and thanks for such great project.

I'm tried run this project, but there got some errors.

nodemon

[nodemon] app crashed - waiting for file changes before starting...
[nodemon] restarting due to changes...
[nodemon] starting `node --enable-source-maps --trace-uncaught bench-json-str.js`
[nodemon] app crashed - waiting for file changes before starting...
^Z⏎                                                                                                                          
turbo-json on  master [!?] is 📦 v1.0.0 via ⬢ v13.11.0 took 1m24s 

node + fish shell

✦ ❯ node bench-json-str.js
fish: Job 2, 'node bench-json-str.js' terminated by signal SIGILL (Illegal instruction)
turbo-json on  master [!?] is 📦 v1.0.0 via ⬢ v13.11.0 

node + zsh shell

✦ ❯ zsh
dalisoft@dalisofts-MBP turbo-json % node bench-json-str.js 
zsh: illegal hardware instruction  node bench-json-str.js

node + bash

dalisoft@dalisofts-MBP turbo-json % bash

The default interactive shell is now zsh.
To update your account to use zsh, please run `chsh -s /bin/zsh`.
For more details, please visit https://support.apple.com/kb/HT208050.
bash-3.2$ node bench-json-str.js 
Illegal instruction: 4

instructions

bash-3.2$ sysctl -a | grep machdep.cpu.leaf7_features
machdep.cpu.leaf7_features: RDWRFSGS SMEP ERMS MDCLEAR IBRS STIBP L1DF SSBD

Env

• OS: macOS Catalina 10.15.4
• Machine: MBP Mid-2012 i5-3210M 8GB RAM
• Node: v13-latest

Query syntax example broken?

Upstream, simdjson is going to soon support JSON Pointer queries, but, it is maybe worth pointing out that the current example from the README fails:

Do

git clone https://github.com/luizperes/simdjson_nodejs.git
cd simdjson_nodejs
npm install

Create this file inside simdjson_nodejs...

const simdjson = require('./index.js');
const fs = require('fs');
const apache_builds = fs.readFileSync('jsonexamples/apache_builds.json', 'utf-8');
const JSONbuffer = simdjson.lazyParse(apache_builds);
console.log(JSONbuffer.valueForKeyPath("foo.bar[2].example"));

Then do...

node myfile.js

I get...

libc++abi.dylib: terminating with uncaught exception of type std::invalid_argument: stoi: no conversion
Abort trap: 6

cc @ioioioio @croteaucarine

error in name for valueForKeyPath

I receive a file like this from the backend:

{
  "@avito-core/toggles:6.1.18": {
    "add_model_review_from": true
  }
 }

run standart code and see:

const JSONbuffer2 = simdjson.lazyParse(data); // external (C++) parsed JSON object
console.log(JSONbuffer2.valueForKeyPath('@avito-core/toggles:6.1.18'));

console.log(JSONbuffer2.valueForKeyPath('@avito-core/toggles:6.1.18'));
                        ^
Error: The JSON field referenced does not exist in this object.

Uses V8 even when SSE4.2 is available

Simdjson also supports SSE4.2 - but this library still falls back to V8's parser when it's available.

Consider updating to simdjson 0.2.0

The library simdjson has a new major release (0.2.0). Major changes:

• Support for 64-bit ARM processors, can run under iOS (iPhone).
• Runtime dispatching on x64 processors (switches to SSE on older x64 processors, uses AVX2 when available). Supports processors as far back as the Intel Westmere.
• More accurate number parsing.
• Fixes most warnings under Visual Studio.
• Several small bugs have been fixed.
• Better performance in some cases.
• Introduces a JSON Pointer interface https://tools.ietf.org/html/rfc6901
• Better and more specific error messages (with optional textual descriptions).
• valgrind clean.
• Unified code style (LLVM).

Expose the underlying tapes as ArrayBuffer

Reading the code I see the fast part of simdjson is parsing the bytes of json and creating two buffers/tapes. One is the json tape that marks starting, ending and types for various elements. The other is a string tape that contains the parsed strings in utf-8 format.

JavaScript offers a nice way of fast buffer indexing and getting our values via TypedArrays and ArrayBuffers. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Typed_arrays

This would mean that the iteration part of getting values out could be done in pure js. I.e it would be technically possible to stream the buffers as binary data to the browser and have the iteration of json part work there too.

Or one could dump the tapes as files and get zero cost parsing by simply mmaping a file and iterating over gigabyes of json tape like Flat buffers.

https://google.github.io/flatbuffers/

I also don’t think lazyParse as the only function is a great interface. Underlying simdjson has a concept of elements and iterators. JavaScript has similar concept of iterators too. One would need to resort to proxy hacks which are a bit too magical and sometimes. I think we can expose a much nicer object/array iterator based interface for underlying tape.

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Iterators_and_Generators

https://codeburst.io/a-simple-guide-to-es6-iterators-in-javascript-with-examples-189d052c3d8e

This would mean there’s be two sub modules. One that is a fast jsonStr -> {jsonTape, strsTape}

The other that takes {jsonTape, strsTape} -> elemIterator.

Hopefully I’m making sense.

I’m happy to write the js part of the code. Just need to figure out how to export the buffers using napi api.

Typescript types as index.d.ts file please

Great library, although the interface is somewhat unintuitive. Is isValid the only method?

Would love to see typescript types so the library interface is fully documented. I can help here. Just don't really know what the interface right now is, and how to make most of the library.

Performance warning?

(Apologies in advance for this issue being kind of all over the place...)

I'm playing with code that is processing a potentially large input which is basically a single array of small-ish objects (a performance dump). I saw #28, and therefore kept it only to uses of lazyParse: read one line at a time, lazy parse it, inspect one field, and dump the object back out.

The result is still significantly slower than JSON.parse. So it seems like just invoking the parser has some significant overhead, which makes this possibly related to #35 too--? If that's the case, then it would be useful to add this in the README so people would no that this thing is a good choice for parsing large objects, but it's not a good choice for lots of smaller objects.

Of course it would be even better if there was a way to parse large files without keeping the whole thing in memory. (I saw one closed issue about a streaming interface on simdjson, but unclear if it's implemented or not...)

NOT AN ISSUE, QUESTION : What is the difference between on demand parser and dom :: parser

What was the intuition behind developing on demand parser present in quick start, since dom::parser provides faster parsing?

Reporting timings in GB/s

It would be more interesting if we could report the timings (in the README.md) in GB/s, or some other normalized metric.

Unable to run on Apple ARM chips

Hi there,

I'm getting illegal hardware instruction when I try to run a benchmark. Judging by https://conan.io/center/recipes/simdjson?version=3.2.2 and simdjson/simdjson#197, simdjson (the c++ version) supports ARM chips. Is this an issue with simdjson_nodejs?

System Information:
macOS Version: 14.0
Chip: Apple M2 Pro
Architecture: x86_64

JavaScript/TypeScript Project Information:
Node.js Version: v20.4.0
TypeScript Version: Version 5.2.2