flitbit / json-ptr

A complete implementation of JSON Pointer (RFC 6901) for nodejs and modern browsers.

json-ptr

A complete implementation of JSON Pointer (RFC 6901) for nodejs and modern browsers.

Supports Relative JSON Pointers. (Example)

Background

I wrote this a few years back when I was unable to find a complete implementation of RFC 6901. It turns out that I now use the hell out of it. I hope you also find it useful.

Install

npm install json-ptr

Release Bundles

As of v3.0.0, we provide CJS, ESM, and UMD builds under the dist/ folder when you install the package from NPM, we also have all appropriate references in our package.json file, so your code should just work. If you need a CDN reference for a website or the like, try UNPKG, which picks up our releases automatically.

CDN: https://unpkg.com/browse/[email protected]/dist/json-ptr.min.js.

Use

Both CJS and ESM are supported.

const { JsonPointer } = require('json-ptr');

import { JsonPointer } from 'json-ptr';

API Documentation

The API documentation is generated from code by typedoc and hosted here. Read the docs.

Documentation is always a work in progress, let us know by creating an issue if you need a scenario documented.

Example

There are many uses for JSON Pointers, here's one we encountered when we updated a public API and suddenly had clients sending two different message bodies to our APIs. This example is contrived to illustrate how we supported both new and old incoming messages:

// examples/versions.ts
import { JsonPointer } from 'json-ptr';

export type SupportedVersion = '1.0' | '1.1';

interface PrimaryGuestNamePointers {
  name: JsonPointer;
  surname: JsonPointer;
  honorific: JsonPointer;
}
const versions: Record<SupportedVersion, PrimaryGuestNamePointers> = {
  '1.0': {
    name: JsonPointer.create('/guests/0/name'),
    surname: JsonPointer.create('/guests/0/surname'),
    honorific: JsonPointer.create('/guests/0/honorific'),
  },
  '1.1': {
    name: JsonPointer.create('/primary/primaryGuest/name'),
    surname: JsonPointer.create('/primary/primaryGuest/surname'),
    honorific: JsonPointer.create('/primary/primaryGuest/honorific'),
  },
};

interface Reservation extends Record<string, unknown> {
  version?: SupportedVersion;
}

/**
 * Gets the primary guest's name from the specified reservation.
 * @param reservation a reservation, either version 1.0 or bearing a `version`
 * property indicating the version.
 */
function primaryGuestName(reservation: Reservation): string {
  const pointers = versions[reservation.version || '1.0'];
  if (!pointers) {
    throw new Error(`Unsupported reservation version: ${reservation.version}`);
  }
  const name = pointers.name.get(reservation) as string;
  const surname = pointers.surname.get(reservation) as string;
  const honorific = pointers.honorific.get(reservation) as string;
  const names: string[] = [];
  if (honorific) names.push(honorific);
  if (name) names.push(name);
  if (surname) names.push(surname);
  return names.join(' ');
}

// The original layout of a reservation (only the parts relevant to our example)
const reservationV1: Reservation = {
  guests: [
    {
      name: 'Wilbur',
      surname: 'Finkle',
      honorific: 'Mr.',
    },
    {
      name: 'Wanda',
      surname: 'Finkle',
      honorific: 'Mrs.',
    },
    {
      name: 'Wilma',
      surname: 'Finkle',
      honorific: 'Miss',
      child: true,
      age: 12,
    },
  ],
  // ...
};

// The new layout of a reservation (only the parts relevant to our example)
const reservationV1_1: Reservation = {
  version: '1.1',
  primary: {
    primaryGuest: {
      name: 'Wilbur',
      surname: 'Finkle',
      honorific: 'Mr.',
    },
    additionalGuests: [
      {
        name: 'Wanda',
        surname: 'Finkle',
        honorific: 'Mrs.',
      },
      {
        name: 'Wilma',
        surname: 'Finkle',
        honorific: 'Miss',
        child: true,
        age: 12,
      },
    ],
    // ...
  },
  // ...
};

console.log(primaryGuestName(reservationV1));
console.log(primaryGuestName(reservationV1_1));

Security Vulnerabilities (Resolved)

• prior to v3.0.0 there was a security vulnerability which allowed a developer to perform prototype pollution by sending malformed path segments to json-ptr. If you were one of these developers, you should upgrade to v3.0.0 immediately, and stop using json-ptr to pollute an object's prototype. If you feel you have a legitimate reason to do so, please use another method and leave json-ptr out of it. Such behavior has been disallowed since it can easily be done using plain ol javascript by those determined to violate common best practice.

• prior to v2.1.0 there was a security vulnerability which allowed an unscrupulous actor to execute arbitrary code if developers failed to sanitize user input before sending it to json-ptr. If your code does not sanitize user input before sending it to json-ptr, your project is vulnerable and you should upgrade to v3.0.0 immediately. And while your at it, start sanitized user input before sending it to any library!

Breaking Changes at v1.3.0

As was rightly pointed out in this issue, I should have rolled the major version at v1.3.0 instead of the minor version due to breaking changes to the API. Not the worst blunder I've made, but my apologies all the same. Since the ship has sailed, I'm boosting the visibility of these breaking changes.

Where did the Global Functions Go?

In version v1.3.0 of the library, global functions were moved to static functions of the JsonPointer class. There should be no difference in arguments or behavior. If you were previously importing the global functions it is a small change to destructure them and have compatible code.

Global Fn	Static Fn	Documentation
create()	JsonPointer.create()	Factory function that creates a JsonPointer
decode()	JsonPointer.decode()	Decodes the specified pointer into path segments.
flatten()	JsonPointer.flatten()	DEvaluates the target's object graph, returning a Record<Pointer, unknown> populated with pointers and the corresponding values from the graph..
get()	JsonPointer.get()	Gets the target object's value at the pointer's location.
has()	JsonPointer.has()	Determines if the specified target's object graph has a value at the pointer's location.
list()		Replaced by JsonPointer.listFragmentIds() and JsonPointer.listPointers().
listFragmentIds()	JsonPointer.listFragmentIds()	Evaluates the target's object graph, returning a UriFragmentIdentifierPointerListItem for each location in the graph.
listPointers()	JsonPointer.listPointers()	Evaluates the target's object graph, returning a JsonStringPointerListItem for each location in the graph.
map()	JsonPointer.map()	Evaluates the target's object graph, returning a Map<Pointer,unknown> populated with pointers and the corresponding values form the graph.
set()	JsonPointer.set()	Sets the target object's value, as specified, at the pointer's location.
	JsonPointer.unset()	Removes the target object's value at the pointer's location.
visit()	JsonPointer.visit()	Evaluates the target's object graph, calling the specified visitor for every unique pointer location discovered while walking the graph.

Tests

We're maintaining near 100% test coverage. Visit our circleci build page and drill down on a recent build's build and test step to see where we're at. It should look something like:

=============================== Coverage summary ===============================
Statements   : 100% ( 270/270 )
Branches     : 100% ( 172/172 )
Functions    : 100% ( 49/49 )
Lines        : 100% ( 265/265 )
================================================================================

We use mocha so you can also clone the code and:

$ npm install
$ npm test

Once you've run the tests on the command line you can open up ./tests.html in the browser of your choice.

Performance

WARNING! These performance metrics are quite outdated. We'll be updating as soon as we have time.

This repository has a companion repository that makes some performance comparisons between json-ptr, jsonpointer and json-pointer.

All timings are expressed as nanoseconds:

.flatten(obj)
...
MODULE       | METHOD  | COMPILED | SAMPLES |       AVG | SLOWER
json-pointer | dict    |          | 10      | 464455181 |
json-ptr     | flatten |          | 10      | 770424039 | 65.88%
jsonpointer  | n/a     |          | -       |         - |

.has(obj, pointer)
...
MODULE       | METHOD | COMPILED | SAMPLES | AVG  | SLOWER
json-ptr     | has    | compiled | 1000000 | 822  |
json-ptr     | has    |          | 1000000 | 1747 | 112.53%
json-pointer | has    |          | 1000000 | 2683 | 226.4%
jsonpointer  | n/a    |          | -       | -    |

.has(obj, fragmentId)
...
MODULE       | METHOD | COMPILED | SAMPLES | AVG  | SLOWER
json-ptr     | has    | compiled | 1000000 | 602  |
json-ptr     | has    |          | 1000000 | 1664 | 176.41%
json-pointer | has    |          | 1000000 | 2569 | 326.74%
jsonpointer  | n/a    |          | -       | -    |

.get(obj, pointer)
...
MODULE       | METHOD | COMPILED | SAMPLES | AVG  | SLOWER
json-ptr     | get    | compiled | 1000000 | 590  |
json-ptr     | get    |          | 1000000 | 1676 | 184.07%
jsonpointer  | get    | compiled | 1000000 | 2102 | 256.27%
jsonpointer  | get    |          | 1000000 | 2377 | 302.88%
json-pointer | get    |          | 1000000 | 2585 | 338.14%

.get(obj, fragmentId)
...
MODULE       | METHOD | COMPILED | SAMPLES | AVG  | SLOWER
json-ptr     | get    | compiled | 1000000 | 587  |
json-ptr     | get    |          | 1000000 | 1673 | 185.01%
jsonpointer  | get    | compiled | 1000000 | 2105 | 258.6%
jsonpointer  | get    |          | 1000000 | 2451 | 317.55%
json-pointer | get    |          | 1000000 | 2619 | 346.17%

These results have been elided because there is too much detail in the actual. Your results will vary slightly depending on the resources available where you run it.

It is important to recognize in the performance results that compiled options are faster. As a general rule, you should compile any pointers you'll be using repeatedly.

Releases

• 2022-02-02 — 3.1.0

  • fixed issue #48 wherein, when calling any of the .set() methods, which in turn rely on setValueAtPath(), and using the force argument to enable creating an object graph, one character path segments were interpreted as numbers, which resulted in unintended object graphs being created when the character was not an integer.
  • fixed borked documentation as reported in issue #44

• 2021-10-26 — 3.0.0 Potential Security Vulnerability Patched

  • When setting a value on an object graph, a developer could purposely use json-ptr to pollute an object's prototype by passing invalid path segments to the set/unset operations. This behavior has been disallowed.

• 2021-05-14 — 2.2.0 Added Handling for Relative JSON Pointers

  • Example usage

• 2021-05-12 — 2.1.1 Bug fix for #36

  • @CarolynWebster reported an unintentional behavior change starting at v1.3.0. An operation involving a pointer/path that crossed a null value in the object graph resulted in an exception. In versions prior to v1.3.0 it returned undefined as intended. The original behavior has been restored.

• 2021-05-12 — 2.1.0 Bug fixes for #28 and #30; Security Vulnerability Patched

  • When compiling the accessors for quickly accessing points in an object graph, the .get() method was not properly delimiting single quotes. This error caused the get operation to throw an exception in during normal usage. Worse, in cases where malicious user input was sent directly to json-ptr, the failure to delimit single quotes allowed the execution of arbitrary code (an injection attack). The first of these issues was reported in #28 by @mprast, the second (vulnerability) by @zpbrent. Thanks also to @elimumford for the actual code used for the fix.

  • If your code sent un-sanitized user input to the .get() method of json-ptr, your project was susceptible to this security vulnerability!

• 2020-10-21 — 2.0.0 Breaking Change

  • Prototype pollution using this library is now disallowed and will throw an error. I've been looking into the origin of this issue and it seems to have been disclosed by mohan on huntr.dev. I received a PR from @luci-m-666, but found another PR by @alromh87 that looks like the origin of the solution. Don't know who to thank, but thanks all -- somebody is due a bounty.
  • Just in case somebody was relying on json-ptr to support pointers across the prototype, I'm rolling the major version number because you're now broken.

BEWARE of Breaking Changes at v1.3.0!

• 2020-07-20 — 1.3.2

  • Added missing tslib dependency.
  • Documented where the global functions are now located; moving them broke compatibility at v1.3.0.

• 2020-07-10 — 1.3.0 BREAKING CHANGES

  • BREAKING CHANGE: Global functions are now static functions on the JsonPointer type. See Where did the Global Functions Go?
  • Merged new .unset() function contributed by @chrishalbert, updated dependencies.
  • Migrated to typescript and retooled build/test/deploy pipeline. Definitely typed.
  • 100% test coverage which illuminated some idiosyncrasies; maybe we killed unobserved bugs, nobody knows.

• 2019-09-14 — 1.2.0

  • Merged new .concat function contributed by @vuwuv, updated dependencies.

• 2019-03-10 — 1.1.2

  • Updated packages to remove critical security concern among dev dependencies'

• 2016-07-26 — 1.0.1

  • Fixed a problem with the Babel configuration

• 2016-01-12 — 1.0.0

  • Rolled major version to 1 to reflect breaking change in .list(obj, fragmentId).

• 2016-01-02 — 0.3.0

  • Retooled for node 4+
  • Better compiled pointers
  • Unrolled recursive .list function
  • Added .map function
  • Fully linted
  • Lots more tests and examples.
  • Documented many previously undocumented features.

• 2014-10-21 — 0.2.0 Added #list function to enumerate all properties in a graph, producing fragmentId/value pairs.

License

MIT

json-ptr's Issues

Retag 1.0 or release patch version

The added JSPM configuration can't be used with semantic versioning from this repo until the tag includes the new changes. I did this on my fork by creating a new tag for v1.0.0. That was all I had to do.

Getting property from array

Getting a property from a named property from an array doesn't seem to work. Example;

const
    jsonpointer = require( "jsonpointer" ),
    thing = Object.assign([ 1, 2, 3 ], { foo : "bar" } );

// What I would expect to happen
"bar" === jsonpointer.get( thing, "/foo" );

// But the result is
undefined === jsonpointer.get( thing, "/foo" );

Some background; I'm currently working with the wordpress api client for node which returns arrays extended with some properties.

get blows up when trying to extract path from a null value - failed gracefully in 1.2.0

I recently upgraded to 1.3.1 so I could use this library with Typescript, but it seems like what used to fail gracefully is now blowing up.

If I have an object like:

{
    id: 1234,
    employee: null,
    created_on: 2021-05-11
}

With 1.2.0 - if I tried to get the field "/employee/st_price" - it would fail gracefully and return undefined
With 1.3.1 - it blows up saying it can't find "st_price" in null

Uncaught TypeError: Cannot read property 'st_price' of null
    at JsonPointer.eval (eval at compilePointerDereference (util.js?06b3:1), <anonymous>:5:17)
    at JsonPointer.get (pointer.js?05e6:131)
    at Function.get (pointer.js?05e6:81)
    ....

Can't get basic example to work.

For some reason this does not work:

const target = {}
JsonPointer.set(target, '/third', 'tenth')
console.log(target) // still empty {}

Creation of non-existent elements in array doesn't work

This is related to what #10 addresses. That issue fixes when arrays don't exist to begin with but there is also the problem that when you set a value using force = true, and the path leads through an array element that doesn't exist yet, you get the final value inserted into the array rather than objects being created as expected.

ptr = require('json-ptr')

a = {
    Foo: []
}

p = ptr.create('/Foo/0/Bar/Baz');

// Succeeds but with incorrect results.
p.set(a, 5, true);
// a is now { Foo: [5] }
// a should be
// {
//   Foo: [
//     {
//       Bar: {
//         Baz: 5
//       }
//     }
//   ]
// }
console.log(a);

p = ptr.create('/Foo/Bar/Baz');

b = {
    Foo: {}
}

p.set(b, 5, true);
// Works as you'd expect.
// b is now
// {
//   Foo: {
//     Bar: {
//       Baz: 5
//     }
//   }
// }
console.log(b);

Provide @type for typescript

This project is so helpful, it's better to provide the @type for using in typescript.

Type safe JsonPointer.get

I made this type because I wanted a type safe way to pull properties from an object via a path, and I thought it might be useful to this library. I chose to make any properties not found in the object never, but you could use unknown to keep current functionality if you wanted. Just thought I'd share.

type TakeProp<T extends string> = T extends `${infer Prop}/${string}` ? Prop : never

type TakeRest<T extends string> = T extends `${string}/${infer Rest}` ? Rest : T

type PathExtract<Value, Path extends string> = 
    TakeProp<Path> extends never //if path type is not a path, get the prop from Value
        ? Path extends keyof Value
            ? Value[Path]
            : never
        : TakeProp<Path> extends keyof Value //check if prop is in value
            ? PathExtract<Value[TakeProp<Path>], TakeRest<Path>> //recurse
            : never //failure
            
const num: number = PathExtract<{a: {b: number}}, "a/b">

This currently does not cover ~0 and ~1 but i think with a little more work that could be figured out.

Multiple escaped slashes not correctly unescaped

e.g. in the following snippet, the path is decoded incorrectly:

node
> var pointer = "/a/http:~1~1www.test.com~1b/c";
> require( "json-ptr" ).decodePointer( pointer );

Actual:

[ 'a', 'http:/~1www.test.com~1b', 'c' ]

Expected:

[ 'a', 'http://www.test.com/b', 'c' ]

Note that this is critical use case when processing json-ld documents with JSON pointers

Many functions are missing in 1.3.1, despite this not being a major version

My library relies on json-ptr 1.1.1, but now that 1.3.1 is out, it's unfortunately broken my library. It seems a number of exported function have been removed, even though this is not a semver major release (create, has, decode to name a few). Could we add these back in and/or republish the last version that had these as 1.3.2, and then republish 1.3.1 as 2.0.0?

missing tslib dependency since 1.3.0

since version 1.3.0 node_modules/json-ptr/dist/index.js does require tslib but package.json does not declare this dependency. Any package depending on json-ptr that does not directly or indirectly bring tslib itself fail because of this problem.
I've noticed your package-lock.json contains tslib so it's possibly why you didn't notice this issue.

Support relative json pointers

I discovered that these exist when i was looking through the ajv code.

Spec: https://tools.ietf.org/id/draft-handrews-relative-json-pointer-00.html

Support for delete() operation

First off, I just want to say thank you for your contributions. I was looking for something that abides to the JSON Pointer RFC and this fulfills all of the specs.

I did want to request an added feature. The delete() operation would simply delete that pointer from the object.

I'll have a PR available shortly if this doesn't bloat the codebase too much. Thanks again!

Updating from 0.2.0 to 0.3.1 introduces a breaking change to the list method

$ npm install [email protected]
$ node
> require('json-ptr').list({'key1': 'value1', 'key2': { 'key3': 'value3' } })
[ { fragmentId: '#', value: { key1: 'value1', key2: [Object] } },
  { fragmentId: '#/key1', value: 'value1' },
  { fragmentId: '#/key2', value: { key3: 'value3' } },
  { fragmentId: '#/key2/key3', value: 'value3' } ]
...
$ npm install [email protected]
$ node
> require('json-ptr').list({'key1': 'value1', 'key2': { 'key3': 'value3' } })
[ { pointer: '', value: { key1: 'value1', key2: [Object] } },
  { pointer: '/key1', value: 'value1' },
  { pointer: '/key2', value: { key3: 'value3' } },
  { pointer: '/key2/key3', value: 'value3' } ]

Undocumented exported functions?

The functions defined in src/util.ts are exported, but are not mentioned in README.md and only their signatures are documented by TypeDoc. Could you clarify whether these are intended to be part of the public API?

Thanks,
Kevin

`get()` blows up when single quotes are in the path

Short repro via REPL using json-ptr 2.0.0:

> const ptr = require('json-ptr')
undefined

> badPointer = new JsonPointer("/I'm/bad")
JsonPointer { path: [ 'I\'m', 'bad' ] }

> badPointer.get({}) // expecting this to return undefined
Thrown:
SyntaxError: Unexpected identifier

> badPointer = new JsonPointer(["I'm", "also", "bad"])
JsonPointer { path: [ 'I\'m', 'also', 'bad' ] }

> badPointer.get({}) // expecting this to return undefined
Thrown:
SyntaxError: Unexpected identifier

Looks like RFC 6901 mentions that quotes must be escaped in paths, but I couldn't find anything in the json-ptr docs that mentioned what is or isn't escaped by the library. Worth mentioning that it seems to work with other tricky inputs - for example, forward slashes in paths seem to be handled correctly.

Literal property value 'e' is interpreted as a number

It appears that 'e' in a pointer path is interpreted as a number, which seems wrong since JS does not generally treat 'e' as a constant number and exponential notation requires a preceding digit ('1e7'). Adding an extra non-numeric character to the path, i.e. '/hi/eX', makes it behave as expected.

const { JsonPointer } = require('json-ptr');

// run `node index.js` in the terminal to repro

const obj = {};

const ptr = JsonPointer.create('/hi/e');

ptr.set(obj, 'hello', true);

console.log(`result`, obj);

// expected:
// { hi: {e: 'hello'} }

// got:
// { hi: [] }

Run it here:
https://stackblitz.com/edit/node-a9wd7r?file=index.js

Arbitrary Code Execution in JsonPointer.get

Hey maintainers @cehoffman @mortonfox @treybrisbane @chrishalbert , I have opened a PR (418sec#3) which will fix the potential arbitrary code execution vulnerability in json-ptr. Please take a review.

If you are fine with that fix, please comment @huntr-helper - LGTM at 418sec#3, or if you need any modifications, please also comment on that PR. Thanks.

Ref: 418sec#3

Mark https://www.npmjs.com/advisories/1706 as fixed

This library, as you know, has an outstanding vulnerability notice:
https://www.npmjs.com/advisories/1706

However I believe you actually patched/fixed this in 2.1.0, it's just that NPM does not seem to know. I don't know how these things work, but could you get NPM to mark this as resolved at the proper version?

Document supported Node.js versions

It appears that [email protected] requires Node.js 14 or later due to use of optional chaining. Presumably this is intentional? It would be nice if this were mentioned in the README or in the engines field in package.json.

Thanks for considering,
Kevin

Documentation unavailable (http 404)

Documentation for both the 'JsonPointer' and 'JsonReference' class appears to be unavailable.
To reproduce:
Go to the documentation website https://flitbit.github.io/json-ptr/
Click 'JsonPointer' on the right hand side
See HTTP 404.

filtering query by index returns undefined result

Hi, thanks for this project and for json-path! I have to admit I'm not 100% clear on the difference/relation between the two, so if I've reported this issue in the wrong queue, please let me know ;)

I've got a number of queries with paths like #/parent_array_field[*]/child_array_field/0, meaning that I want to fetch the first element from the child array that may be found in any of the parent fields (common in GraphQL result sets). However, if I add an additional array filter on the end to try to filter the entire result set, à la "#/parent_array_field[*]/child_array_field/0[0]", the resolution returns undefined. Is this a bug, or am I doing it wrong?

Thanks again!

Use of ES6 prevents minification with UglifyJS

The variable statements like let and the use of template strings here are unable to be minified as tracked in mishoo/UglifyJS#448.

It would be nice if json-ptr in the releases directory at least was transpiled to ES5 for wider browser compatibility too. I notice you have babel in the dependencies and creating releases, but it letting the let, string templates, and class definitions through.