apidevtools / json-schema-ref-parser

Parse, Resolve, and Dereference JSON Schema $ref pointers in Node and browsers

JSON Schema $Ref Parser

Parse, Resolve, and Dereference JSON Schema $ref pointers

Installation

Install using npm:

npm install @apidevtools/json-schema-ref-parser
yarn add @apidevtools/json-schema-ref-parser
bun add @apidevtools/json-schema-ref-parser

The Problem:

You've got a JSON Schema with $ref pointers to other files and/or URLs. Maybe you know all the referenced files ahead of time. Maybe you don't. Maybe some are local files, and others are remote URLs. Maybe they are a mix of JSON and YAML format. Maybe some of the files contain cross-references to each other.

{
  "definitions": {
    "person": {
      // references an external file
      "$ref": "schemas/people/Bruce-Wayne.json"
    },
    "place": {
      // references a sub-schema in an external file
      "$ref": "schemas/places.yaml#/definitions/Gotham-City"
    },
    "thing": {
      // references a URL
      "$ref": "http://wayne-enterprises.com/things/batmobile"
    },
    "color": {
      // references a value in an external file via an internal reference
      "$ref": "#/definitions/thing/properties/colors/black-as-the-night"
    }
  }
}

The Solution:

JSON Schema $Ref Parser is a full JSON Reference and JSON Pointer implementation that crawls even the most complex JSON Schemas and gives you simple, straightforward JavaScript objects.

• Use JSON or YAML schemas — or even a mix of both!
• Supports $ref pointers to external files and URLs, as well as custom sources such as databases
• Can bundle multiple files into a single schema that only has internal $ref pointers
• Can dereference your schema, producing a plain-old JavaScript object that's easy to work with
• Supports circular references, nested references, back-references, and cross-references between files
• Maintains object reference equality — $ref pointers to the same value always resolve to the same object instance
• Compatible with Node LTS and beyond, and all major web browsers on Windows, Mac, and Linux

Example

import $RefParser from "@apidevtools/json-schema-ref-parser";

try {
  await $RefParser.dereference(mySchema);
  // note - by default, mySchema is modified in place, and the returned value is a reference to the same object
  console.log(mySchema.definitions.person.properties.firstName);

  // if you want to avoid modifying the original schema, you can disable the `mutateInputSchema` option
  let clonedSchema = await $RefParser.dereference(mySchema, { mutateInputSchema: false });
  console.log(clonedSchema.definitions.person.properties.firstName);
} catch (err) {
  console.error(err);
}

For more detailed examples, please see the API Documentation

Polyfills

If you are using Node.js < 18, you'll need a polyfill for fetch, like node-fetch:

import fetch from "node-fetch";

globalThis.fetch = fetch;

Browser support

JSON Schema $Ref Parser supports recent versions of every major web browser. Older browsers may require Babel and/or polyfills.

To use JSON Schema $Ref Parser in a browser, you'll need to use a bundling tool such as Webpack, Rollup, Parcel, or Browserify. Some bundlers may require a bit of configuration, such as setting browser: true in rollup-plugin-resolve.

Webpack 5

Webpack 5 has dropped the default export of node core modules in favour of polyfills, you'll need to set them up yourself ( after npm-installing them ) Edit your webpack.config.js :

config.resolve.fallback = {
  "path": require.resolve("path-browserify"),
  'fs': require.resolve('browserify-fs')
}

config.plugins.push(
  new webpack.ProvidePlugin({
    Buffer: ['buffer', 'Buffer']
  })
)

API Documentation

Full API documentation is available right here

Contributing

I welcome any contributions, enhancements, and bug-fixes. Open an issue on GitHub and submit a pull request.

Building/Testing

To build/test the project locally on your computer:

1. Clone this repo
   git clone https://github.com/APIDevTools/json-schema-ref-parser.git

2. Install dependencies
   yarn install

3. Run the tests
   yarn test

License

JSON Schema $Ref Parser is 100% free and open-source, under the MIT license. Use it however you want.

This package is Treeware. If you use it in production, then we ask that you buy the world a tree to thank us for our work. By contributing to the Treeware forest you’ll be creating employment for local families and restoring wildlife habitats.

Big Thanks To

Thanks to these awesome companies for their support of Open Source developers ❤

json-schema-ref-parser's Issues

Loading over https doesn't work

https protocol is ignored and specs are always loaded over http

Check out the attached picture. Despite spec url starts from https:// Chrome Debugger Network tab says that spec was loaded over http:

I've done some research and found out that the issue is caused by this issue: https://github.com/substack/https-browserify/issues/6

So I believe to fix this issue you just need to update browserify version to v13 in this and maybe in BigstickCarpet/simplifyify package

Bug: Root level $ref not being parsed

var input = { "$ref":"http://192.168.99.100/schema/sandbox#/definitions/item" };
$RefParser.dereference(input, function(err, schema) {
        if (err) {
            console.error(err);
            return done = true;
        }
         console.log(schema); 
});
// console:
// { '$ref': 'http://192.168.99.100/schema/sandbox#/definitions/item' }

Maximum call stack size exceeded with self-referencing objects on .resolve

The following causes RangeError: Maximum call stack size exceeded:

const obj = {
  definitions: {
    Self: {
      properties: {
        self: { $ref: "#/definitions/Self" }
      }
    }
  }
}

require("json-schema-ref-parser").resolve(obj, (err, schema) => console.log(err, schema))

Ref parser is not resolving direct references.

Following schema will not be resolved.

{
  "$schema": "http://json-schema.org/schema#",
  "$id": "test-schema",
  "definitions": {
    "i_ref_obj": {
      "type": "object",
      "properties": {
        "file": {
          "type": "string"
        }
      },
      "required": [
        "file"
      ]
    }
  },
  "$ref": "#/definitions/i_ref_obj"
}

[v1.4.1] Webpack with CORS

When the fetched URL exposes the CORS header 'Access-Control-Allow-Origin': '*', the request fails.

A wildcard '*' cannot be used in the 'Access-Control-Allow-Origin' header when the credentials flag is true.

Webpack is using this lib as a http module https://github.com/substack/http-browserify. And for some reason, the lib set by default xhr.withCredentials to true.

Adding withCredentials: false on params given to http module solves the issue https://github.com/BigstickCarpet/json-schema-ref-parser/blob/v1.4.1/lib/read.js#L176

var req = protocol.get(
  {
    hostname: u.hostname,
    port: u.port,
    path: u.path,
    auth: u.auth,
    withCredentials: false,
  },
  onResponse
);

References through other references being produced

Sorry if this is a bit confusing to understand, but I have a complicated schema with many (circular) references, and I'm using the bundle() method then passing the results to z-schema which can't understand it.

My main schema (minus a lot of irrelevant detail) is:

model.yaml

$schema: http://json-schema.org/draft-04/schema#
type: object
title: Model
properties:
    body:
        type: array
        items:
            $ref: section.yaml
        minItems: 1
    body2:
        type: object
        properties:
            content:
                type: array
                items:
                    $ref: any.yaml
                minItems: 1

Then the following are referenced:

any.yaml

oneOf:
  - $ref: list.yaml
  - $ref: section.yaml
  - $ref: text.yaml

list.yaml

$schema: http://json-schema.org/draft-04/schema#
type: object
title: List
properties:
    items:
        type: array
        items:
            oneOf:
              - $ref: list.yaml
              - $ref: text.yaml
        minItems: 1
required:
  - items

section.yaml

$schema: http://json-schema.org/draft-04/schema#
type: object
title: Section
properties:
    content:
        type: array
        items:
            $ref: any.yaml
        minItems: 1
required:
  - content

text.yaml

$schema: http://json-schema.org/draft-04/schema#
type: object
title: Text
properties:
    text:
        type: string
required:
  - text

The bundle() method of this library returns:

{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "type": "object",
    "title": "Model",
    "properties": {
        "body": {
            "type": "array",
            "items": {
                "$schema": "http://json-schema.org/draft-04/schema#",
                "type": "object",
                "title": "Section",
                "properties": {
                    "content": {
                        "type": "array",
                        "items": {
                            "$ref": "#/properties/body2/properties/content/items"
                        },
                        "minItems": 1
                    }
                },
                "required": [
                    "content"
                ]
            },
            "minItems": 1
        },
        "body2": {
            "type": "object",
            "properties": {
                "content": {
                    "type": "array",
                    "items": {
                        "oneOf": [
                            {
                                "$schema": "http://json-schema.org/draft-04/schema#",
                                "type": "object",
                                "title": "List",
                                "properties": {
                                    "items": {
                                        "type": "array",
                                        "items": {
                                            "oneOf": [
                                                {
                                                    "$ref": "#/properties/body/items/properties/content/items/oneOf/0"
                                                },
                                                {
                                                    "$ref": "#/properties/body/items/properties/content/items/oneOf/2"
                                                }
                                            ]
                                        },
                                        "minItems": 1
                                    }
                                },
                                "required": [
                                    "items"
                                ]
                            },
                            {
                                "$ref": "#/properties/body/items"
                            },
                            {
                                "$schema": "http://json-schema.org/draft-04/schema#",
                                "type": "object",
                                "title": "Text",
                                "properties": {
                                    "text": {
                                        "type": "string"
                                    }
                                },
                                "required": [
                                    "text"
                                ]
                            }
                        ]
                    },
                    "minItems": 1
                }
            }
        }
    }
}

When using z-schema to parse (any) JSON it errors with:

[
    {
        "code": "UNRESOLVABLE_REFERENCE",
        "params": [
            "#/properties/body/items/properties/content/items/oneOf/0"
        ],
        "message": "Reference could not be resolved: #/properties/body/items/properties/content/items/oneOf/0",
        "path": "#/properties/body2/properties/content/items/oneOf/0/properties/items/items/oneOf/0"
    },
    {
        "code": "UNRESOLVABLE_REFERENCE",
        "params": [
            "#/properties/body/items/properties/content/items/oneOf/2"
        ],
        "message": "Reference could not be resolved: #/properties/body/items/properties/content/items/oneOf/2",
        "path": "#/properties/body2/properties/content/items/oneOf/0/properties/items/items/oneOf/1"
    }
]

The (first) problem is that the reference at #/properties/body2/properties/content/items/oneOf/0/properties/items/items/oneOf/0 ("$ref": "#/properties/body/items/properties/content/items/oneOf/0") goes through another $ref. According to https://github.com/json-schema/json-schema/wiki/$ref-traps#references-through-other-references this should still be resolvable (so the bug could be in z-schema not understanding it), but the bundle() method could have produced "$ref": "#/properties/body2/properties/content/items/oneOf/0" which would avoid the problem.

I hit this problem yesterday and reordering some properties (hence reordering some $refs) fixed it a couple of times, but I can't find a way around the latest instance.

dereference() fails to deference circular ref

Hi!

I'm using json-schema-ref-parser to power ref resolution in https://github.com/bcherny/json-schema-to-typescript. Thank you for the great work - it makes my life much easier.

I am running into an issue with circular $refs. Here is the simplest repro case:

Input:

{
  properties: {
    foo: {
      $ref: '#/properties/foo'
    }
  }
}

Output (actual):

{
  properties: {
    foo: {
      $ref: '#/properties/foo'
    }
  }
}

Output (expected):

{
  properties: {
    foo: {
      $ref: (reference to properties/foo)
    }
  }
}

This is likely related to #36.

Inline dereferencing and fragments

json-schema-ref-parser appears to not support a portion of the JSON Schema spec regarding inline dereferencing (see http://json-schema.org/latest/json-schema-core.html#anchor31)

Reproducer:
`
var test = {
"id": "http://some.site/schema#",
"not": { "$ref": "#inner" },
"definitions": {
"schema1": {
"id": "#inner",
"type": "boolean"
}
}
};

deref.dereference(test, function(err, schema){
console.log(err, schema);
});
`

Output:

"name": "SyntaxError", "message": "Invalid $ref pointer \"inner\". Pointers must begin with \"#/\"", "stack": "SyntaxError: Invalid $ref pointer \"inner\". Pointers must begin wit h \"#/\"

I understand that this is not a required part of the spec, however it is a SHOULD given that the implementation does support inline dereferencing in general.

Max call stack if schema already has circular references

If the schema includes circular references already a max call stack is hit.

let x = { y: 1, z: {} };
x.z.x = x;
await jsref.dereference(x);
// Error
``

$ref attribute containing forward slashes

When trying to $ref an element, that contains '/' in it's name the pointer cannot be found.
I found the specification for escaping '/' characters here: https://tools.ietf.org/html/rfc6901#page-3

I'm trying to reference the following element:

paths:
  /airport/HAM:
    get:
      ...

of a YML file and reference it from another YML by using:

$ref: "#/paths/~1aircraft~1HAM/get"

yet, I cannot get it to stay escaped properly, as the second '~1' gets evaluated too early giving me the error message:

Token \"/aircraft/HAM\" does not exist.",
  "stack": "SyntaxError: Error resolving $ref pointer \"/Users/someUser/airport.yaml#/paths/~1aircraft~HAM/get\".

I would appreciate any hints or help!

Enhancement: Include a command-line version to use json-schema-ref-parser via CLI

Include a command-line version with the npm package so that json-schema-ref-parser can be simply used via CLI. This will make it easier to use this tool via CICD/build systems and also eliminate the need for writing a wrapper js script to consume this functionality.

Something like:

json-schema-ref-parser bundle --dereference {infile} {outfile}

internal reference in schema

I am trying to dereference resolved schema object. But it is throwing error as "Error resolving $ref pointer "#/definitions/ApiCallList". Token "definitions" does not exist."

Is this a bug or What is the correct way to define $ref to refer internal definition within the schema object.

$RefParser = require('json-schema-ref-parser') $RefParser.dereference(schema, { $refs: { circular: "ignore" } }) .then(function(resolvedSchema) { res.send(resolvedSchema); })

Sample Schema
{ "definitions":{ "ApiCallList":{ "description":"A list of Api Call Resources", "id":"ApiCallList" }, "MaskCallList":{ "description":"A list of Mask Call Resources", "id":"MaskCallList" } }, "resources":{ "api-call":{ "methods":{ "request":{ "$ref":"#/definitions/ApiCallList }, "response":{ "$ref":"#/definitions/MaskCallList } } } } }

Some instructions are missing in Readme, Can't run karma tests

Hi,
I want to potentially use this library in an enterprise environment, and having a security flag ( even on a debug library ) is not a good thing from their point of view.

So I wanted to upgrade 'debug' to ^3.1.0, and I ran into some build issues from master.

Running on:
node v6.11.3
npm v3.10.10

Missing requirements from Readme:
npm install -g karma mocha karma-cli

When running npm test, the karma step fails due to missing plugins.
I have had to modify karma.conf.js

I have made a build that fixes those issues, I can open a PR with the changes .
I can submit another PR for debug library version change.

Cheers,

Dereferencing fails on Windows with CWD C:

To reproduce, create test.js somewhere:

var parser = require( 'json-schema-ref-parser' );
var path = require( 'path' );

parser.dereference( path.join( __dirname, 'test.yaml' ) );

Running test.js from cmd in working directory C:\ fails, running in C:\Users behaves correctly:

C:\>node C:\Users\me\workspace\json-schema-ref-parser-bug-cwd-c\test.js
(node:8048) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejec
tion id: 5): SyntaxError: Unable to resolve $ref pointer "c:///Users/me/
workspace/json-schema-ref-parser-bug-cwd-c/test.yaml"

C:\>cd Users

C:\Users>node C:\Users\me\workspace\json-schema-ref-parser-bug-cwd-c\tes
t.js
(node:9092) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejec
tion id: 4): SyntaxError: "c:/Users/me/workspace/json-schema-ref-parser-
bug-cwd-c/test.yaml" is not a valid JSON Schema

Fails to resolve multi-file #/definitions/ references

I have my definitions in separate files, and the schemas in my paths link to them through the definitions object.

Swagger index file:
... definitions: $ref: './definitions/index.yaml' ...

definitions/index.yaml:
... user: $ref: './main/user.yaml' ...

random path file:
... get: responses: '200': schema: $ref: '#/definitions/user' ...

When I try to dereference this, I get the following error:

SyntaxError: Error resolving $ref pointer "paths/main/user.yaml#/definitions/user". ,
"Token "definitions" does not exist.",

Is this supported? What am I doing wrong?

Add possibility to specify location when you pass object

If you pass object it's impossible to resolve external relative references.
It would be cool if I can add option where I can specify base path for relative refs.

Error resolving $ref for property alongside top-level $ref

All,

I'm having an issue resolving a reference. Having tested a couple scenarios, I was happy to see that providing a top-level reference and included some other properties alongside the $ref netted me a merged/combined object, for example:

Dereferencing this:

{
    "$ref":"http://someuri/myOtherSchema",
    "anotherProperty":"whatever"
}

Results in this (imagining myOtherSchema contains the added properties):

{
    "somePropertyFromMyOtherSchema": "misc",
    "anotherPropertyFromMyOtherSchema":"more",
    "anotherProperty": "whatever"
}

However, if I attempt to dereference the following:

{
    "$ref": "http://someuri/foo",
    "anotherProperty": {"$ref":"http://someuri/blah/some/field/value"}
}

It results in an error ('Error resolving $ref: http://someuri/blah/some/field/value'). It's worth noting that I am using a custom resolver and I changed the $ref urls for this post but removing the top-level $ref results in success i.e. my resolver is working as expected and in fact is a simple in-memory resolver. When this error happens the $ref mentioned in the error never even hits my resolver and its resolution is never attempted. Is having a top-level $ref alongside a property that points to a $ref expressly forbidden by the specification, is this a flaw or a feature of json-schema-ref-parser?

Syncronous API

Hey!

You did a great job here!

I need to use your API syncronously ( because I need it in a syncronous framework). Is there any plan to do it in the future?

Thank you

Module not found: Error: Cannot resolve module 'fs'

I am using the babel generator boilerplate for a library I am building. I added this library and imported it into my project. Now when I run gulp build I get the following error. Any assistance would be highly appreciated.

stream.js:74
      throw er; // Unhandled stream error in pipe.
      ^
Error: ./~/json-schema-ref-parser/lib/resolvers/file.js
Module not found: Error: Cannot resolve module 'fs' in /home/raphael/work/projects/packages/json-schema-form-generator/node_modules/json-schema-ref-parser/lib/resolvers
resolve module fs in /home/raphael/work/projects/packages/json-schema-form-generator/node_modules/json-schema-ref-parser/lib/resolvers
  looking for modules in /home/raphael/work/projects/packages/json-schema-form-generator/node_modules
    /home/raphael/work/projects/packages/json-schema-form-generator/node_modules/fs doesn't exist (module as directory)
    resolve 'file' fs in /home/raphael/work/projects/packages/json-schema-form-generator/node_modules
      resolve file
        /home/raphael/work/projects/packages/json-schema-form-generator/node_modules/fs doesn't exist
        /home/raphael/work/projects/packages/json-schema-form-generator/node_modules/fs.webpack.js doesn't exist
        /home/raphael/work/projects/packages/json-schema-form-generator/node_modules/fs.web.js doesn't exist
        /home/raphael/work/projects/packages/json-schema-form-generator/node_modules/fs.js doesn't exist
        /home/raphael/work/projects/packages/json-schema-form-generator/node_modules/fs.json doesn't exist
[/home/raphael/work/projects/packages/json-schema-form-generator/node_modules/fs]
[/home/raphael/work/projects/packages/json-schema-form-generator/node_modules/fs]
[/home/raphael/work/projects/packages/json-schema-form-generator/node_modules/fs.webpack.js]
[/home/raphael/work/projects/packages/json-schema-form-generator/node_modules/fs.web.js]
[/home/raphael/work/projects/packages/json-schema-form-generator/node_modules/fs.js]
[/home/raphael/work/projects/packages/json-schema-form-generator/node_modules/fs.json]
 @ ./~/json-schema-ref-parser/lib/resolvers/file.js 2:14-27

"refresh" a ref?

Is there any way I can "refresh" a ref that has been already dereferenced?

I have a json file that dereferences from an external API, and the data might change over time...

Fails to deref https schemas, strips https off URL, ignores 301 redirect to https

url == //json.readingpl.us/schemas/alertType.json

url SHOULD equal https://json.readingpl.us/schemas/alertType.json

When redirected to HTTPS resource, it instead requests the HTTP resource again

Relative $ref pointer not working for files

I have a nodejs application and a folder where all schema reside .
The folder are different for both as given below :

Node js application run here
-- /home/ubuntu/work/node_webhook/webhook.js

Json schema files are stored here :

-- /home/ubuntu/work/analytics/data_types/
------- mobile_schema
------------- test_schema.json
------------- test_2_schema.json
-------- enums.json
-------- common_data_types.json

test_schema.json looks like this :

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "object",
  "properties": {
    "amount": {
      "type": "integer"
    },
    "created_at_date": {
      "$ref": "../common_data_types.json#/common_types/date_time"
    },
    "current_stage": {
       "$ref": "../enum_server.json#/all_enum/CURRENT_STAGE"
    }
  },
  "required": [
    "amount",
    "created_at_date",
    "current_stage"
  ]
}

common_data_types.json looks like this :

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "object",
  "common_types": {
    "date_time": {
      "type": "string",
      "format": "date-time"
    }
  }
}

enums.json looks like this

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "all_enum": {
    "PAYMENT_METHOD": {
      "type": "string",
      "enum": [
        "ONLINE",
        "OFFLINE"
      ]
    },
    "CURRENT_STAGE": {
      "type": "string",
      "enum": [
        "ONE",
        "TWO",
        "THREE"
      ]
    }
  }
}

AND NODE JS APLICATION FILE : webhook.js looks like this :

var express = require('express');
var app = express(); 
var $RefParser = require('json-schema-ref-parser');
var Validator = require('jsonschema').Validator;

var v = new Validator();
var parser = new $RefParser();

app.use(bodyParser.urlencoded({ extended: true }));
app.use(bodyParser.json());
var router = express.Router();              // get an instance of the express Router


// middleware to use for all requests
router.use(function(req, res, next) {
    console.log('Got a request.');
    next(); // make sure we go to the next routes and don't stop here
});


function validateEventSchema(req,res){
//console.log(req.body); 
var event_data = req.body.properties;
var parser = new $RefParser();
var event_schema = parser.bundle("/home/travel/work/trips3m/lib/analytics/data_types/mobile_schema/test_schema.json");

parser.dereference(event_schema, function(err, schema) {
  console.log(parser.$refs.paths())
  valid = false;
  if (err) {
    console.error(err);
  }
  else {
    //console.log(schema.properties.ts.type);
    c = v.validate(event_data, schema);
    if(c.errors.length > 0 ){
      console.log("ERROR OCCUR");
    }else{
      valid = true ; 
    }
  }

  console.log(valid)
}

router.route('/dump_thy_data')
  .post(function(req, res) {
      validateEventSchema(req,res)
  });

app.use('/', router);
app.listen(process.env.APP_PORT);

Getting error as it try to resolve ref pointers in current directory in which nodejs application is running . That is : /home/ubuntu/work/node_webhook/
So it cannot find
/home/ubuntu/work/node_webhook/enums.json
/home/ubuntu/work/node_webhook/common_data_types.json

How will i provide it a base directory to load all json files from . Or how to use relative file path ?

parsers/resolvers get partial file URL for json-pointer refs

description

when a schema definition includes $refs with (non-relative) json-pointer values, any parsers/resolvers hook functions (namely, read(), parse(), canRead() and canParse()) are called with a file info object that has a partial URL.

"partial URL" here refers to the original json-pointer URI, stripped of its anchor segment, for example, http://mydomain/schema#/inner would become http://mydomain/schema.

this behavior is the same for both bundle() and dereference() (other API methods had not been tested).

it makes handling of such json-pointer $refs harder when writing custom resolvers/parsers (my use case is returning 'mock' or 'placeholder' schemas for some user defined $ref matches in a generated bundle).

steps to reproduce

1. create a schema definition with a json-pointer $ref, e.g.:

   const mySchema = {
     "$ref": "http://mydomain.com/schema#/inner"
   };

2. implement any of the custom resolvers/parsers hook functions, and test the file.url value:

   parser.bundle(mySchema, {
     resolve: {
       myResolver: {
         order: 1,
         canRead: (file) => {
           console.log(file.url); // -> http://mydomain.com/schema
           return true;
         },
       },
     },
   });

see a runnable demo on runkit.

expected result

the file info object's url field should hold the $ref value as found in the original schema definition.

actual result

the file info object's url field hold a partial value, in case of json-pointer URI values - the anchor segment is stripped out.

workarounds

• i think one can retrieve the last value of parser.$refs.paths() and have the full URL for a specific invocation of parse(), canRead() etc., but that's quite brittle - as the $refs.paths() API may change in future versions, and also not very pure (in a FP sense).
  i haven't tested this approach, though. apparently, this approach doesn't work; parser.$refs.paths() returns the same stripped path.

[v1.4.1] On browser, doesn't use the right protocol and port.

When using this lib served on localhost, the lib doesn't fetch json urls with the right port and protocol.

Some params aren't passed to http module which makes http lib resolve url to fetch according to current protocol and port.

Replacing this block at https://github.com/BigstickCarpet/json-schema-ref-parser/blob/v1.4.1/lib/read.js#L176

var req = protocol.get(
  {
    hostname: u.hostname,
    port: u.port,
    path: u.path,
    auth: u.auth
  },
  onResponse
);

by the following, solve the issue:

var req = protocol.get(
  {
    protocol: u.protocol,
    host: u.host,
    hostname: u.hostname,
    port: u.port,
    path: u.path,
    auth: u.auth
  },
  onResponse
);

No possibility to set xhr.withCredentials to false

CORS requests to servers with wildcarded Access-Control-Allow-Origin fail. For example the following code:

SwaggerParser.bundle("http://petstore.swagger.io/v2/swagger.json")

fails with the following error:

XMLHttpRequest cannot load http://petstore.swagger.io/v2/swagger.json. A wildcard '*' cannot be used in the 'Access-Control-Allow-Origin' header when the credentials flag is true. Origin 'http://127.0.0.1:9000' is therefore not allowed access.

The same time the following code loads schema:

var xhr= new XMLHttpRequest();
xhr.open("GET", "http://petstore.swagger.io/v2/swagger.json", true);
xhr.send();
// xhr.response contains schema after request finishes

I've some investigation and have found out that the problem is in the browserify http module. By default it sets withCredentials option of XMLHttpRequest to true.

It would be great to have some $RefParser option to control this behavior.

$refs files to definitions

Hello,
Is there a way to use this tool to replaces all $ref pointing to files by definitions ?

Intermittent failures resolving filesystem $ref's

We have discovered that json-schema-ref-parser will intermittently fail resolving $ref's from the local filesystem. The failure symptom is attempting to resolve the $ref relative to the current working directory, instead of the referring document. This is estimated to happen about 1 of every 10 times.

  1) PUT /users/{UserId} -> 200 "before all" hook: bound :
     Uncaught Error: Unable to expand schema:
{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "$ref": "../definitions/user.json#"
}
Error:
Error: Error opening file "/Users/coltonlw/projects/definitions/user.json"
ENOENT: no such file or directory, open '/Users/coltonlw/projects/definitions/user.json'
      at /usr/local/lib/node_modules/abao/lib/test.coffee:134:19
      at /usr/local/lib/node_modules/abao/node_modules/call-me-maybe/index.js:13:28
      at _combinedTickCallback (internal/process/next_tick.js:67:7)
      at process._tickCallback (internal/process/next_tick.js:98:9)

json-schema-ref-parser is attempting to load "/Users/coltonlw/projects/definitions/user.json", however the directory structure is like this:

/Users/coltonlw/projects/core/raml/schemas/definitions/user.json

With the referring document located at

/Users/coltonlw/projects/core/raml/schemas/output/user.json

Exclude in-file $ref definitions?

Is it possible to limit only the $ref resolutions to external files only? I wish to avoid references that are within the file itself.

i.e. ignore those '#/definitions/MyDef' lines which refer to definitions within the file itself.

Is this possible?

Custom resolver casing change

Hi, thanks for the really great library. However, I think I have spotted a bug and can't seem to figure out where/why it's happening. Given an object like this:

{
  "$ref": "custom://Path/Is/Case/Sensitive"
}

And a custom resolver like the following:

module.exports = {
  order: 1,

  canRead: /^custom:\/\//i,

  read: function(file) {
    console.log(file.url);
    ...
  },
  ...
}

The url value logged to the console would be: "custom://path/Is/Case/Sensitive", note the p casing change. Is this something I'm doing or possibly a bug?

Syntax erro - Pointers must begin with "#/"

Hi there, I just started getting this error today. Not sure what changed or how to fix it.

Any ideas?

'new' is a reserved keyword

Hey,
first up, great work on this monster! :) Thanks a lot.
I was running some code with it on an old Samsung phone, and the JS parser in that browser exploded because of the object property new, for example here: https://github.com/BigstickCarpet/json-schema-ref-parser/blob/master/dist/ref-parser.js#L51

Turns out the browser thinks you cannot use it there because it's a reserved keyword. Is there any chance that you could rename this property?

Thanks!
Tim

Circular reference causes "Maximum call stack size exceeded"

Here's my schema (simplified from the Gmail API's Message object)

{
  "definitions": {
    "MessagePart": {
      "type": "object",
      "properties": {
        "parts": {
          "type": "array",
          "items": {"$ref": "#/definitions/MessagePart"}
        }
      }
    }
  },
  "title": "Message",
  "type": "object",
  "properties": {
    "part": {"$ref": "#/definitions/MessagePart"}
  }
}

> let rp = require('json-schema-ref-parser')
> rp.dereference(sch).then(r => console.log(r))
Promise { <pending> }
> (node:11443) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 4): RangeError: Maximum call stack size exceeded

Overriding port for absolute refs when run from browser [3.1.2]

I've got a schema with absolute urls to resources with no port number in the URL...

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "id": "http://localhost/schema/01/plantdb.json",
  "type": "object",
  "properties": {
    "names": {"$ref": "http://localhost/schema/01/names.json"}
  }
}

(greatly simplified for clarity)

and I'm loading this on the client from an app hosted at localhost:3000. Trying to dereference a schema file hosted on the same hostname, but without specifying a port number...

parser.dereference('http://localhost/schema/01/plantdb.json')

actually tries to load the schema from http://localhost:3000/schema/01/plantdb.json and fails with an invalid schema error. If I explicitly add the port number to the schema URL:

parser.dereference('http://localhost:80/schema/01/plantdb.json')

then that schema file loads, but all refs within are still loaded from port http://localhost:3000.

Is this a bug, or is there a config somewhere I'm missing to force the port number on ref requests?

Expand schemas

Original spec from here: jdorn/json-editor#49

{
  "type": "object",
  "allOf": [
    {
      "properties": {
        "key1": {
          "type": "string"
        }
      },
      "required": ["key1"]
    },
    {
      "properties": {
        "key2": {
          "type": "string"
        }
      },
      "required": ["key2"]
    }
  ]
}

Should produce the following after it's expanded:

{
  "type": "object",
  "properties": {
    "key1": {
      "type": "string"
    },
    "key2": {
      "type": "string"
    }
  },
  "required": ["key1","key2"]
}

Moved from APIDevTools/swagger-parser#15

How to specify $ref base url?

I serve a frontend locally with node at localhost:8000. I pull the json schema from an API.
The backend returning the json schema seats at localhost:8888 (http://127.0.0.1:8888/api/controllers/schema).

The schema has arrays of $ref fields in form of relative urls :
var schema = [{$ref:"/api/controllers/123"},{...}].

json-schema-ref-parser resolves references relatively to the frontend url rather than the backend url. $RefParser.resolve(schema); throws the following errors:

• request.js:114 GET http://localhost:8000/api/controllers/123 404 (Not Found)
• index.js:67 Uncaught (in promise) Error: Error downloading http://localhost:8000/api/controllers/123.

How can I pass localhost:8888 to resolve as base url?

Redirect (301) aren't followed

Hi,

the server is configured to redirect all traffic from http to https.

Example:
[http://v22017084844652188.powersrv.de/schemas/test.json](HTTP address).

Calling this

parser.dereference("http://v22017084844652188.powersrv.de/schemas/test.json")
      .then((value) => {
        console.log("Schema fetched...");
        this.parserSchema = value;
      })
      .catch(error => {
        console.log("Schema not fetched...");
        console.log(error);
        this.parserError = error;
      });

I'm getting the following error (running into a CORS problem):

Failed to load http://v22017084844652188.powersrv.de/schemas/test.json: Redirect from 'http://v22017084844652188.powersrv.de/schemas/test.json' to 'https://v22017084844652188.powersrv.de/schemas/test.json' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:8100' is therefore not allowed access. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

A redirect has to be detected and the the ressource should be dereferenced!
@BigstickCarpet Should the redirects handled in http.js?

Change reference key name

Currently the library can only parse keys with the name $ref. This matches the JSON Schema specifications, but sometimes it may be useful to rename the key to for example $ref-parse.

One use case is with Swagger. The OpenAPI specification allows references in some places but disallows them in others. By using a special reference tag you could parse some of the references beforehand with json-schema-ref-parser and then allow some of them to pass through to the OpenAPI which can handle the rest of them.

dereferencing ref of refs in the same path

I'm trying to dereference some schemas, but seems that it does not resolve the urls the way I need... I'm probably missing some option here.

I have all the schemas sitting on /schemas:
/schemas/a.json
/schemas/b.json
/schemas/c.json

So /schemas/a.json:

{
    "b": { "$ref": "b.json" }
}

dereferences well, but /schemas/b.json:

{
    "c": { "$ref": "c.json" }
}

is trying to find the schema under /schemas/b.json/:

• GET http://localhost:8080/schemas/b.json/c.json/ 404 (Not Found)
• Error downloading http://localhost:8080/schemas/b.json/c.json/

Cannot find name 'Buffer' when running in Browser

I assume this is look for the node type 'Buffer'? Including type: ['node'] in my tsconfig causes too many other conflicts. Is there a way around this?

Invalidate cache when a file is modified by `dereference()` or `bundle()`

The dereference() and bundle() methods both modify the contents of cached files. That's fine, but it should also invalidate (expire) the cache of those files, so that they are re-read the next time a method is called. Otherwise, they may contain stale data that is out-of-sync with other files.

$ref in $ref causes path hash to be repeated.

Hey, first of all, thanks for the great library. I think I spot an edge case bug when using it.
Here is the JSON file that I have simplified a lot to demo the problem.

{
  "$ref": "#/definitions/doc_node",
  "definitions": {
    "top_level_node": {
      "type": "object"
    },
    "doc_node": {
      "type": "object",
      "properties": {
        "content": { "$ref": "#/definitions/top_level_node" }
      }
    }
  }
}

And when I tried to bundle it

const parser = new RefParser();
parser
  .bundle('./bug-schema.json')
  .then((s) => {
    console.log(JSON.stringify(s, null, 2))
  });

it generates output

{
  "$ref": "#/definitions/doc_node",
  "definitions": {
    "top_level_node": {
      "type": "object"
    },
    "doc_node": {
      "type": "object",
      "properties": {
        "content": {
          "$ref": "#/definitions/top_level_node/definitions/top_level_node"
        }
      }
    }
  }
}

As you can see, the 'definitions/top_level_node' is duplicated in doc_node. And I also did some debugging myself. It turns out that problem happens around here

Pointer.prototype.resolve = function(obj, options) {
  var tokens = Pointer.parse(this.path);

  // Crawl the object, one token at a time
  this.value = obj;
  for (var i = 0; i < tokens.length; i++) {
    if (resolveIf$Ref(this, options)) {
      // The $ref path has changed, so append the remaining tokens to the path
      this.path = Pointer.join(this.path, tokens.slice(i));  <== This line
    }

    var token = tokens[i];
    if (this.value[token] === undefined) {
      throw ono.syntax('Error resolving $ref pointer "%s". \nToken "%s" does not exist.', this.path, token);
    }
    else {
      this.value = this.value[token];
    }
  }

  // Resolve the final value
  resolveIf$Ref(this, options);
  return this;
};

I was trying to figure out what is the code doing here. My immature assumption is that resolveIf$Ref is checking if current object has a $ref and if it has one, it will be resolved. However, one thing I don't understand is the reason behind this path updating. May be it has something todo with line here ? Can we not update the path?

I understand you are busy and hope you could shed some light on this. I am happy to create a PR if you could understand the purpose of the path updating logic.

Cheers

CORS/HTTP/HTTPS Problem

Hi,

only as a side node:
I upgraded my Ionic project to the new WKWebView and after that I'm running into
an issue with the ref parser:

I'm getting the following errors in Safari/Chrome:

[Error] Cross-origin redirection to https://v22017084844652188.powersrv.de/schemas/test.json denied by Cross-Origin Resource Sharing policy: Origin http://localhost:8100 is not allowed by Access-Control-Allow-Origin.
[Error] Failed to load resource: Cross-origin redirection to https://v22017084844652188.powersrv.de/schemas/test.json denied by Cross-Origin Resource Sharing policy: Origin http://localhost:8100 is not allowed by Access-Control-Allow-Origin. (test.json, line 0)
[Error] Fetch API cannot load http://v22017084844652188.powersrv.de/schemas/test.json. Cross-origin redirection to https://v22017084844652188.powersrv.de/schemas/test.json denied by Cross-Origin Resource Sharing policy: Origin http://localhost:8100 is not allowed by Access-Control-Allow-Origin.

This is the code:

export class HomePage {

  public httpSchema: Object;

  public parserSchema: Object;

  public httpError: Object;

  public parserError: Object;

  constructor(public navCtrl: NavController, public http: Http) {
    /*
    this.http.get('https://v22017084844652188.powersrv.de/schemas/test.json')
      .toPromise()
      .then((res) => {
        this.httpSchema = res.json();
      })
      .catch((error) => {
        this.httpError = error;
      });
      */
    this.getFlatSchema("https://v22017084844652188.powersrv.de/schemas/test.json")
      .then((value) => {
        console.log("Schema fetched...");
        this.parserSchema = value;
      })
      .catch(error => {
        console.log("Schema not fetched...");
        console.log(error);
        this.parserError = error;
      });
  }

  public getFlatSchema(uri: string): Promise<Object> {
    let parser = new $RefParser();
    let httpOptions = {
      resolve: {
        http: {
          headers: {
            Origin: "http://localhost:8100"
          }
        }
      }
    }
    return parser.dereference(uri, httpOptions);
  }
}

So since RefParser is not adding an "Origin" request header automatically, I added it manual.
But I'm getting the same error. It seems that in between RefParser tries to load the schema over http instead of https. The server is configured to redirect to https and the "Access-Control-Allow-Origin" header is set, if the request header is sent.

So i'm reading the error messages if there is a problem in between fetching the schemas and changing the protocol from https to http and back..

Can you help me?

Maybe related: #46

Reference local files, by schema/property id

I'm currently using the ajv lib and some custom logic to dereference my schemas, and then I found this lib which will probably do a better job. Is there a way to add schemas so this lib is aware of them? My JSON schemas reference by id, so for example in ajv I'm loading all my schemas schemas.forEach(schema => ajv.addSchema(schema)) and then trying to parse the $refs. Is there something similar I can do for this lib? Or is the example in the docs the only way "$ref": "schemas/people/Bruce-Wayne.json"?

Thanks

Weird circular ref causes endless loop

Something weird about the Bungie.net API causes the dereference() method to spin forever in an endless loop. I haven't had time to fully investigate why yet.

Code to reproduce the bug:

jsonSchemaRefParser.dereference('https://api.apis.guru/v2/specs/bungie.net/2.0.0/swagger.yaml')
    .then(() => {
        console.log('this line will never execute');
    })
    .catch(() => {
        console.log('neither will this line');
    });

File read edge-case bug

hi there,

I stumbled upon a small bug which occurs when all the following are true (I am sure there are other variations of loading multiple schemas that trigger the problem).

Explaination of the problem

1. you use bundle() (I have not tried with other methods but I assume they suffer in the same way)
2. you use a custom file resolver that returns Objects and caches/reuses it's responses (node's own require in my case)
3. you load/dereference 2 files in the same process
4. those 2 files both have a reference to a 3rd schema file
5. the 3rd schema file contains 2 references to a 4th schema.

Script & Schemas to reproduce

Here is a gist with a small repro-case (note that the pass.js uses lodash/deepClone to circumvent the issue):

https://gist.github.com/iamjochem/8fcb056301c4c846773f62022b9831cd

If I run fail.js then the error that I get is this:

{
  "name": "SyntaxError",
  "message": "Error resolving $ref pointer \"/work/demos/bug/b.json#/oneOf/1/properties/x\".
Token \"oneOf\" does not exist.",
}

The built-in file resolver does not have this problem because it uses fs.readFile (so you are getting a string that is JSON.parse()d everytime the reference is resolved.)

Proposed Solution

I believe the thing to would be to deep clone anything that is already an object in the JSON parser on the following line:

https://github.com/BigstickCarpet/json-schema-ref-parser/blob/master/lib/parsers/json.js#L56

e.g.

resolve(clonedeep(data));

where clonedeep is var clonedeep = require('lodash.clonedeep'); ... I use lodash in this example but you may wish to avoid another depedency and write your own version.

Info over lodash.clonedeep v4.5.0 size:

> du -sckh node_modules/lodash.clonedeep/*
4.0K	node_modules/lodash.clonedeep/LICENSE
4.0K	node_modules/lodash.clonedeep/README.md
 44K	node_modules/lodash.clonedeep/index.js
4.0K	node_modules/lodash.clonedeep/package.json
 56K	total

Support HTTP redirects

JSON Schema $Ref Parser should automatically follow HTTP redirects. For example, the following code points to a URL that returns an HTTP 302 redirect. If the redirect were followed, then it would result in a valid JSON document. But JSON Schema $Ref Parser doesn't follow the redirect, so it throws an error because the HTTP 302 response is not a valid JSON document.

$RefParser.dereference('http://httpbin.org/redirect/5')

Dereference allOf

While we wait for v5, allOf is the best way to combine multiple schemas. It would be nice if there was an option which would dereference this for you. For now I'm going to try and use https://www.npmjs.com/package/json-schema-resolve-allof

Bundle/package directory

The bundle method lets us pass in a file which then bundles all referenced files into a single schema and converts to local $refs (though I'm encountering an issue with this which I think is the same as #62).

How difficult would it be to have something similar, but allow passing in a directory which merges all schemas and then converts to local $refs?

I could merge all files into a single JSON quite easily; the challenging part would be converting the $refs to their local equivalent.

bundle() incorrectly escapes certain characters in JSON pointers in $ref's

This package is awesome, so thank you! However, I believe I have discovered a bug with the bundle() method when it creates a pointer to a location that contains special characters.

When I bundle my schema, I end up with something like this (reduced for simplicity):

{
  "paths":
    "/api/menus/{id}" : {
      "get": {
        "responses": {
           "200": {
              "schema": {
                  "type": "object",
                  "description": "Bundle successfully added my schema object here!"
                  ... etc...
              }
           }
        }
      }
    },
    "/api/menus/{id}/tree": {
       "get": {
          "responses": {
             "200": {
                 "description": "OK",
                 "schema": {
                    "$ref": "#/paths/~1api~1menus~1%7Bid%7D/get/responses/200/schema"
                  }
             }
           }
       }
    }
}

The problematic line is this:

    "$ref": "#/paths/~1api~1menus~1%7Bid%7D/get/responses/200/schema"

When I paste the bundled Swagger data into the online Swagger editor, it cannot resolve that reference (the error shows the line number). However, if I change %7B to { and %7D to }, the reference works and the error goes away.

The spec for JSON pointers is at https://tools.ietf.org/html/rfc6901 -- it's pretty heady, but I'm wondering if maybe things are in the weeds with the %x00-2E / %x30-7D / %x7F-10FFFF et al characters?

Is this a bug? Should { and } remain unescaped?

Invalid references created in `bundle` method

I am experiencing issues with the bundle method in a way that it creates broken references. See the following minimal example:

foo.json

{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "definitions": {
        "foodef": { "$ref": "bar.json#/definitions/bardef" }
    },
    "type": "object",
    "properties": {
        "prop": {
            "$ref": "#/definitions/foodef"
        }
    }
}

bar.json

{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "definitions": {
        "bardef": { "type": "string" }
    }
}

run.js

var $RefParser = require('json-schema-ref-parser');

$RefParser
    .bundle("foo.json")
    .then(function(schema) {
        console.log(JSON.stringify(schema, null, '    '));
    });

Running node run.js > result.json results in

result.json

{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "definitions": {
        "foodef": {
            "type": "string"
        }
    },
    "type": "object",
    "properties": {
        "prop": {
            "$ref": "#/definitions/foodef/definitions/bardef"
        }
    }
}

There #/definitions/foodef/definitions/bardef is pointing nowhere.

Am I doing something wrong or is this a bug?