netflix / falcor Goto Github PK

View Code? Open in Web Editor NEW

10.4K 648.0 441.0 23.05 MB

A JavaScript library for efficient data fetching

Home Page: http://netflix.github.io/falcor

License: Apache License 2.0

JavaScript 99.94% Shell 0.06%

falcor's Introduction

Falcor

2.0

2.0 is the current stable Falcor release. 0.x and 1.x users are welcome to upgrade.

Roadmap

Issues we're tracking as part of our roadmap are tagged with the roadmap label. They are split into enhancement, stability, performance, tooling, infrastructure and documentation categories, with near, medium and longer term labels to convey a broader sense of the order in which we plan to approach them.

Getting Started

You can check out a working example server for Netflix-like application right now. Alternately, you can go through this barebones tutorial in which we use the Falcor Router to create a Virtual JSON resource. In this tutorial we will use Falcor's express middleware to serve the Virtual JSON resource on an application server at the URL /model.json. We will also host a static web page on the same server which retrieves data from the Virtual JSON resource.

Creating a Virtual JSON Resource

In this example we will use the falcor Router to build a Virtual JSON resource on an app server and host it at /model.json. The JSON resource will contain the following contents:

{
  "greeting": "Hello World"
}

Normally, Routers retrieve the data for their Virtual JSON resource from backend datastores or other web services on-demand. However, in this simple tutorial, the Router will simply return static data for a single key.

First we create a folder for our application server.

$ mkdir falcor-app-server
$ cd falcor-app-server
$ npm init

Now we install the falcor Router.

$ npm install falcor-router --save

Then install express and falcor-express. Support for restify is also available, as is support for hapi via a third-party implementation.

$ npm install express --save
$ npm install falcor-express --save

Now we create an index.js file with the following contents:

// index.js
const falcorExpress = require("falcor-express");
const Router = require("falcor-router");

const express = require("express");
const app = express();

app.use(
    "/model.json",
    falcorExpress.dataSourceRoute(function (req, res) {
        // create a Virtual JSON resource with single key ('greeting')
        return new Router([
            {
                // match a request for the key 'greeting'
                route: "greeting",
                // respond with a PathValue with the value of 'Hello World.'
                get: () => ({ path: ["greeting"], value: "Hello World" }),
            },
        ]);
    })
);

// serve static files from current directory
app.use(express.static(__dirname + "/"));

app.listen(3000);

Now we run the server, which will listen on port 3000 for requests for /model.json.

$ node index.js

Retrieving Data from the Virtual JSON resource

Now that we've built a simple virtual JSON document with a single read-only key greeting, we will create a test web page and retrieve this key from the server.

Create an index.html file with the following contents:

<!-- index.html -->
<html>
    <head>
        <!-- Do _not_  rely on this URL in production. Use only during development.  -->
        <script src="https://netflix.github.io/falcor/build/falcor.browser.js"></script>
        <!-- For production use. -->
        <!-- <script src="https://cdn.jsdelivr.net/falcor/{VERSION}/falcor.browser.min.js"></script> -->
        <script>
            var model = falcor({
                source: new falcor.HttpDataSource("/model.json"),
            });

            // retrieve the "greeting" key from the root of the Virtual JSON resource
            model.get("greeting").then(function (response) {
                document.write(response.json.greeting);
            });
        </script>
    </head>
    <body></body>
</html>

Now visit http://localhost:3000/index.html and you should see the message retrieved from the server:

Hello World

Steps to publish new version

Make pull request with feature/bug fix and tests
Merge pull request into master after code review and passing Travis CI checks
Run git checkout master to open master branch locally
Run git pull to merge latest code, including built dist/ and docs/ by Travis
Run npm run dist to build dist/ locally
- Ensure the built files are not different from those built by Travis CI, hence creating no change to commit
Update CHANGELOG with features/bug fixes to be released in the new version and commit
Run npm version patch (or minor, major, etc) to create a new git commit and tag
Run git push origin master && git push --tags to push code and tags to github
Run npm publish to publish the latest version to NPM

Additional Resources

For detailed high-level documentation explaining the Model, the Router, and JSON Graph check out the Falcor website.
API documentation
For a working example of a Router, check out the falcor-router-demo.
For questions and discussion, use Stack Overflow.

falcor's People

Contributors

Stargazers

Watchers

Forkers

haegyung r05al c2y3dc nandokstronet schwad megawac silky companyontheworld miguelmota pdehaan steve0341 sgriffiths bijoyv daannijkamp ajayputti nicolasartman cibernox logic855 theprimeagen drooids jontewks fbcouras skipshot4 adi-ads viniciusferreira newsomc dana-kirsanov josejamilena jondlm justomiguel constantin1992 nangal guahanweb trashsydowdev ewinslow jcouyang bithound lumiqai priestd09 vishal-h ismailmechbal victso reactivize voxlol tylerlh nimamehanian luiseduardohdbackup carsonmcdonald danlamarca sposam samarpanda wonderdogone mitesh91 vanloc0301 agilemobiledev sxhao tomzhang delkyd agileurbanite arrmac kuguobing bbss fuchao2012 mausvargas ning-github bobvanluijt ecogels arashbi kublaj nickmerwin greyhwndz bigmeech edwardt sdesai cybernetics sourcecode sigmus jhamlet oaleynik akashdevaraju rasata kk9599 davinirjr spakala dm04806 ananta1 sorrycc swatinem darioajr orneryhippo mcanthony coreygrunewald framewr luciancor daliandragon justnull yosoyubik sheeley fbmw tbranyen

falcor's Issues

set.toPathValues should dedupe

After talking through how set.toPathValues.progressively should work with @jhusain we realized that core.set should dedupe values. When the first set happens, with an onNext function provided, the core should call onNext every value it finds. If the model has _dedupePathValues flag set, the onNext function should only onNext if the value coming in from set is different that the value in the cache.

@trxcllnt does that sound reasonable?

Parser tests and escaped quotes.

The dot syntax is missing the ability to have quotes within quotes and the unit test coverage is bare. In fact any nested quote does not work right now, but it was not needed for the talk. Should be a small task.

Browser Build should produce file with lowercase "f"

Several external developers have incorrectly assumed the namespace was capitalized based on the file capitalization. This is a small change but it will help with ergonomics

Disallow null in middle of path.

By disallowing null in any position but last we clarify what null means (follow ref). We also make the router much easier because we don't need to accommodate nulls.

Model should continue to send missing two data source until no more paths are missing

Currently the model only makes one request to the data source for missing paths. This is a bug because the model must continue to request missing paths from the data source until no more missing paths are found.

This behavior might seem strange. Why should you request missing paths from a data source that the data source was just unable to serve a moment ago? This question represents a fundamental misunderstanding of the responsibility of the data source. It is always the data source's responsibility to respond with a value for every single path requested. If a data source cannot find a path, it must return an undefined atom as a place holder. Failing to return anything is a bug in the data source.

Another way of stating the same requirement is to save it if you request several paths from a model with a data source, and then request the same paths after the first request is served, you should always hit the cache.

There are a few situations in which this might not occur, but none of them have anything to do with the data source. The only thing which can cause missing paths to persist after a request to a (well-behaved) data source are optimistic concurrency violations.

Imagine we have the following cache:

{
genres: {
"0": { $type: "ref", value: ["lists", 3637] },
}
}

Let's say you make a request for [genres, 0, name]. This request is optimized into [lists, 3637, name] and sent to the data source. While the Data source is processing the request, the reference at [genres, 0] is deleted, either because of a push notification, a cache collection, an explicit delete, or some other concurrent action. When the data source responds, it will return the following message:

{
paths: [["lists", 3637]],
value: {
lists: {
"3637"; {
"name": "Thrillers",
}
}
}

When merging into the cache, the model cannot use the path returned by the data source, but must instead use the original requested path. This is necessary because the model must find a way to translate the optimized path back to the request to pass in order to serve it to the caller. When attempting to merge the message in using the original path ["genres", 0, "name"] the registers the path as missing because there is nothing at [genres, 0]. In other words even though the data source was successful in serving the path, a concurrent action still caused it to be registered as missing. In order to resolve the conflict, we simply reissue the missing path (this time unoptimized) to the data source and The request is served successfully.

This does create the hazard that if a data source is not implemented properly you could find yourself in an infinite loop. That is a risk we are willing to take for now.

PathSets that contain empty ranges or empty arrays are considered empty sets.

The following Path sets are empty and gets for them will never return results.

["lolomo", {length:0}, "name"] -> empty set
["lolomo", {from:1, to: 0}, "name"] -> empty set
["lolomo", 5, []] -> empty set

This is allowed because PathSets may be created by algorithms.

No getCache method

There is currently no getCache method to match the setCache method. The getCache method is necessary to take a snapshot of the current cache and store it to local storage. This is an important scenario.

@trxcllnt can you comment?

Implement setPathMaps* and stop using legacy setCache

Understand we're still using this method.

Discuss: do not unbox references and errors

@ktrott @michaelbpaulson @trxcllnt @sdesai

Originally only sentinels were unboxed by default. Then sometime while I was in Paris the decision was made to unbox errors and references. When this was explained to me remotely, it seemed like a reasonable enough idea. From an implementation perspective it's very simple to treat all Atoms the same way. Now that I have had more time to consider this change, it seems like the decision to unbox references and errors causes big problems in mainline scenarios and we should go back to only unboxing sentinels by default.

Dropping the context around whether a value is an error is never useful. The reason why we have treatErrorsAsValues is that a developer might want to display error messages in their view. Without also using box values developers will not be able to discern errors from values, and this is an very common scenario (display errors in red). Unfortunately if developers turn on box values they would have to deal with sentinels at every single field, making their template much more complicated. I don't think we properly considered ergonomics before we made the decision to unbox errors, but rather we were thinking too much about implementation simplicity. The boxValues method is difficult to understand, and it should only be needed for advanced scenarios, not mainline scenarios.

The same problem applies (to a lesser degree) to paths. If developers retrieve a path from the model, and then turn around and set it back in, it becomes a branch node. This is less hazardous than errors, because developers rarely retrieve paths. However we do see this internally.

We should make sure that box values has no effect on references and errors. It should only affect sentinels as it did before. Errors can still be unboxed in error function handlers though, because error context has been established.

DeferBind method

Based on community feedback, we will need to add some sort of deferBind method. This method synchronously returns a model, unlike bind. The model optimizes the path after after the very first value request, but then the bound path never changes again. From this point on, the model behaves like any other model. For example it throws errors if The object it is bound to disappears.

Performance of get and set

2 items here.

Internally we are pushing every requested and optimized path onto an internal tracking structure. This means if you requested 100 paths we would push 200 times. Curious to see the effects that this has on performance.
Currently, for every complex key the requested, optimized, and positional arrays| are being copied which for lolomo, to:9, to:9, summary will copy 211 times, which is a lot. Instead we could not copy the arrays, except in the case of toPathValue and toJSONG, but instead push and pop keys. This should work.

TimeBox: Should take no longer than a day to prove out performance.

Browser Build should produce file with lowercase "f"

Several external developers have incorrectly assumed the namespace was capitalized based on the file capitalization. This is a small change but it will help with ergonomics

Need Better private property prefix than two underscores

We need to come up with a better prefix for private properties than two underscores. User generated strings are often inserted into paths. We need a string prefix that is unlikely to conflict. Note that we have already run into this problem inside of Netflix.

set with value of 0 error

hasValue is false when the value is 0
inside of
function processOperations() ->
results = operation.op(model, operation, operation.values, operation.errorSelector, operation.boundPath); ->

setPathsAsJSONG() {
    values && (values[0] = hasValue && {
        paths: requestedPaths,
        jsong: jsons[-1]
    } || undefined);
    if (values[0] === undefined && values.length === 1) {
        // values[0] is being set to [undefined] because hasValue is false when it should be true
        debugger;
       // if we force the change back there is another error later on
        values[0] = {
            paths: requestedPaths,
            jsong: jsons[-1]
        };
    }
    return {
        'values': values,
        'errors': errors,
        'requestedPaths': requestedPaths,
        'optimizedPaths': optimizedPaths,
        'requestedMissingPaths': requestedMissingPaths,
        'optimizedMissingPaths': optimizedMissingPaths
    };
}

If we make the edits above we now an error for SetRequest.flush

SetRequest.prototype
  flush: function() {
        var incomingValues, query, op, len;
        var self = this;
        var jsongs = self.jsongEnvs;
        var observers = self.observers;
        var model = self._jsongModel;
        self.pending = true;

        // TODO: Set does not batch.
        return model._dataSource.
            set(jsongs[0]).
            subscribe(function(response) {
                incomingValues = response;
                if (!incomingValues) {
                    // incomingValues becomes undefined now
                    debugger;
                }
            }, function(err) {
                var i = -1;
                var n = observers.length;
                while (++i < n) {
                    obs = observers[i];
                    obs.onError && obs.onError(err);
                }
            }, function() {
                var i, n, obs;
                self._queue.remove(self);
                i = -1;
                n = observers.length;
                while (++i < n) {
                    obs = observers[i];
                    // incomingValues is undefined on complete
                    obs.onNext && obs.onNext({
                        jsong: incomingValues.jsong || incomingValues.value,
                        paths: incomingValues.paths
                    });
                    obs.onCompleted && obs.onCompleted();
                }
            });

I'm using set with a selector function

model.set(
  {
    path: title._path.concat(['rating']),
    value: Number(title.rating) - 1
  },
  selectorFunc
)
.forEach(noop, noop, noop);

I'm also using a mock source

function noOp() {}

class LocalDataSource {
  constructor(cache, options = {}) {
    this._options = Object.assign({
      miss: 0,
      onGet: noOp,
      onSet: noOp,
      onResults: noOp,
      wait: false
    }, options);
    this._missCount = 0;
    this.model = new Model({ cache: cache });
  }

  setModel(modelOrCache) {
    if (modelOrCache instanceof Model) {
      this.model = modelOrCache;
    } else {
      this.model = new Model({ cache: modelOrCache });
    }
  }

  get(paths) {
    var self = this;

    var {
      miss,
      onGet,
      onResults,
      wait,
      errorSelector
    } = this._options;

    return Observable.create(function(observer) {
      function exec() {
        // var results;
        var values = [{}];
        if (self._missCount >= miss) {
          onGet(self, paths);
          self.model._getPathsAsJSONG(self.model, paths, values, errorSelector);
        } else {
          self._missCount++;
        }

        // always output all the paths
        var output = {
          paths: paths,
          jsong: {}
        };
        if (values[0]) {
          output.jsong = values[0].jsong;
        }

        onResults(output);
        observer.onNext(output);
        observer.onCompleted();
      }

      if (+wait > 0) {
        setTimeout(exec, +wait);
      } else {
        exec();
      }
    });
  }

  set(jsongEnv) {
    var self = this;

    var {
      miss,
      onSet,
      onResults,
      wait,
      errorSelector
    } = this._options;

    return Observable.create(function(observer) {
      function exec() {
        var values = [{}];
        onSet(self, jsongEnv);
        self.model._setJSONGsAsJSONG(self.model, [jsongEnv], values, errorSelector);

        // always output all the paths

        onResults(values[0]);
        observer.onNext(values[0]);
        observer.onCompleted();
      }

      if ((+wait) > 0) {
        setTimeout(exec, wait);
      } else {
        exec();
      }
    });
  }

  call(path, args, suffixes, paths) {
    return Observable.empty();
  }

}

and also setting the cache in my Model and MockSource

  var model = new Model({
      source: new LocalDataSource({
        // list of user's genres, modeled as a map with ordinal keys
        "genreList": {
           "0": ["genres", 123],
           "1": ["genres", 522],
           "length": 2
        },
        // map of all genres, organized by ID
        "genres": {
           // genre list modeled as map with ordinal keys
           "123": {
               "name": "Drama",
               "0": ["titles", 23],
               "1": ["titles", 99],
               "length": 2
           },
           // genre list modeled as map with ordinal keys
           "522": {
               "name": "Comedy",
               "0": ["titles", 23],
               "1": ["titles", 44],
               "length": 2
           }
        },
        // map of all titles by ID
        "titles": {
          "99": {
               "name": "House of Cards",
               "rating": 5
           },
           "23": {
               "name": "Orange is the New Black",
               "rating": 4
           },
           "44": {
               "name": "Arrested Development",
               "rating": 3
           }
        }
      }), //mockDataSource,
      cache: {
        // list of user's genres, modeled as a map with ordinal keys
        "genreList": {
           "0": ["genres", 123],
           "1": ["genres", 522],
           "length": 2
        },
        // map of all genres, organized by ID
        "genres": {
           // genre list modeled as map with ordinal keys
           "123": {
               "name": "Drama",
               "0": ["titles", 23],
               "1": ["titles", 99],
               "length": 2
           },
           // genre list modeled as map with ordinal keys
           "522": {
               "name": "Comedy",
               "0": ["titles", 23],
               "1": ["titles", 44],
               "length": 2
           }
        },
        "titles": {
          "99": {
            "name": "House of Cards",
            "rating": 5
          },
          "23": {
            "name": "Orange is the New Black",
            "rating": 4
          },
          "44": {
            "name": "Arrested Development",
            "rating": 3
          }
        }

      } // end cache
    });

Can we avoid sentinel cloning when setting JSON Graph into cache?

Sentinel cloning can't be avoided for PathValues because same object may be reused for several locations in cache. Same is not true of JSON Graph, where same sentinel will always be added to same location.

Should be able to avoid cloning sentinels here because the same sentinel will always be set in the same place. Might improve performance in mainline scenario of loading data from server.

Throw on attempt to retrieve a branch node

This will almost certainly be the most common rookie mistake. It will be really helpful to provide an error message if an attempt is made to retrieve a branch note. Unfortunately we can't do anything on an attempt retrieve a branch node on the server. Would love to discuss any ideas here.

@trxcllnt @michaelbpaulson

Create a router which implements the data source interface.

The router Constructor should accept an array of Route descriptor objects.

new Router(routes: Array<RouteDescriptor>)
A route descriptor object has the following signature:

type RouteDescriptor = {
  route: RouteSyntax | RouteSet
  get?: (PathSet, context:object) => Observable<JSONGEnvelope | PathValue> | Promise<JSONGEnvelope>
  set?: (JSONGEnvelope, Context) => Observable<JSONGEnvelope | PathValue>
  call?: (callPath: PathSet, arguments: Array, pathSuffixes: Array<PathSet>, paths: Array<PathSet>) => Observable<PathValue>
}

RouteSyntax is identical to the PathSyntax with the exception of the following special pattern matchers:

{ranges}, {integers}, {keys}

RouteSyntax is parsed into RouteSets by the router. RouteSets are identical to PathSets, except pattern objects are allowed in any KeySet position:

A pattern object is an object with a "type" key, where privateprefix is the obscure character being used as a field prefix in order to store implementation information in JSON Graph models.

A pattern object looks like this:

{"<privateprefix>type": "ranges" | "integers" | "keys" }

The Router object contains constants that map to the three supported pattern objects:

Router.RANGES, Router.INTEGERS, Router.KEYS

Each of these constants map to the {ranges}, {integers}, and {keys} patterns in the RouteSyntax respectively. These constants exists solely to make it more convenient to include pattern matchers in RouteSet syntax. They are not used for comparison purposes when parsing. Pattern objects are identified by the presence of the "pattern" key.

Router.RANGES is { "<privateprefix>pattern": "ranges" }, Router.INTEGERS is { "<privateprefix>pattern": "integers" }, Router.KEYS { "<privateprefix>pattern": "keys" }. Using an object with a type property allows us to make only a single lookup to determine if an object is a pattern or not.

In the RouteSyntax, the pattern matchers above can only be included inside of an indexer.
There this...

genreLists[{integers}].name

...parses to...

["genreLists", Router.INTEGERS, "name"]

Furthermore only one pattern match may be included inside of an indexer. In other words this is illegal:

genreLists[{integers}, {keys}].name

The main advantage of using the route syntax, besides similarity to PathSyntax, is that you can use aliased matches. You can alias a pattern match like so:

genreLists[{integers:genreIds}].name

The route syntax above is parsed into a RouteSet. The RouteSet's structure looks like this:

["genreLists", {"pattern": "integers", alias: "genreIds"}, "name"]

Recall that a RouteSet is an Array. The parsing process will use the name of the pattern ("genreIds") as a key in the array so it is accessible with a named lookup rather than a position one.

Therefore route syntax...

genreLists[{integers:genreIds}].name

...matches this path syntax...

genreLists[0,2,4].name

...and the internal structure of the parsed RouteSet will look like this:

{
  "0": "genreLists",
  "1": [0,2,4],
  "2": "name",
  "length": 3,
  "genreIds": [0,2,4] // (reference equal to key "1")
}

In RouteSets, Pattern matches are specified in a KeySet position (not nested in an indexer): Therefore this ["genreLists", Router.INTEGERS, "name"] is legal, but this is not: ["genreLists", [Router.INTEGERS], "name"]. Nor is this legal: ["genreLists", [Router.INTEGERS, Router.KEYS], "name"].

Here is a table of RouteSyntax expressions, the RouteSet they parse to, PathSyntax they match, and the resulting parsed PathSet:

genreList[{integers}].name ->
["genreLists", Router.INTEGERS, "name"] // which matches 
genreLists[0,1,1..4].name // and parses to 
["genreLists", [0,1,2,3,4], "name"]

genreList[{ranges}].name ->
["genreLists", Router.RANGES, "name"] // which matches 
genreLists[0,1,1..4].name // and parses to 
["genreLists", [{from:0,to:0},{from:1,to:1},{from:1,to:4}], "name"]

genreList[{keys}].name ->
["genreLists", Router.KEYS, "name"] // which matches 
genreLists[0,1,1..4, "length"].name // and parses to 
["genreLists", [0,1,2,3,4,"length"], "name"]

You can instantiate a Router like this:

new Router([
  { 
    route: "genreLists[{ranges}][{ranges}]",
    get: (pathSet, context, nodeCallback) => {
       // return Promise<JSONGraphEnvelope> | Observable<JSONGraphEnvelope | PathValue>
    },
    set: (jsonGraph, context, nodeCallback) => {
      // return Promise<JSONGraphEnvelope> | Observable<JSONGraphEnvelope | PathValue>
    },
    call: (callPath, args, pathSuffixes, paths) => {
       // return Promise<JSONGraphEnvelope> | Observable<JSONGraphEnvelope | PathValue>
    }
  }
]);

The router recursively matches pathsets until the path is exhausted or path evaluation is short-circuited.


Routes are matched based on highest precedence. If two rules have equal precedence the router constructor should throw with a descriptive error message. The following rules apply in order of precedence:

Longer path beats shorter path

lolomo[{range}] < lolomo[{range}][{range}]


Key beats pattern (range, integers, keys):

lolomo[{range}] < lolomo[0]


Integers or Ranges beats keys:

lolomo[{keys}] < lolomo[{ranges}]
lolomo[{keys}] < lolomo[{integers}]


There may be two paths with equal precedence (same length, same pattern vs key balance). in these cases, the path with the first key of higher precedence wins.

lolomo[{keys}][0] < lolomo[0][{keys}]


In cases where two paths have identical precedence and keys an error should be thrown at router construction time.

lolomo[{ranges}][0] has same precedence as lolomo[0][{integers}]

Discuss introducing Path Syntax

@trxcllnt @michaelbpaulson @ktrott @sdesai

I still think we should have this in place before open-sourcing. The Path Syntax is to Falcor as JSX is to React. JSX is just a thin wrapper for JSON objects. FB wisely understood when you're introducing people to vastly different technologies, you want to give them all the familiar touchpoints you can.

Can we cost this work out? I'd be surprised if it was more than three days. I think we should just push the release date to get it in.

Need to support Arrays in Branch position in JSON Graph

Now that we are explicitly calling out references, There's no reason why we can't use Arrays in JSON Graph branch node positions.

This change is vital because it will make it much easier to mock server models locally. Remember that this is one of the key selling points of Falcor: you can mock the server and then code it up later on. The current requirement to use maps for lists has always been one of the most complex things for new developers to understand. It's very unnatural.

I want to write this:

new Model({cache: {
users: [
{ $type: "ref", value: ["usersById", 22] },
{ $type: "ref", value: ["usersById", 97] }
],
usersById: {
"22": { name: "Jim" },
"97": { name: "John" }
}
}});

Not this:

new Model({cache: {
users: {
"0": { $type: "ref", value: ["usersById", 22] },
"1": { $type: "ref", value: ["usersById", 97],
"length": 2
}
],
usersById: {
"22": { name: "Jim" },
"97": { name: "John" }
}
}});

It's important to bear in mind that the win here is not less typing, it's easier prototyping. Note that developers can call the array methods using call to add things to the server.

model.call(["users", "push"], [[{ $type: "ref", value: ["usersById", 22]}]);

If they were using maps to model collections they would have to write push over maps - for no reason.

This is one of the big wins of "ref." We should capitalize on it!

See failing test:
'get should go right through arrays in branch position'

Relative expiration not working on nodes passed into Model constructor

See failing test: 'should retrieve a value from the cache that has a relative expiration that has not expired yet'

Server 500s are put in as values.

When a dataSource throws an error the results for each of the requested paths are put in as sentinels / atoms. They should be put in as errors and onError'd to the user.

Remove unnecessary "/" and "./" values from Falcor branch nodes

Can't find any reason for these keys to be added to every node and setting them is non-zero cost. The ".." key is necessary from what I can see, but not the other keys.

Get and Set Progressively

Currently there is no way to get / set progressively, both the method on the prototype and the logic in the request function.

Dev mode assertions

Currently it is very to learn the software because if you do anything wrong you get no feedback whatsoever. We should add assertions to the build that is not minified so that we can give helpful error messages whenever possible, including things like invalid paths, and expected but not found items in the JSONG,

Browser Build should produce file with lowercase "f"

Several external developers have incorrectly assumed the namespace was capitalized based on the file capitalization. This is a small change but it will help with ergonomics

Undefined sentinels should not be written into the cache if the model does not have a data source

Title says it all placeholder issue.

Updating README.md

From @mannyhenri
"
A quick suggestion on the readme file, the Netflix exemples in context to Falcor's inner workings are excellent and will help new developers to understand its concept over a RESTFUL API! I'd suggest to keep them for official release.

Also the sanction Why not use REST (line 300) is already explained a bit above (line 217). :)
"
Maybe we could collapse the 2 explanations of rest.

Browser build

We need a build for the browser which includes XmlHttpSource. This is a vital component on this platform and there's no reason that developers should need to download another file.

Zero length ranges produce A single result

Running A get operation on a range with the lower bound at zero and the length as zero produces a single result at index zero. This is incorrect behavior. A path with a length of zero always produces no results.

This is particularly important because on the server-side we will be gradually stripping paths out of path sets.

withoutDataSource vs. withoutSource

We renamed the DataSource to ModelSource in the API rewrite. We usually just abbreviate to Source when it's a member on the Model, like in the constructor:

new Model({
source: ...
})

Can we S&R withoutDataSource to withoutSource to match the constructor?

Support Node-style callbacks

Should be easy to add an on method alongside then method. The on method follows the usual Node.js convention:

model.get(["lolomo",0,0,"name"]).on(function(error, data) {

});

Set should not insert sentinels for undefined values if operating without data source

If a Model does not have source it is the source or truth. If it does, it is a proxy to a Model that is a source of truth. When operating as a proxy it is necessary to store sentinels to remember that a value is undefined. By storing a Sentinel, we are able to remember the value is undefined and avoid making another call to the server just to retrieve that same information again. When a Model is a source of truth (has no data source) storing sentinels for undefined values is a memory leak.

Once we make this change, will be able to express invalidateSync as a helper method that just invokes withoutSource and sets all paths to undefined.

Consider adopting community-driven JSON over the wire format

JSON Graph solves some problems that, in my biased opinion, other community projects have solved better; e.g. http://jsonapi.org/ being the most prominent I know of. Most obviously dedup refs, filters.

I'm curious what factors went into creating it vs. working with an existing community format. If something in them wasn't ideal for falcor, was there any effort to discuss these with the community/maintainers? Can (or should) the falcor team be taking a vested interest in driving a standard, whether that be jumping into an existing one or cutting JSON Graph out?

More of a long-term vision question than a priority topic.

Errors and sentinels included in JSON message post-deserialization

Everything almost worked perfectly when I put together a worker sample demonstrating how models and model sources interact. However I ran across a problem as I tried sentinels and errors.

// worker code
importScripts('./Falcor.js');  

function WorkerServer(model) {
    this.model = model;
}

// create a server model
var model = 
    new falcor.
        Model({
            cache: {
                user: {
                    name: "Jim",
                    location: {$type: "error", value: "Something broke!"}
                }
            }
        }).
        boxValues().
        treatErrorsAsValues().
        materialize();

WorkerServer.prototype.onmessage = function(action) {
  var method = action[0],
    jsonGraphEnvelope,
    callPath,
    pathSuffixes,
    paths;

    switch(method) {
        case "get": {
            paths = action[1];

            return model.get.apply(model, paths).
                toJSONG();
        }
        case "set": {
            jsonGraphEnvelope = action[1];

            return model.set(jsonGraphEnvelope).
                toJSONG();            
        }
        case "call": {
            callPath = action[1];
            args = action[2];
            pathSuffixes = action[3];
            paths = action[4];

            return model.call(callPath, args, pathSuffixes, paths).
                toJSONG(); 
        }
    }
}

var workerServer = new WorkerServer(model);

onmessage = function(e) {
    var data = e.data,
        id = data[0];

    workerServer.
        onmessage(data.slice(1)).
        subscribe(
            function(result) {
                postMessage([id, null, result]);
            },
            function(error) {
                postMessage([id, error]);
            });
}

Now here's the web page code:

function WebWorkerSource(worker){
    this._worker = worker;
}

WebWorkerSource.prototype = {
    id: 0,
    get: function(paths) {     
        return this._getResponse(['get', paths]);
    },
    set: function(jsonGraphEnvelope) {   
        return this._getResponse(['set', jsonGraphEnvelope]);
    },
    call: function(callPath, arguments, pathSuffixes, paths) {
        return this._getResponse(['call', callPath, arguments, pathSuffixes, paths]);
    },
    _getResponse: function(action) {
        var self = this;
        return falcor.Observable.create(function(observer) {
            var id = self.id++,
            handler = function(e) {
                var response = e.data,
                    error,
                    value;
debugger;
                if (response[0] === id) {
                    error = response[1];
                    if (error) {
                        observer.onError(error);
                    }
                    else {
                        value = response[2];
                        observer.onNext(value);
                        observer.onCompleted();
                    }
                }
            };

            action.unshift(id);

            self._worker.postMessage(action);
            self._worker.addEventListener('message', handler);

            return function(){
                self._worker.removeEventListener('message', handler);
            };
        });
    }
};

var worker = new Worker('worker.js');

var model = new falcor.Model({ source: new WebWorkerSource(worker) });

model.
    get(["user", ["name", "age", "location"]]).
    subscribe(
        function(json) {
            debugger;
            console.log(JSON.stringify(json, null, 4));
        },
        function(errors) {
            console.error('ERRORS:', JSON.stringify(errors));
        });

The following output is expected:

//{
//    json: {
//        "user": {           
//            "99": {
//               "name": "Jim"
//               // age not included because it is undefined
//               // location not included in message because it resulted in error
//            }
//        }
//    }
//}
//ERRORS: [{"path":["user","location"],"value":"Something broke!"}]

Unfortunately we get the sentinel and error in the message:

{
    "json": {
        "user": {
            "name": "Jim",
            "age": {
                "$type": "sentinel"
            },
            "location": "Something broke!"
        }
    }
}
testfalcor.html:68 ERRORS: [{"path":["user","location"],"value":"Something broke!"}]

Path Syntax doesn't correctly stringify numbers

model.get("titlesById[07]") should parse to this ["titlesById", 7] but it parses to ["titlesById", "07"]

Initializing cache with Model constructor leaves cache in inconsistent state

When you create a Model and pass in a cache as a parameter, sentinel values are not created to wrap regular values.

In other words this...
var model = new falcor.Model({
cache: {
users: [
{
age: 4
}
]
}
});
...should result in a cache that looks like this internally:
{
users: [
{
age: {$type:'sentinel', value: 4}
}
]
}

Instead the value is left unwrapped in a sentinel internally.

add an observe method to the model

Internal customers and external customers have asked for an observe method on the model.The observe method has the exact same signature as the get method, but the observable that it returns never completes.

The simplest implementation I can imagine is simply to keep a list of observed paths inside of the model and run a get operation for them after every single set operation. This implementation can be optimized somewhat using the version flag to avoid full traversals.

Factory function for reference, error, Sentinel

Currently building graphs is unnecessarily verbose. We could make it much more terse by simply adding 3 factory functions for building each of the atom types.

Falcor.ref
Falcor.error
Falcor.sentinel

Each of these methods should be static so that they can be called with their this parameter as null. Each one of them accepts a single parameter, which is just the value key on the resulting Atom.

Reentrancy issue

When calling a selector function we set _allowSync to true. At the end of the selector function wet set _allowSync to false.

var model = new falcor.Model({ cache: {
user: {
name: "Jim"
}
}});

model.get(["user","name"], name => {
// allowSync -> true

// get that happens to run synchronously because info is in cache
model.get(["user","name"]).subscribe();
// allowSync is NOW FALSE!!

model.getValueSync(["user", "age"]) // throws because allowSync is TRUE
// allowSync -> false
});

Solutions appears to be using a counter for allowSync.

We are currently running in unsafe mode by default

Should definitely be fixed before we inner source any further.

Run invalidate synchronously

@ktrott @michaelbpaulson @trxcllnt @sdesai

Proposal: rename invalidate to invalidateSync and have it return void.

Currently invalidate returns an Observable. This doesn't make sense because it always executes synchronously. It should be like getValueSync, setValueSync, and bindSync. The documentation clearly states that all 'Sync' methods will execute synchronously and can only execute when there is no dataSource or in unsafeMode. Invalidate always runs asynchronously, and should therefore be like everything other sync method. Sync methods draw a very clear line between methods that operate locally, and those that make modifications on the server. Having invalidate return an Observable gives the erroneous impression that it invalidates remotely.

The argument for keeping it returning an Observable is that invalidate can return the data it removes, and can do so in any format (JSON Graph, PathValue, JSON). IMO this is a anti-pattern. Falcor is designed around a simple principle: keep your data in the async model. As soon as developers follow a pattern of keeping their data outside the cache for a time, RENO-style server push cache invalidatations doesn't work. I understand Dylan is doing this, probably to avoid errors in the cache overwriting good data. I've sent him a mail to figure out what he's trying to accomplish so that we can accommodate his use-case in a sanctioned way that works with RENO. If there is a use-case for doing this, we should address it with a mode (.ignoreErrors()), not a pattern that requires pulling data out of the cache and then mutating it back in later.

Started this thread so we could reach a conclusion on this. Please comment if you feel we shouldn't make this change.

Introduce constant for null.

Model.REF_TARGET = null

Add a dedupe mode to Model

When in the Dedupe mode, A model checks a path request against all currently outstanding requests, and avoids sending a duplicated request to server.

Discuss: should this be a mode? Should it be default behavior? What should the default be?

Path Syntax router syntax [{range|integers|keys}]

The path syntax needs to be able to support the router syntax. Further, the router syntax should allow for named identifiers.

titlesById[{integers:id}].name

This should return the proper path with the named identifiers.

Change name of Sentinel to Atom and vice versa

@trxcllnt @michaelbpaulson @sdesai

The current name "Sentinel" doesn't really match up correctly with its definition of JSON graph. Currently we have "atom" which is the term which we apply to a reference, a Sentinel, and an error. It doesn't appear in code anywhere, just in the documentation.

Here is the definition of Sentinel from Wikipedia:

A sentinel node is a specifically designated node used with linked lists and trees as a traversal path terminator

Technically references, sentinels, and errors are all sentinels. We need to differentiate what is currently called Sentinel today and describe what it does. Currently I'm finding this really hard to document, which is a real smell.

I propose we swap "sentinel" to "atom." This is much more descriptive of what the sentinel is used for: ensuring that objects with multiple properties are treated as atomic values.

Make ModelResponse support then method.

Making a ModelResponse support a Promise-style then method will make it possible for Falcor to be used with Promise libraries.

Much more extensive and detailed comments

Every single function needs at least a simple description describing what it does. More complex functions like the core algorithms need significantly more comments inside the code.

Refs should not be allowed in JSON

Take the following cache:

{
  genres: [
    {
      name: "Thrillers",
      titles: [
        {$type: 'ref', value: ['titlesById', 234] }
      ]
    }
  ],
  titlesById: {
    "234": "Die Hard"
  }
}

model.get(["genres",0,0,null]) current outputs "Die Hard". More specifically it outputs:

{
  json: {
    genres: {
      "0": {
        "0": {
          "null": "Die Hard"
        }
     }
   }
}

Note that the null key is used to follow references in Falcor. This is a big divergence from JavaScript path evaluation behavior, in which nulls are implicitly converted to the string "null".

x[null] === x["null"]

We need this null key semantic in Falcor, because JSON Graph has Reference objects and JSON does not. The problem is that when we pull this information into a JSON tree, we need to differentiate between whether the object at a location is a reference or the data the reference is pointing at. That's why we have the awkward "null" key in the JSON message:

{
  json: {
    genres: {
      "0": {
        "0": {
          "null": "Die Hard"
        }
     }
   }
}

The immediate problem is that this is ambiguous. JSON is not just used to output paths, but to select them:

model.get({
    genres: {
      "0": {
        "0": {
          "null": "Die Hard"
        }
     }
   }
});

Note that in the example above we can not disambiguate between whether the path is ["genres",0,0,null] or ["genres",0,0,"null"].

The solution to this problem is relatively straightforward: don't return references in JSON. Developers don't expect to deal with references in JSON - they're requesting tree information. The current behavior has no valid use-cases. Developers are expected to request JSON when they are trying to create data they intend to show on-screen. Developers should never be displaying paths on screen. This is an anti-pattern, because it encourages developers to make display considerations when designing their path structures. Paths should be determined by one thing only: efficiency and correctness. We don't want developers to use "pretty" paths instead of efficient ones. It is always possible to create a branch node which contains a pretty version of a path alongside the real one. This cleanly separates presentation concerns from modeling concerns.

The change can succinctly be described thusly: when the output format is JSON, all input paths are assumed to have an implicit trailing [null] key. This would mean that all references encountered at paths would be followed and we would never have a ref in JSON. Then we could omit "null" key from JSON path maps altogether. That would give us this:

{
  json: {
    genres: {
      "0": {
        "0":  "Die Hard"
     }
   }
}

I submit this is consistent with developer expectation. If they are looking to use JSON, it is a tree format and they should not expect references to be returned. If they want references they can always use PathValues or JSON Graph.

@trxclint @michaelbpaulson @sdesai @ktrott

Document Falcor governance / community contribution model

Call implementation currently depends on RX

We need to remove any dependencies on RX before open sourcing.

netflix / falcor Goto Github PK

falcor's Introduction

Falcor

2.0

Roadmap

Getting Started

Creating a Virtual JSON Resource

Retrieving Data from the Virtual JSON resource

Steps to publish new version

Additional Resources

falcor's People

Contributors

Stargazers

Watchers

Forkers

falcor's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs