netflix / falcor-router Goto Github PK

Call is not implemented.

I am going to adopt the rules from falcor and have them as fail since code base is so much smaller.

This would make things even better for debugging when out in production.

Rx's reduce operator throws an error if the stream is Empty. There is no good reason for this other than matching JS impl, which has this behavior because it returns a scalar, not an array. As a result of we want to be tolerant of empty streams we need to use scan().finalValue()

@cerealbox can you add details?

Currently the authorize call back is invoked regardless of which Data source method is invoked. The problem is that some methods may be supported on a route without authorization, and others require authorization.

The solution should be to add the optional authorizeGet, authorizeSet, and authorizeCall methods to the route. Depending on the method invoked, it's specific authorize method should be executed. If a specific authorize method is not present, we should execute the generic authorize method if it exists. If no authorize method exists, access is automatically granted.

Note that all of the authorize methods should always be invoked with the Router as the this object.

Routes should be able to return multiple data types:

Observable of JSON graph

Observable of path value

Observable of array of Path value

Promise<Array>

Promise

Each onNext notification should be processed separately. Collapses should only occur when a single onNext delivers multiple values in an Array or a JSON Graph.

Promises should not in anyway be disadvantaged over observables.

The router will needs its simple merge strategy that does not consider boxValues, materialize, or treatErrorsAsValues but instead builds up the cache from the messages coming in from routes and building up the missing paths.

Should be called for each Node error discovered. This will allow us to send error information to the logs. We want the original node error, not the error we generate to serialize.

There is a need to ensure that errors being thrown from a route are properly handled.

There needs to be a set of keys and in what positions that can be taken out to reduce the scope of collapse. This could be done in between the match and execute function.

It turns out that this is a much harder problem than i initially thought. But the matching function should return the set of missing paths as well as all the matched paths. This will be used by the Router to either emit errors or emit materialized missing paths.

Ran into this issue while installing.

implement set as described between @jhusain and I.

Incoming JSONG will be matched and converted to pathValues.

Examples.

var incomingJSONG = {
    jsong: {
        videos: {
            1234: {
                rating: 5
            }
        }
    },
    paths: [
        ['videos', 1234, 'rating']
        ['videos', 555, 'rating']
    ]
};

// router
...
route: 'videos[{integers:ids}].rating',
set: function(pathValues) {
    // there are 2 pathValues -
    // 1. [videos, [1234], rating]
    // 2. [videos, [555], rating]
    var idList = pathValues.reduce(function(acc, pV) {
        return acc.concat(pV.ids);
    }, []);
    ... more code around setting ... 
}
...

// But there could be ranges

...
route: 'videos[{ranges:indices}].rating',
set: function(pathValues) {
    // there are 2 pathValues -
    // 1. [videos, [{from: 1234, to: 1234}], rating]
    // 2. [videos, [{from: 555, to: 555}], rating]
    var idList = pathValues.reduce(function(acc, pV) {
        return convertRangeToArray(pV.indices)
    }, []);
    ... more code around setting ... 
}
...

@jhusain does that look correct?

One invalid path parameter will bring down the Node server. We need to catch errors when handling user input, and return a descriptive error in the errors array. Error should not have a path, like call.

see disabled unit tests in:
https://github.com/Netflix/falcor-router/blob/master/test/unit/core/call.spec.js

Each individual path being evaluated by the router should allow 50 redirects maximum. By default the number should be 50. However we should be able to override that number in the constructor.

with this example path:

'titlesById[0..1]["title", "year"]'

when multiple specific keys are used in a route rule:

"titlesById[{integers:titleIds}]['title','year','description','rating','boxshot']"

instead of a generic keys rule:

"titlesById[{integers:titleIds}][{keys:titleKeys}]"

the route is called multiple times with a pathSet for each key:

[ 'titlesById', [ 0, 1 ], 'title']
[ 'titlesById', [ 0, 1 ], 'year']

instead of once with a single pathSet for every key:

[ 'titlesById', [ 0, 1 ], [ 'title', 'year' ]]

@michaelbpaulson I understand we're casting strings to ints, and it would seem this key could be a candidate for such a cast, but since my route is matching Router.keys, shouldn't the matcher skip the cast? I suppose if we have to cast, we should verify the new integer's digit count is the same as the original string.

var router = new Router([{
    route: 'lolomo',
    get: function(path) {
        return Rx.Observable.return({
            paths: [path],
            jsonGraph: {
                lolomo: $ref(["lolomos", "0123456789"]) // <-- bug part 1
            }
        })
        .do(function(x) {
            console.log(JSON.stringify(x));
        });
    }
}, {
    route: 'lolomos[{keys:lolomoID}][{keys:lists}]',
    get: function(path) {
        var lolomoID = path.lolomoID[0];
        console.log(lolomoID) // <-- bug part 2: should print String "0123456789" -- actually prints Number 123456789
        return Rx.Observable
            .from(path.lists)
            .reduce(function(lists, i) {
                lists[i] = $ref(["lists", i]);
                return lists;
            }, {})
            .map(function(lists) {
                var lolomos = {};
                lolomos[lolomoID] = lists;
                return {
                    paths: [path],
                    jsonGraph: { lolomos: lolomos }
                };
            })
            .do(function(x) {
                console.log(JSON.stringify(x));
            });
    }
}]);

from Netflix/falcor#289

the falcor-router-demo was a huge help for me getting a working server up, I was having trouble before using the example from the falcor repo readme, I'm guessing it's a bit out of date. I got a basic server going and connecting to a mysql db, but I haven't been able to figure out how to combine fields so they can make a single sql request.

I remember from older examples/screencasts I watched that all the fields would come in to the get function in the router, and could then be combined into a query, but now it seems it is calling a separate get function for each field, which makes combining them more difficult.

I thought maybe a solution with combining promises would work, but it seems like it would be needlessly complicated and I am probably just missing something.

In any case I think the falcor-router-demo repo could do with another router function to demonstrate how this should be done, assuming it still can be.

after talking to jafar on the phone it became apparent this might be a bug with the router

here's the route code I was testing:

{
    route: "recipesById[{integers:recipeIds}]['title','url_title']",
    get: function (pathSet) {
        console.log((new Date()).getTime(), pathSet);
        /*var key = pathSet[2];
        var stuff = [];
        _.each(pathSet.recipeIds, function(id) {
            stuff.push(dbquery('SELECT '+key+' from km.recipes where id='+id).then(function(data) {
                var rows = data[0];
                return {
                    path: ['recipesById', id, key],
                    value: rows[0][key]
                };
            }));
        });
        return Q.all(stuff);*/
    }
}

on the client side:

model.get('recipesById[1..10]["title","url_title"]').then(/* ... */);

log output from the get function on the server happens twice, once for each field

1435606139423 [ 'recipesById', [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ], 'title', recipeIds: [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ] ]
1435606139497 [ 'recipesById', [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ], 'url_title', recipeIds: [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ] ]

falcor-router at 7b2b28
falcor-restify at e97e5c
falcor at c8dbdd
falcor-path-syntax at 60253d

Currently if you attempt to execute a function that does not exist, everything (including the function name) is materialized. The appropriate response should be to have the response from the Router onError (with no onNext)

The specific message in the error should be "function does not exist".

Unit test and PR coming.

The more I dig into the router code the more I wonder why the router has not been modeled like an http framework such as Hapi around how it handles requests.

Well given the focus on solving a particular problem for Netflix I understand and it explains the FP approach of the code.

But to future proof so to speak the router and to make sure it can be used by a broader audience, I wonder if it wouldn't be a good idea to do a little refactoring so it becomes more idiomatic and provides extension points so users can easily plug it into their infrastructure.

The problem right now is the Router is expected to be a singleton but in reality it can't really be one because most likely you need it to be stateful and there is not really a well defined object for the request or the response that would allow people to hook into the router request/response lifecycles such as http://www.hapijs.com/api#request-lifecycle

I believe it would help greatly if the router would become a singleton by contract and if we had FalcorRequest and FalcorResponse objects that would help to capture the state in well defined expected objects and provide extension points to hook into the router.

We should return a 500 here to avoid retrying endless numbers of times for a request that will never be satisfied.

Note that this error only occurs when a route is matched but the route returns no data.

We should also log to the console exactly which were expected to be found, but not found.

When the jsong response includes arrays, the router mistakenly converts these to undefined atoms. I have created a test to demonstrate the issue. I wanted input from the team before submitting a pull request with my test.

it('should allow arrays in jsong response', function (done) {
    var routeResponse = {
       "jsong":{
          "ProductsById":{
             "KMW39254":{
                "SellingPoints":[
                   "Charges USB powered devices with micro or mini ports.",
                   "Charges in the car or on AC wall outlet.",
                   "Ultra-small and low profile.",
                   "Car charger is one amp-ready for quick smartphone charging."
                ]
             }
          }
       }
    };
    var router = new R([
        {
            route: "ProductsById[{keys}][{keys}]",
            get: function (pathSet) {
                return Observable.of(routeResponse);
            }
        }
    ]);
    var obs = router.get([["ProductsById", "KMW39254", "SellingPoints"]]);
    var called = false;
    obs.subscribe(function (res) {
        expect(res).to.deep.equals(routeResponse);
        called = true;
    }, done, function () {
        expect(called, 'expect onNext called 1 time.').to.equal(true);
        done();
    });
});

When I run the test, it fails as shown below.

1) Router Unit Core Specific should allow arrays in jsong response:

  AssertionError: expected { Object (jsong) } to deeply equal { Object (jsong) }
  + expected - actual

   {
     "jsong": {
       "ProductsById": {
         "KMW39254": {
  -        "SellingPoints": {
  -          "undefined": {
  -            "$type": "atom"
  -          }
  -        }
  +        "SellingPoints": [
  +          "Charges USB powered devices with micro or mini ports."
  +          "Charges in the car or on AC wall outlet."
  +          "Ultra-small and low profile."
  +          "Car charger is one amp-ready for quick smartphone charging."
  +        ]
         }
       }
     }
   }

Each route should have an optional authorize callback. This method is called with the matched pathSet, and it returns either a synchronous Boolean or a Observable/Promise. If the Boolean is true, the pathSet is passed to the route handler. If false, a error object with a value of "unauthorized" is materialized for every path in the pathSet.

By default, if no authorize callback is provided universal access is assumed.

via @jhusain

when route returns a promise of pathvalue, each individual path is evaluated separately.
the intention was that whenever pathValues where returned together in a single callback they should all be evaluated and collapsed before being sent to the route handlers.

in other words, the only time in which i would expect paths to be evaluated separately is when they were returned progressively in an observable.
when returned as an array of pathValues i would expect the behaviour to be the same as a JSONGraph envelope.

to see this behaviour in action,
download falcor-router-demo: https://github.com/netflix/falcor-router-demo
and evaluate the following path:
'genrelist[0].titles[0..1]["title"]'

the last route:
"titlesById[{integers:titleIds}]['title','year','description','rating','boxshot']"
gets matched with the following individual paths:
["titlesById",[1],"title"]
["titlesById",[2],"title"]

this is the case because the previous route:
"genrelist[{integers:indices}].titles[{integers:titleIndices}]"
returns a Promise<Array> and each individual path is evaluated separately.

if we change the previous route to return a Promise the paths are properly collapsed before being passed to the route.

this function was used to convert a list of path values to a JSONGraphEvelope for the purposes of testing:

function pathValuesTOJSONGraphEvelope(pathValues) {
    var jsonGraph = {}
    pathValues.forEach(function(pathValue) {
        var path = pathValue.path
        var value = pathValue.value
        var node = jsonGraph
        var parent = jsonGraph
        path.slice(0, -1).forEach(function(key) {
            node = node[key]
            if (node == null || typeof node !== "object") {
                node = parent[key] = {}
            }
            parent = node
        })
        node[path[path.length - 1]] = value
    })
    return {
        jsonGraph: jsonGraph
    }
}

I was wondering if it could be useful to have meta attached to routes ?
Something like:

{
  route: "",
  get: function() {},
  meta: {
     foo: "bar"
  }
}

This could be pretty interesting to have a declarative approach to assist for example the authorize callback instead of inlining the information in the code.

What do you think ?

When 2 paths of different types, get, set, or call are set in, even amoung different routes, no error should be thrown. Only when 2 paths of same types are in a route that should cause the error.

The router needs to take in missing paths and self resolve instead of relying on the model to re-request from the dataSource the paths.

The matched set that finishes with suffix will need to have its next set of nextPaths to be collapsed.

The router needs to validate the out-going jsong and ensure that the paths that are missing should be materialized.

router.get([['videos', [123, 456, 999], 'title']])

// output is:
{
    jsong: {
        videos: {
            123: {
                title: 'HoC'
            },
            456: {
                title: 'OITNB'
            }
        }
    }
};

With this the router should validate the jsong and attach the following information.

{
    jsong: {
        videos: {
            123: {
                title: 'HoC'
            },
            456: {
                title: 'OITNB'
            },
            999: {
                title: {$type: 'atom'}
            }
        }
    }
};

Builds need to break if there are the lints. There were 60ish lint issues. Fix them and ensure correct build.

This covers the input and output for the router. This would make a good unit test. @jhusain this looks correct?

// callPath, args, suffix, paths
var callPath = ['lolomo', 'add'];
var args = [$ref('listsById[29]')];
var suffixes = [['name']];
var paths = [['length']];

// The route info
{
    lolomo: ['lolomos', 123],
    lolomos: {
        123: {
            add: (callPath, listRef) => {
                return {
                    path: ['lolomos', 123, 7],
                    value: $ref('listsById[29]')
                };
            }
        }
    }
}

// The output jsonGraph
{
    jsong: {
        lolomo: ['lolomos', 123],
        lolomos: {
            123: {
                add: {$type: 'atom', $expires: 0},
                7: $ref('listsById[29]'),
                length: 8
            }
        },
        listsById: {
            29: {
                name: 'Thrillers'
            }
        }
    },
    paths: [
        // The call path
        ['lolomo', 'add'],

        // The suffix
        ['lolomo', 7, 'name'],

        // The paths
        ['lolomo', 'length']
    ]
}

The big take away from are the following:

• Paths (jsongEnv paths) from the server must be returned (different from get/set)
• Paths must be specified from the callPath, not from the optimized path.
  • This means suffixes must be relative to the callPath, not the optimized path.
  • This should be easy:
    • If the callPath is different than the path used to reach the call function, then its been optimized
    • If the path to the call function has been optimized then the references returned from the call path need to be sliced like the following:

var optimizedLength = optimizedCallPath.length;

// Returns an array of array syntax paths
// From the above example this should make [lolomos, 123, 7, name] 
// and transform it into [[lolomo, 7, name]], from the following values:
// callPath = [lolomo, add]
// suffixes = [[name]]
// basePath = [lolomo, 7]
// normalizedSuffixes = [[lolomo, 7, name]]
var normalizedSuffixes = 
    getReferencesFromCallResults(callResults).
        reduce(function(normalizedSuffix, ref) {
            var basePath = callPath.concat(ref.slice(optimizedLength));
            suffixes.forEach(function(suffix) {
                normalizedSuffix.push(basePath.concat(suffix));
            });
            return normalizedSuffix;
        }, []);

// If there was such a function.
callSuffixes(normalizedSuffixes);

Finally: Router.prototype.get will have to take in a jsonGraphContex as a second argument. This will make it so that the cache calculated from the callPath optimizations are not lost when performing the get (as it could cause a cache fracture).

Despite the fact that there appears to be a test for this, it looks like this bug does exist. I found this while attempting to match two contiguous indices using an integers pattern. The two contiguous integers got converted into a range on the client, and were subsequently not matched by the route. Unit test and PR to follow.

Note that the reverse does not appear to be true. Integers are successfully matched by the ranges pattern. Will make a separate PR for a test I made that confirms this. Feel free to reject this PR if you feel that this case is adequately covered in your unit tests.

As of right now the following problem can happen.

// The router
{
    ...,
    {
        route: 'videos[1234].summary',
        get: function() { ... }
    },
    {
        route: 'videos[{integers}].summary',
        get: function() { ... }
    }

when the following paths come in, only the specific path will be matched.

var incoming = ['videos', [1234, 444], 'summary'];
var matched = [{
    precedence: 444,
    path: ['videos', 1234, 'summary']
}, {
    precedence: 424,
    path: ['videos', 1234, 'summary']
}]

Only the top match will get matched and no path stripping will occur. What needs to happen is the most specific needs to be executed, its matched paths stripped from the overall incomingPaths and the next lower precedence executed until there are no paths left.

The results of those should all be merged into one observable and when that completes the collapse / re-match / execute should happen.

Call does not consider errors for now. To get happy path out of the way this is something i have chosen to do afterwords and this is a ticket to not forget it.

example client request:

model.get('todosById[0..3,6..9].name').then(function (data) {
    console.log(JSON.stringify(data, null, 4));
});

example server router:

var router = new Router([{
    route: "todosById[{integers:ids}].name",
    get: function(pathSet) {
        return pathSet.ids.map(function(id) {
            return {
                path: ['todosById', id, 'name'],
                value: "get milk from corner store."
            }
        })
    }
}]);

falcor sends 3 server requests. the path of the first is:

[["todosById","[object Object]","name"]]

When there are references and no knowledge of the server then set does not work. The path values retrieved by set have to be done with the requested paths, not the current path.

{
    jsong: {
        genreLists: {
            0: {
                0: {
                    rating: 5
                }
            }
        }
    },
    paths: [['genreLists', 0, 0, 'rating']]
}

That will attempt to retrieve pathValues with ['videos', 1234, 'rating'] which will be undefined. In other words, set that passes through references (unoptimized logic) will not work.

Let's say we have the following call:

router.get([["titlesById", 23, "title"], ["titlesById", 99, "rating"]])

Let's also say the router to find the following routes:

"titlesById[{integers}].title"

"titlesById[{integers}].rating"

Each one of these routes calls the same service: serviceLayer.getTitlesByIds(ids). This service is turn batched, meaning that if both the paths are evaluated in the same turn, A single call will be made to the underlying service for both the ID 23 and ID 99.

Currently multiple paths are not matched against routes in the same turn. There is some sort of sequential instead of concurrent RX operation going on here. It is defeating the batching in the service there.

If a route intends to match and handle all paths, there is no need for it to return a paths key in the JSON Graph message. The router should assume that a missing paths key on a JSON Graph Envelope implies that a route has matched all paths. In other words, if a JSONGraphEnvelope has no paths key, the Router can set the paths key on the JSONGraphEnvelope to all the paths passed in.

Note that this only applies to set and get, not to call which must return a paths key.

@michaelbpaulson

When two routes with same constant keys and same precedence are inserted then an error should be thrown.

e.g.:

{
    ...
    route: 'videos[{integers}].rating',
    ...
    route: 'videos[{ranges}].rating'
    ...
}

This should throw an error.

After the set operation we need to return the post set property value. If there is no set handler, a property is assumed to be read only. And therefore in order to ensure that the property is set to the correct value, we need to execute the get handler to return the correct value to the Falcor client, and overwrite the optimistically set value in the cache with the actual value.

Unit test and PR coming.

Failing test with repro checked in.

This is just a clean up task to stop using bind.

associated pull request:
#89

As the server evaluates the incoming paths, there could be overlap. All the cache that is being created needs to be checked against and optimized. This way if videos, 1234 has already been computed, then there is no reason for it to be recomputed.

This should not really happen when it comes to get, but will happen with call.

Hi All,

So far, all the examples I have seen assume that there are no join being made by the database and most examples use services and expect the application to join.

If the router need to retrieve Foo.a and Foo.Bar.a given a Foo ID, how can this be done with only one round trip to the database ?