Oboe.js helps web applications respond quicker by wrapping http's request-response model with a progressively streamed interface. It glues a transport that sits somewhere between streaming and downloading to a JSON parser that sits somewhere between SAX and DOM. It is small enough to be a micro-library, doesn't have any external dependencies and doesn't care which other libraries you need it to speak to.

Oboe makes it really easy to start using json from a response before the ajax request completes. Or even if it never completes.

In Node any stream can be read, not just http.

Use cases
Examples
Installing
API
Why I made this
Getting the most from oboe
Browser support

Use cases

Oboe.js isn't specific to my application domain, or even to solving the big-small download compromise. Here are some more use cases that I can think of:

Sarah is sitting on a train using her mobile phone to check her email. The phone has almost finished downloading her inbox when her train goes under a tunnel. Luckily, her webmail developers used Oboe.js so instead of the request failing she can still read most of her emails. When the connection comes back again later the webapp is smart enough to just re-request the part that failed.

Arnold is using a programmable stock screener. The query is many-dimensional so screening all possible companies sometimes takes a long time. To speed things up, Oboe.js, means each result can be streamed and displayed as soon as it is found. Later, he revisits the same query page. Since Oboe isn't true streaming it plays nice with the browser cache so now he see the same results instantly from cache.

Janet is working on a single-page modular webapp. When the page changes she wants to ajax in a single, aggregated json for all of her modules. Unfortunately, one of the services being aggregated is slower than the others and under traditional ajax she is forced to wait for the slowest module to load before she can show any of them. Oboe.js is better, the fast modules load quickly and the slow modules load later. Her users are happy because they can navigate page-to-page more fluidly and not all of them cared about the slow module anyway.

John is developing internally on a fast network so he doesn't really care about progressive loading. Oboe.js provides a neat way to route different parts of a json response to different parts of his application. One less bit to write.

Examples

A simple download

It isn't really what Oboe is for but you can use it as a simple AJAX library. This might be good to drop it into an existing application so you can refactor later to make it progressive.

oboe('/myapp/things.json')
   .done( function(things) {
   
      // we got it
   })
   .fail(function() {
   
      // we don't got it
   });

Using objects from the JSON stream

Say we have a resource called things.json that we need to fetch over AJAX:

{
   "foods": [
      {"name":"aubergine",    "colour":"purple"},
      {"name":"apple",        "colour":"red"},
      {"name":"nuts",         "colour":"brown"}
   ],
   "badThings": [
      {"name":"poison",       "colour":"pink"},
      {"name":"broken_glass", "colour":"green"}
   ]
}

In our webapp we want to download the foods and show them in a webpage. We aren't showing the non-foods here so we won't wait for them to be loaded:

oboe('/myapp/things.json')
   .node('foods.*', function( foodThing ){
   
      // This callback will be called everytime a new object is found in the 
      // foods array. Oboe won't wait for the download to finish first.
       
      console.log( foodThing.name + ' is ' + foodThing.colour );
   })
   .node('badThings.*', function( badThing ){
          
      console.log( 'Danger! stay away from ' + badThings.name );
   })   
   .done( function(things){
      console.log( 'there are ' + things.foods.length + ' things you can eat ' +
                   'and ' + things.nonFoods.length + ' that you shouldn\'t.' ); 
   });

Hanging up when we have what we need

We can improve on the example above. Since we only care about the foods object and not the non-foods we can hang up as soon as we have the foods, reducing our precious download footprint.

oboe('/myapp/things.json')
   .node({
      'foods.*': function( foodObject ){
   
         alert('go ahead and eat some ' + foodObject.name);
      },
      'foods': function(){
         this.abort();
      }
   });

Duck typing

Sometimes it is more useful to say what you are trying to find than where you'd like to find it. In these cases, duck typing is more useful than a specifier based on paths.

oboe('/myapp/things.json')
   .node('{name colour}', function( foodObject ) {   
      // I'll get called for every object found that 
      // has both a name and a colour   
   };

Detecting strings, numbers

Want to detect strings or numbers instead of objects? Oboe doesn't care about the types in the json so the syntax is the same:

oboe('/myapp/socialgraph.json')
   .node({
      'name': function( name ){
         // do something with the name
      },
      'friends.*.name':function( friendsName ){
         // etc etc
      });

Reacting before we get the whole object

As well as .node, you can use .path to be notified when the path is first found, even though we don't yet know what will be found there. We might want to eagerly create elements before we have all the content to get them on the page as soon as possible.

var currentPersonElement;
oboe('//people.json')
   .path('people.*', function(){
      // we don't have the person's details yet but we know we found someone 
      // in the json stream. We can eagerly put their div to the page and 
      // then fill it with whatever other data we find:
      currentPersonElement = jQuery('<div class="person">');
      jQuery('#people').append(personDiv);
   })
   .node({
      'people.*.name': function( name ){
         // we just found out that person's name, lets add it to their div:
         currentPersonElement.append('<span class="name"> + name + </span>');
      },
      'people.*.email': function( email ){
         // we just found out this person has email, lets add it to their div:
         currentPersonElement.append('<span class="email"> + email + </span>');
      }
   });

Giving some visual feedback as a page is updating

If we're doing progressive rendering to go to a new page in a single-page web app, we probably want to put some kind of indication on the page as the parts load.

Let's provide some visual feedback that one area of the page is loading and remove it when we have data, no matter what else we get at the same time

I'll assume you already implemented a spinner

MyApp.showSpinner('#foods');

oboe('/myapp/things.json')
   .node({
      '!.foods.*': function( foodThing ){
         jQuery('#foods').append('<div>').text('it is safe to eat ' + foodThing.name);
      },
      '!.foods': function(){
         // Will be called when the whole foods array has loaded. We've already
         // wrote the DOM for each item in this array above so we don't need to 
         // use the items anymore, just hide the spinner:
         MyApp.hideSpinner('#foods');
      }
   });

The path parameter

The callback is also given the path to the node that it found in the json. It is sometimes preferable to register a wide-matching pattern and use the path parameter to decide what to do instead of

// JSON from the server side. 
// Each top-level object is for a different module on the page.
{  "notifications":{
      "newNotifications": 5,
      "totalNotifications": 4
   },
   "messages": [
      {"from":"Joe", "subject":"blah blah", "url":"messages/1"},
      {"from":"Baz", "subject":"blah blah blah", "url":"messages/2"}
   ],
   "photos": {
      "new": [
         {"title": "party", "url":"/photos/5", "peopleTagged":["Joe","Baz"]}
      ]
   }
   // ... other modules ...
}

oboe('http://mysocialsite.example.com/homepage.json')
   .node('!.*', function( moduleJson, path ){
   
      // This callback will be called with every direct child of the root
      // object but not the sub-objects therein. Because we're coming off
      // the root, the path argument is a single-element array with the 
      // module name like ['messages'] or ['photos']
      var moduleName = path[0];
      
      My.App.getModuleCalled(moduleName).showNewData(moduleJson);
   });

Deregistering a callback

Calling this.forget() from inside a callback deregisters that listener.

// We have a list of items to plot on a map. We want to draw the first
// ten while they're loading. After that we want to store the rest in a
// model to be drawn later. 

oboe('/listOfPlaces')
   .node('list.*', function( item, path ){
      var itemIndex = path[path.length-1];
      
      model.addItemToModel(item);      
      view.drawItem(item);
              
      if( itemIndex == 10 ) {
         this.forget();
      }
   })
   .done(function( fullJson ){
      var undrawnItems = fullJson.list.slice(10);
            
      model.addItemsToModel(undrawnItems);
   });

Css4 style patterns

Sometimes when downloading an array of items it isn't very useful to be given each element individually. It is easier to integrate with libraries like Angular if you're given an array repeatedly whenever a new element is concatenated onto it.

Oboe supports css4-style selectors and gives them much the same meaning as in the proposed css level 4 selector spec.

If a term is prefixed with a dollar sign, instead of the element that matched, an element further up the parsed object tree will be given instead to the callback.

// the json from the server side looks like this:
{"people": [
   {"name":"Baz", "age":34, "email": "[email protected]"}
   {"name":"Boz", "age":24}
   {"name":"Bax", "age":98, "email": "[email protected]"}}
]}

// we are using Angular and have a controller:
function PeopleListCtrl($scope) {

   oboe('/myapp/things.json')
      .node('$people[*]', function( peopleLoadedSoFar ){
         
         // This callback will be called with a 1-length array, a 2-length
         // array, a 3-length array etc until the whole thing is loaded 
         // (actually, the same array with extra people objects pushed onto
         // it) You can put this array on the scope object if you're using 
         // Angular and it will nicely re-render your list of people.
         
         $scope.people = peopleLoadedSoFar;
      });
}

Like css4 stylesheets, this can also be used to express a 'containing' operator.

oboe('/myapp/things.json')
   .node('people.$*.email', function( personWithAnEmailAddress ){
      
      // here we'll be called back with baz and bax but not Boz.
      
   });

Using Oboe with d3.js

// Oboe works very nicely with d3. http://d3js.org/

// get a (probably empty) d3 selection:
var things = d3.selectAll('rect.thing');

// Start downloading some data.
// Every time we see a new thing in the data stream, use d3 to add an element to our 
// visualisation. This basic pattern should work for most visualistions built in d3.
oboe('/data/things.json')
   .node('$things.*', function( thingsArray ){
            
      things.data(thingsArray)
         .enter().append('svg:rect')
            .classed('thing', true)
            .attr(x, function(d){ return d.x })
            .attr(y, function(d){ return d.x })
            .attr(width, function(d){ return d.w })
            .attr(height, function(d){ return d.h })
            
      // no need to handle update or exit set here since downloading is purely additive
   });

Reading from any stream (Node.js only)

Instead of giving a url you can pass any ReadableStream. To load from a local file you'd do this:

oboe( fs.createReadStream( '/home/me/secretPlans.json' ) )
   .on('node', {
      'schemes.*': function(scheme){
         console.log('Aha! ' + scheme);
      },
      'plottings.*': function(deviousPlot){
         console.log('Hmmm! ' + deviousPlot);
      }   
   })
   .on('done', function(){
      console.log("*twiddles mustache*");
   })
   .on('fail', function(){
      console.log("Drat! Foiled again!");   
   });

Because explicit loops are replaced with declarations the code is usually about the same length as if you'd done JSON.parse:

fs.readFile('/home/me/secretPlans.json', function( err, plansJson ){     
   if( err ) {
      console.log("Drat! Foiled again!");
      return;
   }
   var plans = JSON.parse(err, plansJson);
   
   plans.schemes.forEach(function( scheme ){
      console.log('Aha! ' + scheme);   
   });   
   plans.plottings.forEach(function(deviousPlot){
      console.log('Hmmm! ' + deviousPlot);
   });
      
   console.log("*twiddles mustache*");   
});

Rolling back on error

The fail function gives a callback for when something goes wrong. If you started putting elements on the page and the connection goes down you have a few options

If the new elements you added are useful without the rest, leave them
If they are useful but you need the rest, make a new request
Rollback any half-done changes you made

var currentPersonElement;
oboe('everyone')
   .path('people.*', function(){
      // we don't have the person's details yet but we know we found 
      // someone in the json stream, we can use this to eagerly add them to 
      // the page:
      personDiv = jQuery('<div class="person">');
      jQuery('#people').append(personDiv);
   })
   .node('people.*.name', function( name ){
      // we just found out that person's name, lets add it to their div:
      currentPersonElement.append('<span class="name"> + name + </span>');
   })
   .fail(function(){
      if( currentPersonElement ) {
         // oops, that didn't go so well. instead of leaving this dude half 
         // on the page, remove them altogether      
         currentPersonElement.remove();
      }
   })

More patterns

!.foods.colour the colours of the foods
person.emails[1] the first element in the email array for each person {name email} any object with a name and an email property, regardless of where it is in the document
person.emails[*] any element in the email array for each person
person.$emails[*] any element in the email array for each person, but the callback will be passed the array so far rather than the array elements as they are found.
person all people in the json, nested at any depth
person.friends.*.name detecting friend names in a social network
person.friends..{name} detecting friends with names in a social network
person..email email addresses anywhere as descendent of a person object
person..{email} any object with an email address relating to a person in the stream
$person..email any person in the json stream with an email address
* every object, string, number etc found in the json stream
! the root object (fired when the whole response is available, like JSON.parse())

Installing

Browser

For the client-side grab either oboe-browser.js or oboe-browser.min.js, or use bower like:

bower install oboe

If AMD is detected Oboe will define itself. Otherwise it adds oboe to the global namespace. Either load the module using require.js, almond etc or just use it directly.

If using with Require some config is needed so Require knows to load a file named oboe-browser.js for the oboe module. Alternatively, you could rename oboe-browser.js to oboe.js.

require.config({
    paths: {
        oboe: 'oboe-browser'
    }
});

Node.js

npm install oboe

Then load as usual:

var oboe = require('oboe');

API

If you are in Node.js or are in a browser and have AMD loaded Oboe won't register a global so do this:

var oboe = require('oboe')

The oboe object

Start a new AJAX request by calling one of these methods:

oboe( String url ) // makes a GET request
      
oboe({
   method: String,  // defaults to GET
   url: String,
   headers:{ key: value, ... },
   body: Object|String   
})

// DEPRECATED: The oboe.doFoo() methods will be removed for oboe v2.0.0:
oboe.doGet(    String url )
oboe.doDelete( String url )
oboe.doPost(   String url, Object|String body )
oboe.doPut(    String url, Object|String body )
oboe.doPatch(  String url, Object|String body )              

oboe.doGet(    {url:String, headers:{ key: value, ... }}, cached:Boolean )       
oboe.doDelete( {url:String, headers:{ key: value, ... }} )   
oboe.doPost(   {url:String, headers:{ key: value, ... }, body:Object|String} )
oboe.doPut(    {url:String, headers:{ key: value, ... }, body:Object|String} )       
oboe.doPatch(  {url:String, headers:{ key: value, ... }, body:Object|String} )

If the body is given as an object it will be serialised using JSON.stringify. Method, body and headers arguments are all optional.

If the cached option is set to false caching will be avoided by appending "_={timestamp}" to the URL's query string.

The above are supported under Browsers or Node.js. Under Node you can also give Oboe a ReadableStream:

oboe( ReadableStream source ) // Node.js only

When reading from a stream http headers and status code will not be available via the start event or the .header() method.

.node() and .path()

When you make a request the returned Oboe instance exposes a few chainable methods:

.node( String pattern, 
       Function callback(node, String[] path, Object[] ancestors)
)

.on( 'node', 
     String pattern, 
     Function callback(node, String[] path, Object[] ancestors)
)

// 2-argument style .on() which is compatible with Node.js EventEmitter#on
.on( 'node:{pattern}',  
     Function callback(node, String[] path, Object[] ancestors)
)

Listening for nodes registers an interest in JSON nodes which match the given pattern so that when the pattern is matched the callback is given the matching node. Inside the callback this will be the Oboe instance (unless you bound the callback)

The parameters to callback are:

node - the node that was found in the JSON stream. This can be any valid JSON type - Array, Object, String, true, false or null.
path - an array of strings describing the path from the root of the JSON to the location where the node was found
ancestors - an array of node's ancestors. ancestors[ancestors.length-1] is the parent object, ancestors[ancestors.length-2] is the grandparent and so on.

.path( String pattern, 
       Function callback( thingFound, String[] path, Object[] ancestors)
)

.on( 'path', 
     String pattern, 
     Function callback(thingFound, String[] path, Object[] ancestors)
)

// 2-argument style .on() which is compatible with Node.js EventEmitter#on
.on( 'path:{pattern}',  
     Function callback(thingFound, String[] path, Object[] ancestors)
)

.path() is the same as .node() except the callback is fired when we know about the matching path, before we know about the thing at the path.

Alternatively, several patterns may be registered at once using either .path or .node:

.node({
   pattern1 : Function callback,
   pattern2 : Function callback
});

.path({
   pattern3 : Function callback,
   pattern4 : Function callback
});

.done()

.done(Function callback(Object wholeJson))

.on('done', Function callback(Object wholeJson))

Register a callback for when the response is complete. Gets passed the entire JSON. Usually it is better to read the json in small parts than waiting for it to completely download but this is there when you need the whole picture.

.start()

.start(Function callback(Object json))

.on('start', Function callback(Number statusCode, Object headers))

Registers a listener for when the http response starts. When the callback is called we have the status code and the headers but no content yet.

.header([name])

.header()

.header(name)

Get http response headers if we have received them yet. When a parameter name is given the named header will be returned, or undefined if it does not exist. When no name is given all headers will be returned as an Object. The headers will be available from inside node, path, start or done callbacks, or after any of those callbacks have been called. If have not recieved the headers yet .header() returns undefined.

.root()

.root()

At any time, call .root() on the oboe instance to get the JSON received so far. If nothing has been received yet this will return undefined, otherwise it will give the root Object.

.forget()

.node('*', function(){
   this.forget();
})

Calling .forget() on the Oboe instance from inside a node or path callback de-registers the currently executing callback.

.removeListener()

.removeListener('node', String pattern, callback)
.removeListener('node:{pattern}', String pattern, callback)

.removeListener('start', callback)
.removeListener('done', callback)
.removeListener('fail', callback)

Remove a node, path, start, done, or fail listener. From inside the listener itself .forget() is usually more convenient but this works from anywhere.

.abort()

.abort() Stops the http call at any time. This is useful if you want to read a json response only as far as is necessary. You are guaranteed not to get any further .path() or .node() callbacks, even if the underlying xhr already has additional content buffered and the .done() callback will not fire. See example above.

.fail()

Fetching a resource could fail for several reasons:

non-2xx status code
connection lost
invalid JSON from the server
error thrown by a callback

   .fail(Function callback(Object errorReport))
   
   .on('fail', Function callback(Object errorReport))

An object is given to the callback with fields:

thrown: The error, if one was thrown
statusCode: The status code, if the request got that far
body: The response body for the error, if any
jsonBody: If the server's error response was json, the parsed body.

Pattern matching

Oboe's pattern matching is a variation on JSONPath. It supports these clauses:

! root object
. path separator
person an element under the key 'person'
{name email} an element with attributes name and email
* any element at any name
[2] the second element (of an array)
['foo'] equivalent to .foo
[*] equivalent to .*
.. any number of intermediate nodes (non-greedy) $ explicitly specify an intermediate clause in the jsonpath spec the callback should be applied to

The pattern engine supports CSS-4 style node selection using the dollar ($) symbol. See also some example patterns.

Why I made this

Early in 2013 I was working on complementing some Flash financial charts with a more modern html5/d3 based web application. The Flash app started by making http requests for a very large set of initial data. It took a long time to load but once it was started the client-side model was so completely primed that it wouldn't need to request again unless the user scrolled waaaay into the past.

People hate waiting on the web so naturally I want my html5 app to be light and nimble and load in the merest blink of an eye. Instead of starting with one huge request I set about making lots of smaller ones just-in-time as the user moves throughout the data. This gave a big improvement in load times but also some new challenges.

Firstly, with so many small requests there is an increased http overhead. Worse, not having a model full of data early means the user is likely to need more quite soon. Over the mobile internet, 'quite soon' might mean 'when you no longer have a good connection'.

I made Oboe to break out of this big-small compromise. We requested relatively large data but started rendering as soon as the first datum arrived. We have enough for a screenfull when the request is about 10% complete. 10% into the download and the app is already fully interactive while the other 90% steams silently in the background.

Sure, I could have implemented this using some kind of streaming framework (socket.io, perhaps?) but then we'd have to rewrite the server-side and the legacy charts would have no idea how to connect to the new server. It is nice to just have one, simple service for everything.

Getting the most from oboe

Asynchronous parsing is better if the data is written out progressively from the server side (think node or Netty) because we're sending and parsing everything at the earliest possible opportunity. If you can, send small bits of the output asynchronously as soon as it is ready instead of waiting before everything is ready to start sending.

Browser support

Browsers with Full support are:

Recent Chrome
Recent Firefox
Internet Explorer 10
Recent Safaris

Browsers that work but don't stream:

Internet explorer 8 and 9, given appropriate shims for ECMAScript 5

Unfortunately, IE before version 10 doesn't provide any convenient way to read an http request while it is in progress.

The good news is that in older versions of IE Oboe gracefully degrades, it'll just fall back to waiting for the whole response to return, then fire all the events together. You don't get streaming but it isn't any worse than if you'd have designed your code to non-streaming AJAX.

mtou091 / oboe.js Goto Github PK

oboe.js's Introduction