jimhigson / oboe.js Goto Github PK

I am try this but doesn't work
var demo = "verylarge.json"

function start() {

oboe.fetch(demo).onFind('id', function( id ){
    alert(id);
});

}

currently I am trying to read an existing json file with this structure

[ {"Product" : "Test", "Type" : "Beta"}, {"Product" : "Test2", "Type" : "Beta"}, {"Product" : "Test3", "Type" : "Alpha"} ]

The file is huge itself so imagine thousands of them.

What I want is to have some logic applied to this data for instance.

When oboe reads an object {object1} if object1.Type = beta, push the entirety of object1 to a new array/stream. Essentially constructing a new json object.

Also I would need the ability to know let oboe drain the object from the stream so it does not stay in memory once it has done this check. I want to essentially create a rudimentary search feature for reading from json streams.

so I tried this and got some interesting results.

oboe(fs.createReadStream("./testfile"))
.node( "{}", function(data) {
console.log(data);
});
});

it would log each property of the object then the entire object itself, which means it is emitting extra fields.

So in this example it console.logs it like this.

Test
Beta
{ Product : 'Test',
Type : 'Beta' }
Test2
Beta
{ Product : 'Test2',
Type : 'Beta' }
Test3
Alpha
{ Product : 'Test3',
Type : 'Alpha' }
{ Product : 'Test',
Type : 'Beta' },
{ Product : 'Test2',
Type : 'Beta' },
{ Product : 'Test3',
Type : 'Alpha' }

Any ideas?

It would be cool if oboe supported json lines streams.
http://jsonlines.org/

It would be helpful to know which browsers OOTB support streaming of JSON without a polyfill. Could you perhaps add such a table?

Hopefully I'm just doing something simple wrong and you can point me in the right direction, but I can't seem to get the cookie to send with the request.

Here are my attempts: https://gist.github.com/brian-gates/c5456101401ca2d4409d

Any help would be appreciated. Thanks!

Is there any way to refer to the Oboe instance if I'm working with a bounded callback?

oboe({
    url: this.props.url
}).node('user', (user) => {
    this.setState({
        user: user
    });

    // And I'd like to do .abort() here
})

Is there a chance to use oboe in a HTML5 web worker? If I try to load use it I get a

ReferenceError: window is not defined

because in web workers you can't access the window object. My JSON file is extremely large so I can't process all this stuff in the GUI thread.

Supposed I send some JSON data to the client using Oboe.js that is created dynamically by the server. From time to time, creation of this data takes some time, so after 120 seconds of inactivity, the connection is closed.

How can I force the connection to stay open infinitely?

The most obvious approach is using a timer and sending some fake data, but this means that my client app has to filter out the fake data.

Is there a better approach?

Using request in node (https://github.com/request/request)

var stream = request(blah..);
oboe(stream) // Doesn't think stream is a stream
.on("farb",....)

This is because it checked for 'unpipe' and 'unshift' to determine if it had a stream or not.
I removed the check for those functions as below and all worked fine.

var nodeStreamMethodNames = list('resume', 'pause', 'pipe' ); //,'unpipe','unshift'),

As I know browsers' WebSocket APIs are message-based so we can only receive messages not frames (and not streams respectively cause there is no actual chunks of data). Does it mean that there is no profit using oboe with WebSockets? Will passing a ReadableStream to oboe under browser environment be possible after browsers will support Streams and Streams API?

When using method: POST and where data is an object would headers: {'Content-Type' : 'application/json'} be a sane default? Or is that outside the scope of this library?

When receiving an error response with a payload, oboe fires the done callback, followed by the fail callback.

This is the response I'm receiving:

HTTP/1.1 401 Unauthorized
Cache-Control: private
Transfer-Encoding: chunked
Content-Type: application/json; charset=utf-8
Date: Mon, 06 Apr 2015 09:13:01 GMT

{"code":1,"message":"Invalid credentials"}

The done method fires first and is passed the parsed JSON response object. The fail method fires shortly thereafter with the statusCode correctly set.

Unless I'm mistaken, done should not be fired here because the response contains an error. This is what happens when the webserver returns just an error code with no JSON response.

pubsub.js was written to be an ultra-minimal event lib in the browser.

In Node, no reason not to use the normal EventEmitter.

Might need some methods like .once adding to pubsub.js to match.

Also, separate internal methods for node found and node matched.

All is in the title :)

At oboe-node.js, the query string is stripped from the URL (given to oboe()).

var req = http.request({
         hostname: parsedUrl.hostname,
         port: parsedUrl.port, 
         path: parsedUrl.pathname, // This line
         method: method,
         headers: headers 
});

Should that not be parsedUrl.path? From the node docs:

• pathname: The path section of the URL, that comes after the host and before the query, including the initial slash if present.
• search: The 'query string' portion of the URL, including the leading question mark.
• path: Concatenation of pathname and search.

Or is this wanted behavior?

Do you plan to support AMD ?

I have oboe pulling out the objects I want from the readstream but I dont know how to pipe it back to res?

app.get('/foo', function(req, res) {
var rstream = oboe(fs.createReadStream("./test");
rstream.node( "{}", function(data) {
if ( data.test != undefined ) {
>>>> now what?

rstream has no method pipe so I cant simple
rstream.pipe(res) at this point.

If i try the example with streaming HTML with JSON service, I get an error saying no engine or extension provided when creating a new view.

I am really not sure how to proceed here

Hi,
first of all thanks for this great lib!
Just a question, i tried to stream a HUGE JSON from node to oboe, but chrome keeps crashing because it reaches 1GB+ of memory occupation in less then 10sec. I think the problem is that oboe keep collecting the data to create the complete JSON.

Is there any way to avoid the complete object?

Just in case my code is here:

var item = 0;
oboe( {
  url: '/data/'+social,
  method: 'GET',
  cache: false
} )
.node( 'location', function( location ) {
  item++;
} )
.done( function() {
  console.log( item );
} );

Sorry if this is a ridiculous question, but how can I use this with

res.send({"items": items})

in an Express route, where items represents the JSON data (already rec'd from an authorization api that retrieves the user's list) - which is large and I'd like to beginning showing as soon as the first item is available.

Thanks in advance!

Using this Github API to get the latest release it'd go:

oboe('/repos/jimhigson/oboe.js/releases')
   .node('![last].name', function( releaseName ){
      console.log('The most recent release was', releaseName);
   })

This is tricky because you don't know if a an array element is the last when you see it from the stream because you haven't seen all of the parent yet.

This is partly a question, but might also be a possible bug or point of confusion, so I figured I'd ask here rather than the google group. I didn't see any guidelines on where to ask stuff, so this seemed like the best place. If you'd like to move the conversation there, let me know.

I just started using oboe with a server that streams json responses. It streams each object separately.

So, something like:

{
  "foo": 3,
  "bar": "BAR!!"
}

So far oboe is able to work with the results and each object separately, and most of the examples work well. However, when trying to follow an example with a done callback, it seems to fire the done event every time it receives an object. Is this how it's supposed to be?

My interpretation of it was that the event is only fired once the entire response is completed. But it seems like it's treating each object as its own response. Is it behaving differently because the server is actually streaming json rather than returning a single json object? Or could something else be at play here? Am I using it improperly?

Let me know if there's any information you might need.

I have this JS on my page:

oboe(requestUri)
.node('*', function( Thing ){
alert('data!');
})
.fail(function() {
alert('problems');
});
});

The requestUri returns an open connection (which never closes) and sends JSON. Here's an example of what it would send:

[{"Attribute":"att1","ID":"[email protected]","Value":"value1"},{"Attribute":"att2","ID":"[email protected]","Value":"value2\r\n"}]

It'll send a series of these, as things change. Each time it sends a [ ] containing 1 or more { } (the example shows 2), followed by a line break. Is all that OK for Oboe?

(The \r\n is in the data I'm using - I just noticed it but don't think it would cause a problem?)

I'm using Chrome. Using Dev Tools and Fiddler, I see the initial OPTIONS call (in fact, I had to code the response for it as I didn't have one!), then the GET being made. The data is returned (I've tested using a console app), but Oboe doesn't show anything. No errors in Console window...just nothing.

Am I wrong in trying to use Oboe for streaming HTTP data, or have I missed something?

Thanks for any help here :)

-tom

Sometimes you need to access the http response header, rather than just the json content, for example, Github API pagination utilize HTTP header Link: http://developer.github.com/v3/#pagination .

So, propose an API to get response headers from an oboe instance, or make it an event.

Github README.md got us so far

I have seen that there already was an issue on memory leaks (it's #45), and that things should be resolved by returning oboe.drop as documented here.

Unfortunately, 2.1.1 seems to still have memory issues (or I am getting it absolutely wrong how to use oboe.drop correctly).

My setup is as follows: I have a server based on Express that delivers an endless JSON array, and I have a client that uses Oboe to stream these data. I measure the memory consumption of this client, and within a few hours it uses hundreds of MBytes, and GC obviously does not clean up as expected.

The server looks like this:

var http = require('http');

var express = require('express');

var app = express();

app.get('/', function (req, res) {
  var i = 0;

  var sendEventToClient = function (event) {
    res.write(JSON.stringify(event) + ',');
  };

  req.setTimeout(0);
  res.setTimeout(0);

  res.writeHead(200, {
    'content-type': 'application/json'
  });
  res.write('[');

  setInterval(function () {
    sendEventToClient({
      foo: 'bar',
      bar: 'baz',
      nufta: 23,
      counter: i++
    });
  }, 10);
});

http.createServer(app).listen(3000);

The client looks like this:

var url = require('url');

var oboe = require('oboe'),
    Stethoskop = require('stethoskop');

var stethoskop = new Stethoskop({
  from: {
    application: 'client.js'
  },
  to: {
    host: 'localhost',
    port: 8125
  },
  enabled: true
});

oboe({
  url: url.format({
    protocol: 'http',
    hostname: 'localhost',
    port: 3000,
    pathname: '/'
  }),
  cached: false
}).on('node:!.*', function (event) {
  console.log(event);
  return oboe.drop;
}).on('fail', function (err) {
  console.log('Error!', err.thrown);
});

So, the server sends a new object every 10 ms, and the client should do nothing with it but drop it. I am using stethoskop to measure the client's fitness: It sends the CPU and memory data to a StatsD server.

I have run these two processes for 4.5 hours, and the memory consumption looks like this:

I also tried to run it returning null instead of oboe.drop, same result (the left part is the same test as above, the right part is with null, so basically both options show the very same behavior. The drop to 0 in the middle is not because of GC, it's because I stopped and restarted the processes):

So, to cut a long story short, basically I have two questions:

• Am I doing something wrong in the client? If so, what do I need to correct? Any hints?
• If not, this seems to be a bug in Oboe. Can I do something to help fix it? Pointing out and giving some hints where to look might already be quite helpful :-)

Moreover, the documentation states:

Dropping from an array will result in an array with holes in it. This is intentional so that the array indices from the original JSON are preserved:

My guess is that dropping nodes only works (wrt memory consumption) for objects, but not for arrays. Since I am using an array as outer container here, and since sparse arrays seem to capture more memory than dense ones, this might be the cause of the problem. Please note that I'm not too sure about memory behavior of sparse arrays, so any answer in this direction will be appreciated.

When talking to an API, its pretty common for the API to generate an error status code but also provide a more human friendly error in the response body.

I don't see a way to get the response object from Oboe in a case where a server didn't give a 2XX status. Is this possible currently?

I am still working to implement oboe.js for stream-based JSON upload in my web application. The uploaded JSON file will be very large (some gigabyte) and processed directly into MongoDB without buffering all of it inside memory (that ain't fit)

I found out that oboe.js detected http.IncomingMessage (which is an implementation of ReadableStream used to receive the uploaded data) as an ordinary URL because it has field called "url" and oboe.js assuming that it is an ordinary URL, thus my web application return an error like this :

/mnt/data/Dropbox/Journal/2013B/SISTER/DewaSister/FP/node_modules/oboe/dist/oboe-node.js:1093
         throw new Error(
               ^
Error: Supported protocols when passing a URL into Oboe are http and https. If you wish to use another protocol, please pass a ReadableStream (ht
tp://nodejs.org/api/stream.html#stream_class_stream_readable) like oboe(fs.createReadStream("my_file")). I was given the URL: /
    at fetchUrl (/mnt/data/Dropbox/Journal/2013B/SISTER/DewaSister/FP/node_modules/oboe/dist/oboe-node.js:1093:16)
    at streamingHttp (/mnt/data/Dropbox/Journal/2013B/SISTER/DewaSister/FP/node_modules/oboe/dist/oboe-node.js:1141:7)
    at wire (/mnt/data/Dropbox/Journal/2013B/SISTER/DewaSister/FP/node_modules/oboe/dist/oboe-node.js:2349:7)
    at applyDefaults (/mnt/data/Dropbox/Journal/2013B/SISTER/DewaSister/FP/node_modules/oboe/dist/oboe-node.js:2407:11)
    at oboe (/mnt/data/Dropbox/Journal/2013B/SISTER/DewaSister/FP/node_modules/oboe/dist/oboe-node.js:2419:17)
    at Server.<anonymous> (/mnt/data/Dropbox/Journal/2013B/SISTER/DewaSister/FP/bermainStream.js:5:5)
    at Server.EventEmitter.emit (events.js:104:17)
    at HTTPParser.parserOnIncoming [as onIncoming] (_http_server.js:505:12)
    at HTTPParser.parserOnHeadersComplete (_http_common.js:111:23)
    at Socket.socketOnData (_http_server.js:358:22)

Hi,
i have a problem with the fail handler. I use oboe in the browser and the streams are served through an node app (express). When i stop my node process, or the server is unreachable, the fail handler gets called with this meaningless error object:

{
  statusCode: 0,
  body: "",
  jsonBody: undefined,
  thrown: undefined
}

How i can detect what kind of error rised from oboe? like CONNECTION_REFUSED or ABORTED?

Using oboe v1.14.2 installed using bower.

Hi,

The fail() handler is supposed to be called on connection loss but it doesn't seem so. I get it called ok for example when the connection couldn't establish at all but not when it is terminated in the middle of a stream.

For example I have a service the receives JSON array of IDs (using POST) and returns streamed JSON array of requested objects:

oboe({
          method: "POST",
          url: "http://my_url",
          body: [id1, id2, id3, id4, id5]
        })
          .node('!.*', function (my_obj) {
             console.obj('obj ' + my_obj['id'] + 'returned');
           })
          .fail(function (error) {
            console.log('error');
          });

If I kill the service for example after receiving second object the fail() is not called and I have no way to handle that situation. This works ok if I run it when the service is already dead - the fail() handler gets called in this case.

Thanks, Antony.

Currently oboe catches errors thrown in callbacks and triggers the fail event

/** 
*  wrap a callback so that if it throws, Oboe.js doesn't crash but instead
*  handles it like a normal error
*/
function protectedCallback( callback ) {
  return function() {
    try {      
      return callback.apply(oboeApi, arguments);   
    } catch(e)  {

      // An error occured during the callback, publish it on the event bus 
      oboeBus(FAIL_EVENT).emit( errorReport(undefined, undefined, e));
    }      
  }   
}

I agree that oboe.js should not crash if there is an error thrown in a callback. However, I disagree that the error should be caught and passed to the fail event handler. Firstly, catching errors in functions it receives as arguments should not be a responsibility of oboe. Secondly, the fail event handler should just be for unsuccessful requests

Thus I suggest the code be updated to the following:

/** 
*  execute the callback in its own event loop so that if it throws, Oboe.js doesn't crash
*/
function protectedCallback( callback ) {
  return function() {
    setTimeout(callback.bind(oboeApi, arguments));     
  }   
}

This way errors in callbacks do not cause oboe to crash and errors thrown by callbacks are not caught.

I am happy to implement this change, but want to get feedback before I spend time updating the tests.

https://github.com/jimhigson/oboe.js/blob/master/src/functional.js#L5

The example in comments is probably not right, IMHO. The boundArgs is used as array inside the function, but it should pass value, not array, when the curried function is called.

Although it's not a big problem, I hope it can be fixed to make oboejs more awesome :)

Are there plans to support the for bower package manager? Thanks

I'm trying to use the Oboe 2.0 syntax to make a POST request:

oboe({
   method: 'POST',
   url: 'http://gariety.xxx.dev',
   headers: { 'x-key': 'value' },
   body: 'foo'  
})

But oboe performs a GET request, with the correctly url, headers and body.

Shouldn't it perform a POST?

As far as I can tell, the interface requires a path or URL. However in Node.js, it is very common to already have a buffer or stream of data. Is it possible to use that buffer or stream as input to the interface?

For example, let's say I have created a REST API server which needs to handle a POST request. This POST request has a very large JSON body and I would like to use oboe.js to start processing the POST body immediately. In this case, there is no URL or path, you are given a buffer/stream from the HTTP module. Is it possible to give oboe.js this buffer/stream?

While CORS GET is working great I'm having troubles with CORS POST... When doing CORS POST, cookie was not being sent along with the headers and the request seemed to be stuck on the preflight part. Is CORS POST supported yet?

Hi there,

Are there any plans to make oboe stream the request as well as the response? Currently, oboe JSON.stringify'es the request body, which works for small requests. However, for large requests (~150MB), it causes the node process to hang.

See this stackoverflow question for more discussion: http://stackoverflow.com/questions/28288896/node-js-posting-a-large-request-using-request-module-hangs

It would be great if you could set cached: false when making an Oboe request so that it never pulls from the cache, always grabs the server's copy.

This is part of the $.ajax() spec and it's very useful, especially when benchmarking. At the moment in my benchmarks I can only run one test at a time, but with a cached: false property I can grab a number of samples and average them.

When loading json with relative path from file:// url, fail(undefined,0,"",undefined) will always be called at the end.

This is a case of many JSON objects concatenated. (Generated by a service on the internet, not much I can do about it). In my case I'm interested in knowing when the HTTP request is done (marks an end of a 'list'), but there appears to be no way to tell. There is a "start" at the beginning of the request, with "done" after each object in the stream, but nothing after the HTTP request has finished. Everything else seems to work well.

The only reason I'm using oboe is that it was the only thing I could find that would even parse the data to begin with...

I'm using oboe and appreciate the work that went into something so essential. I'm having a very good experience with it overall but I think I'm seeing behavior that should not be occurring. I could be wrong but I think I am using it properly.

Here's my oboe.js code:

oboe({
   url: "data/" + startTime + "/" + endtime,
   method: "GET",
   headers: {contentType: "application/json"},
   cache: false
  }).node('data.*', function(element) {
    // BLAH BLAH BLAH
    $("#status").text("retrieving: " + data.length + " records " + visible.length + " displayed");
  }).done(function(data){
    $("#status").text(data.length + " records retrieved " + visible.length + " displayed");
  });

The idea is that I want a different message to display after all the data is retrieved than when I am in the middle of the stream. The response from the restful service can be a bit jerky (data, pause, data, etc.) and I've had issues with both chrome and firefox "giving up" on really long responses part way through so I want to be able to tell when the stream is truly ended.

I thought the above would work but I the message will flip back and forth many times (dozens) before the end of the stream is reached. Am I misunderstanding what triggers the done event?

thanks,

-Matt

I'm advocating for XHR streaming since a long time and i'm pretty happy to see the lines moving on that subject.

But why staying stucked with JSON ? Most JSON APIs are ouputting JSON synchronously (JavaScript JSON.stringify, PHP JSON.encode etc...) in a manner that there will not have a real streaming chain from the server to the browser. By example, while using JSON, no easy way to output databases entries as they come from the DB server with many server side languages.

I'd suggest you to recommend using at least CSV wich is a well known and streamable format or the VarStream format i made for that purpose precisely ( https://github.com/nfroidure/VarStream ).

In Safari 4.0.5 and Firefox 3.6 I get:

TypeError: Result of expression 'regex.exec.bind' [undefined] is not a function.

Coming from function regexDescriptor(regex) in oboe-browser.js

If you can't fix this, can you at least make it fallback to using regular ajax? This is a show stopper for using oboe in production.

Forking and pulling oboe.js, I attempt an npm install and get the following:

my-MBP:oboe.js user$ npm install
npm ERR! Darwin 14.0.0
npm ERR! argv "node" "/usr/local/bin/npm" "install"
npm ERR! node v0.10.22
npm ERR! npm  v2.1.14
npm ERR! code EPEERINVALID

npm ERR! peerinvalid The package karma does not satisfy its siblings' peerDependencies requirements!
npm ERR! peerinvalid Peer [email protected] wants karma@~0.10.0
npm ERR! peerinvalid Peer [email protected] wants karma@>=0.9
npm ERR! peerinvalid Peer [email protected] wants karma@>=0.9
npm ERR! peerinvalid Peer [email protected] wants karma@>=0.9

npm ERR! Please include the following file with any support request:
npm ERR!     /Users/jahyoung/Projects/oboe.js/npm-debug.log

The problem here is that we're using older versions of libraries that have since been upgraded. I was able to update package.json to the latest versions, like so:

"karma": "^0.12.31",
"karma-firefox-launcher": "~0.1.0",
"karma-jasmine": "~0.3.4",
"karma-safari-launcher": "~0.1.1",

However, this breaks the existing tests as Jasmine has changed a lot of their conventions since the currently-used version.

Suggest updating package dependencies and updating the tests to use new Jasmine syntax.

If I'm processing an array of elements, and returning oboe.drop, then all of the nodes claim to be at index 0 in the path argument. It would be more useful if these were at the correct index in the source JSON.

I'm working with Oboe on my current project, and I'm noticing that the 204 responses from my API are triggering a fail event. I took a look at the code, but it looks pretty tricky:

I don't have the time to dig into it now, but I found a way around it by simply checking to see if the response code was a 204 in the start and fail callbacks.

Change oboe.doGet, oboe.doPost etc to oboe({url:'blah', method:'get'})

When working with Cross domain enabled services often X-Requested-With isn't added to the allowed headers and Chrome will throw an error.

I've removed the offending line but I don't know why it was added in the first place.

I'm streaming more than one JSON file simultaneously with Oboe, in Firefox 28.0. One of the streams contains information about a UI event, so for now I'm trying to pop up an alert() box when that event is seen in the stream. But adding the alert() produces an exception in oboe:
"Bad array\nLn: 1\nCol: 132\nChr: ["

Neither stream has a [ on column 132, confirmed with firefox's console.

When I add a console.log(c) at the beginning of clarinet.write() in oboe-browser.js, it looks like it's mixing the various streams up, but I could be wrong...

I'm not sure my interpretation of this problem is right, I'll dig in further is someone says that is impossible.

Full stack is
emitError@https://rawgithub.com/jimhigson/oboe.js/master/dist/oboe-browser.js:628
write@https://rawgithub.com/jimhigson/oboe.js/master/dist/oboe-browser.js:802
applyEach@https://rawgithub.com/jimhigson/oboe.js/master/dist/oboe-browser.js:497
singleEventPubSub/<.emit@https://rawgithub.com/jimhigson/oboe.js/master/dist/oboe-browser.js:2005
handleProgress@https://rawgithub.com/jimhigson/oboe.js/master/dist/oboe-browser.js:1219