mbostock / file-source

Read files as standard WhatWG streams in Node.

file-source

A sliceable, readable stream reader implementation on top of a Node file read stream. This library allows you to write code that takes a source as input, and can work with either native readable streams or Node streams. For example:

var file = require("file-source");

file("README.md")
  .then(function read(source) {
    return source.slice(40).then(value => {
      if (value == null) return;
      process.stdout.write(value);
      return read(source);
    });
  })
  .catch(error => console.error(error.stack));

API Reference

# file(path[, options]) <>

Returns a Promise that yields a source for the file at the specified path when the underlying file is open. The following options are allowed:

• highWaterMark - the stream’s internal buffer size; defaults to 65,536

# source.slice(length) <>

Returns a Promise for the next chunk of data from the underlying stream, yielding a Uint8Array (a Buffer) of length bytes, or the remaining bytes of the underlying stream if the underlying stream has more than zero but fewer than length bytes remaining, or null when no bytes remain in the stream.

# source.read() <>

Returns a Promise for the next chunk of data from the underlying stream. The yielded result is an object with the following properties:

• value - a Uint8Array (a Buffer), or undefined if the stream ended
• done - a boolean which is true if the stream ended

# source.cancel() <>

Returns a Promise which is resolved when the underlying stream has been destroyed.

file-source's Issues

Cross-platform API?

This API is currently Node-specific since the source.read interface returns a Buffer. If we changed it to return an ArrayBuffer (or DataView) instead, then it could be used in both Node and the browser.

Unfortunately, Node doesn’t support TextDecoder yet, which is part of the Encoding Living Standard. But then, neither do Safari or IE. 😦 I’m not currently using the source.readString feature—I’m using iconv-lite for dbf parsing—so one option might be to drop this method from the source interface. Though that would simply punt the work to shapefile if the goal is to get that working in the browser.

Expose options?

Should perhaps allow options = {highWaterMark: 65536} to file(path[, options]). The other options don’t seem applicable.

Require node engine versions that have `Promise` natively?

I was trying to run shp2json from your shapefile package and I ran into an error that Promise is not defined in file-source:

/usr/local/lib/node_modules/shapefile/node_modules/path-source/node_modules/file-source/index.js:7
  return new Promise(function(resolve, reject) {
             ^
ReferenceError: Promise is not defined

According to this stackoverflow answer it looks like Node didn't add Promise until version 0.12, and I'm stuck on an earlier version because it's the version we use to build our package:

$ node --version
v0.10.25

Would it be possible to update the package.json to specify the engines supported, so it'll fail at install time? It looks like this is the way to do that.

Thanks,
Ryan