feross / multistream

A stream that emits multiple other streams one after another (streams3)

multistream

A stream that emits multiple other streams one after another (streams3)

Simple, robust streams3 version of combined-stream. Allows you to combine multiple streams into a single stream. When the first stream ends, the next one starts, and so on, until all streams are consumed.

This module is used by WebTorrent, specifically create-torrent.

install

npm install multistream

usage

Use multistream like this:

var MultiStream = require('multistream')
var fs = require('fs')

var streams = [
  fs.createReadStream(__dirname + '/numbers/1.txt'),
  fs.createReadStream(__dirname + '/numbers/2.txt'),
  fs.createReadStream(__dirname + '/numbers/3.txt')
]

new MultiStream(streams).pipe(process.stdout) // => 123

You can also create an object-mode stream with MultiStream.obj(streams).

To lazily create the streams, wrap them in a function:

var streams = [
  fs.createReadStream(__dirname + '/numbers/1.txt'),
  function () { // will be executed when the stream is active
    return fs.createReadStream(__dirname + '/numbers/2.txt')
  },
  function () { // same
    return fs.createReadStream(__dirname + '/numbers/3.txt')
  }
]

new MultiStream(streams).pipe(process.stdout) // => 123

Alternatively, streams may be created by an asynchronous "factory" function:

var count = 0
function factory (cb) {
  if (count > 3) return cb(null, null)
  count++
  setTimeout(function () {
    cb(null, fs.createReadStream(__dirname + '/numbers/' + count + '.txt'))
  }, 100)
}

new MultiStream(factory).pipe(process.stdout) // => 123

contributors

• Feross Aboukhadijeh
• Mathias Buus
• Yuri Astrakhan

license

MIT. Copyright (c) Feross Aboukhadijeh.

multistream's Issues

Error ERR_STREAM_WRITE_AFTER_END on second run

I have this code:

  async run(unsortedCsvFilesList: string[]): Promise<string> {
    const csvFilesList = unsortedCsvFilesList.sort(naturalCompare);
    const jobId = getReqId();
    const jobUnzipTargetFilePath = `${this.defaultTransformerSourceFilePath}/${jobId}`;

    const Multistream = require('multistream');
    const fullFileNamePathsToStream = (filenamePaths: string[]) => {
      return new Multistream(filenamePaths.map(path => fs.createReadStream(path)));
    };

    const filesStream = fullFileNamePathsToStream(csvFilesList.map(fileName => `${jobUnzipTargetFilePath}/${fileName}`));

    await mkdirPromised(`${this.defaultTransformerDestinationFilePath}/${jobId}`, { recursive: true });

    const { getSchemaTransformer, stringifier } = require('transformers/src');

    try {

      getSchemaTransformer('mBiznesExCsv')
        .reduce((acc, transformer) => acc.pipe(transformer), filesStream)
        .pipe(stringifier)
        .pipe(
          fs.createWriteStream(`${this.defaultTransformerDestinationFilePath}/${jobId}/download.csv`),
          // await getDestinationStream(
          // `${this.defaultTransformerDestinationFilePath}/${jobId}/download.csv`)
        );
      // filesStream.destroy();
    } catch (error) {
      this.logger.error(error,'oh crap');
    }

    return jobId;
  }

It is the method stored in use case class inside tsoa NodeJS application with InversifyJS.
getSchemaTransformer returns transformer made with mississippi module. stringifier is csv-stringify module.

It works ok for the first time.

But every second run it crashes with

Error [ERR_STREAM_WRITE_AFTER_END]: write after end
    at writeAfterEnd (_stream_writable.js:248:12)
    at Parser.Writable.write (_stream_writable.js:296:5)
    at MultiStream.ondata (/home/user/workspace/bank-statement-csv-processor/node_modules/multistream/node_modules/readable-stream/lib/_stream_readable.js:681:20)
    at MultiStream.emit (events.js:198:13)
    at addChunk (/home/user/workspace/bank-statement-csv-processor/node_modules/multistream/node_modules/readable-stream/lib/_stream_readable.js:298:12)
    at readableAddChunk (/home/user/workspace/bank-statement-csv-processor/node_modules/multistream/node_modules/readable-stream/lib/_stream_readable.js:280:11)
    at MultiStream.Readable.push (/home/user/workspace/bank-statement-csv-processor/node_modules/multistream/node_modules/readable-stream/lib/_stream_readable.js:241:10)
    at MultiStream._forward (/home/user/workspace/bank-statement-csv-processor/node_modules/multistream/index.js:56:28)
    at ReadStream.onReadable (/home/user/workspace/bank-statement-csv-processor/node_modules/multistream/index.js:108:12)
    at ReadStream.emit (events.js:198:13)
    at emitReadable_ (_stream_readable.js:554:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
error Command failed with exit code 1.      

I tried to pipe the output to process.stdout, I tried to require('multistream') inside the method body.

I don't know how to debug this issue. More, the try{}catch clause does not catch this error - I don't know why.

Please help

An in-range update of concat-stream is breaking the build 🚨

☝️ Greenkeeper’s updated Terms of Service will come into effect on April 6th, 2018.

Version 1.6.1 of concat-stream was just published.

Branch	Build failing 🚨
Dependency	concat-stream
Current Version	1.6.0
Type	devDependency

This version is covered by your current version range and after updating it in your project the build failed.

concat-stream is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details

• ❌ continuous-integration/travis-ci/push The Travis CI build failed Details

---

Commits

The new version differs by 1 commits.

• bb2b8d0 1.6.1

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

---

Your Greenkeeper Bot 🌴

stream errors are not handled correctly on streams other than the first

When multiple streams are passed in directly, and a stream other than the first one immediately has an error (file read permission, perhaps), that errors throws instead of being handled. Here's an example:

multistream([
  fs.createReadStream('file/that/works/fine'),
  fs.createReadStream('file/with/no/read/permission')
]);

This will throw the read permission error. I can change the code to this to make it work:

multistream([
  fs.createReadStream('file/that/works/fine'),
  function() { return fs.createReadStream('file/with/no/read/permission'); }
]);

However, it feels like I shouldn't have to do that. Not a big deal, since it's better not to open all of the files at the same time anyway, but still, the docs suggest that it should work just fine and it probably needs to.

file without "new line" at the end

Hi!

When I use multistream for file 1 and file 2 and consume the stream line by line I see this:

aaa
bbb
cccxxx
yyy
zzz

file 1

aaa
bbb
ccc

file2

xxx
yyy
zzz

Is this the correct behaviour?

I would prefer that a new line is added at the end of the files that don't have it. :)

multistream not consuming entire stream due to race condition. reproduction included.

I'm running into an issue where multistream doesn't consume the entire stream. Not sure why. If you follow the instructions and run node multistream.js lots of times, you'll see intermittent content length mismatches.

I included node passthrough.js to provide an example that never fails. multistream.js and passthrough.js are nearly identical, only difference being piping to multistream vs piping to a passthrough stream.

https://www.diffchecker.com/bi5wMRCW

Reproduction steps

$ git clone https://github.com/guanzo/multistream-debug
$ cd multistream-debug
$ npm i 

# Run 20+ times, no failures
$ node passthrough.js

# Run 20+ times, intermittent failures
$ node multistream.js

here's what i'm seeing

Edit:
I found this bug after seeing webtorrent make never ending requests due to an invalid torrentFile. create-torrent uses multistream, and whenever this bug occurs, an invalid torrentFile is returned. Webseed tries to verify a valid chunk against an invalid torrentFile's chunk hash. The verification fails, so webtorrent retries endlessly.

Edit:
$ node multistream.js works with [email protected]. This commit seems to be the problem.

0813df4

Removing && this._drained fixes the bug. Thoughts?

Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close

Hey,

I seem to repeatedly get Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close when connecting 100-1000 fs streams and piping over a tcp pipe.

Does anyone know what can cause this?

Feature request: append

CombinedStream has the following:

var combinedStream = CombinedStream.create();
combinedStream.append(function(next) {
  next(fs.createReadStream('file1.txt'));
});
combinedStream.append(function(next) {
  next(fs.createReadStream('file2.txt'));
});

It would be useful to support append in multistream as well.

TypeScript types

Hi,

I'm using this project from within TypeScript and created types for it.
Would you be interested in a PR which put the types into the repo?
All it would require is a single index.d.ts in the root of the module containing the types below.
You can see how redux does it here

Thanks

import { Readable, ReadableOptions } from 'stream';

type StreamGenerator = () => Readable;
type AsyncStreamFactory = (callback: AsyncResultCallback<Readable, Error>) => void;
type Streams = (Readable | StreamGenerator)[] | AsyncStreamFactory;

interface MultiStream {
  new (streams: Streams, opts?: ReadableOptions): Readable;
  (streams: Streams, opts?: ReadableOptions): Readable;
  obj(streams: Streams): Readable;
}
declare const _MultiStream: MultiStream;
export = _MultiStream;

v3 migration guide? changelog?

There seems to be no changelog and no migration guide for v2 -> v3.
What are the breaking changes?

Can this package be used as a conductor for scheduling online streams?

Hi
I need to create a conductor to schedule and prioritize multiple video streams at the same time, so that streamer 1 and streamer 2 stream simultaneously. Only it should be displayed on the platform, even if streamer number two is streaming, but when streamer number one goes offline, streamer number two will start playing automatically.
Can this package help me? Please let me know if you know a tool to create this process

trying to pipe multiple streams to client player, only first stream plays

I have a gridfs bucket of video files, using the following server function, I can easily play any single video on my player of choice (react-player)

const fetchVideoStream = async (req, res) => {
    try {
      gfs.find({ _id: mongoose.Types.ObjectId(req.params.id) }).toArray((err, files) => {
        if (!files[0] || files.length === 0) {

          return res.status(200).json({
            success: false,
            message: "No files available",
          });
        }
        if (files[0].contentType === 'video/mkv' || files[0].contentType === 'video/webm') {
          // render video to browser
          gfs.openDownloadStream(mongoose.Types.ObjectId(req.params.id)).pipe(res);
        } else {
          res.status(404).json({
            err: "Not an image",
          });
        }
      });
    }
    catch (error) {
      res.status(400).json({
        err: error,
        success: false
      })
    }
  };

what I'm trying to do, is to play multiple files one after the other in the following function:

  const fetchMultiStream = async (req, res) => {
    try {
      let streams = [];
      const recordingId = req.params.id;
      let container;
      let ids;
      try {
        container = await Recordings.findById(recordingId); //The file ids are stored in a container for just such reference needs
        ids = container.recording.map((record) => mongoose.Types.ObjectId(record.id));
      } catch (error) {
        return res.status(500).json({
          message:
            "getting file Ids failed, please make sure you have the correct recordingsId.",
        });
      }
      try {
        gfs.find({ '_id': { $in: ids } }).toArray((err, files) => {
          if (!files[0] || files.length === 0) {
            return res.status(200).json({
              success: false,
              message: "No files available",
            });
          }
          files.forEach(file => {
            if (file && (file.contentType === 'video/mkv' || file.contentType === 'video/webm')) {
             let stream =  gfs.openDownloadStream(file._id)
             streams.push(stream);
            }
          })
          new MultiStream(streams).pipe(res);
        });
      } catch (error) {
        console.log(error)
      } 
    } catch (error) {
      console.log(error)
    }
  }

However, when I try to stream a video using this function only the first video seams to playback, and then there is some odd audio scrunching, like code is trying to shove all the remaining data left to play at the end of the first playback

why the initial self.next() ?

I'm creating a class that extends MultiStream with a factory

class F extends MultiStream {
  constructor() {
    // Factory
    super(cb => {
      this.doSomething(cb)
    })
  }

  doSomething(cb) {
    // ...
  }
}

const f = new F()

But this throws an error:

Must call super constructor in derived class before accessing 'this' or returning from derived constructor

I'm wondering why you try to do work when there is nothing that is consuming the stream.
Shouldn't the factory be called when it actually need to? Meaning after p gets piped to something or calling p.read()

does multistream have an on 'end' event?

eg:

var write_stream = fs.createWriteStream(`${relPath}.mp3`);

var streams = [];

array_of_readable_audio_streams.forEach((audio_stream) => {
streams.push(audio_stream);
});

new MultiStream(streams).pipe(write_stream);

// when finished piping to write_stream, do something here...

Sending multiple files in single response

I am using Express router to process and respond to requests from browsers. I tried using multistream to send multiple files in a single response, but I get only one file in the client. Do you have any code samples for achieving the same?

Work with Video MP4?

I wonder if it works with streaming video is MP4?

Thanks!

An in-range update of readable-stream is breaking the build 🚨

☝️ Greenkeeper’s updated Terms of Service will come into effect on April 6th, 2018.

Version 2.3.5 of readable-stream was just published.

Branch	Build failing 🚨
Dependency	readable-stream
Current Version	2.3.4
Type	dependency

This version is covered by your current version range and after updating it in your project the build failed.

readable-stream is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Status Details

• ❌ continuous-integration/travis-ci/push The Travis CI build failed Details

---

Release Notes

v2.3.5

• Fixed lolodex integration #325 #324 #323

Commits

The new version differs by 4 commits ahead by 4, behind by 1.

• a47240c Bumped v2.3.5
• 2f933db Fixed broken lolex integration.
• fbb0d25 Support for the Foundation ci.
• 862ab4f Bumped v2.3.4

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

---

Your Greenkeeper Bot 🌴

TypeScript Error: 'new' expression, whose target lacks a construct signature, implicitly has an 'any' type.

I am getting this type error, the @type is pointing to 2.1, is it an outdated package or am I doing something wrong?

import MultiStream from 'multistream'
....
return new MultiStream([stdout, stderr])

'new' expression, whose target lacks a construct signature, implicitly has an 'any' type.