rlugojr / streamsaver.js

StreamSaver writes stream to the filesystem directly asynchronous

StreamSaver.js

First I want to thank Eli Grey for a fantastic work implementing the FileSaver.js to save files & blobs so easily! But there is one obstacle - The RAM it can hold and the max blob size limitation

StreamSaver.js takes a different approach. Instead of saving data in client-side storage or in memory you could now actually create a writable stream directly to the file system (I'm not talking about chromes sandboxed file system)

StreamSaver.js is the solution to saving streams on the client-side. It is perfect for webapps that need to save really large amounts of data created on the client-side, where the RAM is really limited, like on mobile devices.

Supported browsers

Browser	Supported	Missing
Opera 39+	Yes
Chrome 52+	Yes
Firefox	No	Streams
Safari TP	No	Streams, Fetch, SW
Edge	No	Streams, SW
IE	No	Everything (IE is dead)

Aha moments

• Chrome don't show that the file is being download and won't give you a dialog to choose where to save it until you have written at least 1024 bytes or so (think headers are included)... Or until you close the stream
  But that only applies when you have the "ask where to save each time" turned on in your browser settings
• Chrome was capable of writing more than 15 GB of data without any memory issues

Getting started

It's important to test browser support before you include the web stream polyfill
because the serviceWorker needs to respondWith a native version of the ReadableStream

<script src="StreamSaver.js"></script> <!-- load before streams polyfill to detect support -->
<script src="https://cdn.rawgit.com/creatorrr/web-streams-polyfill/master/dist/polyfill.min.js"></script>
<script>
	// it also support commonJs and amd
	import { createWriteStream, supported, version } from 'StreamSaver'
	const { createWriteStream, supported, version } = require('StreamSaver')
	const { createWriteStream, supported, version } = window.streamSaver
	alert( supported )
</script>

Syntax

// If you know what the size is going to be then you can specify
// that as 2nd arguments and it will use that as Content-Length header
const fileStream = streamSaver.createWriteStream('filename.txt', size)
const writer = fileStream.getWriter()
// WriteStream is a whatwg standard writable stream
// https://streams.spec.whatwg.org/

// and the write fn only accepts uint8array
writer.write(uint8array)
// when you are done: you close it
writer.close()
// when you want to cancel the download: you abort
writer.abort(reason) // ATM Canary only recognize if the stream has been errored

// it's also possible to pipe a readableStream stream to the fileStream
// but then you shouldn't call .getWriter() or .close()
readableStream.pipeTo(fileStream)

That is pretty much all StreamSaver.js dose :)

Examples

Writing some plain text

const fileStream = streamSaver.createWriteStream('filename.txt')
const writer = fileStream.getWriter()
const encoder = new TextEncoder
let data = 'a'.repeat(1024)
let uint8array = encoder.encode(data + "\n\n")

writer.write(uint8array)
writer.close()

Read blob as a stream and pipe it (see: Screw FileReader)

require('screw-filereader')
const fileStream = streamSaver.createWriteStream('filename.txt')
const blob = new Blob([ 'a'.repeat(1E9*5) ]) // 1*5 MB

blob.stream().pipeTo(fileStream)

Save a media stream

get_user_media_stream_somehow().then(mediaStream => {
	let fr = new FileReader
	let mediaRecorder = new MediaRecorder(mediaStream)
	let chunks = Promise.resolve()
	let fileStream = streamSaver.createWriteStream('filename.mp4')
	let writer = fileStream.getWriter()
	// use .mp4 for video(camera & screen) and .wav for audio(microphone)

	// Start recording
	mediaRecorder.start()

	closeBtn.onclick = event => {
		mediaRecorder.stop()
		setTimeout(() =>
			chunks.then(evt => writer.close())
		, 1000)
	}

	mediaRecorder.ondataavailable = ({blob}) => {
		chunks = chunks.then(() => new Promise(resolve => {
			fr.onload = () => {
				writer.write(new Uint8Array(fr.result))
				resolve()
			}
			fr.readAsArrayBuffer(blob)
		}))
	}

})

Get a "stream" from ajax

res.body is a readableByteStream, but don't have pipeTo yet
So we have to use the reader instead which is the underlying method in streams

fetch(url).then(res => {
	const fileStream = streamSaver.createWriteStream('filename.txt')
	const writer = fileStream.getWriter()
	// Later you will be able to just simply do
	// res.body.pipeTo(fileStream)

	const reader = res.body.getReader()
	const pump = () => reader.read()
		.then(({ value, done }) => done
			// close the stream so we stop writing
			? writer.close()
			// Write one chunk, then get the next one
			: writer.write(value).then(pump)
		)

	// Start the reader
	pump().then(() =>
		console.log('Closed the stream, Done writing')
	)
})

Get a node-stream from webtorrent

Note it still keeps the data in memory. A more correct way to do this would be to use some kind of Custom chunk store (must follow abstract-chunk-store API)

const client = new WebTorrent()
const torrentId = 'magnet:?xt=urn:btih:6a9759bffd5c0af65319979fb7832189f4f3c35d&dn=sintel.mp4&tr=wss%3A%2F%2Ftracker.btorrent.xyz&tr=wss%3A%2F%2Ftracker.fastcast.nz&tr=wss%3A%2F%2Ftracker.openwebtorrent.com&tr=wss%3A%2F%2Ftracker.webtorrent.io&ws=https%3A%2F%2Fwebtorrent.io%2Ftorrents%2Fsintel-1024-surround.mp4'
// Sintel, a free, Creative Commons movie

client.add(torrentId, torrent => {
	// Download the first file

	const file = torrent.files[0]
	let fileStream = streamSaver.createWriteStream(file.name, file.size)
	let writer = fileStream.getWriter()

	// Unfortunately we have two different stream protocol so we can't pipe.
	file.createReadStream()
		.on('data', data => writer.write(data))
		.on('end', () => writer.close())
})

How is this possible?

There is not any magical saveAs() function that saves a stream, file or blob. The way we mostly save Blobs/Files today is with the help of a[download] attribute FileSaver.js takes advantage of this and create a convenient saveAs(blob, filename) function, very fantastic, but you can't create a objectUrl from a stream and attach it to a link...

link = document.createElement('a')
link.href = URL.createObjectURL(stream) // DOES NOT WORK
link.download = 'filename'
link.click() // Save

So the one and only other solution is to do what the server does: Send a stream with Content-Disposition header to tell the browser to save the file. But we don't have a server! So the only solution is to create a service worker that can intercept links and use respondWith() This will scream high restriction just by mentioning service worker. It's such a powerful tool that it need to run on https but there is a workaround for http sites: popups + 3rd party https site. Who would have guess that? But I won't go into details on how that works. (The idea is to use a middle man to send a dataChannel from http to a serviceWorker that runs on https).

So it all boils down to using serviceWorker, MessageChannel, postMessage, fetch, respondWith, iframes, popups (for http -> https -> serviceWorker), Response and also WritableStream for convenience and backpressure

Test locally

# A simple php or python server is enough
php -S localhost:3001
python -m SimpleHTTPServer 3001
# then open localhost:3001/example.html

Consensus

Go ahead and vote for how important this feature is

• serviceWorker MS Edge status: In Development
• streams Firefox Status: ASSIGNED