web-std / io

Web standard IO APIs implemented for nodejs

Web Standard IO

Set of libraries that provide standard web IO APIs so they can be imported and used across web and node runtimes without any configurations.

io's Issues

Improve headers types

Headers supports an array of values new Headers({ "foo": ["a", "b"] }) but this isn't reflected in the types.
It results in { "foo": "a,b" }.

Would you accept a PR?

Rollup bundle error trying to import default from `web-streams-polyfill`

Using rollup to bundle a module that depends on @web-std/fetch I'm hitting an error:

@web-std/blob (imported by src/platform.js, node_modules/@web-std/fetch/src/index.js, node_modules/@web-std/fetch/src/body.js)
[!] Error: 'default' is not exported by node_modules/web-streams-polyfill/dist/polyfill.mjs, imported by node_modules/@web-std/fetch/src/body.js
https://rollupjs.org/guide/en/#error-name-is-not-exported-by-module
node_modules/@web-std/fetch/src/body.js (12:7)
10: 
11: import {Blob} from '@web-std/blob';
12: import WebStreams from 'web-streams-polyfill';

Looking at

io/fetch/src/index.js

Lines 21 to 26 in 68e89a8

	import WebStreams from 'web-streams-polyfill';
	import {pipeline as pump, PassThrough} from 'stream';
	import * as Stream from 'stream';
	import {Blob} from '@web-std/blob';
	const {ReadableStream} = WebStreams;

and https://github.com/MattiasBuelens/web-streams-polyfill/blob/956de3d56a1c4565563f85f9180d4bac39571b74/src/polyfill.ts#L19-L38

it looks like there is no default export from web-streams-polyfill

Request.formData() doesn't support application/x-www-form-urlencoded

Per the spec the Request.formData should support parsing body of Content-Type: application/x-www-form-urlencoded, but it currently only supports multipart/form-data. I have an update locally and can open a PR shortly

fetch seems to error on 301 response

fetch(url, { method: 'POST', headers })
Promise { <pending> }
> f(node:66034) UnhandledPromiseRejectionWarning: FetchError: request to ... failed, reason: connect ENETUNREACH ...:443
    at ClientRequest.<anonymous> (file:///Users/gozala/Projects/nft.storage/node_modules/@web-std/fetch/src/index.js:100:11)
    at ClientRequest.emit (events.js:315:20)
    at ClientRequest.EventEmitter.emit (domain.js:467:12)
    at TLSSocket.socketErrorListener (_http_client.js:469:9)
    at TLSSocket.emit (events.js:315:20)
    at TLSSocket.EventEmitter.emit (domain.js:467:12)
    at emitErrorNT (internal/streams/destroy.js:106:8)
    at emitErrorCloseNT (internal/streams/destroy.js:74:3)
    at processTicksAndRejections (internal/process/task_queues.js:80:21)
(node:66034) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 3)

Promise {
  <rejected> FetchError: request to ... failed, reason: connect ENETUNREACH ....:443
      at ClientRequest.<anonymous> (file:///Users/gozala/Projects/nft.storage/node_modules/@web-std/fetch/src/index.js:100:11)
      at ClientRequest.emit (events.js:315:20)
      at ClientRequest.EventEmitter.emit (domain.js:467:12)
      at TLSSocket.socketErrorListener (_http_client.js:469:9)
      at TLSSocket.emit (events.js:315:20)
      at TLSSocket.EventEmitter.emit (domain.js:467:12)
      at emitErrorNT (internal/streams/destroy.js:106:8)
      at emitErrorCloseNT (internal/streams/destroy.js:74:3)
      at processTicksAndRejections (internal/process/task_queues.js:80:21) {
    type: 'system',
    errno: 'ENETUNREACH',
    code: 'ENETUNREACH',
    erroredSysCall: 'connect'
  }
}

Instead request should follow a redirect.

Converging with remix-run/web-std-io

We got ton of great contributions from @MichaelDeBoey backported from https://github.com/remix-run/web-std-io. I assume fork was created so remix would not be blocked on me.

I think it would be wonderful if remix team felt empowered to drive changes without having to maintain the fork, I would love to support this. I have invited @MichaelDeBoey to the @jacob-ebey contributors team that have maintainers role.

Also starting this thread to discuss whether converging would make sense and if so what should we do to make it happen. For what it's worth my hope was that it would be a community project so getting more teams driving it it more than welcome.

`fetch` is at `2.0.1` but package.json in repo is `2.0.0`

@web-std/fetch is at 2.0.1 on the registry, but the repo package.json is at 2.0.0

io/fetch/package.json

Line 3 in 68e89a8

"version": "2.0.0",

fetch blob update

fetch-blob
The reason this library exists is because fetch-blob chooses to compromise Web API compatibility of blob.stream() by using nodejs native Readable stream. We found this to be problematic when sharing code across nodejs and browser runtimes. Instead this library stays true to the specification by using ReadableStream implementation from web-streams-polyfill library even if that is less convenient in nodejs context.

This is no longer true as fetch-blob@3 uses web stream as well now...

Support file:// url in fetch APIs

I often find myself needing to load assets e.g. in tests in a platform agnostic way. Today this is tricky because in node you need to perform FS read and in browser you have fetch.

It should be fairly trivial to make support file URLs in node making fetch a universal API for loading assets.

web-std/fetch is regressed in cjs environments

Change #35 introduced a regression by leaving out dist/index.cjs from publishing to npm as reported in nftstorage/nft.storage#1166

Form data util does not use @web-std/file

io/packages/fetch/src/utils/form-data.js

Line 107 in c88170b

form.append(name, new File([data], filename, { type: contentType }))

The File instance being constructed here is not imported from @web-std/file causing it to fail in node with

ReferenceError: File is not defined

Export native `fetch` from `@web-std/fetch`

As per #51 it would be help to use fetch for loading assets in tests across node/browser. Which in turn would be straight forward if we @web-std/fetch could be used in test without having to use alternative code paths across node and web.

Release @web-std/streams v1.0.2

Looks like CI or something failed and 1.0.2 of streams was not released:

Should be able to pass a URL to fetch

The current typs require us to pass a string to @web-std/fetch as the resource parameter. In the browser you can pass a URL instance as the resource, as it defines a stringifier attribute for it's href property https://url.spec.whatwg.org/#idl-index

The typings here are leading us to write fetch(urlInstance.toString()) which is unappealing.

io/fetch/@types/index.d.ts

Line 119 in 68e89a8

type RequestInfo = string | Body;

Please release 3.0.1

Seems that the release yesterday did not make it's way to NPM.

Usage of experimental APIs

Both the blob and the stream package use the not yet stable Blob and Web Streams API from Node. This leads to warnings when used with newer node versions as found in remix-run/remix#1141

It seems unsafe to me to use these APIs yet so I would opt for removing them until they become stable.

Error: export 'default' (imported as 'streams') was not found in 'web-streams-polyfill' with Webpack 5.42.0

Hello @Gozala, thanks for your answer. I've recreated the issue, since web-file-polyfill project has been archived.

Original issue

Problem
I have Angular Universal project and I use this package to create File and Blob polyfills on server side. After updating to Angular v12 I faced some problem with building app.

My server.ts file:

import { createWindow } from 'domino';
import { File, Blob } from 'web-file-polyfill';

const win = createWindow(template);

global.window = win as Window & typeof globalThis;
global.document = win.document;

global.localStorage = localStorage;
global.File = File;
global.Blob = Blob;

When I try to build project, an error throws:

./node_modules/@web-std/blob/src/package.js:4:51-58 - Error: export 'default' (imported as 'streams') was not found in 'web-streams-polyfill' (possible exports: ByteLengthQueuingStrategy, CountQueuingStrategy, ReadableByteStreamController, ReadableStream, ReadableStreamBYOBReader, ReadableStreamBYOBRequest, ReadableStreamDefaultController, ReadableStreamDefaultReader, TransformStream, TransformStreamDefaultController, WritableStream, WritableStreamDefaultController, WritableStreamDefaultWriter)

If I change import in the @web-std/blob/src/package.js:

import * as streams from "web-streams-polyfill"

It works well.

Steps to reproduce:

1. Clone this project
2. Run npm i and npm run dev:ssr

File in which I use @web-std/file

'require() of ES not supported Module' in 2.1.0

I have @web-std/fetch as a sub-dependency. Starting yesterday I started having this issue during build. Downgrading to @web-std/[email protected] resolves this.

[path]/node_modules/ts-node/dist/index.js:729
            return old(m, filename);
                   ^
Error [ERR_REQUIRE_ESM]: require() of ES Module [path]/node_modules/@ssttevee/multipart-parser/lib/index.js from [path]/node_modules/@web-std/fetch/dist/index.cjs not supported.
Instead change the require of index.js in [path]/node_modules/@web-std/fetch/dist/index.cjs to a dynamic import() which is available in all CommonJS modules.
    at Object.require.extensions.<computed> [as .js] ([path]/node_modules/ts-node/dist/index.js:729:20)
    at Object.<anonymous> ([path]/node_modules/@web-std/fetch/dist/index.cjs:14:25)
    at Object.require.extensions.<computed> [as .js] ([path]/node_modules/ts-node/dist/index.js:729:20)
    at Object.<anonymous> ([path]/node_modules/nft.storage/dist/src/token.cjs:5:1)
    at Object.require.extensions.<computed> [as .js] ([path]/node_modules/ts-node/dist/index.js:729:20)
    at Object.<anonymous> ([path]/node_modules/nft.storage/dist/src/lib.cjs:8:13)
    at Object.require.extensions.<computed> [as .js] ([path]/node_modules/ts-node/dist/index.js:729:20)
    at Object.<anonymous> ([path]/src/services/ipfs.ts:28:23)
    at Module.m._compile ([path]/node_modules/ts-node/dist/index.js:735:29)
    at Object.require.extensions.<computed> [as .ts] ([path]/node_modules/ts-node/dist/index.js:737:16)
...