denoland / eszip

A compact file format to losslessly serialize an ECMAScript module graph into a single file

eszip

The eszip format lets you losslessly serialize an ECMAScript module graph (represented by deno_graph::ModuleGraph) into a single compact file.

The eszip file format is designed to be compact and streaming capable. This allows for efficient loading of large ECMAScript module graphs.

https://eszip-viewer.deno.dev/ is a tool for inspecting eszip files.

Examples

Creating an eszip

cargo run --example eszip_builder https://deno.land/std/http/file_server.ts file_server.eszip2

Viewing the contents of an eszip

cargo run --example eszip_viewer file_server.eszip2

Loading the eszip into V8

cargo run --example eszip_load file_server.eszip2 https://deno.land/std/http/file_server.ts

File format

The file format looks as follows:

Eszip:
| Magic (8) | Header size (4) | Header (n) | Header hash (32) | Sources size (4) | Sources (n) | SourceMaps size (4) | SourceMaps (n) |

Header:
( | Specifier size (4) | Specifier (n) | Entry type (1) | Entry (n) | )*

Entry (redirect):
| Specifier size (4) | Specifier (n) |

Entry (module):
| Source offset (4) | Source size (4) | SourceMap offset (4) | SourceMap size (4) | Module type (1) |

Sources:
( | Source (n) | Hash (32) | )*

SourceMaps:
( | SourceMap (n) | Hash (32) | )*

There is one optimization for empty source / source map entries. If both the offset and size are set to 0, no entry and no hash is present in the data sections for that module.

Development

When opening a PR make sure to rebuild Wasm by running:

deno task build

eszip's Issues

Q: How to load modules from eszip files?

Similar to #15, I'm also looking for more information about actual usage of this library, especially on how to load modules from generated eszip files. My understanding is that the JS code inside lib/ only provides the primitives to a) turn ES modules into the eszip container format and b) parse generated eszip files. What I found a bit confusing is that b) seems to be limited to reading a container's content without being able to access modules themselves.

As far as I know, the only way to actually use/load modules from a eszip file is to write a custom loader using Rust, which is what deno compile does. Please correct me if I'm wrong.

The main reason why I'm asking: Netlify seems to use eszip to bundle code in different stages, which makes me think that they must be using some custom Deno loader code behind the scenes. Without it, there seems to be no way to make use of the generated stage files other than writing my own loader in Rust, right? 🤔

Module.specifier should be a `ModuleSpecifier`?

Can specifier be a ModuleSpecifier?

pub struct Module {
  pub specifier: String,
  pub kind: ModuleKind,
  inner: ModuleInner,
}

let module = Module {
            specifier: specifier.to_string(),

`eszip extract` fails when a specifier in the eszip has no path or is a directory of another specifier

The code here:

eszip/lib/eszip.ts

Line 89 in c9e9144

await write(join(dest, "source", url2path(specifier)), module);

...is very basic and looks at the pathname only to figure out how to extract.

This will fail in two scenarios:

1. Where there is no path, like https://fresh.deno.dev
2. A path is a directory of another specifier. For exmaple: something/ then something/other.ts

Support for npm specifiers

Since Deno 1.28 supports npm specifiers, ESZIPs should be able to resolve npm packages and add them to the generated module graph.

Consider using a better `Accept` header when fetching modules

Currently the following header is sent when fetching modules:

Accept: */*

Which makes it very difficult for a server to do content negotiation. Either a header of:

Accept: application/javascript, application/typescript, text/javascript, text/typescript, text/jsx, text/tsx, application/json;q=0.9, text/plain;q=0.8, application/octet-stream;q=0.8

Or:

Accept: application/javascript, application/typescript, text/javascript, text/typescript, text/jsx, text/tsx, application/json;q=0.9, text/plain;q=0.8, application/octet-stream;q=0.8, */*;q=0.7

Would allow better negotiation of content for a web server.

Ref: denoland/deno#13287

bug: redirects leaves dangling/invalid specifiers

The root cause of denoland/deno#13704

mod.ts -> a.js -> https://deno.land/std/testing/asserts.ts

https://deno.land/std/testing/asserts.ts redirects to https://deno.land/[email protected]/testing/asserts.ts
but its dependencies do not redirect in the graph:
https://deno.land/[email protected]/testing/asserts.ts -> https://deno.land/std/fmt/colors.ts.

This breaks loading the module since std/fmt/colors.ts (unversioned) does not exist on the graph.

It should be https://deno.land/[email protected]/fmt/colors.ts instead.

repro

// mod.ts
import "./a.js";

// a.js
import {
  assertNotEquals,
  assertStrictEquals,
} from 'https://deno.land/std/testing/asserts.ts';

Graph:

~> deno run -A lib/eszip.ts list src/testdata/issue13704.eszip

Loading eszip v2
file:///Users/divy/gh/eszip_pain/mod.ts
file:///Users/divy/gh/eszip_pain/a.js
https://deno.land/[email protected]/testing/asserts.ts
https://deno.land/[email protected]/fmt/colors.ts
https://deno.land/std/testing/asserts.ts
https://deno.land/[email protected]/testing/_diff.ts

Add docs for using CLI

JS API is not idiomatic or documented

It seems like the only way to know how to even use the JS API is to dig into the Rust code. The generated documentation at https://deno.land/x/[email protected]/mod.ts is completely useless.

I think the following changes should be made:

• Make Parser wrap InternalParser, rather than extend it.
• Since all of the methods of (Internal)Parser return promises, make the Parser constructor synchronous and initialize the wasm module on the first method call. We can still keep createInstance for backward compatibility.
• Can we make Parser::parse take a ReadableStream<Uint8Array> and have it acquire a BYOB reader, rather than take the BYOB reader? Or maybe have the JS-side parse method wrapper take a ReadableStream<Uint8Array> | ReadableStreamByobReader.
• Making Parser a wrapper class would take care of having the methods and their argument/return types in the documentation, but there needs to be more documentation on top of that.

eszip 2.2

We need to be able to store the raw untranspiled bytes of TypeScript in eszip. For example:

import bytes from "./mod.ts" with { "type": "bytes" }; // raw file bytes
import * as mod from "./mod.ts" // js source stored in eszip

Luca's suggestion (something along these lines):

Entry (module with raw source):
| Source offset (4) | Source size (4) | Raw source offset (4) | Raw source size (4) | SourceMap offset (4) | SourceMap size (4) | Module type (1) |

Entry (module with identical raw source):
| Source offset (4) | Source size (4) | SourceMap offset (4) | SourceMap size (4) | Module type (1) |

Then we never emit Entry (module) anymore.

Edit: Nevermind. Need to rethink for this scenario:

import json from "https://deno.land/file.json" with { "type": "json" }; // json bytes
import data from "https://deno.land/file.json" with { "type": "bytes" }; // some ts file source
import mod from "https://deno.land/file.json"; // some ts file that gets transpiled

Fails with private repos, does not respect DENO_AUTH_TOKENS

Per the title, this will fail on files that are from private repositories. The files load without issue in Deno thanks to DENO_AUTH_TOKENS.

Looking into a bit it seems this tool handles no auth whatsoever in fetching https modules here

eszip/lib/loader.ts

Line 43 in 1086f52

const response = await fetch(String(url), { redirect: "follow" });

Specifically this is failing for me as a user of Supabase edge functions (which run on Deno Deploy). I can't deploy my functions because they're bundled using this tool and they have deps in private repositories.

eszip build fails with: Error: Blocked by null entry for ""https://deno.land/x/""

Hello!

When I attempt to build an eszip with the following importMap and a very simple module. I receive the error below.

mod.ts

import 'std/fs/mod.ts';
import 'https://deno.land/x/fresh/server.ts'

importMap.json

{
    "imports": {
        "std/": "https://deno.land/std/",
        "https://deno.land/x/": "./vendor/deno.land/x/"
    }
}

error: Error: Blocked by null entry for ""https://deno.land/x/""
      const ret = new Error(getStringFromWasm0(arg0, arg1));
                  ^
    at __wbg_new_651776e932b7e9c7 (file:///workspaces/eszip/lib/eszip_wasm.generated.js:313:19)
    at <anonymous> (wasm://wasm/00a90a36:1:78732)
    at <anonymous> (wasm://wasm/00a90a36:1:1463894)
    at <anonymous> (wasm://wasm/00a90a36:1:1957066)
    at __wbg_adapter_18 (file:///workspaces/eszip/lib/eszip_wasm.generated.js:144:6)
    at real (file:///workspaces/eszip/lib/eszip_wasm.generated.js:128:14)

export `fromGraph` to JavaScript module

There are situations where you want full control over the generation of a module graph, and it is useful to be able to use eszip from_graph directly.

Please expose `add_opaque_data` in the JS api

...plus equivalent getters in Parser

Thanks!

Reduce global commands necessary for contributing

Currently we require a few globally configured and installed commands:

• wasm-bindgen-cli
• wasm-opt (binaryen)
• wasmbuild
• make - doesn't work on Windows out of the box
• cargo
• deno

This requires a decent amount of developer time setup to build this repo and when the versions of the global commands change it might cause problems. For example, wasm-bindgen-cli gives a very cryptic error when building with the wrong globally installed version (we even have this mentioned in a troubleshooting section, which could be removed if we didn't have this as a global command https://github.com/denoland/eszip/tree/d9d77ec2bf0db0e87a93ba1a759affd8cc070997#troubleshooting).

We should reduce this list to:

• cargo
• deno

This could be achieved for example by:

1. wasmbuild - Don't use deno install and instead use deno run in the makefile. Done
2. wasm-bindgen-cli - denoland/wasmbuild#6
3. wasm-opt - denoland/wasmbuild#7
4. make - Move away once we have deno script/deno task or switch to a deno script file with a shebang (not a big deal for windows users because they can read the shebang and execute the script)

Invalid eszip is generated if an import map is imported from a program

Files

main.js

import m from "./import_map.json" assert { type: "json" };

import_map.json

{
  "imports": {
  }
}

Reproduction

Given the two files above, run the following command:

$ cargo run --example eszip_builder file://$(pwd)/main.js out.eszip file://$(pwd)/import_map.json
source: file:///Users/yusuktan/Repo/github.com/magurotuna/eszip/import_map.json
source: file:///Users/yusuktan/Repo/github.com/magurotuna/eszip/main.js
source: file:///Users/yusuktan/Repo/github.com/magurotuna/eszip/import_map.json

This succeeds, but the log indicates that import_map.json is contained in the output twice.

After that, if we run the following command to take a peek at the eszip, it panics.

$ cargo run --example eszip_viewer out.eszip
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: InvalidV2SourceOffset(0)', src/examples/viewer.rs:40:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

How do you use this?

How do you use the zip file with Deno? I don't see that mentioned in the repo page.

Dynamic import support

It seems that modules specified by dynamic import are not included in the eszip target.

Add back examples

In #63 I removed the examples because of the circular dependency between eszip and deno_core. We probably want to add some sort of examples back in the future.