wasmerio / wasmer-pack Goto Github PK

We need to generate Python and JavaScript bindings for all modules in a Pirita file and write them to their own directories.

Requires #14.

At the moment, we can't generate JavaScript bindings for wasm32-wasi modules because both @wasmer/wasi and the bindings generated by wit-bindgen want to be the ones calling WebAssembly.instantiate().

We need to figure out how to compose the two.

See the conversation on Slack for more details.

We should be able to use the webc crate to parse Pirita files.

Determining available WASI commands and the default entrypoint should be relatively straightforward.

The WIT file metadata is embedded in its own "metadata" volume (see e43482d from https://github.com/wasmerio/pirita/pull/25) so we should be able to read it.

Note that this strategy will need to change once we allow WIT files which may depend on other WIT files. When we get to that point, it's better if the package author can place all *.wit files in the filesystem associated with a particular module. That way we can pass a custom callback to Interface::parse_with() which will look up the requested entry in the volume (cc: @fschutt).

I've found that wapm-to-webc (wapm2pirita on WAPM) will rename a WebAssembly file to use the atom name.

That means when writing Python's MANIFEST.in, we can't just include **/*.wasm because the WebAssembly file will have been renamed from hello_world.wasm to hello_world and subsequently excluded from the package.

For context, the error message looks something like this:

Traceback (most recent call last):
  File "D:\Projects\WIT\Python-Tests-Tutorial01\test.py", line 5, in <module>
    instance2 = bindings2.strings_and_lists()                                                              ib\\si
  File "D:\Projects\WIT\Python-Tests-Tutorial01\env\lib\site-packages\tutorial_02\bindings\__init__.py", line 44, in strings_and_lists
    module = self._get_module(filename)
  File "D:\Projects\WIT\Python-Tests-Tutorial01\env\lib\site-packages\tutorial_02\bindings\__init__.py", line 24, in _get_module
    wasm = Path(__file__).parent.joinpath(filename).read_bytes()
  File "C:\Python310\lib\pathlib.py", line 1124, in read_bytes
    with self.open(mode='rb') as f:
  File "C:\Python310\lib\pathlib.py", line 1117, in open
    return self._accessor.open(self, mode, buffering, encoding, errors,
FileNotFoundError: [Errno 2] No such file or directory: 'D:\\Projects\\WIT\\Python-Tests-Tutorial01\\env\\lib\\site-packages\\tutorial_02\\bindings\\strings_and_lists\\tutorial-02'

That way, users can provide their own inputs to generate a package for js/python.

Example usage:

wax wit-pack --module mymodule.wasm --exports mymodule.wit --package-name=mypackage --package-version=1.2.3

We need to create some sort of facade class which exposes all the commands and bindings under one object. In theory, this should be easier than #18 because Python has metaclasses and better reflection capabilities than JavaScript.

@ayys mentioned that the WAPM backend now has a dependency on wit-pack, so it'll need to be be available on both the production and development registries.

Implement the Records tutorial with a suitable example.

• Use an easy example, like a student or a point.
• Don't use variants or resources

At the moment, the wapm CLI expects each set of bindings to be completely self-contained, however it's not uncommon for one WIT file to import types or functionality from another. When packaging a WAPM package, the wapm CLI should recursively include any bindings that are used.

For example, it’s pretty common for the functionality exposed by the host and the guest to share the same types (e.g. errno in WASI).That means you’ll have three WIT files:

• guest.wit
• host.wit
• common.wit

... where both guest.wit and host.wit import records/resources/whatever from common.wit.

At the moment, the only WIT file that gets added to the bundled *.tar.gz is the one specified in your wapm.toml. Therefore, if we tried to generate bindings for a package like the one above, wit-pack would error out with “unable to import common.wit: file not found”.

Given a library with bindings (defined as wit), it should create a package for the given language (JS, Python) with the following sturcture.

This wit-pack generator should also live as a package in wapm.io/wapm.dev

JS

This should be a tar.gz file that can be installed in npm/node.

With the contents:

- package.json // This depends on wasmer-js?
- src
   - module.wasm
   - init.js / init.ts // Has the bindings

Python

This should be a tar.gz file that can be installed in npm/node.

With the contents:

- setup.py # this depends on wasmer-python
- PACKAGE_NAME
  - module.wasm
  - __init__.py # with the bindings generated for wasmer-python from wit-bindgen

Eventually, the setup.py and package.json will define after-install hooks that will automatically download the precompile artifacts for the used version of wasmer

We need to generate a JavaScript package that loads some module.wasm on import and exposes its functionality to the user as native JavaScript functions.

• Generate the JavaScript bindings using wit-bindgen-gen-js
• Create a index.js that will automatically instantiate the JavaScript bindings on import using the wasmer-js package
  • Not sure how we could do this - WebAssembly compiling is async and JavaScript doesn't allow top-level await
  • Maybe just provide a factory function that people can call so they get an object representing the WebAssembly module's exports?
  • The runtime & build environments will change how we load module.wasm - NodeJS can just load with the fs module, but browsers would either need to do fetch() or make the bundler figure it out
  • Just support NodeJS environments to start with because it's easiest
• Generate a package.json

At the moment, the CLI blindly assumes all atoms are commands, which is actually not true (libraries count as atoms, too!). Instead, when discovering libraries and commands, we should read the manifest directly and iterate over each Atom, matching its kind field against some known constants.

When trying to generate bindings for a package aysjha/597d361e-f431-4960-9b2a-7e78ec0dbfeb, I get the following error

Error: Unable to parse the package name

Caused by:
    0: "597d361e-f431-4960-9b2a-7e78ec0dbfeb" is not a valid package name
    1: Identifiers must start with an ascii letter

I am not sure if this is expected behavior. If it is not, then we should document the validation checks being performed on package-name, so it is consistent across projects (backend, wasmer-cli, wit-pack).

Npm generated packages expose a binding wrapper which expose the module that has the functions in them.

import { bindings } from "@dynamite-bud/tutorial-02";

(async () => {
  const mod = await bindings.strings_and_lists();
  console.log(mod.greet("Michael"));
  console.log(mod.greetMany(["Michael, Rudra, Syrus"]));
})();

Bindings Exposed via the tutorial-02 package then re-expose the module with the instance to be used.

• Announce the wai fork
• Explain what it does and brief example of how to use it
  • Note: should we mention you need to install the wapm CLI from git
• Mention packages already using wasmer-pack
• Point to the tutorial

It'd be cool if we can migrate brave-intl/wasm-thumbnail over to wit-pack.

The implementation is available in the wit-pack branch of wasmerio/wasm-thumbnail.

• Switch the underlying crate over to wit-bindgen
• Add examples for using the Python and JavaScript bindings
• Add instructions for building/publishing bindings
• Publish to wapm.dev (see Michael-F-Bryan/wasm-thumbnail)
• Make a PR upstream

It looks like we blindly assume all commands are WASI when inspecting a *.webc file.

You can reproduce this using the f0rodo/[email protected] package.

$ wasmer --version --verbose
wasmer 3.0.0 (3a105 2022-11-20)
binary: wasmer-cli
commit-hash: 3a105194470a955ba663509fc8dae60d3085717c
commit-date: 2022-11-20
host: x86_64-unknown-linux-gnu
compiler: singlepass,cranelift,llvm

$ wget https://registry-cdn.wapm.io/packages/f0rodo/rusty/rusty-1.0.0.webc

$ wasmer run wasmer/[email protected] --dir . -- show ./rusty-1.0.0.webc
Error: Unable to load the package

Caused by:
    no "atom" or "wasi.atom" or "emscripten.atom" found in command Command {
        runner: "https://webc.org/runner/wasm4/command@unstable_",
        annotations: {
            "wasm4": Map(
                {
                    Text(
                        "atom",
                    ): Text(
                        "rusty",
                    ),
                    Text(
                        "package",
                    ): Null,
                    Text(
                        "main_args",
                    ): Null,
                },
            ),
        },
    }

Ideally, the webc crate's get_atom_name_for_command() function would use the manifest to figure out the atom's ABI automatically, but in the meantime we should skip wasm4 commands when generating bindings.

• Update the wabt project from using the old *.wtix format to a newer wit-bindgen and the *.wit format (wapm-packages/wabt#1)
• Update the vscode-wasm plugin to use wabt bindings from WAPM instead of the NPM package

We need to generate a Python package that loads some module.wasm on import and exposes its functionality to the user as native Python functions.

• Generate the Python bindings using wit-bindgen-gen-wasmer-py
• Create a __init__.py that will automatically instantiate the Python bindings on import using the wasmer-python package
• Generate a pyproject.toml file and requirements.txt

Right now we are using await fs.readFile for reading the modules content. However, that API will never work in a browser.
We need to move back to the new URL(base, imports.meta....) to make sure the code is usable in both the browser and servers

Add support for providing a custom package name when generating the bindings

We need to inject async wrappers around every function/method wit-bindgen generates so we can eventually execute them remotely.

At the moment, our integration tests work by importing the files as they exist as a directory on disk, rather than the tarball you would get from WAPM. It'd be nice if we could update our tests to do a tar -czvf some-package.tar.gz ./out and import that ./some-package.tar.gz to more closely match what end users will be doing (this would have helped detect #59).

I initially implemented this in #65, but ran into issues where both pip and yarn treat a tarball as something that has been properly released. That meant they'll add integrity checks every time the tests try to install the package, and those checks were always failing.

For example, we got the following error with Yarn:

error Integrity check failed for "@wasmer/wabt" (computed integrity doesn't match our records, got "sha512-P4mwSUIu12z/9A/MF89lH0pKzQEFKVzc9oNNlfklyrUaYUlBkzKVethIKf9IJSmp/a6B7U4e6w5QchiK6b8nZQ== sha1-Ji/XdoPfCLiJpROBc/pTVkBPUpw=")

Pip generated a similar error message

Creating virtualenv python-DVkIVhwW-py3.9 in /home/runner/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies...

Writing lock file

Package operations: 14 installs, 0 updates, 0 removals

  • Installing pyparsing (3.0.9)
  • Installing attrs (22.1.0)
  • Installing exceptiongroup (1.0.0)
  • Installing iniconfig (1.1.1)
  • Installing mypy-extensions (0.4.3)
  • Installing packaging (21.3)
  • Installing pluggy (1.0.0)
  • Installing tomli (2.0.1)
  • Installing typing-extensions (4.4.0)
  • Installing wasmer (1.1.0)
  • Installing wasmer-compiler-cranelift (1.1.0)

  RuntimeError

  Hash for wasmer (1.1.0) from archive wasmer-1.1.0-cp39-cp39-manylinux_2_24_x86_64.whl not found in known hashes (was: sha[256](https://github.com/wasmerio/wasmer-pack/actions/runs/3348066190/jobs/5546725994#step:12:257):76bdd17a89dd652d39d789c138fff7236bb38da3e0ad54c716e2e72a90efef2e)

  at ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/poetry/installation/executor.py:681 in _validate_archive_hash
      677│         archive_hash: str = "sha256:" + file_dep.hash()
      678│         known_hashes = {f["hash"] for f in package.files}
      679│ 
      680│         if archive_hash not in known_hashes:
    → 681│             raise RuntimeError(
      682│                 f"Hash for {package} from archive {archive.name} not found in"
      683│                 f" known hashes (was: {archive_hash})"
      684│             )
      685│

NPM and Python packages should automatically add the available commands and

Similarly to Zig in Python, where people can do:

python -m ziglang {YOURCOMMANDTOZIG}

Or similarly to cowasm

~$ npx python-wasm@latest
Python 3.11.0 (main, Oct 27 2022, 10:03:11) [Clang 15.0.3 ([email protected]:ziglang/zig-bootstrap.git 0ce789d0f7a4d89fdc4d9571 on wasi
Type "help", "copyright", "credits" or "license" for more information.
>>> 2 + 3
5
>>> import sys; sys.version
'3.11.0 (main, Oct 27 2022, 10:03:11) [Clang 15.0.3 ([email protected]:ziglang/zig-bootstrap.git 0ce789d0f7a4d89fdc4d9571'
>>> sys.platform
'wasi'

How this should work?

If the package has an entrypoint command, that should be run by default relying the args.

JS

In JS, we should add all the runnable webc commands inside of the bin:

package.json

{
  ...
  "bin": { "COMMAND_NAME": "./bin/COMMAND_NAME" },
}

Python

In Python, we can rely on the

setup.py

setup(
    ...
    entry_points={'console_scripts': [
        'COMMAND_NAME = PACKAGE_NAME.commands:FUNCTION_COMMAND',
    ]},
)

In today's retro, we were playing around with the bindings for dynamite-bud/hello_world (note the _) and ran into an error with the generated bindings.

The JavaScript code:

// index.mjs

import { bindings } from "@dynamite-bud/hello_world";

async function main() {
	const wasm = await bindings.hello_world();
	console.log(wasm.greet("World"));
}

main();

The error:

$ node ./index.mjs
node:internal/modules/cjs/loader:995
  const err = new Error(message);
              ^

Error: Cannot find module './hello_world/hello_world.js'
Require stack:
- /tmp/node_modules/@dynamite-bud/hello_world/src/bindings/index.js
- /tmp/node_modules/@dynamite-bud/hello_world/src/index.js
    at Module._resolveFilename (node:internal/modules/cjs/loader:995:15)
    at Module._load (node:internal/modules/cjs/loader:841:27)
    at Module.require (node:internal/modules/cjs/loader:1061:19)
    at require (node:internal/modules/cjs/helpers:103:18)
    at Object.<anonymous> (/tmp/node_modules/@dynamite-bud/hello_world/src/bindings/index.js:2:37)
    at Module._compile (node:internal/modules/cjs/loader:1159:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1213:10)
    at Module.load (node:internal/modules/cjs/loader:1037:32)
    at Module._load (node:internal/modules/cjs/loader:878:12)
    at Module.require (node:internal/modules/cjs/loader:1061:19) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/tmp/node_modules/@dynamite-bud/hello_world/src/bindings/index.js',
    '/tmp/node_modules/@dynamite-bud/hello_world/src/index.js'
  ]
}

Node.js v19.0.0

The generated files:

$ tree ./node_modules/@dynamite-bud/hello_world
./node_modules/@dynamite-bud/hello_world
├── package.json
└── src
    ├── bindings
    │   ├── hello_world
    │   │   ├── hello-world.d.ts
    │   │   ├── hello-world.js
    │   │   ├── hello_world.wasm
    │   │   └── intrinsics.js
    │   ├── index.d.ts
    │   └── index.js
    ├── index.d.ts
    └── index.js

3 directories, 9 files

If you look at the generated src/bindings/index.js, we see that it assumes the file generated by wai-bindgen is called hello_world.js (same as the package name) instead of hello-world.js.

$ head ./node_modules/@dynamite-bud/hello_world/src/bindings/index.js
const fs = require("fs/promises");
const { HelloWorld: _HelloWorld } = require("./hello_world/hello_world.js");
...

The WAPM backend has a mutation which lets you update the bindings generator and automatically re-generate bindings for every package on WAPM. It'd be really cool if our "Releases" job in CI ran this automatically whenever a new version is released.

mutation changeGenerator {
  generateBindingsForAllPackages(input: {
    bindingsGeneratorId: "pkv_NnzrI7t5SwVn",
    bindingsGeneratorCommand:"wasmer-pack",
  }) {
    message
  }
}

The bindingsGeneratorId comes from the package version you are wanting to use.

{
  getPackageVersion(name:"wasmer/wasmer-pack-cli", version:"0.5.0") {
    id
  }
}

See Slack for more details.

It would be great to include a comment on the generated code (for js and python) with the wit-pack version used to generate the bindings.

Kind of:

Javascript (main.js)

// Code generated with wit-pack 0.2.1

Python (init.py)

# Code generated with wit-pack 0.2.1

The code would be more maintainable if we created a single Context object per set of language bindings instead of making a new ad-hoc object with minijinja::context! {} for each template. If we pull it out into structs, that'll also give us somewhere to document various assumptions (e.g. "this String is actually a Python identifier").

For the MVP, we were using write!() as a quick'n'dirty solution for generating code, but we should switch to a proper templating engine now we need to include logic (see #12).

In #78, @syrusakbary mentioned we should let users generate bindings from a package directory (i.e. something containing wapm.toml and all the *.wasm files).

Assuming we have some package called Michael-F-Bryan/foo which contains a bar library (with bar.wit bindings), a baz command, and a quuz command (set as the entrypoint), I would expect an interface something like this:

// src/index.d.ts

import { Wasi } from "@wasmer/wasi";

export function load(options?: Partial<LoadOptions>): Foo;

export type LoadOptions = {
  wasi: Wasi,
};

export interface Foo {
  bindings: {
    bar: {
      greet: (who: string) => Promise<void>,
      readFile: (filename: string) => Promise<Result<string, SomeError>>,
      ...
    },
  },
  commands: {
    baz: (options?: Partial<RunOptions>) => Promise<ExitStatus>,
    quux: (options?: Partial<RunOptions>) => Promise<ExitStatus>,
  },
  entrypoint: (options?: Partial<RunOptions>) => Promise<ExitStatus>,
}

The commands will just be an object containing the run() function from each command (see #16) and should be straightforward enough.

The bindings field will be a bit more difficult to implement because

1. All functions should be asynchronous so we can allow things like web workers or remote execution
2. We want people to call functions directly without (explicitly) instantiating the WebAssembly module

I was thinking we do something like this:

// src/bar/index.ts

// First, import the Bar class that wit-bindgen generated for us
import { Bar } from "./bar.ts";

// Read our WebAssembly bytes from somewhere and compile them.
const module: Promise<WebAssembly.Module> = fs.readFile(...).then(WebAssembly.compile);

// Create a wrapper which exposes each method on the generated type.
export default function(options: LoadOptions) {
  let instance: Bar | undefined;

  // A getter which will lazily instantiate the Bar that was generated by wit-bindgen
  async function getInstance(): Promise<Bar> {
    if (instance) {
      return instance;
    }

    const pending = new Bar();
    const imports = { };
    await pending.instantiate(await module, imports);
    instance = pending;
    return instance;
  }  

  return {
    // wrap each function we've generated bindings for with something that will lazily 
    // get the instance before deferring to the appropriate method. We can probably
    // automate this using reflection or something.
    greet: async (args...: any[]) {
      const instance = await getInstance();
      return instance.greet(args...);
    },
  };
}

When tarball is generated via WAPM bindings. It seems to be unusable due to the directory structure.

$ npm install .\tutorial01.tar.gz
npm WARN config global `--global`, `--local` are deprecated. Use `--location=global` instead.
npm ERR! code ENOENT
npm ERR! syscall open
npm ERR! path C:\Users\Rudra\AppData\Local\npm-cache\_cacache\tmp\T2YZea/package.json
npm ERR! errno -4058
npm ERR! enoent ENOENT: no such file or directory, open 'C:\Users\Rudra\AppData\Local\npm-cache\_cacache\tmp\T2YZea\package.json'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent 

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\Rudra\AppData\Local\npm-cache\_logs\2022-10-25T15_18_22_086Z-debug-0.log 

Directory Structure of tutorial.tar.gz

$ tar -ztvf tutorial-01-0.1.0.tar.gz
tar: Removing leading `/' from member names
drwxr-xr-x root/root         0 2022-10-24 20:48 /
-rw-r--r-- root/root       160 2022-10-24 20:48 package.json
drwxr-xr-x root/root         0 2022-10-24 20:48 src/
drwxr-xr-x root/root         0 2022-10-24 20:48 src/bindings/
drwxr-xr-x root/root         0 2022-10-24 20:48 src/bindings/hello-world/
-rw-r--r-- root/root      2520 2022-10-24 20:48 src/bindings/hello-world/hello-world.d.ts
-rw-r--r-- root/root      1086 2022-10-24 20:48 src/bindings/hello-world/hello-world.js
-rw-r--r-- root/root       213 2022-10-24 20:48 src/bindings/hello-world/intrinsics.js
-rw-r--r-- root/root   1725550 2022-10-24 20:48 src/bindings/hello-world/tutorial-01.wasm
-rw-r--r-- root/root       731 2022-10-24 20:48 src/bindings/index.d.ts
-rw-r--r-- root/root       958 2022-10-24 20:48 src/bindings/index.js
-rw-r--r-- root/root       102 2022-10-24 20:48 src/index.d.ts
-rw-r--r-- root/root       116 2022-10-24 20:48 src/index.js

Directory Structure of the camelcase npm package.

$ tar -ztvf ~/Downloads/camelcase-7.0.0.tar.gz
drwxrwxr-x root/root         0 2022-06-06 13:07 camelcase-7.0.0/
-rw-rw-r-- root/root       175 2022-06-06 13:07 camelcase-7.0.0/.editorconfig
-rw-rw-r-- root/root        19 2022-06-06 13:07 camelcase-7.0.0/.gitattributes
drwxrwxr-x root/root         0 2022-06-06 13:07 camelcase-7.0.0/.github/
-rw-rw-r-- root/root       115 2022-06-06 13:07 camelcase-7.0.0/.github/funding.yml
-rw-rw-r-- root/root       179 2022-06-06 13:07 camelcase-7.0.0/.github/security.md
drwxrwxr-x root/root         0 2022-06-06 13:07 camelcase-7.0.0/.github/workflows/
-rw-rw-r-- root/root       436 2022-06-06 13:07 camelcase-7.0.0/.github/workflows/main.yml
-rw-rw-r-- root/root        23 2022-06-06 13:07 camelcase-7.0.0/.gitignore
-rw-rw-r-- root/root        19 2022-06-06 13:07 camelcase-7.0.0/.npmrc
-rw-rw-r-- root/root      2449 2022-06-06 13:07 camelcase-7.0.0/index.d.ts
-rw-rw-r-- root/root      3214 2022-06-06 13:07 camelcase-7.0.0/index.js
-rw-rw-r-- root/root       684 2022-06-06 13:07 camelcase-7.0.0/index.test-d.ts
-rw-rw-r-- root/root      1117 2022-06-06 13:07 camelcase-7.0.0/license
-rw-rw-r-- root/root       886 2022-06-06 13:07 camelcase-7.0.0/package.json
-rw-rw-r-- root/root      3812 2022-06-06 13:07 camelcase-7.0.0/readme.md
-rw-rw-r-- root/root     15059 2022-06-06 13:07 camelcase-7.0.0/test.js

It'd be nice if we automatically deployed wit-pack to WAPM whenever cargo release tags a commit.

It's possible for someone to name their module the same as a type/function that is defined by our templated code, causing duplicate name errors when importing (see wasmerio/vscode-wasm#40 (comment) for an example).

We should make sure any packages that are imported programmatically use some sort of prefix/namespacing so we avoid naming conflicts.

This is currently blocking wasmerio/vscode-wasm#40 from being merged.

Intro

• A promise statement
  • We're going to introduce the concept of "resources"
  • Gives the host access to an "object" with associated behaviour without
    exposing how that object is implemented, or even which language it was
    written in
• A preview of what's to come
  • We'll turn our calculator into an object

Body

Define the WIT file:

// calculator-v2.wit
resource calculator {
    static new: func(initial-value: f32) -> calculator
    current-value: func() -> f32
    add: func(value: f32)
    multiply: func(value: f32)
    divide: func(value: f32)
}

Implement the guest:

• Explain the need for interior mutability
• Publish

Use the bindings:

• Create a Python project
  • Use pytest to test the calculator is correct
• Create a JavaScript project
  • Similar tests, but with Jest

Conclusion

• Reminder of how helpful the guide is
• Reiterate how important your topic is
• Call-to-action
  • Create a Regex library using the regex crate

The various code generators should copy static assets like README.md and LICENSE.md into the generated package.

We want to update the ruby.wasm project so it can use bindings generated from their WAPM package instead of their manually building ruby.wasm and wrapping it.

This requires:

• Figuring out ruby.wasm's build process
• Find the ruby.wasm file which implements rb-abi-guest.wit
• Update wapm.toml to also expose the library and bindings
• Update the corresponding build-package.sh to put binaries in the right place
• Publish to wapm.dev under the wasmer organisation (or maybe Michael-F-Bryan for now?)
• Update the JavaScript package to use the wapm.dev package instead of their version
• Make a PR upstream

Building

You can start up their development environment by running the following command from the repository's root directory:

$ docker run --rm -it \
    --volume $(pwd):/src -w "/src" \
    ghcr.io/ruby/ruby.wasm-builder:wasm32-unknown-wasi \
    /bin/bash

(note: this means all files and folders created during the build process will be owned by root on your host machine, but we can't use the --user $(id -u) trick because it breaks npm later on)

Full build instructions are available here, but you can use that Docker environment to compile from source. On my computer, this takes 5-15 minutes.

$ rake npm:ruby-head-wasm-wasi
...
$ tree -L 4 rubies/head-wasm32-unknown-wasi-full-js-debug
rubies/head-wasm32-unknown-wasi-full-js-debug/
└── usr/
    └── local/
        ├── bin/
        ...
        │   ├── irb
        │   └── ruby
        ├── include/
        │   └── ruby-3.2.0+2/
        ├── lib/
        │   ├── libruby-static.a
        │   ├── pkgconfig/
        │   └── ruby/
        └── share/
            └── man/

10 directories, 14 files
$ tree packages/npm-packages/ruby-head-wasm-wasi/dist
packages/npm-packages/ruby-head-wasm-wasi/dist/
├── bindgen/
│   ├── intrinsics.js
│   ├── rb-abi-guest.d.ts
│   ├── rb-abi-guest.js
│   ├── rb-js-abi-host.d.ts
│   └── rb-js-abi-host.js
...
├── index.cjs.js
├── index.d.ts
├── index.esm.js
├── index.js
├── index.umd.js
...
├── ruby.debug+stdlib.wasm
├── ruby+stdlib.wasm
└── ruby.wasm

1 directory, 28 files

Hopefully you won't need to call scripts manually, but the code that actually compiles ruby.wasm (the library, not the executable) and copies it to the ruby-wasm-wasi package's dist/ directory is here, however it only gets invoked from ruby-head-wasm-wasi's build script 🤷

Architecture

Their packaging system seems a bit convoluted because they've got a "template" JavaScript package under packages/npm-packages/ruby-wasm-wasi/ which contains the actual bindings and a "HEAD" package (ruby-head-wasm-wasi) which seems to just re-export code from the template.

The API being exported can be found in rb-abi-guest.wit, while the functionality imported by the host is defined in rb-js-abi-host.wit. They've also created a hand-written wrapper which manages the library instantiation and makes calling library functions slightly easier.

The executables that have already been uploaded to WAPM are under packages/wapm-packages/ and just provide access to the ruby and irb commands.

It would be nice if the wit-pack CLI could differentiate between different types of errors (invalid input, IO, rendering error, etc.) so it can provide feedback to users or automatically retry.

This will involve switching the wit-pack crate from anyhow::Error to a proper error type, then match on it in the CLI.

Cc: @ayys

Several crates have integration tests which exercise the bindings generated by wasmer-pack. Actually writing the integration test code can be kinda brittle because it relies on external programs (e.g. python and wapm) which need to be installed, and paths tend to change depending on the OS.

It'd be nice to pull all this integration test logic out into a single helper crate that lets you,

• Compile a project to a WAPM package
• Generate bindings for that package
• Set up a Python/Yarn/NPM environment with the bindings installed
• Run a set of integration tests defined in that language
• Cache dependencies from previous runs to avoid re-downloading them

I'm imagining being able to write code like this:

#[test]
fn sha2_integration_tests() -> Result<(), Error> {
  let env = TestEnvironment::for_crate("./path/to/Cargo.toml")?;

  env.python("./my_tests.py")
     .javascript("./my_test.js")
     .typescript("./my_tests.ts")
     .execute()?;
}

We want to invoke WASI executables programmatically in JavaScript.

I'm imagining each WASI executable exposing an interface like this:

import Wasi from "@wasmer/wasi";

export async function run(args: string, options?: Partial<RunOptions>): Promise<ExitStatus>;

type RunOptions = {
  wasi: Wasi,
  ...
};

type ExitStatus = {
  code: number,
}

We can reuse the @wasmer/wasi object to do things like the file system, stdin/stdout, environment variables, etc.

It'd be more user-friendly if, instead of this...

https://github.com/wasmerio/wit-pack/blob/0b12fe02ca741cb34a74977aaf7b459220fb68fc/crates/wit-pack/tests/python-wasi/test_wabt.py#L6

https://github.com/wasmerio/wit-pack/blob/0b12fe02ca741cb34a74977aaf7b459220fb68fc/crates/wit-pack/tests/python-wasi/test_wabt.py#L18-L21

... people just needed to write this...

from wabt import bindings

wabt_library_instance = bindings.wabt()
wasm_result = wabt_library_instance.wat2wasm("(module)", WasmFeature.MUTABLE_GLOBALS)

CC: @syrusakbary

(Both the cli and bindings)

There should be a high-level CLI built on top of the wit-pack crate for generating bindings.

$ wit-pack npm ./path/to/wapm.toml --out-dir ./js-bindings
$ tree js-bindings
.
├── package.json
└── src
    ├── module.wasm
    ├── index.d.ts
    └── index.js

$ wit-pack python ./path/to/wapm.toml --out-dir ./python-bindings
$ tree python-bindings
.
├── setup.py
└── my-package
    ├── module.wasm
    └── __init__.py

Some packages will require functionality from the host in order to function. At the moment, wit-pack only generates code for exposing an interface, and any imports are passed in via an untyped imports option.

Implementing this requires the following:

• Add some sort of imports field to wapm.toml
• Update the wapm CLI to embed these imports in the *.tar.gz file it generates
• Update the generated code to expect imports to be provided (will probably mean each library gets its own argument type with the specific imports it expects)
  • Python
  • javaScript
• Make sure imports are displayed in the WAPM UI

We need to plan and write out a tutorial for importing host functions (doc/tutorial/06-host-functions.md).

Similar to #16, we need a way to invoke WASI executables from Python code.

I'm imagining an interface something like this:

from dataclass import dataclass
from wasmer import Wasi

async def run(args: str, options: Optional[RunOptions] = None) -> ExitStatus:
    pass

@dataclass
class RunOptions:
    wasi: Optional[Wasi]
    ...

@dataclass
class ExitStatus:
  code: int

  @property
  def success(self):
    return self.code == 0

The run function shouldn't complete until the WASI executable has exited.