bytecodealliance / jco Goto Github PK

Current behaviour

When using jco transpile with --instantiation, the generated code loads and compiles the necessary WASM modules for every created instance:

export async function instantiate(compileCore, imports, instantiateCore = WebAssembly.instantiate) {
  const module0 = compileCore('component.core.wasm');
  const module1 = compileCore('component.core2.wasm');
  const module2 = compileCore('component.core3.wasm');
  // ...
}

A better way

It would be better to load the modules one, and re-use the loaded modules. You could even load them the same way they are loaded without the --instantiation flag, so that the user doesn't have to provide the compileCore function.

A full example can be seen here with the usage here, but here is the code snippet:

// TODO: should be Promise.all
const module0 = await fetchCompile(
  new URL("./component.core.wasm", import.meta.url)
);
const module1 = await fetchCompile(
  new URL("./component.core2.wasm", import.meta.url)
);
const module2 = await fetchCompile(
  new URL("./component.core3.wasm", import.meta.url)
);

export async function instantiate(
  // compileCore, // not needed anymore
  imports,
  instantiateCore = WebAssembly.instantiate
) {
  // Not needed anymore either
  // const module0 = compileCore('component.core.wasm');
  // const module1 = compileCore('component.core2.wasm');
  // const module2 = compileCore('component.core3.wasm');
  // ... same code as before
}

This is not only better for performance (doesn't re-fetch and re-compile modules for every instance), but also better for the user, since they don't need to worry about the compileCore function.

Things to iron out

This was kind of "hacked together", but it shows that it can work. The modules should be loaded in parallel with Promise.all, and there might be other problems with a top level await, but I assume both of these issues can be easily resolved.

In bytecodealliance/wit-bindgen#406, having imports treated as ES module imports, results in their types no longer being asserted.

The problem is how to assert the imported types, as opposed to the TypeScript compiler checking them against what is being imported.

A type-level assertion type like type AssertType<Expected, Actual extends Expected> = void may be possible if we were outputting a .ts file for the main interface instead of just a declaration, but isn't permitted in the ambient declaration file.

From #102 (comment):

In the replication repo at Repro, with the installation steps:

cargo build -p typegraph_core --target wasm32-unknown-unknown -F wasm
wasm-tools component new target/wasm32-unknown-unknown/debug/typegraph_core.wasm -o target/debug/typegraph_core.wasm
jco transpile target/debug/typegraph_core.wasm -o typegraph/deno/gen --no-nodejs-compat --map "*=../src/imports.ts"

In exports/metatype-typegraph-runtimes.d.ts, this generates an invalid typing file for the WIT that tries to load from a relative typing path incorrectly classifying it as an import instead of another export and using the wrong name.

Types like string | URL are pretty common in JS, so would it be a good idea for unions to map to that rather than being the same as variants?

Implementing this could be quite tricky. When passing a type like that back to another module, the variant would have to be figured out based on the value. Some types would be impossible to distinguish from one another in JS, e.g. a union of u8 and u16. That would either have to be disallowed (probably a bad idea), or fall back on the default variant representation. The latter would be kind of weird and magical, though.

So, I'm not really sure if this is actually a good idea, but I thought I'd bring it up.

When generating many functions with similar bindgen, there tends to be a lot of repeated code. This is by design of the bindgen process, and output size is not currently being optimized.

If we had either a post-analysis which could determine the shared functions, or a bindgen approach based on outputting functions or shared objects (eg enum switches for wasi error codes), that could reduce the bindgen output size considerably.

Currently we generate different slightly different type variations based on the position being an argument or return. The reason being that, for example, an optional field can be omitted for an input type from user code, but on a result type will always be included. Similarly there are cases where array buffer coercions being more flexible on inputs is useful to loosen the input typing for.

After discussing this at the recent CTW event, @dbaeumer convinced me we should rather:

1. Remove the special casing of ArrayBuffer inputs and just use a strict input type, ie instead of trying to be "javascripty" to be "typescripty" and expect the right type
2. Treat all optionals as undefined | val instead of null | val to remove null and therefore ensure equivalence between records of of optionals and records of optional fields. Then it should be possible to unify the input and return types properly.
3. In any other cases where there are decisions like this to be made - unify on the stricter option, instead of trying to allow more loose conventions.

This should be relatively straightforward to PR, I will aim to pick it up in a week or so.

For jco WIT parsing to support WIT directories, we will need to support readfile ops inside of the main WIT parsing functions in the wasm-tools components, so we would depend on WASI to do this, and as a JS project, hence the WASI shim. So the self-build would rely on the preview2-shim.

Better WASI support is therefore not only a feature for this project, but necessary to properly support WIT.

It could be worth formally supporting error tuples of the form tuple<string, any> to indicate ComponentError { msg: string, payload: any } when converting the error into a JS error. This would allow having a nice readable string while also supporting custom payload data.

This would be a new component-specific convention, but could be worth discussing, either in this form or another, to get nice errors while still having serialized data available.

Since TypeScript supports first-class JSDoc comments, it would be useful to explore this as a first-class typing approach in the bindgen.

These have since been added to the spec and it'll ideally be more obvious where to implement these once bytecodealliance/wit-bindgen#314 is done. Once implemented this should also lead to the implementation of "lockdown semantics" where if a component traps it prevents the entire component from being reentered.

When transpiling into JS files, specifier names should use hyphens which are more specific to the JS ecosystem in places where dashes would otherwise be output.

For interop with strings that expect null termination, it could be beneficial to just always encode all strings with null terminators.

Otherwise in these use cases, an additional internal copy is always needed for these strings to pass them to APIs that assume null termination.

This one caught me off-guard and seems like an easy footgun. Passing [1,2,3] into a Vec<u32> by mistake should give a clear error (or coercion?).

Hi,

I am a developer on the VS Code team and I am working on adding WASM-WASI support to VS Code. Our current implementation is based on WASI preview 1 and there is a blog post describing its state.

Over the summer I looked into supporting the new component model and WASI preview 2 inside VS Code and I made quite some progress. I will attend WASMCON in Bellevue. I saw that some of you participate as well and I would really like to meet you in person there to discuss, if there are synergies between what you are working on and what I am doing.

Here is what I have been working on so far:

• implemented the canonical ABI as described here: https://github.com/WebAssembly/component-model/blob/main/design/mvp/CanonicalABI.md. The corresponding TypeScript implementation can be found here: https://insiders.vscode.dev/github/microsoft/vscode-wasi/blob/dbaeumer/monetary-ox-cyan/wasm-component-model/src/common/componentModel.ts#L1

• implemented a Wit parser in TS. Major reason is that I have limited Rust skills :-). The grammar and AST are here: https://insiders.vscode.dev/github/microsoft/vscode-wasi/blob/dbaeumer/monetary-ox-cyan/wasm-component-model/src/tools/wit.pegjs#L1 and https://insiders.vscode.dev/github/microsoft/vscode-wasi/blob/dbaeumer/monetary-ox-cyan/wasm-component-model/src/tools/wit-ast.ts#L1

• a code generator that emits TypeScript code for Wit input files together with a meta model to actually handle storing, loading, lowering and lifting of user defined data types generically. It also has meta information for function signatures to handle lifting of function params and lowering of results. The emitted TypeScript doesn't conform fully to the proposal how the ABI should be mapped to JS (see https://github.com/WebAssembly/component-model/blob/main/design/mvp/Explainer.md#js-api). Main reason was to leverage TypeScript as much as possible.

• I also generated TypeScript code for some of the WASI packages. Example code, including the meta model, can be found here: https://insiders.vscode.dev/github/microsoft/vscode-wasi/blob/dbaeumer/monetary-ox-cyan/wasi/src/clocks/monotonic-clock.ts#L1

Hope we can get in touch at WASMCON.

Best Regards

Dirk

Great work! The library seems overall to work well with Deno, some syntax updates are nonetheless required:

• add file extension like .d.ts
• the main components.d.ts files syntax is not recognized

import { Core as CoreExports } from './exports/core';
'core': typeof CoreExports, // ← that line

Currently they trap on out-of-bounds integers and booleans, but current component model semantics are such that they should be allowed. The invalid.rs runtime test needs to be updated to ensure the same behavior across JS/Rust/Python.

Whenever trying to use the interfaces from the preview2-shim directly, you get the following compilation error:

ERROR in /project/imports/wasi-poll-poll.d.ts 3:29-31
[tsl] ERROR in /project/imports/wasi-poll-poll.d.ts(3,30)
TS1359: Identifier expected. 'in' is a reserved word that cannot be used here.

I've run into an issue when running jco transpile which can be reproduced using the following command on the main branch:

$ ./jco.sh transpile -o dist2 ./test/fixtures/records.component.wasm 
(jco transpile) ComponentError: failed to parse the input component

Caused by:
    missing component instantiation argument named `import-type-f1` (at offset 0xca87)
    at generate (file:///home/danielbevenius/work/wasm/jco/node_modules/@bytecodealliance/jco/cli.mjs:43508:13)
    at transpileComponent (file:///home/danielbevenius/work/wasm/jco/node_modules/@bytecodealliance/jco/cli.mjs:44561:37)
    at transpile (file:///home/danielbevenius/work/wasm/jco/node_modules/@bytecodealliance/jco/cli.mjs:44480:27)
    at async file:///home/danielbevenius/work/wasm/jco/node_modules/@bytecodealliance/jco/cli.mjs:45243:9

I get a similar error for my own .wasm which is referring to a record type used as the return type of a function that is exported.

I noticed that I was able to get the command to run successfully if I use the following command:

$ node src/jco.js transpile -o dist2 ./test/fixtures/records.component.wasm 

Transpiled JS Component Files:

 - dist2/exports/exports.d.ts         2.71 KiB
 - dist2/imports/imports.d.ts         2.71 KiB
 - dist2/imports/testwasi.d.ts        0.13 KiB
 - dist2/records.component.core.wasm  44.8 KiB
 - dist2/records.component.d.ts       0.26 KiB
 - dist2/records.component.js         31.4 KiB

And using that command with my .wasm file works. Node version:

$ node --version
v18.14.0

I'm trying to figure out the difference in the commands and why I'm seeing this error and wanted to open this issue as perhaps others know what the issue is, or might be able to point me in the right direction.

See eb82375a44ec866ab1e01f5fc86145f7a32963de

bytecodealliance/wit-bindgen#355

Found myself recently doing this for WASI:

jsct transpile x.wasm --map wasi-exit=../wasi/exit.js --map wasi-logging=../wasi/logging.js --map wasi-default-clocks=../wasi/default-clocks.js --map wasi-poll=../wasi/poll.js --map wasi-filesystem=../wasi/filesystem.js --map wasi-clocks=../wasi/clocks.js --map wasi-random=../wasi/random.js

one simple way to achieve this would be based on the Node.js exports mapping rules to have:

jsct transpile x.wasm --map "wasi-*=../wasi/*.js"

The rules of this substitution can work exactly like the Node.js exports path rules.

If the generated code created a base "class" then users would be able to extend and override the code easily.

One of my main pain points with emscripten generated JS was that while it had lots of options that you could tweak and the ability to pre/post append JS, there were always parts of the wrapper that you couldn't get at.

Also, all those options ended up bloating the generated code.

IMO the class abstraction is ideal for this scenario...

Currently it seems like docs are being carried through only when going from a guest generator into a component Wasm file. When running the JS host generator against a component with an interface, I don't currently see any of the tests outputting TypeScript documentation in the generated TS file.

This would be a really really great feature to see, and do share if there are any implementation pointers on what is missing here.

Right now

foo: func(a: option<u32>)

generates:

export interface Input {
  foo(a: number | null): void;
}

where it should ideally instead generate:

export interface Input {
  foo(a?: number): void;
}

If we had the capability to convert adapter core modules that only do transcoding (eg in #79) and other simple binding attachment operations into the associated JS code, this would unblock some composition use cases.

We already covert Wasm to JS via jco transpile --js but that doesn't support multiple memories. Figuring out the core module subset this conversion applies to would be the first step here I think to determine what structure of JS needs to be output (all Wasm ops, or just a subset?).

Instance flags support came up recently with the composed.wasm component in https://github.com/macovedj/composition/tree/main.

Multi memories still seems to be the main todo to focus on, but instance flags will also need some degree of support next.

This component is currently giving an identifier naming collision when transplied.
operator_log-opt.wasm.gz

The PR in bytecodealliance/wit-bindgen#409 adds singular result type error handling and wrapping of errors when calling imported functions. This is done by creating intermediate { tag, val } objects in the output. It should be possible to more efficiently construct the output to avoid any intermediate object creations at all and directly initiate the correct control flow at the corresponding points in the generated code.

For example,

  const ret = instance0.exports.fn(ptr0, len0, variant2_0, variant2_1, variant2_2);  
  let variant12;
  switch (dataView(memory0).getUint8(ret + 0, true)) {
    case 0: {
      variant12 = { tag: 'ok', val: ... };
    }
    case 1: {
      const ptr11 = dataView(memory0).getInt32(ret + 4, true);
      const len11 = dataView(memory0).getInt32(ret + 8, true);
      const result11 = utf8Decoder.decode(new Uint8Array(memory0.buffer, ptr11, len11));
      variant12 = {
        tag: 'err',
        val: result11
      };
      break;
    }
    default: {
      throw new RangeError('invalid variant discriminant for expected');
    }
  }
  postReturn0(ret);
  if (variant12.tag === 'err') {
    throw new ComponentError(variant12.val);
  }
  return variant12.val;

should be possible to simplify to:

  const ret = instance0.exports.fn(ptr0, len0, variant2_0, variant2_1, variant2_2);
  switch (dataView(memory0).getUint8(ret + 0, true)) {
    case 0: {
      const result12 = ...;
      postReturn0(ret);
      return result12;
    }
    case 1: {
      const ptr11 = dataView(memory0).getInt32(ret + 4, true);
      const len11 = dataView(memory0).getInt32(ret + 8, true);
      const err = utf8Decoder.decode(new Uint8Array(memory0.buffer, ptr11, len11));
      postReturn0(ret);
      throw new ComponentError(err);
    }
    default: {
      throw new RangeError('invalid variant discriminant for expected');
    }
  }

In my generated js, I see the following function and variable,

let utf8EncodedLen = 0;
function utf8Encode(s, realloc, memory) {
  if (typeof s !== 'string') throw new TypeError('expected a string');
  if (s.length === 0) {
    utf8EncodedLen = 0;
    return 1;
  }
  let allocLen = 0;
  let ptr = 0;
  let writtenTotal = 0;
  while (s.length > 0) {
    ptr = realloc(ptr, allocLen, 1, allocLen + s.length);
    allocLen += s.length;
    const { read, written } = utf8Encoder.encodeInto(
    s,
    new Uint8Array(memory.buffer, ptr + writtenTotal, allocLen - writtenTotal),
    );
    writtenTotal += written;
    s = s.slice(read);
  }
  if (allocLen > writtenTotal)
  ptr = realloc(ptr, allocLen, 1, writtenTotal);
  utf8EncodedLen = writtenTotal;
  return ptr;
}

used as follows

 const ptr1 = utf8Encode(v0_0, realloc1, memory0);
    const len1 = utf8EncodedLen;
    const ptr2 = utf8Encode(v0_1, realloc1, memory0);
    const len2 = utf8EncodedLen;
    const {logRoot: v3_0, logLength: v3_1, mapRoot: v3_2 } = arg1;
    const ptr4 = utf8Encode(v3_0, realloc1, memory0);
    const len4 = utf8EncodedLen;
    const ptr5 = utf8Encode(v3_2, realloc1, memory0);
    const len5 = utf8EncodedLen;
    exports1['protocol#prove-inclusion'](ptr1, len1, ptr2, len2, ptr4, len4, toUint32(v3_1), ptr5, len5);

When using this in practice all length vars match the length of the most recently encoded string, I believe because utf8EncodedLen is a global var.
I believe the issue would be alleviated if utf8Encode were to return an object with a pointer and a length field, and the global variable were eliminated, though perhaps I'm missing something that makes the global variable make more sense? Happy to try and open a PR if others think this would be a good change.

Just like Node.js WASI supports instancing, we should make our preview2 shim implementation instancing based.

That is, still have the exact same exports we have right now, but have a new entry point like @bytecodealliance/preview2-shim/factory that provides a high-level class that instantiates into the various interfaces. Internally we should ideally then use this as the base dependency, while trying to keep cross-dependence to a minimum.

Perhaps each subsection will be its own factory - @bytecodealliance/preview2-shim/factory/filesystem say.

This will enable custom virtualization / reinstancing workflows. Eg for testing etc.

For some reason, when installing on Windows, the binary paths seem to not work out properly. It may be the ".mjs" extension on the CLI bin file, so it might be worth converting this file to CommonJS instead.

In the validation phase for component optimization, validation parsing must enable all new Wasm features, otherwise WASI preview 2 adaption gives errors like bulk memory options are not enabled.

Similar to how it is being done in wasmtime (code here), we should add tracing details for the host's generated binding code.

Thank you for creating jco!

I wanted to propose a formal documentation consolidation effort for API reference & the project itself.

P.S. As a current docs maintainer for multiple projects & a former technical writer, I'd be happy to chip in and/or work together to see this through.

Background

When I run jco transpile it builds a root .js file that contains:

import {
    cliBaseEnvironment as interface14,
    cliBaseExit as interface1,
    cliBasePreopens as interface9,
    cliBaseStderr as interface4,
    cliBaseStdin as interface2,
    cliBaseStdout as interface3,
} from "@bytecodealliance/preview2-shim/cli-base"
import { filesystemFilesystem as interface0 } from "@bytecodealliance/preview2-shim/filesystem"
import { ioStreams as interface5 } from "@bytecodealliance/preview2-shim/io"
import { randomRandom as interface13 } from "@bytecodealliance/preview2-shim/random"
import { smokeImports as interface7 } from "smoke"

Issue

I get 2 issues with this:

1. NodeJS resolve.

Nodejs cannot resolve the "@bytecodealliance/preview2-shim/cli-base", I believe because the package.json is using ./* instead of what the docs use ./features/* which has a keyword of some sort after the slash.

I can confirm that when I add ./features/* and "@bytecodealliance/preview2-shim/features/cli-base", it resolves as expected:

import {
    cliBaseEnvironment as interface14,
    cliBaseExit as interface1,
    cliBasePreopens as interface9,
    cliBaseStderr as interface4,
    cliBaseStdin as interface2,
    cliBaseStdout as interface3,
} from "@bytecodealliance/preview2-shim/feature/cli-base"
import { filesystemFilesystem as interface0 } from "@bytecodealliance/preview2-shim/feature/filesystem"
import { ioStreams as interface5 } from "@bytecodealliance/preview2-shim/feature/io"
import { randomRandom as interface13 } from "@bytecodealliance/preview2-shim/feature/random"
import { smokeImports as interface7 } from "smoke"

2. Phantom Import

The second issue is this generated file imports 'smoke' which is my world name so that makes sense, except again NodeJS cannot find this import because it doesn't exist. It expects to find it in npm, but of course it;s not there. I looked for it everywhere but cannot seem to find it. What is this import supposed to be?

When wrapping a component with jco transpile so that it can run on the web, the generated import functions are wrong, and the generated ts bindings are also wrong. See this repo at the jco-wrong-import-type-and-usage tag for full example.

Steps to reproduce

1. Create a wit protocol with multiple import functions. For example, use protocol.wit:

package example:protocol

world my-world {
  record point {
    x: s32,
    y: s32,
  }

  import print: func(msg: string)
  import import-point: func(pnt: point) -> point

  export run: func()
  export move-point: func(pnt: point) -> point
}

2. Implement the protocol in Rust using the wit-bindgen crate, see plugin-rust.
3. Convert the wit-protocol WASM module to a WASM component with wasm-tools component new plugin_rust.wasm -o component.wasm
4. Convert the WASM component back to a WASM module so that it can run in the browser with jco transpile component.wasm --instantiation -o out-dir

Expected behavior

The generated typescript bindings should export an idiomatic interface with the import functions, like this:

export interface ImportObject {
  print: (msg: string): void,
  importPoint: (pnt: Point): Point,
}

And the generated js implementation should use this import object correctly, like this:

  const importPoint = imports.importPoint;
  const print = imports.print;

Actual behaviour

The generated typescript bindings look like this:

export interface ImportObject {
  print: {
    default(msg: string): void,
  },
  'import-point': {
    default(pnt: Point): Point,
  },
}

And the generated JS code looks like this:

  const importPoint = imports.default;
  const print = imports.default;

This is wrong for several reasons:

1. The import-point function name in TS types isn't converted to use proper JS naming convention, which would be importPoint.
2. The print and import-point functions in the import object should be functions, not objects containing a default function.
3. The generated JS code is flat out wrong. According to the bindings, it should be const importPoint = imports['importPoint'].default, but ideally, it should be const importPoint = imports.importPoint.
4. Even ignoring the TS bindings, the JS code can never work, since both importPoint and print are being set to the same value.

Existing workaround

A fix for the JS code can be done with inline sed, see build-plugin.sh:

sed -E -i 's/const print = imports.default;/const print = imports.print;/' out-dir/component.js
sed -E -i 's/const importPoint = imports.default;/const importPoint = imports.importPoint;/' out-dir/component.js

This expression could be generalized, and maybe similar "fixup" could be made for the TS bindings, but ideally, the generated code would be correct without a need to fix it.

Software used

OS: Windows 10 64bit
jco: 0.9.3
wasm-tools: 1.0.35
wit-bindgen (crate): 0.8.0

There are certain cases where the host/guest implementation needs to pass an asynchronous function. Can this be supported in the JavaScript codegen even when the WIT definition has its signature as synchronous?

Summary

When I have a wasm component with a wit world that has "inline interfaces" (see bellow), when I transpile it with jco transpile component.wasm --instantiation -o out-dir, the generated TS bindings for the exported "inline interface" are wrong. Other TS types seem to be correct.

Full example

Full reproducible example can be found here, see the plugin-rust guest and runtime-rust-web host.

Wit file

package example:protocol

interface host-imports {
  print-line: func(msg: string)
}

interface guest-exports {
  run: func()
}

world my-world {
  import host-imports
  export guest-exports

  import inline-imports: interface {
    add-one: func(num: s32) -> s32
  }
  export inline-exports: interface {
    add-three: func(num: s32) -> s32
  }
}

Generated TS bindings

export interface Root {
  'example:protocol/guest-exports': typeof ExportsExampleProtocolGuestExports,
  'inline-exports': typeof ExportsInlineExports,
  guestExports: typeof ExportsExampleProtocolGuestExports,
  inlineExports: typeof ExportsInlineExports,
}

Actual behaviour

The actual "instance" object is missing the 'inline-exports' object, it only has inlineExports.

Ideally, it would have both to make it consistent with how "named" exports also have both variants.

Edit

I changed the name, because it seems that the TS bindings are "correct" and the JS code is wrong. In other words, it would be better to update the JS code to match the TS bindings and not the other way around.

Currently not inlining the latest version per npm.

It should be possible to pass a binary to extract wit from, particularly for the componentEmbed use case when there is not wit specified.

For generating types and stub JS code for componentization, it could be nice to have a simple gen command.

Perhaps something like:

jco gen --wit file.wit

to generate the source code for componentizing a world, outputting world.js, with stub interfaces and functions and imports.

TypeScript generation is in theory a subset of the above, so a jco gen --wit file.wit --types might be useful.

Options should otherwise be similar to transpile, supporting different wit inputs, and world selection, naming and possibly even mapping.

Hello,

I am trying to transpile the hello world from bytecodealliance/cargo-component but it seems to be incompatible?

$ jco wit ./target/wasm32-wasi/debug/cargo_comp.wasm

(jco componentWit) ComponentError: Failed to decode wit component
unknown component version: 0xd (at offset 0x0)

I get the same error when I try to jco transpile ... wasm components I've built from wit-bingen and adapted using the preview2 adapter.

Am I missing something?

This thread provides an example that should be easy to reproduce. The component generated from wasmbuilder can be found here

Because the preview2 version is set by jco, and we use a variable jco range in componentize, it can be tricky to figure out the version lock workflow.

Effectively it is something like:

{
  "dependencies": {
    "@bytecodealliance/jco": "0.5.3",
    "@bytecodealliance/componentize-js": "0.0.3"
  },
  "resolutions": {
    "@bytecodealliance/jco": "0.5.3",
    "@bytecodealliance/componentize-js": "0.0.3"
  }
}

then using the local ./node_modules/.bin/jco runner.

We should make sure there is a clearer / easier story here.

Currently it just uses slices of the underlying array buffer and doesn't attempt to swap endianness, which means that little-endian integers from wasm will be exposed as native-endian integers in JS, which means that on big-endian this will all be backwards.

Currently if @bytecodealliance/jco and @bytecodealliance/componentize-js are installed this happens locally:

$ jco componentize -w foo.wit -o bar.wasm foo.js
(jco componentize) Error: componentize-js must first be installed separately via "npm install @bytecodealliance/componentize-js".
    at componentize (file:///home/acrichto/node-v19.8.1-linux-arm64/lib/node_modules/@bytecodealliance/jco/cli.mjs:44427:11)
    at async file:///home/acrichto/node-v19.8.1-linux-arm64/lib/node_modules/@bytecodealliance/jco/cli.mjs:45420:9

I believe the reason for this is that I'm on an AArch64 Linux machine which doesn't have precompiled binaries for Wizer, but it'd have been more helpful for me if the error had mentioned that.

I did like to use jco for handling wasi preview2 in browser_wasi_shim. Is it possible to use jco directly in the browser to take a wasm component, run the transpilation step and directly run it as a shim until browsers natively support the wasm component model? Or does it only run in nodejs and does it only support writing out javascript files to the filesystem that can later be imported rather than directly evaluated using new Function or eval?

Add documentation that explains the runtime/codegen tests and how to add new ones.

Ideally, this will both help users trying to contribute to jco and also people trying to create minimal tests/reproductions of bugs.

When I compile a rust wasi component with cargo component build --release and I run jco on the result with

jco transpile plugin-wasi/target/wasm32-wasi/release/plugin_wasi.wasm --instantiation -o out-dir/

Then the generated TS bindings do not match what the JS actually does in the instantiate function.

Genrated TS bindings:

export interface ImportObject {
  'import-point': {
    default(pnt: Point): Point,
  },
  print: {
    default(msg: string): void,
  },
  'wasi:cli-base/environment': typeof ImportsEnvironment,
  'wasi:cli-base/exit': typeof ImportsExit,
  'wasi:cli-base/preopens': typeof ImportsPreopens,
  'wasi:cli-base/stderr': typeof ImportsStderr,
  'wasi:cli-base/stdin': typeof ImportsStdin,
  'wasi:cli-base/stdout': typeof ImportsStdout,
  'wasi:filesystem/filesystem': typeof ImportsFilesystem,
  'wasi:io/streams': typeof ImportsStreams,
}

Actual JS code:

export async function instantiate(compileCore, imports, instantiateCore = WebAssembly.instantiate) {
  const module0 = compileCore('plugin_wasi.core.wasm');
  const module1 = compileCore('plugin_wasi.core2.wasm');
  const module2 = compileCore('plugin_wasi.core3.wasm');
  const module3 = compileCore('plugin_wasi.core4.wasm');
  
  const { environment, exit: exit$1, preopens, stderr, stdin, stdout } = imports['@bytecodealliance/preview2-shim/cli-base'];
  const { filesystem } = imports['@bytecodealliance/preview2-shim/filesystem'];
  const { streams } = imports['@bytecodealliance/preview2-shim/io'];
  const importPoint = imports['import-point'].default;
  const print = imports.print.default;

The custom imported functions are fine, but the wasi import object does not match.

Edit: jco version 0.9.4

After discussion with @lukewagner about the current design, we determined some tweaks may still be necessary to the namespace conventions implemented in #86.

Consider:

world test {
  export wasi:filesystem/types
  export my:app/iface
}

We should actually be generating the canonical string names, instead of the combined kebab name of the package and interface:

// initial interface definitions are now just local variables
// with internal naming (ie not exposed to consumers)
const wasiFilesystemTypes = {
  // ... fns
};
const myAppIface = {
  // ... fns
};

// these are the "canonical export name" exports of these interfaces
export { wasiFilesystemTypes as 'wasi:filesystem/types', myAppIface as 'my:app/iface' }

// interface names can still be exported directly as
// being nested under the namespace when there is no conflict
// (as currently)
export { wasiFilesystemTypes as types, myAppIface as iface }

Future additions may include package or function lowering or aliasing, but the above should capture the major use case for now, while remaining compatible on the direct interface exports anyway.

For imports of namespaces, strictly speaking the ESM integration model should be based on the full string - ns:pkg/iface, and not do any handling of this. For our own handling of WASI shims, we do custom rewriting here in order to support a nice integration model, but this is more of a build exception than a standard interface model.

For non-WASI interfaces, we currently convert:

world test {
  import my:package/iface
}

into a JS import of the rough form:

import { iface } from 'my:package'

where the my:package module effectively exports multiple interfaces or functions corresponding to the implementation. This structure should instead be reverted to:

import * as iface from 'my:package/iface'

While this breaks reciprocal dependence for namespace linkage graphs (ie it is not possible for a component to link against another component in the implicit ESM integration without kebab names), instead we will separately have techniques for constructing implementation dependence graphs, which are instead not possible with the current namespacing model by design since it is primarily focused on host-level interfaces instead of implementation interfaces.

//cc @lukewagner let me know if this seems like a good summary of our discussion to you.

In regards to the Preview2 shim (more specifically wasi:filesystem) for browsers, should we leverage the File System Standard as defined by the WHATWG?