bytecodealliance / wasm-tools

CLI and Rust libraries for low-level manipulation of WebAssembly modules

wasm-tools

A Bytecode Alliance project

CLI and Rust libraries for low-level manipulation of WebAssembly modules

Installation

Precompiled artifacts built on CI are available for download for each release.

If you'd prefer to build from source then first install Rust for your platform and then use the included Cargo package manager to install:

$ cargo install wasm-tools

Installation can be confirmed with:

$ wasm-tools --version

Subcommands can be explored with:

$ wasm-tools help

Examples

Basic validation/printing:

# Validate a WebAssembly file
$ wasm-tools validate foo.wasm

# Validate a WebAssembly module in the text format, automatically converting to
# binary.
$ wasm-tools validate foo.wat

# Validate a WebAssembly file enabling an off-by-default feature
$ wasm-tools validate foo.wasm --features=exception-handling

# Validate a WebAssembly file with a default-enabled feature disabled
$ wasm-tools validate foo.wasm --features=-simd

# Print the text format of a module to stdout
$ wasm-tools print foo.wasm

# Convert a binary module to text
$ wasm-tools print foo.wasm -o foo.wat

Simple mutation as well as piping commands together:

# Mutate a WebAssembly module and print its text representation to stdout
$ wasm-tools mutate foo.wasm -t

# Mutate a WebAssembly module with a non-default seed and validate that the
# output is a valid module.
$ wasm-tools mutate foo.wasm --seed 192 | wasm-tools validate

# Demangle Rust/C++ symbol names in the `name` section, strip all other custom
# sections, and then print out what binary sections remain.
$ wasm-tools demangle foo.wasm | wasm-tools strip | wasm-tools objdump

Working with components:

# Print the WIT interface of a component
$ wasm-tools component wit component.wasm

# Convert WIT text files to a binary-encoded WIT package, printing the result to
# stdout
$ wasm-tools component wit ./wit -t

# Convert a WIT document to JSON
$ wasm-tools component wit ./wit --json

# Round trip WIT through the binary-encoded format to stdout.
$ wasm-tools component wit ./wit --wasm | wasm-tools component wit

# Convert a core WebAssembly binary into a component. Note that this requires
# WIT metadata having previously been embedded in the core wasm module.
$ wasm-tools component new my-core.wasm -o my-component.wasm

# Convert a core WebAssembly binary which uses WASI to a component.
$ wasm-tools component new my-core.wasm -o my-component.wasm --adapt wasi_snapshot_preview1.reactor.wasm

CLI Conventions

There are a few conventions that all CLI commands adhere to:

• All subcommands print "short help" with -h and "long help" with --help.
• Input is by default read from stdin if no file input is specified (when applicable).
• Output is by default sent to stdout if a -o or --output flag is not provided. Binary WebAssembly is not printed to a tty by default, however.
• Commands which output WebAssembly binaries all support a -t or --wat flag to generate the WebAssembly text format instead.
• A -v or --verbose flag can be passed to enable log messages throughout the tooling. Verbosity can be turned up by passing the flag multiple times such as -vvv.
• Color in error messages and console output is enabled by default for TTY based outputs and can be configured with a --color argument.

Tools included

The wasm-tools binary internally contains a number of subcommands for working with wasm modules and component. Many subcommands also come with Rust crates that can be use programmatically as well:

CLI	Rust Crate	Description
wasm-tools validate	wasmparser	Validate a WebAssembly file
wasm-tools parse	wat and wast	Translate the WebAssembly text format to binary
wasm-tools print	wasmprinter	Translate the WebAssembly binary format to text
wasm-tools smith	wasm-smith	Generate a valid WebAssembly module from an input seed
wasm-tools mutate	wasm-mutate	Mutate an input wasm file into a new valid wasm file
wasm-tools shrink	wasm-shrink	Shrink a wasm file while preserving a predicate
wasm-tools dump		Print debugging information about the binary format
wasm-tools objdump		Print debugging information about section headers
wasm-tools strip		Remove custom sections from a WebAssembly file
wasm-tools demangle		Demangle Rust and C++ symbol names in the name section
wasm-tools compose	wasm-compose	Compose wasm components together
wasm-tools component new	wit-component	Create a component from a core wasm binary
wasm-tools component wit		Extract a *.wit interface from a component
wasm-tools component embed		Embed a component-type custom section in a core wasm binary
wasm-tools metadata show	wasm-metadata	Show name and producer metadata in a component or module
wasm-tools metadata add		Add name or producer metadata to a component or module
wasm-tools addr2line		Translate wasm offsets to filename/line numbers with DWARF
wasm-tools completion		Generate shell completion scripts for wasm-tools
wasm-tools json-from-wast		Convert a *.wast file into JSON commands

The wasm-tools CLI contains useful tools for debugging WebAssembly modules and components. The various subcommands all have --help explainer texts to describe more about their functionality as well.

Libraries

As mentioned above many of the tools of the wasm-tools CLI have libraries implemented in this repository as well. These libraries are:

• wasmparser - a library to parse WebAssembly binaries
• wat - a library to parse the WebAssembly text format
• wast - like wat, except provides an AST
• wasmprinter - prints WebAssembly binaries in their string form
• wasm-mutate - a WebAssembly test case mutator
• wasm-shrink - a WebAssembly test case shrinker
• wasm-smith - a WebAssembly test case generator
• wasm-encoder - a crate to generate a binary WebAssembly module
• wit-parser - a crate to parse and manage *.wit files and interfaces.
• wit-component - a crate to create components from core wasm modules.
• wasm-metadata - a crate to manipulate name and producer metadata (custom sections) in a wasm module or component.

It's recommended to use the libraries directly rather than the CLI tooling when embedding into a separate project.

C/C++ bindings

Using the CMakeLists.txt in crates/c-api, wasm-tools can be used from the wasm-tools.h header. Note that these bindings do not comprehensively cover all the functionality of this repository at this time, but please feel free to contribute more if you find functions useful!

Versioning and Releases

This repository has both a CLI and a suite of crates that is published to crates.io (Rust's package manager). The versioning scheme used by this repository looks like:

• wasm-tools - the CLI follows the versioning pattern of 1.X.Y. Frequently Y is 0 and X is bumped as part of a release for this repository.
• wat - this Rust crate is versioned at 1.X.Y as well and matches the wasm-tools version.
• wast - this Rust crate is versioned as X.0.Y. The X here matches the X in 1.X.Y of wasm-tools.
• All other crates - all other crates in this repository are versioned at 0.X.Y where X matches the 1.X.Y of wasm-tools.

Note that the Y of all the versions above will also match for any release of this repository. This versioning scheme is intended to reflect the stable nature of the CLI and the wat crate in terms of API stability. Other crates, however, all receive a major version bump that are not automatically considered API compatible on all releases. This reflects how WebAssembly itself is an evolving standard which is not an unchanging foundation. All of the crates in this repository are suitable for "production use" but be aware that API stability is not guaranteed over time. If you have difficulty upgrading versions please feel free to file an issue and we can help out.

Also, this repository does not currently have a strict release cadence. Releases are done on an as-needed basis. If you'd like a release done please feel free to reach out on Zulip, file an issue, leave a comment on a PR, or otherwise contact a maintainer.

For maintainers, the release process looks like:

• Go to this link
• Click on "Run workflow" in the UI.
• Use the default bump argument and hit "Run workflow"
• Wait for a PR to be created by CI. You can watch the "Actions" tab for if things go wrong.
• When the PR opens, close it then reopen it. Don't ask questions.
• Review the PR, approve it, then queue it for merge.

That should be it, but be sure to keep an eye on CI in case anything goes wrong.

Contributing

See CONTRIBUTING.md for more information about contributing to this repository.

License

This project is licensed under the Apache 2.0 license with the LLVM exception. See LICENSE for more details.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this project by you, as defined in the Apache-2.0 license, shall be licensed as above, without any additional terms or conditions.

wasm-tools's Issues

wasmparser: get function name

Is it possible to get function names when parsing WA with wasmparser? It seems FunctionSectionReader doesn't have such ability.

Make wasmparser no_std again

There is still intent to support "no_std" feature for this crate. There is no need to support allocation hungry API such as WasmParser/WasmDecoder, so it will be beneficial to make this legacy API under feature -- the section readers is more performant API and has to be used at this moment.

TODOs:

• Make WasmParser and friends optional via feature (and/or deprecate)
• Cleanup sections readers to not use dynamic memory Box, Vec, HashMap

I think that will allow us to be no_std without depending on alloc or hashbrown crates.

Implement Display trait for AST types

I want to propose implementing the Display trait for all of the AST types to easily convert from AST to human readable wat/wast. I volunteer to implement this.

I am in the design phase of writing a language which is a strict extension of wat and wast (https://github.com/vitiral/wak-lang) and so would like to use this crate. However, one of the debug implementations of my compiler will be to export commented wat code so it is clear how code was generated. For instance:

32

Might "compile" into

(;(;@1:0;) 32;)
(i32.const 32)

(where (;@1:0;) is a nested comment representing the line number where the source code starts)

In order to implement this, I need all wat types to be printed to human-readable wat source. Obviously I could create my own traits and implement all the types into that, but I think this would be generally useful for other users of this crate.

Thanks!

Yet another valid wasm module that fails validation

test.wasm.gz

(module
  (type (;0;) (func (result i32)))
  (type (;1;) (func (param i32 i32)))
  (import "" "" (table (;0;) 4278190080 funcref))
  (func (;0;) (type 1) (param i32 i32)
    call 1
    return
    local.get 1
    loop (param i32 i32)  ;; label = @1
      loop (param i32 i32)  ;; label = @2
        call 0
      end
    end
    unreachable)
  (func (;1;) (type 0) (result i32)
    unreachable))

fails with:

type mismatch: stack size does not match block type (at offset 119)

Once again, in unreachable code (following the unconditional return).

cc @yurydelendik

Implement `Hash` on types

Currently even data enums and structs (e.g. Type, FuncType, NameEntry, etc.), are missing Hash implementation, which makes it hard to use them in a HashMap for caching.

Would it be possible to add simple #[derive(Hash)] to all non-builder structs?

Change default branch name

As a policy, the Bytecode Alliance is changing the default branch names in all repositories. We would like for all projects to change the default to main by June 26. (We mention June 26th because there is some suggestion that GitHub may be adding the ability to make this process more seamless. Feel free to wait for that, but only up to June 26. We'll provide further support and documentation before that date.)

Please consider this a tracking issue. It is not intended for public debate.

• Change branch name
• Update CI
• Update build scripts
• Update documentation

Parse from `R: Read` instead of byte slice

Even though this parser is streaming, it's currently not possible to actually have it operate on a stream. The API requires a &[u8]. This isn't really necessary though, most of the API could remain identical while supporting any input type R where R: Read. I tried to implement this myself but this codebase is so huge and has so much duplicated code that I gave up, since it's not immediately necessary for my use-case.

Valid `if` with result types and unreachable code fails to validate

test.wasm.gz

(module
  (type (;0;) (func (result i32 i32)))
  (func (;0;) (type 0) (result i32 i32)
    global.get 0
    call 2
    unreachable
    select
    if (result i32 i32)  ;; label = @1
      unreachable
    else
      unreachable
    end)
  (func (;1;) (type 0) (result i32 i32)
    unreachable)
  (func (;2;) (type 0) (result i32 i32)
    unreachable)
  (table (;0;) 0 funcref)
  (global (;0;) (mut i32) (i32.const 808661811)))

Requires multi-value to be enabled.

This fails to validate with the error:

type mismatch: stack size does not match block type (at offset 113)

Fuzz test a 32-bit build

In the read_function_body() function, the code reading the local variable declarations looks like this:

        for _ in 0..local_count {
            let (count, ty) = self.reader.read_local_decl()?;
            locals_total += count as usize;
            if locals_total > MAX_WASM_FUNCTION_LOCALS {
                return Err(BinaryReaderError {
                               message: "local_count is out of bounds",
                               offset: self.reader.position - 1,
                           });
            }
            locals.push((count, ty));
        }

In a 32-bit build, the addition in locals_total += count as usize could overflow which causes a panic only in debug builds. In release builds it silently wraps.

A fuzz tester running on a 32-bit build would probably catch that.

Another valid modules fails validation

test.wasm.gz

(module
  (type (func))
  (type (func (result f64 i32)))
  (func (type 1)
    loop
      br 0
      call 0
      br_if 0
      i32.trunc_f64_u
      unreachable
    end
    unreachable
  )
)

Haven't started investigating yet, but it looks pretty similar to the last one: unreachable code (following the unconditional br) and a br_if instruction.

cc @yurydelendik

License disambiguation

The cargo.toml and README specify the license as Apache-2.0 WITH LLVM-exception but the crate contains both LICENSE-APACHE and LICENSE-MIT files.
Please explain under what license is this program distributed.

Thanks.

wast v35.0.2 unable to complile

Unable to compile wast v35.0.2 under nightly 2020-05-07.

error[E0277]: the trait bound `std::string::String: std::convert::From<char>` is not satisfied
   --> /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/wast-35.0.2/src/lexer.rs:851:28
    |
851 |         '\x20'..='\x7e' => String::from(c),
    |                            ^^^^^^^^^^^^ the trait `std::convert::From<char>` is not implemented for `std::string::String`
    |

Unreachable in block breaks (?) some validation

This wasm file:

(module
  (func
    unreachable
    i32.const 0
    select
    block (param i32)
      unreachable
    end))

validates with wabt but not with wasmparser:

$ cargo run -q --bin wat2wasm-rs test.wat -o test.wasm
$ wasm-validate test.wasm
$ cargo run -q --bin wasm-validate-rs test.wasm --enable-multi-value
Error: type mismatch: stack size does not match block type (at offset 35)

[crash/fuzzing] Resources exhaustion (CPU/MEM) using wasmprinter::print_bytes()

During fuzzing of wasmprinter, I trigger a resources exhaustion bug leading to full CPU usage when parsing a crafted wasm module. The file is only 650 bytes in size.

Download: huge_cpu_usage_wasmprinter.zip

Repo:

use std::env;
use std::fs::File;
use std::io;
use std::io::Read;

/// Read the contents from file path
fn read_contents_from_path(path_str: &String) -> Result<Vec<u8>, io::Error> {
    let mut buffer: Vec<u8> = Vec::new();
    let file_path = std::path::PathBuf::from(path_str);

    println!("file_to_process: {:?}", file_path);

    let mut file = File::open(file_path)?;
    file.read_to_end(&mut buffer)?;
    drop(file);
    Ok(buffer)
}

fn main() {
    println!("Start debugging of wasmprinter_parser");
    let args: Vec<String> = env::args().collect();

    // verify file_to_process is provided
    if args.len() != 2 {
        println!("Usage: wasmprinter_parser <file_to_process>\n");
        return;
    }

    // read data from provided file
    let data = read_contents_from_path(&args[1]).expect("cannot read file content");

    // call the fuzzing target
    wasmprinter::print_bytes(&data).is_ok();

    println!("No crash, everything is OK\n");
}

[wasm-encoder] Multi-value blocks

I have noticed that there seems to be no way to create multi-value blocks as in the example below:

(module
    (func $adder (param $a i32) (result i32)
        local.get $a
        i32.const 10
        i32.gt_s
        (if (result i32 i32) (then
            i32.const 1
            local.get $a
        ) (else
            local.get $a
            local.get $a
        ))
        i32.add
    )

    (func $main (export "_start") (result i32)
        i32.const 5
        (call $adder)
    )
)

Because BlockType only allows one type:

#[derive(Clone, Copy, Debug)]
pub enum BlockType {
    /// `[] -> []`
    Empty,
    /// `[] -> [t]`
    Result(ValType),
    /// The `n`th function type.
    FunctionType(u32),
}

Is there another way of creating multi-value blocks or this feature is just not supported yet?

[wasmparser] Split Operator::BrTable into multiple sub-operators

Currently the Operator enum has a lifetime attached which makes it a bit harder to work with in some contexts.

The only reason for this is its BrTable { table: BrTable<'a> } variant with which it is possible as user of this crate to lazily (but necessarily) parse the targets of the branching table.

Proposed Design

The solution I wanted to propose here is to split the single Operator::BrTable variant into 3 variants:

1. Operator::BrTableStart { count_targets: u32 }
2. Operator::BrTableTarget { depth: u32 }
3. Operator::BrTableDefault { depth: u32 }
   These variants will appear in exactly this order for every BrTable in the Wasm input if parsed by the wasmparser crate.
   The count_targets: u32 field tells the user how many Operator::BrTableTarget variants are to be expected directly afterwards, followed by a single Operator::BrTableDefault variant.

Alternative Design

An alternative design would be to treat the Operator::BrTableDefault as just another Operator::BrTableTarget and therefore the count_targets: u32 field in BrTableStart simply reflects the number of all targets as well as the additional default target.

What does this solve?

1. This proposal would eliminate the lifetime in the very big Operator enum. However, it would also be a breaking change and would probably result in potentially significant changes in how users handle BrTable inputs.

2. Additional benefits are that the Operator enum could then more or less easily derive a lot more useful traits such as:

   • Copy
   • PartialEq and Eq
   • PartialOrd and Ord
   • Hash

3. Also this proposal solves the current double parsing of BrTable as convenience type.
   The current implementation first parses the br_table just for validating the structure but throws all the information away so that the user can use this convenience type to re-parse and store the information. With this proposal the double parsing would be gone.

Streamlined Design

In my opinion this design would streamline the design of the Operator enum since it already splits some entities into multiple operators such as If, Else and End or Block and End or Loop and End etc. The split of BrTable would be just another.

Existing Works

It seems that the widely used wasmi interpreter is already using a similar technique to handle branching tables as can be seen here: (Please note that I am not familiar with its codebase.)
https://github.com/paritytech/wasmi/blob/master/src/isa.rs#L359

I am curious if this idea has already been discussed and what others think about it.

wasmprinter: missing space between "import" and the import string in module linking instantiation

test.wasm.gz

running wasmprinter::print_bytes on this (invalid wasm) file results in this output:

(module
  (module $memory-module (;0;)
    (memory (;0;) 1)
    (export "memory" (memory 0))
    (export "__wizer_memory_0" (memory 0)))
  (instance (;0;)
    (instantiate 0))
  (instance (;1;)
    (instantiate 1
      (import"x" (instance 0))))
  (alias 1 "init" (func (;0;)))
  (alias 0 "memory" (memory (;0;)))
  (type (;0;) (func))
  (type (;1;) (func (result i32)))
  (func (;1;) (type 0)
    call 0)
  (func (;2;) (type 1) (result i32)
    i32.const 0
    i32.load offset=1337)
  (export "wizer.initialize" (func 1))
  (export "run" (func 2)))

Note that (import"x", which should be (import "x".

cc @alexcrichton

Proposed tool: WASM binary output crate

This is like the single wasm-related tool I can't find here: something to take a Webassembly AST or slightly-more-convenient data structure, and output binary wasm. I'm writing a compiler backend that outputs wasm binary format, and having a known-correct crate that does this for me would be super convenient. I'm currently using parity-wasm, which works but is not exactly convenient or up to date, lacking things like multiple value returns. It looks like the wasmtime wast library also has the functionality I want, but is inconvenient in different ways; there's not much point in the compiler backend worrying about function names and code spans and such, and the docs don't really demonstrate how to go straight from AST to binary without starting by parsing text. There's also some cases like the Id type, which apparently can't be constructed at all except by the parser. So trying to use it for conjuring wasm from nothing would result in a pile of PR's to add the needed functionality.

So, is the functionality of taking a more binary-specific AST and outputting wasm bytecode something that would be interesting if I were contribute it?

Thanks in advance.

One more valid wasm module that fails validation

test.wasm.gz

(module
  (type (;0;) (func))
  (type (;1;) (func))
  (type (;2;) (func))
  (type (;3;) (func))
  (type (;4;) (func))
  (type (;5;) (func))
  (type (;6;) (func))
  (type (;7;) (func (param f64)))
  (type (;8;) (func))
  (type (;9;) (func (param f64)))
  (type (;10;) (func))
  (type (;11;) (func))
  (type (;12;) (func))
  (type (;13;) (func))
  (type (;14;) (func))
  (type (;15;) (func))
  (type (;16;) (func))
  (type (;17;) (func (param f64)))
  (type (;18;) (func (param f64)))
  (type (;19;) (func (param f64)))
  (type (;20;) (func (param f64)))
  (type (;21;) (func))
  (type (;22;) (func))
  (type (;23;) (func))
  (type (;24;) (func))
  (type (;25;) (func))
  (type (;26;) (func))
  (type (;27;) (func))
  (type (;28;) (func))
  (type (;29;) (func))
  (type (;30;) (func))
  (type (;31;) (func))
  (type (;32;) (func))
  (type (;33;) (func))
  (type (;34;) (func))
  (type (;35;) (func))
  (type (;36;) (func))
  (type (;37;) (func))
  (type (;38;) (func))
  (type (;39;) (func))
  (type (;40;) (func))
  (type (;41;) (func))
  (type (;42;) (func))
  (type (;43;) (func))
  (type (;44;) (func))
  (type (;45;) (func (param i32) (result i32)))
  (type (;46;) (func (param f64)))
  (type (;47;) (func))
  (type (;48;) (func))
  (type (;49;) (func))
  (type (;50;) (func))
  (type (;51;) (func))
  (type (;52;) (func))
  (type (;53;) (func))
  (type (;54;) (func))
  (type (;55;) (func))
  (type (;56;) (func))
  (type (;57;) (func (param f64)))
  (type (;58;) (func))
  (type (;59;) (func))
  (type (;60;) (func))
  (type (;61;) (func))
  (type (;62;) (func))
  (type (;63;) (func))
  (type (;64;) (func))
  (type (;65;) (func))
  (type (;66;) (func))
  (type (;67;) (func))
  (type (;68;) (func))
  (type (;69;) (func (param i32)))
  (func (;0;) (type 45) (param i32) (result i32)
    (local i32)
    local.get 1
    loop (param i32)  ;; label = @1
      i32.const -458751
      br_table 0 (;@1;) 1 (;@0;) 0 (;@1;)
      unreachable
    end
    unreachable)
  (func (;1;) (type 51)
    unreachable))

Going to try and reduce this a little more.

Performance tests seem to be broken

As already stated in bytecodealliance/wasmparser#216 under Benchmarks I am not able to run benchmarks for wasmparser on the current master.

At least I get the following output on cargo bench with RUST_BACKTRACE=1:

Benchmarking it works benchmark: Warming up for 3.0000 sthread 'main' panicked at 'unexpected error Bad magic number (at offset 0)', benches/benchmark.rs:38:42
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/libunwind.rs:86
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print_fmt
             at src/libstd/sys_common/backtrace.rs:78
   3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
             at src/libstd/sys_common/backtrace.rs:59
   4: core::fmt::write
             at src/libcore/fmt/mod.rs:1069
   5: std::io::Write::write_fmt
             at src/libstd/io/mod.rs:1537
   6: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:62
   7: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:49
   8: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:198
   9: std::panicking::default_hook
             at src/libstd/panicking.rs:218
  10: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:477
  11: rust_begin_unwind
             at src/libstd/panicking.rs:385
  12: std::panicking::begin_panic_fmt
             at src/libstd/panicking.rs:339
  13: criterion::Bencher<M>::iter
  14: <criterion::routine::Function<M,F,T> as criterion::routine::Routine<M,T>>::warm_up
  15: criterion::routine::Routine::sample
  16: criterion::analysis::common
  17: criterion::benchmark_group::BenchmarkGroup<M>::bench_function
  18: criterion::Criterion<M>::bench_function
  19: benchmark::main
  20: std::rt::lang_start::{{closure}}
  21: std::rt::lang_start_internal::{{closure}}
             at src/libstd/rt.rs:52
  22: std::panicking::try::do_call
             at src/libstd/panicking.rs:297
  23: std::panicking::try
             at src/libstd/panicking.rs:274
  24: std::panic::catch_unwind
             at src/libstd/panic.rs:394
  25: std::rt::lang_start_internal
             at src/libstd/rt.rs:51
  26: main
  27: __libc_start_main
  28: _start
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

error: bench failed

Issues with parsing select instructions?

Hi,

I've been building my own system on top of wast and have found that the AST emitted does not include the types for Select, and instead just lists "None". (I also see that "Drop" does not infer the type of the value being dropped automatically either - would be great if the AST node for drop contained this information as well)

Sample WASM:

The node in the AST just says there are no types present

types = SelectTypes {
tys: None,
}

Is there a workaround for this issue? Or do types for Select/Drop instructions have to be resolved dynamically in WASM (It seems like it should be statically available)? I probably could implement the type inference myself in my own AST parsing code, but preferably I could just use the default parser. I can see in the codebase that there is code to do the type inference for this instruction (

wasm-tools/crates/wast/src/ast/expr.rs

Line 1564 in 0462ff6

impl<'a> Parse<'a> for SelectTypes<'a> {

)

I may also be completely mistaken and that code refers to manually annotated types in which case nevermind - but it would be great to have some type inference here!

Issue in looking up event indices in `throw` instruction validation

I think there is a small bug in validating throw in cases like the following:

(assert_invalid
  (module
    (type (func))     ;; func type at index 0
    (func throw 0))   ;; event at index 0, which is missing
  "throw index out of bounds")

I believe this should be invalid, because the throw argument index should index into the events vector of the module, but it's looking up in the function types vector instead. Currently it doesn't throw an error.

(I can submit a PR to fix this, though maybe I should wait to see if there are thoughts on PR #152 first as it affects catch in that PR too)

wast: Ordering of instructions is a bit disordered

wast::Instruction [1] lists all instructions that can be parsed by WebAssembly. Currently it is neither sorted by name, encoding, or proposal. This makes it a bit hard to find where an instruction is, and I'd be happy to open a PR to sort it in some way.

@alexcrichton What do you think about sorting by proposal and then sorting by encoding within proposal?

[1]

wasm-tools/crates/wast/src/ast/expr.rs

Line 341 in e3f6090

Block(BlockType<'a>) : [0x02] : "block",

Request: Annotations for definitions of Wasm proposals/extensions

tl;dr

We might have a discoverability problem with all the Wasm proposals and extensions implemented in this crate.

Problem

When working with the very low level and generic wasmparser crate I often run into the same problem over and over again:
The wasmparser crate provides support not only for the WebAssembly MVP but also for many of the proposals and extensions that are finished or even under development.
For me as a person that does not have the entire set of proposals and extensions in my head it sometimes can be hard to infer what enum variant data structure, routine or even some fields are actually interesting for me and my projects that only supports a defined subset of the entire set of proposals and extensions of even just the MVP. As of today using the wasmparser often makes me feel that I have to know in advance all these proposals and how they extend the MVP set of definitions so that I can successfully filter out what I do not need and eventually provide useful information to the user for why something is (currently) not supported in my project.

Symptoms

My current solution when I find some Wasm definitions that are unknown to me is to google for them (with mostly not so much success) followed by a query through the entire set of proposals and extensions for Wasm which can becomes very frustrating quickly if it isn't very obvious sometimes where some definitions could potentially come from. Also not very productive although I have to admit that you learn a lot in the process, however, not very helpful if you want to progress with your project.

Solution

What would help me?

This is actually pretty simple: Each and every field, parameter, data type or enum variant that is the result of some non Wasm MVP proposal or extension needs to have one line of documentation that states which proposals or extensions imply its existence. This is all information I need to be able to do some further research and be able to easily identify parts that I do or do not need to support in case I do not know about all the proposals and all of their details that might even be under active development.

Example

If for every such definitions there was a line of documentation that states something like This definition is part of the Wasm proposal <proposal_name>. E.g. This definition is part of the Wasm proposal for bulk memory operations.. This allows a user of the library to then simply query this particular spec and directly find out all the details or immediate decide if this definitions is actually useful for the project at hand.

Why wasmparser ?

Now the obvious question is: Why should we add this to wasmparser? This might be interesting for literally all low level Wasm libraries which would be way too much work. The wasmparser crate is just the natural entry point into Wasm and therefore suited very well for an initial implementation of these docs.

I think those documentation strings are not hard to maintain since once a proposal is finished they won't be changing a lot anymore.

Implementation

I am sure we can incrementally get there. But what this issue demands is that in the future for everything we add to wasmparser we have this one line of documentation and eventually update the already existing parts to include it.

Expose internals of wast::Error

I am interested in using wast in one of my projects for reporting parse errors. I need to be able to extract useful information about the errors such as the file name, span, error message, etc. But it seems I can only really get access to the displayed string with wast because all of the internals of wast::Error are private.

Is there any particular reason for this? Can it be changed?

In theory I could reparse the displayed string to collect this info but that's messy and error prone and I'd rather avoid going that route.

crates.io links to old repository

Improve README.

The intro README for this project doesn't say much. It would be great if we highlighted some of the performance and flexibility features of this parser, and why people should use it.

• Highlight performance features.
• Highlight that it's flexible and can be used in a streaming fashion, describe why this is important and why this architectural decision was made.
• Give more context around the example, perhaps show the output of the example code.
• Highlight that it is well tested, and that it is used as part of the Cretonne project.
• Maybe draw a figure that describes the architecture.

Somehow protect against hugely-recursive module types

Currently it's relatively easy to get the validator (and wasm-smith which actually generates modules like this) to go down a rabbit hole in terms of performance. To do this you can create a hugely-nested module type with expands to lots of recursive checks of subtypes:

(module
  (type $m0 (module))
  (type $m1 (module
    (import "1" (module (type $m0)))
    (import "2" (module (type $m0)))
    (import "3" (module (type $m0)))
    (import "4" (module (type $m0)))
    (import "5" (module (type $m0)))
    (import "6" (module (type $m0)))
    (import "7" (module (type $m0)))
    (import "8" (module (type $m0)))
    (import "9" (module (type $m0)))
    (import "10" (module (type $m0)))
  ))
  (type $m2 (module
    (import "1" (module (type $m1)))
    (import "2" (module (type $m1)))
    (import "3" (module (type $m1)))
    (import "4" (module (type $m1)))
    (import "5" (module (type $m1)))
    (import "6" (module (type $m1)))
    (import "7" (module (type $m1)))
    (import "8" (module (type $m1)))
    (import "9" (module (type $m1)))
    (import "10" (module (type $m1)))
  ))
  (type $m3 (module
    (import "1" (module (type $m2)))
    (import "2" (module (type $m2)))
    (import "3" (module (type $m2)))
    (import "4" (module (type $m2)))
    (import "5" (module (type $m2)))
    (import "6" (module (type $m2)))
    (import "7" (module (type $m2)))
    (import "8" (module (type $m2)))
    (import "9" (module (type $m2)))
    (import "10" (module (type $m2)))
  ))
  (type $m4 (module
    (import "1" (module (type $m3)))
    (import "2" (module (type $m3)))
    (import "3" (module (type $m3)))
    (import "4" (module (type $m3)))
    (import "5" (module (type $m3)))
    (import "6" (module (type $m3)))
    (import "7" (module (type $m3)))
    (import "8" (module (type $m3)))
    (import "9" (module (type $m3)))
    (import "10" (module (type $m3)))
  ))
  (type $m5 (module
    (import "1" (module (type $m4)))
    (import "2" (module (type $m4)))
    (import "3" (module (type $m4)))
    (import "4" (module (type $m4)))
    (import "5" (module (type $m4)))
    (import "6" (module (type $m4)))
    (import "7" (module (type $m4)))
    (import "8" (module (type $m4)))
    (import "9" (module (type $m4)))
    (import "10" (module (type $m4)))
  ))
  (type $m6 (module
    (import "1" (module (type $m5)))
    (import "2" (module (type $m5)))
    (import "3" (module (type $m5)))
    (import "4" (module (type $m5)))
    (import "5" (module (type $m5)))
    (import "6" (module (type $m5)))
    (import "7" (module (type $m5)))
    (import "8" (module (type $m5)))
    (import "9" (module (type $m5)))
    (import "10" (module (type $m5)))
  ))
  (type $m7 (module
    (import "1" (module (type $m6)))
    (import "2" (module (type $m6)))
    (import "3" (module (type $m6)))
    (import "4" (module (type $m6)))
    (import "5" (module (type $m6)))
    (import "6" (module (type $m6)))
    (import "7" (module (type $m6)))
    (import "8" (module (type $m6)))
    (import "9" (module (type $m6)))
    (import "10" (module (type $m6)))
  ))
  (type $m8 (module
    (import "1" (module (type $m7)))
    (import "2" (module (type $m7)))
    (import "3" (module (type $m7)))
    (import "4" (module (type $m7)))
    (import "5" (module (type $m7)))
    (import "6" (module (type $m7)))
    (import "7" (module (type $m7)))
    (import "8" (module (type $m7)))
    (import "9" (module (type $m7)))
    (import "10" (module (type $m7)))
  ))
  (type $m9 (module
    (import "1" (module (type $m8)))
    (import "2" (module (type $m8)))
    (import "3" (module (type $m8)))
    (import "4" (module (type $m8)))
    (import "5" (module (type $m8)))
    (import "6" (module (type $m8)))
    (import "7" (module (type $m8)))
    (import "8" (module (type $m8)))
    (import "9" (module (type $m8)))
    (import "10" (module (type $m8)))
  ))

  (type $m (module
    (import "" (module (type $m9)))
  ))
  (import "a" (module $a (type $m9)))
  (import "b" (module $b (type $m)))
  (instance (instantiate $b "" (module $a)))
)

Today that module takes quite a long time in validation (30s!).

I'm not entirely sure how to prevent this just yet, but we shouldn't in any case have modules take 30s in validation when they're so small. Whatever fix is applied here needs to be extended to wasm-smith since sometimes generating modules takes 30+ seconds since it's doing subtyping checks internally.

wast: How do you look up function indices when `Id::new(...)` is not public?

If the following code is the idiomatic way to look up a function index then there's a problem. I can't create an Id:

let wat = "...";
let module = parse::<Wat>(&wat)?.module;
let names = module.resolve()?;
names.resolve_func(&mut Index::Id(
    Id::new( // associated function `new` is private
        "$my-name",
        Span::from_offset(0)
    )
));

Is there a better way to look up the $my-name function index?

wasm-smith: add a "no trapping" mode

I think we could do this with a post-processing pass, similar to what we do with ensure_termination.

We'd walk over each instruction and potentially insert some code right before it:

• We would insert a couple instructions to ensure that a division instruction's denominator is never zero
• We would insert a couple instructions to mask heap addresses to ensure they are within the memory's minimum size
• Similar for table.get and table.set
• Similar for trapping floating point conversion instructions
• Every unreachable would be replaced with code to create dummy result values (ie zeroes) and then br out of the current control frame

We would also have to make sure that active data/elem segments were always in bounds of their memories/tables.

I think that's everything? I might be missing some trapping cases, but I think the approach would work for everything.

cc @alexcrichton

BrTable should not allocate dynamically while parsing

The current interface to read the jump entries of a BrTable operator requires dynamic heap allocation upon parsing. We could remove this by changing the provided interface of BrTable a bit.

Instead of

pub fn read_table(&self) -> Result<(Box<[u32]>, u32)> { .. }

We provide a

pub fn entries(&self) -> impl Iterator<u32> { .. }

Due to creation of BrTable we already have to parse the whole area that is associated to it so we do not have to return a Result here anymore. Also we return an iterator instead of a boxed entity so the user can decide how to process this information and potentially avoid dynamic allocation altogether.

We could even improve upon this by making the returned impl Iterator an ExactSizeIterator.

Consider publishing wasmparser-dump

Hi,

I'm currently in the process of packaging wat and wast crates for Fedora (as rpm packages). In order to execute the tests during build, the dev-dependencies of these crates should be published on crates.io.

Would you consider publishing wasmparser-dump (and, if possible, adding the version identifier to dev-dependencies in all crates)?

Re-enable ./compare-master.sh

The benchmarking was disabled -- it takes hours to perform the tasks.

Looks like we are doing it for all files. Probably we need to identify smaller subset and use it for the task.

ResizableLimits can't represent the maximum memory32 heap size of 4GB

Cf https://github.com/bytecodealliance/wasm-tools/blob/main/crates/wasmparser/src/primitives.rs#L196. The maximum heap size for memory32 is 4GB (64K pages of 64K bytes each), but as this structure uses a u32 for the maximum field, and this field appears to represent a number of bytes and not the largest valid index, the maximum size cannot be represented. The structure should probably use u64 fields.

(Firefox is changing to support 4GB heaps, and we use wasmparser.)

How to get the AST information after parse processing

Hello, here is a simple code snippet:

use wast::Wat;
use wast::parser::{self, ParseBuffer, Result};
use wast::Module;

fn main() -> Result<(), > {
    let wat = "(module (func $add (param $lhs i32) (param $rhs i32) (result i32)local.get $lhs local.get $rhs i32.add)(export \"add\" (func $add)))";
    let buf = ParseBuffer::new(wat)?;
    let module = parser::parse::<Wat>(&buf)?;
    Ok(())
}

After that code, it seems successfully parse the wat text. But, I didn't find any document to how to print or utilize the AST we could get from this parsing.

Hope you can help me figure it out.

Thanks

Enable no-heap-allocation validation

Currently Validator stores many of its types and such in locally-defined vectors, but this necessitates heap allocation and support of all possible wasm modules. We should ideally support a form of generic validation which doesn't require any heap allocation at all, for example having maximums of functions, types, etc.

Why are section ids read as var u7 in wasmparser?

The spec says that a section id is a byte, why are then the section ids in wasmparser read as a var u7? I know there is no practical difference between the two interpretations as the number of valid section ids are too few to matter but was just curious about this difference.

Update encodings for wasm extensions

See also WebAssembly/spec#1228, and in particular WebAssembly/spec#1228 (comment): prefix is a single byte, and what follows is a LEB128, while wasmparser currently accepts a single byte (except for SIMD which accepts a LEB128 already).

Yet another valid wasm module does not validate: `select` and `if` in unreachable code

Originally discovered in fitzgen/wasm-smith#21

We have a select and an if in unreachable code. I've annotated each line with the stack after the instruction is executed below:

(module
  (type (;0;) (func))
  (func (;0;) (type 0)
    i32.const -393040     ;; [i32]
    i32.const -9211021    ;; [i32 i32]
    i32.const -1          ;; [i32 i32 i32]
    br 0 (;@0;)           ;; [i32 i32 i32]
    select                ;; [i32]
    if  ;; label = @1     ;; []
    end))                 ;; []

test.wasm.gz

Reproduce

$ cargo run --bin wasm-validate-rs -- test.wasm

Expected

It validates okay.

Actual

Error: type mismatch: stack size does not match block type (at offset 80)

Fix Linking and Reloc section format

https://github.com/WebAssembly/tool-conventions/blob/master/Linking.md changed. The reader needs to be updated.

Should I implement my own wat AST?

I want to be able to build a data structure in-memory in a Rust program that represents a Wasm module and output that module as a pretty-printed text format (.wat) string. I want to be able to control exactly how the expressions get folded into s-expressions, i.e. I don't want to always fold everything or always unfold. For example, I want to be able to generate code like this:

(module
  ;; snip
    (local.set $err
      (call $some_func
        (i32.const 0)
      )
    )
    (if
      (i32.ne (local.get $err) (i32.const 0))
      (then
        ;; snip
      )
    )
  ;; snip
)

I implemented a proof-of-concept library that can do this. See example here.

My question is: Does my approach make sense? It seems I'm repeating work already done in the wast. But it seems to me the wast crate's AST does not retain information about some instructions "foldedness". (And it's not public API so I can't use it anyways). Should I be implementing my own AST (not sure if I can call what I've done an AST)?

Inconsistent naming in wast::Instructions

Noticed that some instruction names are inconsistent, e.g. I16x8WidenHighI8x16u vs I16x8WidenLowI8x16U (last U), or with wasmparser, e.g. V128Andnot.

Get end of module span

We're now going to use this crate to parse .wast scripts and convert them to specialized .js for testing in SpiderMonkey. [1]

One issue I ran into was how to output modules into JS. Ideally we'd output the module in text form, for easy debugging. The problem is how to get from a wast::Module to a text module.

I looked into two ways of doing this:

1. wasmprinter::print(module.encode()) - this has the disadvantage that we lose info, such as $idNames. Additionally, some spec tests are changed semantically when round-tripped like this.
2. Use module.span and a handwritten scanner to mind the end of the S-expr for the module in the .wast file, then copying the string from the full-span.

(2) is the approach I went with and it works surprisingly well, once the scanner was written (look for closed_module in the phab diff. But I'm a little worried this is fragile, and it feels like getting the end of the span from the parser would be a better solution.

[1] https://phabricator.services.mozilla.com/D111306

Add an "unlimited items" mode to wasm-smith

Wasm-smith is proving very useful at testing whether or not engines resource-constrain wasm modules. For example it has expose #179 as a weakness in validating module-linking modules.

I think it'd be useful to add a SwarmConfig-style generator for wasm-smith where items in the module itself are practically unlimited and may exceed implementation limits. That way we can test that wasmparser doesn't ever have a blowup of resources on these modules, nor does wasmtime itself blow up in resources. We would just need to thread through a boolean that the wasm-smith-generated-module may not be valid.

Please include wasm-encoder in the main README

Hi!

I've noticed that wasm-encoder is not mentioned in the README.

Is there a reason for that?

I was looking for a binary generator and had trouble finding one, and because wasm-encoder is not mentioned in the README of this project, I wasted quite some time trying some other libraries which do not seem to be maintained anymore and are riddled with bugs.

Since I found wasm-encoder, it has worked perfectly for me, so far! Thanks for the good job.

Could you please also mention it in Crates you should know document while you're at it?

Thanks again.

Valid Wasm module fails validation

test.wasm.gz

(module
  (type (;0;) (func))
  (type (;1;) (func))
  (import "" "" (global (;0;) (mut i32)))
  (func (;0;) (type 1)
    (local f64)
    loop  ;; label = @1
      loop  ;; label = @2
        unreachable
        global.get 0
        global.get 0
        br_if 0 (;@2;)
        global.set 0
        unreachable
      end
      unreachable
    end
    unreachable))

I need to double check the spec and how unreachable code is validated, but all of wasm2wat, wasm-validate, and wasm-dis accept this module as valid, while wasmparser's validator returns an error.

Refactor wasmparser to avoid "double parsing"

There's currently two primary reasons that a wasm module has bits and pieces of it which end up being "double parsed":

• First is that the BinaryReader has some skip_* methods which are used. These methods are typically used to conform to the iterator protocol of Rust. For example when looking at ElementSectionReader when you call read it acts as an iterator, repositioning at the next element. The processing of the first element, however, happens by the consumer, which must happen afterwards. This means that the read method must skip to the next element for the next call to `read. Affected locations are:

  • ElementSectionReader::read
  • DataSectionReader::read
  • GlobalSectionReader::read
  • FunctionBody::get_operators_reader
  • FunctionLocalReader::read (and probably more in this file)
  • InstanceSectionReader::read

• Secondly the API design of the Validator type is such that it will always parse "header" sections, and then consuming applications (like wasmtime) are likely to then re-parse content of the section again. For example Validator::import_section will parse the import section, but then wasmtime also will iterate over the import section, re-parsing everything.

In general this isn't a massive concern because the header sections are likely all dwarfed in size by the code section so parsing is quite fast. Nonetheless we should strive to parse everything in a wasm file precisely once. I think to fix this we'll need two features, one for each problem above:

1. For the first issue I think we'll want to move towards a more advancing-style API rather than an iterator-based API. For example we'd have a dedicated type for reading the element section, and you'd say "read the header" followed by "read the elements". We might be able to use Drop and clever trickery to skip over data that wasn't explicitly read, or we could simply panic if methods aren't called in the right order. The downside of this is that consumers are likely going to get a little more complicated, but this may be fixable with clever strategies around APIs. I'm not sure how this would exactly look like.

2. For the second issue we'll want to add more APIs to the validator. For example instead of taking the import section as a whole we'd probably want to add something like "the import section is starting with this many items" which gives you a "sub-validator" which is used to validate each import after parsing. What I'm roughly imagining is that the application does all the parsing and then just after parsing feeds in everything to the validator. Another possible alternative is a "validating parser" which automatically feeds parsed values into the validator before handing them to the application. I'm not sure if this alternative is possible with "parse everything precisely once", however, since for example the element section ideally shouldn't be parsed twice, just once.

Validation of module linking should somehow account for nested resource limits

Currently wasmparser's validation imposes "reasonable" limits on modules so you can't validate a module with a million tables that all have a million table elements. This validation, however, is local to one module and does not span module-linking modules. Module-linking modules can instantiate nested modules, so if you allow 10-layer-deep modules to create at most 10 instances, that's still 10^10 instances which is a bit unreasonable!

Wasmparser's validation should bake in limits related to nested and recursive modules. I'm not entirely sure how to best slot this in and account for things, though. I suspect that what will need to happen is that instantiation of a module is connected to whether it's instantiating a known module or an imported module, and then in that scenario instantiation discounts from the global resource maximums accordingly. For locally-defined modules we know what resources to discount (as it's what the module defines), and for imported modules we could probably conservatively assume that it creates at least one item of each entity.

This would mean that our 10^10 instances wouldn't validate!

Note that we'll also need to update wasm-smith to not generate modules of this shape. I suspect we'll need similar accounting for resources there to ensure this.

validate accepts invalid wasm

The attached wasm file (g
x.wasm.gz
zipped due to github file type detection) triggers no errors with the wasmparser validator, but it has an invalidly large label index (0x8180808021) in the br_table default label.

Wabt detects the error:

$ wabt/build/wasm2wat x.wasm
000001f: error: unable to read u32 leb128: br_table default target depth

Module linking text-to-binary excessively slow with ~400 modules

This was found with a fuzzer so it's not quite reminiscient of what might be reasonable, but this module takes multiple seconds (~6 locally) to convert to a binary. Conversion from binary to text takes ~40ms, so this is a huge discrepancy which is a bug in the resolver.

Some profiling shows that most of the time is going to "expanding" the module which is elaborating types and such. I didn't exactly write it to be fast, but it looks like it's way slower than I thought it was.