gfx-rs / rspirv

Rust implementation of SPIR-V module processing functionalities

rspirv

Rust implementation of SPIR-V module processing functionalities. It aims to provide:

• APIs for processing SPIR-V modules
• Command line tools building on top of the APIs for common processing tasks

rspirv defines a common SPIR-V data representation (DR) as the medium for various purposes. rspirv also provides a builder to build the DR interactively and a parser to parse a given SPIR-V binary module into its DR. A higher level structured representation is currently under developing.

SPIR-V is a common intermediate language for representing graphics shaders and compute kernels for multiple Khronos APIs, such as Vulkan, OpenGL, and OpenCL.

SPIRV-Tools is the Khronos Group's official C++ implementation of SPIR-V binary parser, assembler, disassembler, optimizer, and validator. rspirv is not a Rust language binding for that project; it is a complete rewrite using Rust.

Documentation

The current implementation supports SPIR-V 1.5 (Revision 4).

Multiple crates are published from this project:

Name	Crate	Docs
rspirv
spirv

In total rspirv APIs contains:

• The SPIR-V header (all SPIR-V structs, enums, and constants)
• The whole SPIR-V grammar (instruction layouts and their operands)
• A data representation of SPIR-V modules and its loader and builder
• SPIR-V binary module decoding and parsing functionalities

The Khronos SPIR-V JSON grammar is leveraged to generate parts of the source code using rspirv-autogen.

Please see the links to docs.rs for detailed documentation.

Status

I plan to implement several functionalities:

• SPIR-V data representation (DR)
• SPIR-V module builder
• SPIR-V module assembler
• SPIR-V binary module parser
• SPIR-V binary module disassembler
• HLSL/GLSL to SPIR-V frontend (maybe)
• SPIR-V DR to LLVM IR transformation (maybe)

The DR doesn't handle OpLine and OpNoLine well right now.

The SPIR-V binary module parser is feature complete.

Usage

This project uses associated constants, which became available in the stable channel since 1.20. So to compile with a compiler from the stable channel, please make sure that the version is >= 1.20.

Examples

Building a SPIR-V module, assembling it, parsing it, and then disassembling it:

use rspirv::binary::Assemble;
use rspirv::binary::Disassemble;

fn main() {
    // Building
    let mut b = rspirv::dr::Builder::new();
    b.set_version(1, 0);
    b.capability(spirv::Capability::Shader);
    b.memory_model(spirv::AddressingModel::Logical, spirv::MemoryModel::GLSL450);
    let void = b.type_void();
    let voidf = b.type_function(void, vec![]);
    let fun = b
        .begin_function(
            void,
            None,
            spirv::FunctionControl::DONT_INLINE | spirv::FunctionControl::CONST,
            voidf,
        )
        .unwrap();
    b.begin_basic_block(None).unwrap();
    b.ret().unwrap();
    b.end_function().unwrap();
    b.entry_point(spirv::ExecutionModel::Vertex, fun, "foo", vec![]);
    let module = b.module();

    // Assembling
    let code = module.assemble();
    assert!(code.len() > 20); // Module header contains 5 words
    assert_eq!(spirv::MAGIC_NUMBER, code[0]);

    // Parsing
    let mut loader = rspirv::dr::Loader::new();
    rspirv::binary::parse_words(&code, &mut loader).unwrap();
    let module = loader.module();

    // Disassembling
    assert_eq!(
        module.disassemble(),
        "; SPIR-V\n\
         ; Version: 1.0\n\
         ; Generator: rspirv\n\
         ; Bound: 5\n\
         OpCapability Shader\n\
         OpMemoryModel Logical GLSL450\n\
         OpEntryPoint Vertex %3 \"foo\"\n\
         %1 = OpTypeVoid\n\
         %2 = OpTypeFunction %1\n\
         %3 = OpFunction  %1  DontInline|Const %2\n\
         %4 = OpLabel\n\
         OpReturn\n\
         OpFunctionEnd"
    );
}

Directory Organization

There are multiple crates inside this repo:

• autogen/: Crate to generate various Rust code snippets used in the modules in spirv/ and rspirv/, from SPIR-V's JSON grammar. If you are not modifying spirv/ or rspirv/, you don't need to care about this directory.
• spirv/: The spirv crate.
• rspirv/: The core rspirv crate.
• dis/: A binary SPIR-V disassembler based on the rspirv crate.
• spirv-blobs: SPIR-V blobs provided by the user for testing.

Build

git clone https://github.com/gfx-rs/rspirv.git /path/to/rspirv

If you just want to compile and use the spirv crate:

cd /path/to/rspirv/spirv
cargo build

If you just want to compile and use the rspirv crate:

cd /path/to/rspirv/rspirv
cargo build

If you want to refresh the spirv or rspirv crate with new code snippets generated from SPIR-V's JSON grammar:

cd /path/to/rspirv
# Clone the SPIRV-Headers repo
git submodule update --init
cargo run -p rspirv-autogen

Test

Running cargo test would scan spirv-blobs folder and its subfolders (with symlinks followed) for any SPIR-V binary blobs. The test will try to load them, disassemble them, and then compose back from the internal representations, ensuring the smooth round-trip.

Contributions

This project is licensed under the Apache License, Version 2.0 (LICENSE or http://www.apache.org/licenses/LICENSE-2.0). Please see CONTRIBUTING before contributing.

Authors

This project is initialized Lei Zhang (@antiagainst) and currently developed by the gfx-rs Translators team.

rspirv's Issues

Cargo.lock in tree

I flip-flop randomly between believing that Cargo.lock should be included in a library repo, but I thought I'd raise this for discussion seen as it was mentioned in #100. The rust documentation does advise against it (FAQ), but I occasionally see a good argument for it. In general, I think it's probably a good idea to remove it.

Fuzzy testing infrastructure

Fuzzers are known to be good bug finders, proven useful for compression libraries and GLSL parsers. It would be great to have some fuzzy testing for rspirv as well.

Instruction: Derive Eq and Hash

hey, for a personal project I need to store Instructions in a HashMap.

Can these derives be added to Instruction?

mismatched types using rspirv 0.5.4 on latest nightly rustc

Hi,
When creating a simple project with the latest nightly rustc (1.43.0-nightly (823ff8cf1 2020-03-07)), cargo fails to build rspirv as a dependency due to a type mismatch.

carado@uwu:~/tmp/rust20 $ cargo init
     Created binary (application) package
carado@uwu:~/tmp/rust20 $ echo 'rspirv = "0.5.4"' >> Cargo.toml 
carado@uwu:~/tmp/rust20 $ cargo check
    Updating crates.io index
[…]
    Checking spirv_headers v1.4.2
[…]
    Checking rspirv v0.5.4
error[E0308]: mismatched types
   --> /home/carado/.cargo/registry/src/github.com-1ecc6299db9ec823/rspirv-0.5.4/mr/constructs.rs:190:22
    |
190 |             version: (spirv::MAJOR_VERSION << 16) | (spirv::MINOR_VERSION << 8),
    |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `u32`, found `u8`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0308`.
error: could not compile `rspirv`.

Address access sanitation

We need a way of modifying the SPIR-V so that no resource access goes beyond the bounds, for the sake of GPUWeb security. Some of the checks could be omitted by doing a static analysis, but a general approach could be injecting the code that queries buffers/images dimensions and reset the address offset to 0 using a dynamic check for that address to be in the bounds.

Multiple SPIR-V versions and target environments support

Vulkan and OpenCL targets different versions of SPIR-V (1.0 and 1.1 right now, respectively). rspirv should be able to support multiple versions of SPIR-V and multiple target environments.

SR error handling

Currently, there are situations where rspirv can panic, it should not be possible to cause a panic from inputs to the libraries (any observed panic should be regarded as a bug). It also errors out very quickly for a specific class of error. I am proposing a new error handling strategy:
All issues with user input should be classed into three types: warnings, recoverable errors and unrecoverable errors. These are treated in very different ways:

Warnings

These are for when a single valid alternative exists, e.g. Missing required capabilities. These will be logged to a store (a Vec) which is returned with the module on successful and unsuccessful parsing. These are probably quite rare and in practice may all get subsumed into a validation pass, so this category may not be necessary.

Recoverable Errors

These are for when there is an error with the SPIR-V which means that a valid module cannot be produced, but there is a sensible way to continue with the parsing. They are logged to a store (another Vec), and returned instead of the module. Most errors fall into this category as we can usually just replace the offending instruction/type/constant with a dummy value. These should all be instrumented with the instruction and offset where they occurred.

Unrecoverable Errors

This case should be avoided as it gives the least information to the user, however, if an error occurs and no reasonable corrective action can be taken, then a Result::Err should be propagated up the call stack and returned instead of the module, along with the list of recoverable errors encountered up to that point. This kind of error includes, encountering the end of the stream of words mid instruction, or receiving an instruction where the number of operands specified in the binary doesn't match the number encountered or expected.

Any comments on this approach, or alternatives, would be greatly appreciated.

Should the builder cache types?

For example:

    let void = b.type_void();
    let void1 = b.type_void();

results in

%2 = OpTypeVoid
%3 = OpTypeVoid

I expected just:

%2 = OpTypeVoid

Should the builder cache types? Or should duplicate instructions be removed in a later stage?

Validator

We need a validator for SPIR-V code

`cargo build` for codegen outputs everything on one line

When I open rspirv/sr/type_enum_check.rs, it looks like the following:

// Copyright 2016 Google Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//      http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

// AUTOMATICALLY GENERATED from the SPIR-V JSON grammar:
//   external/spirv.core.grammar.json.
// DO NOT MODIFY!

# [ derive ( Clone , Debug , PartialEq , Eq ) ] pub ( in sr ) enum TypeEnum { Void , Bool , Int { width : u32 , signedness : u32 } , Float { width : u32 } , Vector { component_type : TypeToken , component_count : u32 } , Matrix { column_type : TypeToken , column_count : u32 } , Image { sampled_type : TypeToken , dim : spirv :: Dim , depth : u32 , arrayed : u32 , ms : u32 , sampled : u32 , image_format : spirv :: ImageFormat , access_qualifier : Option < spirv :: AccessQualifier > } , Sampler , SampledImage { image_type : TypeToken } , Array { element_type : TypeToken , length : ConstantToken } , RuntimeArray { element_type : TypeToken } , Struct { field_types : Vec < TypeToken > } , Opaque { type_name : String } , Pointer { storage_class : spirv :: StorageClass , pointee_type : TypeToken } , Function { return_type : TypeToken , parameter_types : Vec < TypeToken > } , Event , DeviceEvent , ReserveId , Queue , Pipe { qualifier : spirv :: AccessQualifier } , ForwardPointer { storage_class : spirv :: StorageClass } , PipeStorage , NamedBarrier } impl Type { pub fn is_void_type ( & self ) -> bool { match self . ty { TypeEnum :: Void => true , _ => false } } pub fn is_bool_type ( & self ) -> bool { match self . ty { TypeEnum :: Bool => true , _ => false } } pub fn is_int_type ( & self ) -> bool { match self . ty { TypeEnum :: Int { .. } => true , _ => false } } pub fn is_float_type ( & self ) -> bool { match self . ty { TypeEnum :: Float { .. } => true , _ => false } } pub fn is_vector_type ( & self ) -> bool { match self . ty { TypeEnum :: Vector { .. } => true , _ => false } } pub fn is_matrix_type ( & self ) -> bool { match self . ty { TypeEnum :: Matrix { .. } => true , _ => false } } pub fn is_image_type ( & self ) -> bool { match self . ty { TypeEnum :: Image { .. } => true , _ => false } } pub fn is_sampler_type ( & self ) -> bool { match self . ty { TypeEnum :: Sampler => true , _ => false } } pub fn is_sampled_image_type ( & self ) -> bool { match self . ty { TypeEnum :: SampledImage { .. } => true , _ => false } } pub fn is_array_type ( & self ) -> bool { match self . ty { TypeEnum :: Array { .. } => true , _ => false } } pub fn is_runtime_array_type ( & self ) -> bool { match self . ty { TypeEnum :: RuntimeArray { .. } => true , _ => false } } pub fn is_structure_type ( & self ) -> bool { match self . ty { TypeEnum :: Struct { .. } => true , _ => false } } pub fn is_opaque_type ( & self ) -> bool { match self . ty { TypeEnum :: Opaque { .. } => true , _ => false } } pub fn is_pointer_type ( & self ) -> bool { match self . ty { TypeEnum :: Pointer { .. } => true , _ => false } } pub fn is_function_type ( & self ) -> bool { match self . ty { TypeEnum :: Function { .. } => true , _ => false } } pub fn is_event_type ( & self ) -> bool { match self . ty { TypeEnum :: Event => true , _ => false } } pub fn is_device_event_type ( & self ) -> bool { match self . ty { TypeEnum :: DeviceEvent => true , _ => false } } pub fn is_reserve_id_type ( & self ) -> bool { match self . ty { TypeEnum :: ReserveId => true , _ => false } } pub fn is_queue_type ( & self ) -> bool { match self . ty { TypeEnum :: Queue => true , _ => false } } pub fn is_pipe_type ( & self ) -> bool { match self . ty { TypeEnum :: Pipe { .. } => true , _ => false } } pub fn is_forward_pointer_type ( & self ) -> bool { match self . ty { TypeEnum :: ForwardPointer { .. } => true , _ => false } } pub fn is_pipe_storage_type ( & self ) -> bool { match self . ty { TypeEnum :: PipeStorage => true , _ => false } } pub fn is_named_barrier_type ( & self ) -> bool { match self . ty { TypeEnum :: NamedBarrier => true , _ => false } } }

No way to make forward references with mr::Builder

When building an if-else–construct using branch_conditional(cond, true_label, false_label, …) I need the OpLabel-ids before they are generated.

If branch_conditional returned &mr::Instruction and forward references were Cell<Word> instead of Word I could patch it later. Though this would allow anyone with a non-mut reference to mess things up.

Update spirv_headers to version 1.3.1

I've tried updating spirv_headers to version 1.3.1 with no luck (I was getting a LiteralContextDependentNumber: this kind is not expected to be handled here)
Now that Vulkan 1.1 supports SPIR-V 1.3 it would be nice if spirv_headers was upgraded to 1.3.1

Licence headers out of date

All of the license headers state Copyright 2016 Google Inc..
I feel like these should be being updated to the year they were last modified at the least. I'm also unsure as to what the licensing situation is with this project: references to Google are slowly disappearing; and it's licensed solely under Apache v2, whereas most of the rust ecosystem is dual-licensed under MIT/Apache v2 (I'm not actually sure if this can cause any issues, it might be worth looking into).
Changing the license is a real pain and a tricky business so it would make sense to get it right now before too many people touch the code.

Improve codegen grammar typing

The biggest gain would be having strongly typed parsing results, e.g.
if let Class::ModeSetting = inst.class { ... } instead of if inst.class = "ModeSetting" { ... }

We should evaluate the cost bringing the dependency, the amount of code removed from here, and the benefits.

Update `derive_more` when it is ready

Spawned out of #108

In order to reduce the number of compilations for syn of different versions, it would be useful to upgrade derive_more 0.15, as it depends on syn 0.15. However derive_more 0.99 which uses syn 1.0 is not usable right now because:

• For derive_more::From, multiple variants with the same fields will have conflicting implementations. In 0.15 it simply didn't generate the impl From<_>. JelteF/derive_more#106
• Getting around that issue (JelteF/derive_more#105) requires a lot of attributes, which is undesirable:

#[derive(Clone, Debug, Eq, PartialEq, PartialOrd, Ord, From)]
pub enum Decoration {
    #[from(ignore)]
    RelaxedPrecision,
    #[from(ignore)]
    SpecId(u32),
    #[from(ignore)]
    Block,
    #[from(ignore)]
    BufferBlock,
    #[from(ignore)]
    RowMajor,
    #[from(ignore)]
    ColMajor,
    #[from(ignore)]
    ArrayStride(u32),
    // .. etc
}

Optimize autogen build

On my machine, touching autogen module comes with a 35 second waiting penalty. It would be very useful for our productivity to make it faster.

Reflecting on SPIRV

I just want to reflect on SPIRV and I am not completely sure if I am doing the correct thing here.

I am trying to parse the following glsl code compiled as SPIRV.

....
layout (location = 0) in vec4 pos;
layout (location = 1) in vec3 normal;
layout (location = 2) in vec2 uv;

layout (location = 0) out vec2 o_uv;
layout (location = 1) out vec3 o_normal;
layout (location = 2) out vec3 o_pos;
....

I want to extract all information of all variables, but for this example I am only parsing the vector type.

extern crate rspirv;
extern crate spirv_headers as spirv;

use spirv::Op;
use rspirv::binary::Parser;
use rspirv::mr::{Loader, Operand, Instruction, Module};

use std::fs::File;
use std::path::Path;
use std::io::Read;
#[derive(Debug)]
pub enum Primitive {
    F32,
}
#[derive(Debug)]
pub enum Type {
    Vec(Primitive, u32),
}

fn get_id_ref(inst: &Instruction) -> Option<u32> {
    inst.operands
        .iter()
        .filter_map(|o| if let &Operand::IdRef(id) = o {
            Some(id)
        } else {
            None
        })
        .nth(0)
}

fn get_literal(inst: &Instruction) -> Option<u32> {
    inst.operands
        .iter()
        .filter_map(|o| if let &Operand::LiteralInt32(val) = o {
            Some(val)
        } else {
            None
        })
        .nth(0)
}

pub fn match_vec(module: &Module, id: u32) -> Option<Type> {
    if let Some(type_ptr_inst) = module.global_inst_iter().find(|i| i.result_id == Some(id)) {
        if type_ptr_inst.class.opcode == Op::TypePointer {
            let vec_id_ref = get_id_ref(&type_ptr_inst);
            if let Some(type_vec) = module
                .global_inst_iter()
                .find(|i| i.result_id == vec_id_ref)
            {
                if type_vec.class.opcode == Op::TypeVector {
                    let type_id_ref = get_id_ref(&type_vec);
                    let ty = module
                        .global_inst_iter()
                        .find(|i| i.result_id == type_id_ref);
                    let dimension = get_literal(&type_vec);
                    if let (Some(ty), Some(dim)) = (ty, dimension) {
                        if ty.class.opcode == Op::TypeFloat {
                            return Some(Type::Vec(Primitive::F32, dim));
                        }
                    }
                }
            }
        }
    }
    None
}
fn main() {
    let mut file = File::open(Path::new(
        "/path/vert.spv",
    )).expect("file");
    let mut bytes: Vec<u8> = Vec::new();
    file.read_to_end(&mut bytes).expect("read");

    let mut loader = Loader::new();
    {
        let p = Parser::new(&bytes, &mut loader);
        p.parse().unwrap();
    }
    let module = loader.module();
    for i in module
        .global_inst_iter()
        .filter(|i| i.class.opcode == Op::Variable)
    {
        if let Some(id) = i.result_type {
            println!("{:?}", match_vec(&module, id));
        }
    }
}

This outputs

Some(Vec(F32, 2))
Some(Vec(F32, 2))
Some(Vec(F32, 3))
Some(Vec(F32, 3))
Some(Vec(F32, 3))
Some(Vec(F32, 4))
None
None

which is exactly what I wanted. But it felt very weird, am I using the API correctly? Or can this be done simpler?

Infer result types

Being explicit about the result type in each instruction is redundant in most cases. There are exceptions, where the result type is a part of the semantics (to be added):

• OpCopyLogical

I wonder if we could strip it away from SR, and just take special care of the exceptional instructions, by adding a more information in the instruction itself (e.g. a field in the instruction enum variant) about what it does.

The expected benefit from this would be less work for transforming and validating SR.

Threaded code generation with mr::Builder

I currently want to multi-thread my code generation, but Builder is inherently mutable. The simple solution would be to just wrap it in a mutex, but that would kill the performance.

One solution that I am thinking of is to create a separate builder object for every node that I traverse, then I would return the builder object and merge them together in the correct order.

The only problem that I see are the ids. If I would want to split the builder object into many small builder objects I would need to keep track of the correct IDs.

#[derive(Default)]
pub struct Builder {
    module: mr::Module,
    next_id: Arc<AtomicUsize>, // <= from u32 to Arc<AtomicUsize>,
    function: Option<mr::Function>,
    basic_block: Option<mr::BasicBlock>,
}

One solution would be to use an Arc<AtomicUsize>, this could also made generic.

And and append function to merge them together. builder.append(other_builder);

What are your thoughts?

function parameters not being assembled

https://github.com/google/rspirv/blob/master/rspirv/binary/assemble.rs#L118-L132

Found this after a few hours of debugging a crash when trying to run a generated spir-v binary in intel's experimental OpenCL 2.1 CPU driver. Wish they had a nice error message rather than just segfaulting!

Unable to run tests

Thomass-MacBook-Pro:Development thomashavlik$ git clone https://github.com/google/rspirv.git
Thomass-MacBook-Pro:Development thomashavlik$ cd rspirv
Thomass-MacBook-Pro:rspirv thomashavlik$ RUST_BACKTRACE=1 cargo test
   Compiling rspirv-codegen v0.1.0 (/Users/thomashavlik/Development/rspirv/codegen)
error: failed to run custom build command for `rspirv-codegen v0.1.0 (/Users/thomashavlik/Development/rspirv/codegen)`
process didn't exit successfully: `/Users/thomashavlik/Development/rspirv/target/debug/build/rspirv-codegen-d054220d5ae98f1e/build-script-build` (exit code: 101)
--- stderr
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/libcore/result.rs:1009:5
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:71
   2: std::panicking::default_hook::{{closure}}
             at src/libstd/sys_common/backtrace.rs:59
             at src/libstd/panicking.rs:211
   3: std::panicking::default_hook
             at src/libstd/panicking.rs:227
   4: <std::panicking::begin_panic::PanicPayload<A> as core::panic::BoxMeUp>::get
             at src/libstd/panicking.rs:491
   5: std::panicking::continue_panic_fmt
             at src/libstd/panicking.rs:398
   6: std::panicking::try::do_call
             at src/libstd/panicking.rs:325
   7: core::char::methods::<impl char>::escape_debug
             at src/libcore/panicking.rs:95
   8: core::result::unwrap_failed
             at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libcore/macros.rs:26
   9: <core::result::Result<T, E>>::unwrap
             at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libcore/result.rs:808
  10: build_script_build::main
             at codegen/build.rs:88
  11: std::rt::lang_start::{{closure}}
             at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libstd/rt.rs:74
  12: std::panicking::try::do_call
             at src/libstd/rt.rs:59
             at src/libstd/panicking.rs:310
  13: panic_unwind::dwarf::eh::read_encoded_pointer
             at src/libpanic_unwind/lib.rs:102
  14: std::panicking::update_count_then_panic
             at src/libstd/panicking.rs:289
             at src/libstd/panic.rs:398
             at src/libstd/rt.rs:58
  15: std::rt::lang_start
             at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libstd/rt.rs:74
  16: build_script_build::main

OS X 10.14.3
Stable branch; cargo 1.32.0 (8610973aa 2019-01-02); rustc 1.32.0 (9fda7c223 2019-01-16)

Builder should probably use slices instead of Vec

https://github.com/google/rspirv/blob/master/rspirv/mr/builder.rs#L270

    pub fn entry_point<T: Into<String>>(&mut self,
                                        execution_model: spirv::ExecutionModel,
                                        entry_point: spirv::Word,
                                        name: T,
                                        interface: Vec<spirv::Word>) {
        let mut operands = vec![mr::Operand::ExecutionModel(execution_model),
                                mr::Operand::IdRef(entry_point),
                                mr::Operand::LiteralString(name.into())];
        for v in interface {
            operands.push(mr::Operand::IdRef(v));
        }

        let inst = mr::Instruction::new(spirv::Op::EntryPoint, None, None, operands);
        self.module.entry_points.push(inst);
    }

Is there a reason why every 'list' is of type interface: Vec<spirv::Word>? I have looked a bit though your code base and it seems every occurrence can be replaced with a slice like interface: &[spirv::Word].

Structured OpPhi representation

This is how I understand OpPhi so far:

1. It's only allowed at the start of a block, could be multiple instructions of this type.
2. Every OpPhi carries a list of pars (source_block_id, value), where:
   • source_block_id is the basic block we get (to the current one) from
   • value - the value to select in case we got here from this source basic block
3. There must be exactly as many pairs (in each OpPhi instruction in the current block) as there are "jumps" to this block.
4. Result of OpPhi can be used in other operations inside this basic block, like any other results

Let's see which of these properties we can express better with the type system in the structured representation.

First of all, it would be great to have OpPhi instructions explicitly being a part of the BasicBlock and not just be variants of Op in the list of operations in a basic block.

Secondly, with the way it's represented in DR, it's easy to make a mistake by missing one of the pieces upon modifying another one. For example, adding a jump to a block requires one to modify all OpPhi instructions of this block to take this jump into account. It seems that moving that responsibility to a jump itself would then make it easy to write correct code, i.e.:

The basic block could know about a list of inputs it receives. Each input can probably just be described by its type:

struct BasicBlock {
  pub inputs: Vec<Token<Type>>,
  ...
}

Whenever there is a branch or jump to this basic block, instead of it being Token<BasicBlock>, we could bundle it with all the inputs required to jump there, ending up with something like (Token<BasicBlock>, Vec<Token<Variable>>). Each element in this variable vector would then have to correspond exactly to an element (with the same index) of inputs list in the basic block. Effectively what this gives us is that a basic block does no longer need to know where the code jumped from, it only knows what it needs to define its own logic.

Speaking of the basic block logic, there needs to be a way to refer to the OpPhi results. We could achieve this declaring the Phi variant specifically to just reference an input in the inputs list of the basic block:

enum Op {
  Phi {
    input_index: u8,
  },
  ...
}

@antiagainst how does this approach sound to you?

Update to Rust2018 edition

Duplicated code in Disassemble

From where is the code duplicated?
https://github.com/gfx-rs/rspirv/blob/master/rspirv/binary/disassemble.rs#L146-L170

0.5.x doesn't compile anymore

   Compiling rspirv v0.5.4
error[E0599]: no variant or associated item named `DecorateStringGOOGLE` found for type `spirv::Op` in the current scope
   --> /home/maik/.cargo/registry/src/github.com-1ecc6299db9ec823/rspirv-0.5.4/grammar/table.rs:433:11
    |
433 |     inst!(DecorateStringGOOGLE, [], [(IdRef, One), (Decoration, One)]),
    |           ^^^^^^^^^^^^^^^^^^^^
    |           |
    |           variant or associated item not found in `spirv::Op`
    |           help: there is a variant with a similar name: `DecorateString`

error[E0599]: no variant or associated item named `DecorateStringGOOGLE` found for type `spirv::Op` in the current scope
  --> /home/maik/.cargo/registry/src/github.com-1ecc6299db9ec823/rspirv-0.5.4/grammar/reflect.rs:53:20
   |
53 |         spirv::Op::DecorateStringGOOGLE |
   |                    ^^^^^^^^^^^^^^^^^^^^
   |                    |
   |                    variant or associated item not found in `spirv::Op`
   |                    help: there is a variant with a similar name: `DecorateString`

error[E0308]: mismatched types
   --> /home/maik/.cargo/registry/src/github.com-1ecc6299db9ec823/rspirv-0.5.4/mr/constructs.rs:190:22
    |
190 |             version: (spirv::MAJOR_VERSION << 16) | (spirv::MINOR_VERSION << 8),
    |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected u32, found u8

error[E0599]: no variant or associated item named `DecorateStringGOOGLE` found for type `spirv::Op` in the current scope
  --> /home/maik/.cargo/registry/src/github.com-1ecc6299db9ec823/rspirv-0.5.4/mr/build_annotation.rs:62:56
   |
62 |         let mut inst = mr::Instruction::new(spirv::Op::DecorateStringGOOGLE, None, None, vec![mr::Operand::IdRef(target), mr::Operand::Decoration(decoration)]);
   |                                                        ^^^^^^^^^^^^^^^^^^^^
   |                                                        |
   |                                                        variant or associated item not found in `spirv::Op`
   |                                                        help: there is a variant with a similar name: `DecorateString`

error: aborting due to 4 previous errors

There was a breaking change in spirv_headers 1.3 to 1.4 where DecorateStringGOOGLE got renamed.

Temporary workaround is to fix the cargo.lock file to point to 1.3.4 instead.

If instead we had specified the version string as ^1.0, cargo should update to 1.1 if it is the latest 1.y release, but not 2.0

unwrap() on ConsumerError(DetachedInstruction)

I'm trying to parse a SPIR-V file generated by DXC, but getting error on parsing. Not sure if this is a rspirv or DXC issue.

extern crate rspirv;

fn main() {
    let spirv = include_bytes!("shader.spirv");

    let mut loader = rspirv::mr::Loader::new();
    rspirv::binary::parse_bytes(&spirv[..], &mut loader).unwrap();
    let module = loader.module();
}

shader.spirv (disassembled)

; SPIR-V
; Version: 1.0
; Generator: Google spiregg; 0
; Bound: 24
; Schema: 0
               OpCapability Shader
          %1 = OpExtInstImport "GLSL.std.450"
               OpMemoryModel Logical GLSL450
               OpEntryPoint Vertex %vs "vs" %in_var_A %gl_Position
               OpSource HLSL 600
               OpName %vs "vs"
               OpName %in_var_A "in.var.A"
               OpDecorate %gl_Position BuiltIn Position
               OpDecorate %in_var_A Location 0
       %void = OpTypeVoid
          %6 = OpTypeFunction %void
      %float = OpTypeFloat 32
    %v4float = OpTypeVector %float 4
%_ptr_Input_v4float = OpTypePointer Input %v4float
%_ptr_Output_v4float = OpTypePointer Output %v4float
    %float_1 = OpConstant %float 1
       %bool = OpTypeBool
   %in_var_A = OpVariable %_ptr_Input_v4float Input
%gl_Position = OpVariable %_ptr_Output_v4float Output
         %vs = OpFunction %void None %6
         %13 = OpLabel
         %14 = OpLoad %v4float %in_var_A
         %15 = OpCompositeExtract %float %14 0
         %16 = OpFOrdEqual %bool %15 %float_1
               OpSelectionMerge %17 None
               OpBranchConditional %16 %18 %19
         %18 = OpLabel
         %20 = OpExtInst %v4float %1 FAbs %14
         %21 = OpFNegate %v4float %20
               OpBranch %17
         %19 = OpLabel
         %22 = OpVectorShuffle %v4float %14 %14 3 2 1 0
               OpBranch %17
         %17 = OpLabel
         %23 = OpPhi %v4float %21 %18 %22 %19
               OpStore %gl_Position %23
               OpReturn
               OpFunctionEnd

shader.hlsl

struct VSInput
{
    float4 a : A;
};

float4 vs(VSInput input) : SV_Position
{
    if (input.a.x == 1.0) {
	return -abs(input.a);
    } else {
        return input.a.wzyx;
    }
}

stacktrace

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ConsumerError(DetachedInstruction)', libcore\res
ult.rs:945:5
stack backtrace:
   0: std::sys::windows::backtrace::set_frames
             at C:\projects\rust\src\libstd\sys\windows\backtrace\mod.rs:104
   1: std::sys::windows::backtrace::set_frames
             at C:\projects\rust\src\libstd\sys\windows\backtrace\mod.rs:104
   2: std::sys_common::backtrace::_print
             at C:\projects\rust\src\libstd\sys_common\backtrace.rs:71
   3: std::sys_common::backtrace::_print
             at C:\projects\rust\src\libstd\sys_common\backtrace.rs:71
   4: std::panicking::default_hook::{{closure}}
             at C:\projects\rust\src\libstd\panicking.rs:211
   5: std::panicking::default_hook
             at C:\projects\rust\src\libstd\panicking.rs:227
   6: std::panicking::rust_panic_with_hook
             at C:\projects\rust\src\libstd\panicking.rs:475
   7: std::panicking::continue_panic_fmt
             at C:\projects\rust\src\libstd\panicking.rs:390
   8: std::panicking::rust_begin_panic
             at C:\projects\rust\src\libstd\panicking.rs:325
   9: core::panicking::panic_fmt
             at C:\projects\rust\src\libcore\panicking.rs:77
  10: core::result::unwrap_failed<rspirv::binary::parser::State>
             at C:\projects\rust\src\libcore\macros.rs:26
  11: core::result::Result<(), rspirv::binary::parser::State>::unwrap<(),rspirv::binary::parser::State>
             at C:\projects\rust\src\libcore\result.rs:782
  12: simple::main
             at .\examples\simple.rs:7
  13: std::rt::lang_start::{{closure}}<()>
             at C:\projects\rust\src\libstd\rt.rs:74
  14: std::rt::lang_start_internal::{{closure}}
             at C:\projects\rust\src\libstd\rt.rs:59
  15: std::rt::lang_start_internal::{{closure}}
             at C:\projects\rust\src\libstd\rt.rs:59
  16: panic_unwind::__rust_maybe_catch_panic
             at C:\projects\rust\src\libpanic_unwind\lib.rs:105
  17: std::panicking::try
             at C:\projects\rust\src\libstd\panicking.rs:289
  18: std::panicking::try
             at C:\projects\rust\src\libstd\panicking.rs:289
  19: std::panicking::try
             at C:\projects\rust\src\libstd\panicking.rs:289
  20: std::rt::lang_start<()>
             at C:\projects\rust\src\libstd\rt.rs:74
  21: main
  22: invoke_main
             at f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:78
  23: invoke_main
             at f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:78
  24: BaseThreadInitThunk
  25: RtlUserThreadStart

Build not dependent on code generation

The way the code generation is currently set up means that it is possible to change the generation code, not build it and end up with the project in an inconsistent state. While this situation is unlikely, it would be really hard to spot (although it would be easy to fix). There are a few ways I can think to deal with this:

1. Document it, simplest and easiest, this problem is unlikely to come up so adding it to the contribution guidelines and/or a PR template would probably be enough.
2. Add a CI check to make sure that the source doesn't change, this is (probably) foolproof and doesn't require any restructuring although I have no idea how easy it would be to do.
3. Restructure the code generation to generate it during the build process. This is the most complex option and could slow down downstream compilations but has the added benefit of getting rid of the weird triple build you have to do if you modify the autogen. It would require some care so that it builds correctly when used as a dependency but it is probably possible.

I was leaning towards (3) (mostly because I mostly motivated by wanting to stop the triple build) when I started to write this but it has some quite strong downsides that would need to be investigated first.

Example from docs doesn't validate

extern crate rspirv;
extern crate spirv_headers as spirv;

use rspirv::binary::Assemble;
use rspirv::binary::Disassemble;

fn main() {
    // Building
    let mut b = rspirv::mr::Builder::new();
    b.memory_model(spirv::AddressingModel::Logical, spirv::MemoryModel::GLSL450);
    let void = b.type_void();
    let voidf = b.type_function(void, vec![void]);
    b.begin_function(void,
                     None,
                     (spirv::FunctionControl::DONT_INLINE |
                      spirv::FunctionControl::CONST),
                     voidf)
     .unwrap();
    b.begin_basic_block(None).unwrap();
    b.ret().unwrap();
    b.end_function().unwrap();
    let module = b.module();

    // Assembling
    let code = module.assemble();
    assert!(code.len() > 20);  // Module header contains 5 words
    assert_eq!(spirv::MAGIC_NUMBER, code[0]);

    // Parsing
    let mut loader = rspirv::mr::Loader::new();
    rspirv::binary::parse_words(&code, &mut loader).unwrap();
    let module = loader.module();
    use std::io::Write;
    let mut f = std::fs::File::create("out.frag").unwrap();
    let len = code[..].len() * 4;
    let buf = unsafe { std::slice::from_raw_parts(code[..].as_ptr() as *const u8, len) };
    f.write(buf);
}

gives the following error when spirv-val out.frag is run.

error: line 1: Operand 2 of MemoryModel requires one of these capabilities: Shader
  OpMemoryModel Logical GLSL450

Builder decorate methods don't have a way to specify extra decorator operands

https://github.com/google/rspirv/blob/a5a5d58385c7495df16407cb566cc51efbbd7213/rspirv/mr/build_annotation.rs#L21-L24

See https://www.khronos.org/registry/spir-v/specs/1.1/SPIRV.html#OpDecorate and https://www.khronos.org/registry/spir-v/specs/1.1/SPIRV.html#Decoration

Many decorations require additional operands. Strange that they don't include the extra operands in the SPIR-V grammar json. In particular, I'm trying to decorate a variable with the GlobalInvocationId BuiltIn.

Move OpPhi into BasicBlock

The spec says that OpPhi has to appear within a block before any other instruction (except for "OpLabel", which is already hard-wired to the BasicBlock). Would it make sense to represent it as this?

/// Data representation of a SPIR-V basic block.
#[derive(Debug, Default)]
pub struct BasicBlock {
    /// The label starting this basic block.
    pub label: Option<Instruction>,
    /// Variable control flow control for this basic block.
    pub phi: Option<Instruction>,
    /// Instructions in this basic block.
    pub instructions: Vec<Instruction>,
}

OperandExceeded when using specialization constants in work group size

Context

Given the following GLSL compute shader which defines its work group size using specialization constant 0:

#version 460

layout(local_size_x_id = 0) in;

void main() {
    int kernelId = int(gl_WorkGroupSize.x);
}

Compiling it using shaderc result in the following (disassembled using spirv-dis) SPIR-V file:

; SPIR-V
; Version: 1.0
; Generator: Google Shaderc over Glslang; 7
; Bound: 17
; Schema: 0
               OpCapability Shader
          %1 = OpExtInstImport "GLSL.std.450"
               OpMemoryModel Logical GLSL450
               OpEntryPoint GLCompute %main "main"
               OpExecutionMode %main LocalSize 1 1 1
               OpSource GLSL 460
               OpSourceExtension "GL_GOOGLE_cpp_style_line_directive"
               OpSourceExtension "GL_GOOGLE_include_directive"
               OpName %main "main"
               OpName %kernelId "kernelId"
               OpDecorate %10 SpecId 0
               OpDecorate %gl_WorkGroupSize BuiltIn WorkgroupSize
       %void = OpTypeVoid
          %3 = OpTypeFunction %void
        %int = OpTypeInt 32 1
%_ptr_Function_int = OpTypePointer Function %int
       %uint = OpTypeInt 32 0
         %10 = OpSpecConstant %uint 1
     %uint_1 = OpConstant %uint 1
     %v3uint = OpTypeVector %uint 3
%gl_WorkGroupSize = OpSpecConstantComposite %v3uint %10 %uint_1 %uint_1
     %uint_0 = OpConstant %uint 0
         %15 = OpSpecConstantOp %uint CompositeExtract %gl_WorkGroupSize 0
         %16 = OpSpecConstantOp %int IAdd %15 %uint_0
       %main = OpFunction %void None %3
          %5 = OpLabel
   %kernelId = OpVariable %_ptr_Function_int Function
               OpStore %kernelId %16
               OpReturn
               OpFunctionEnd

Expected behavior

Parsing the module using rspirv succeeds.

Observed behavior

Trying to parse the resulting binary module with rspirv results in the following error:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: OperandExceeded(436, 23)', src/libcore/result.rs:1188:5
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print_fmt
             at src/libstd/sys_common/backtrace.rs:84
   3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
             at src/libstd/sys_common/backtrace.rs:61
   4: core::fmt::write
             at src/libcore/fmt/mod.rs:1025
   5: std::io::Write::write_fmt
             at src/libstd/io/mod.rs:1426
   6: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:65
   7: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:50
   8: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:193
   9: std::panicking::default_hook
             at src/libstd/panicking.rs:210
  10: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:471
  11: rust_begin_unwind
             at src/libstd/panicking.rs:375
  12: core::panicking::panic_fmt
             at src/libcore/panicking.rs:84
  13: core::result::unwrap_failed
             at src/libcore/result.rs:1188
  14: core::result::Result<T,E>::unwrap
             at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libcore/result.rs:956
  15: rspirv_bug::main
             at src/main.rs:6
  16: std::rt::lang_start::{{closure}}
             at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libstd/rt.rs:67
  17: std::rt::lang_start_internal::{{closure}}
             at src/libstd/rt.rs:52
  18: std::panicking::try::do_call
             at src/libstd/panicking.rs:292
  19: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:78
  20: std::panicking::try
             at src/libstd/panicking.rs:270
  21: std::panic::catch_unwind
             at src/libstd/panic.rs:394
  22: std::rt::lang_start_internal
             at src/libstd/rt.rs:51
  23: std::rt::lang_start
             at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libstd/rt.rs:67
  24: main
  25: __libc_start_main
  26: _start
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Affected versions

• Current state of the master branch (#3905855)
• Latest version published on crates.io (0.5.4)

How to reproduce the bug

See example code at https://github.com/vtavernier/rspirv-bug. Clone and use cargo run to check.

Is rspirv too abstracted?

rspirv is a binary to AST parser, but I'm not sure it looks like one any more. To me, it looks a lot larger and more complex than I would expect, just from an overhead PoV and also knowing how much it actually does atm.

The current complexity of the application makes a lot of things that it should be doing awkward and convoluted. The current error handling story is a bit of a mess (getting a list of issues that occurred would be hard to get working), mostly just because of how layered everything is.

The autogen is also a problem. The level of metaprogramming trying to get it to output something you want seems silly. To me for the changes we've made in the past few months have been more suited to sed than to rust, and we aren't exactly expecting the SPIR-V spec to be changing wildly over the years to come.

After looking through each high-level part of the application (sr, dr, binary) I'm generally happy with how each bit is structured. It's just when viewing the application as a whole that something seems off, which is what has led me to believe that the project is just currently too abstracted.

I've been looking at all these as separate issues for a while but, after stepping back, I think we might need for of a shift in mindset (and maybe get a bit more free with the delete key!) to really tackle these issues. I hope no-one takes me the wrong way with this, I really like rspirvs goal and I truly do think that each individual bit is well designed for its own goal, it's just the project as a whole that has been eating away at me recently.

Derive serialization for the types in spirv-headers

Would be great to have an optional serde feature, which makes all the types from headers to derive serialization.

Case against is_type() methods

There is a large auto-generated section looking like:

impl Type {
  pub fn is_foo(&self) -> bool {...}
  pub fn is_bar(&self) -> bool {...}
  ...
}

It enumerates each and every enum variant.

What is the motivation behind this API? I think we could live without it. The more complex (and semantically rich) checks would stay and just pattern match internally:

impl Type {
    pub fn is_numerical_type(&self) -> bool {
    }
    pub fn is_scalar_type(&self) -> bool {
    }
    pub fn is_aggregate_type(&self) -> bool {
    }
    pub fn is_composite_type(&self) -> bool {
    }
}

Testing with real SPIR-V data

We need a mechanism for testing (locally and on CI) the binary parser and DR -> SR conversion on some real data. Preferably, without including any actual SPIR-V modules in the repository, as requested by @antiagainst .

`Assemble::assemble(&self) -> Vec<u32>` always allocate heap memory

Codegen for extinst.glsl.std.450.grammar.json

https://github.com/KhronosGroup/SPIRV-Headers/blob/master/include/spirv/1.2/extinst.glsl.std.450.grammar.json

Should this also be included in the spirv-headers?

DR and SR duplication and merging

Currently, SR is built on top of DR, as DR is already written and it provides a useful and productive base to work on. However, SR may be simpler as a parser from binary blob form, so if SR is requiring large scale changes in DR code then it is worthwhile considering merging the two at that point to reduce duplicated effort.

Type inference and updates

If at any point you or someone else has implemented any form of type inference, automatic type updates, or large scale type checking, then it is worth considering that it could replace the current store of each operations type. Most operations perform simple type-level manipulations of input to output, so module/function level type inference is feasible. It was deemed simpler to keep the type of each operation inline, however, any code that performs this sort of function could change this calculus so it should be reconsidered.
Supersedes #103, see that for extra context.

Name of the crate

First off, this isn't exactly an issue per se.

I was looking through my crates on crates.io and discovered I had a crate from 2 years ago named spirv - which I created just to support a hobby project before there were any other SPIR-V libraries in Rust. I haven't maintained it in ages and the code/features/testing/etc are quite obviously inferior to the fantastic work being done in this crate. So basically my question is, since I'm going to mark that as 'abandoned' and suggest that nobody should use it, would another crate (eg: this one) like to take that spirv name?

I realise this is perhaps a dumb question, since changing names on crates.io would force all the users of rspirv to change their code, however I guess I'm just letting you know in case somebody feels that spirv is a 'nicer' or more canonical name to use.

Edit: I guess I'm also looking for rust-lang/rust#41616

Autogen customisation and value

Currently, lots of bits of rspirv are generated from the JSON grammar, this is to allow fast and correct updating and bulk feature addition. However, as rspirv grows more and more of this will need to be customised, some of these may be able to be upstreamed into the grammar file, some will not. The goal of this issue is to make sure that the cost of these customisations doesn't exceed the value of the autogen. If that does happen then the auto-generated code will be inlined and all updates will need to be provided manually.

Below is a list of all of the current customisations to auto-generated code, feel free to edit the list if you spot, or add, some more.

• spirv.core.grammar.json: Branch instructions have their class renamed from Terminator to Branch (should be upstreamed)
• spirv.core.grammar.json: OpPhi's third operand renamed from 'Variable, Parent, ...' to 'ValueLabelPairs' (could be upstreamed?)
• OpPhi in general is treated uniquely
• autogen/binary: LiteralInteger has special method name handling
• autogen/binary: All possible pair kinds are hardcoded
• autogen/binary: "IdResultType", "IdResult", "LiteralContextDependentNumber", "LiteralSpecConstantOpInteger" are all handled manually
• autogen/binary: NaN has dodgy snake_casing rules
• autogen/dr: IdResult treated separately
• autogen/dr: OpReturn and OpReturnValue have special function names (as return is a keyword)
• autogen/dr: LiteralString has different get_init_list handling
• autogen/dr: Pair types treated separately again, in get_push_extras
• autogen/dr: All number kinds hardcoded
• autogen/dr: Pointer types filtered out in gen_dr_builder_types
• autogen/header: NaN has special snake casing rules
• autogen/header: Dim types have Dim prepended as types can't start with numbers
• autogen/sr: Lots of operand types have special name and typing rules
• autogen/sr: Decoration ops treated completely separately
• autogen/sr: gen_sr_code_from_instruction_grammar skips constants
• autogen/sr: gen_sr_code_from_instruction_grammar has special casing for types, function ops, branches, terminators and phi instructions
• autogen/util: Literal numbers get unique operand kind
• autogen/util: underlying type for pairs, literals and ids are all special
• autogen/util: Parameter name for IdResultType hardcoded

Javelin requirements

Javelin is the project of gfx-rs to get the shader translation done in pure Rust, announced on the blog some time ago. We want to implement it using the Structured Representation of #5 . Here is a rough list of use cases we need rspirv to support:

1. Load an SPIR-V module and valide that it's valid, gracefully error handling invalid modules
2. Inspect and modify the bindings (e.g. split combined image-samplers)
3. Inspect and modify the global variables (e.g. poor man's specialization)
4. Analyze the uniform-ity of the control flow within any scope
5. Perform generic optimization passes
6. Insert bound checks on SSBO and image access
7. Save result SPIR-V module from the modified SR
8. Generate shaders in other languages, both textual and binary

Structured representation

Currently there is only a plain data representation (DR) implemented in this project. While it introduces little overheads when working with the binary parser and is straightforward for investigating the contents of SPIR-V binaries, it has quite limited functionalities. Instructions are not interconnected; types are not hierarchicalized; metadata (decorations, debug instructions, etc.) are not linked to the targets; etc. It is inconvenient to build future tasks like validation, optimization, and translating from front end languages on top of this DR. So a structured representation (SR) should be introduced, containing:

• A type hierarchy
• A constant hierarchy
• Common language constructs: Module, Function, BasicBlock, Instruction, etc.
• Representing all SPIR-V instructions
• Decoration struct. This will be attached to types, instructions, etc.

Type hierarchy

We can either implement it using 1) enums or 2) marker traits with multiple implementing structs.

1. Types are frequently used, it would be nice to reduce the overheads. This favors enums since marker traits leads to trait objects.
2. It reads better and is more easy to use (without excessive matchs). Also easy to attach data and methods (for like, getting constants from a type, composing types, etc.) on each type.

I'm leaning towards 2. Let's worry about performance later.

// sketch

trait Type {
  is_numerical(&self) -> bool;
  is_aggregate(&self) -> bool;
  is_composite(&self) -> bool;
  ...
}
trait ScalarType { ... }
trait AggregateType { ... }

struct FloatType {
  bitwidth: u32,
}
struct VectorType {
  element_type: &ScalarType,
  size: u32,
}
struct ArraryType { ... }

impl ScalarType for IntType { ... }
impl AggreateType for VectorType { ... }

Representing SPIR-V instructions

A single flexible instruction struct that can represent all SPIR-V instructions is inferior to a dedicated instruction struct for each SPIR-V instruction: it loses the structure naturally encoded in dedicated instruction struct and will need validation for operand layout correctness, etc. A huge enum containing all possible instructions is ugly. It seems there is no way to go around trait objects, which is avoided by the Rust community to the best. But it's the natural way of fulfilling this kind of task.

// Sketch
 
trait Instruction {
  result_id(&self) -> Option<u32>;
  result_type(&self) -> Option<&Type>;
  ...
}

struct IAddInst {
  result_type: &Type,
  result_id: u32,
  op1: &Instruction,
  op2: &Instruction
}

impl Instruction for IAddInst { ... }

There are quite some instructions to implement, but most of them can be generated from the grammar.

Decoration struct

Decorations can have associated data.

Special care should be taken when handling struct types since decorations on struct members cannot penetrate to the underlying base types.

Parse module from VkShaderModuleCreateInfo

[VkShaderModuleCreateInfo](https://www.khronos.org/registry/vulkan/specs/1.0/man/html/VkShaderModuleCreateInfo.html) allow us to get a &[u32]. How can I parse it with rspirv? load only supports Vec<u8>.
For the record, I'm working on https://github.com/Yamakaky/cendre.

Publish 1.3.5 of spirv-headers

Now that #97 and #100 landed I would love to be able to depend on spiv-headers proper again. Would you mind publishing a new version?

Scope of rspirv

For cendre, I'll have to execute the shaders in SPIR-V. Do you want to do something in common or is it out of the scope of what you want to do?
One possibility would be to convert SPIR-V to LLVM IR, run a JIT compiler then execute it. Maybe with a wrapper main which takes the uniforms, inputs, outputs... and calls the real shader main.

Port "mr" generators to tokens

The whole "codegen/mr.rs" is implemented to emit strings. This doesn't look nearly as tidy as quote!-based workflow of "codegen/sr.rs". We should make those consistent.

Design Questions

I've been looking around the codebase so that I can begin to make contributions, however, I have a few questions about the design that aren't immediately obvious:

1. Why are the data and structured representations seperate? As far as I can see sr is intended to be strictly stronger than mr.
2. Are there some specific use cases in mind for this library? I'm coming from gfx-rs so I am thinking about javelin, the dependents listed on crates.io seem to be constructing SPIR-V modules by hand, and there are a few issues with some specific requests, are there any other cases that are influencing the API design?
3. How much validation is the parser aiming to do? Some of the spec probably needs to be left to a validation pass, but there are some bits (i.e. getting the right magic number) that the parser essentially has to do.

Thanks!

Make a pull request template

We need to make the commit message guidelines more visible, see #63 (comment)