maekawatoshiki / altius

Small ONNX inference runtime written in Rust

uint256_t

altius's Issues

Publish to crates.io

Hello!

was looking to play with this as part of a larger rust program. Is it possible to publish the core to crates.io so I can import the inference part of this and build my own tooling around it?

Add more optimizers

• MatMul+Add fusion
  • MatMul+Add → Gemm
    • 
• Reshape+Transpose fusion
  • 
• Automatic fusion for element-wise operations

Support ops in core

core support completed (interpreter not yet):

• LeakyRelu
• Resize
• Concat
• Transpose
• Squeeze
• Div
• ReduceMin
• Round
• Sub
• Loop
• Tile
• Cast
• Slice
• Exp
• Sigmoid
• Unsqueeze
• NonMaxSuppression

Requirements should be expressed in README.

Hi, I tried this repo, looks nice!

I think you had better state explicitly in README that installation of git-lfs is required to clone and try this repo properly.

Support more ops in interpreter

Supported kernels:

• LeakyRelu
• Resize
• Concat
• Transpose
• Squeeze
• Div
• ReduceMin
• Round
• Sub
• Loop
• Tile
• Cast
• Slice
• Exp
• Sigmoid
• Unsqueeze
• NonMaxSuppression

Support dynamic shapes

Add topo sort that achieves minimal memory consumption

Refine tests for core

Support FastViT

• initial support (model: https://huggingface.co/timm/fastvit_s12.apple_in1k)
• improve performance

Fix bench

Implement `ReduceMax` kernel on CPU backend

Add CPU backend

• CPU backend generates C source code, compiles it, and runs it.
• Compared to Interpreter backend, kernel fusion is possible on CPU backend.

---

The following examples have been tested and work well:

• mnist
• deit
• mobilenet
• vit
• deeplab
• fcn
• yolov5
• fugumt
• gpt-2

Add BERT example written in Rust

Add `FastGelu` fusion

Fast `softmax` kernel

• We need a better implementation (for CPU backend) for softmax.

runtime	softmax in gpt2 (ms)
onnxruntime	1.5
altius	2.4

Implement NCHWc Convolution

Refactor `Op::compute_output_shapes()`

It seems to be ideal for the function to have a signature such as:

impl Model {
  fn compute_output_shapes(&self, node: &Node, inputs: &[&Tensor]) ->
    Result<(Op, Vec<TypedFixedShape>)>;
}

[CPU] Add more convenient way to generate C code

Add a useful method for modifying graphs

• Create struct GraphModifier or something like that, to easily implement graph modification code.

Support Identity and Average Pooling Ops

Run tests on cpu backend

• altius/crates/altius-py/tests/test_ops_elemwise.py

  Line 165 in efd8fd3

  altius_sess = altius_py.InferenceSession(filepath)

Support opset version

Current implementation does not care about onnx opset version.

Implement a specialized memory allocator

Implement operators in the CPU backend

Part of #10

• Expand
• Sigmoid (with SIMD exp)
• Resize
• Split (opset < 13)