xenowaza / rusty-deploy

MLOps Deploy Solutions with Rust

Rust Deployment Experiments

My hypothesis is RUST is the ideal language for LLMOPs, NOT PYTHON.

Workflows to Prove Out for Rust MLOps Deployment

Distroless

Rust T5 Model

PyTorch Distroless Pre-Trained Model

• Package PyTorch into a Rust Distroless Container

Hugging Face Distroless Pre-Trained Model

Binary Deployment

• Statically Linked ONNX to Rust Binary
• Statically Linked PyTorch to Rust Binary
• Statically Linked HuggingFace to Rust Binary

References

• huggingface/text-generation-inference a Rust example