mistobaan / model_analyzer

Triton Model Analyzer is a CLI tool to help with better understanding of the compute and memory requirements of the Triton Inference Server models.

Triton Model Analyzer

LATEST RELEASE: You are currently on the main branch which tracks under-development progress towards the next release. The latest release of the Triton Model Analyzer is 1.22.0 and is available on branch r22.12.

Triton Model Analyzer is a CLI tool to help with better understanding of the compute and memory requirements of the Triton Inference Server models. These reports will help the user better understand the trade-offs in different configurations and choose a configuration that maximizes the performance of Triton Inference Server.

Features

• Brute and Quick search: Model Analyzer can help you automatically find the optimal settings for Max Batch Size, Dynamic Batching, and Instance Group parameters of your model configuration. Model Analyzer utilizes Performance Analyzer to test the model with different concurrency and batch sizes of requests. Using Manual Config Search, you can create manual sweeps for every parameter that can be specified in the model configuration.

• Multi-Model Search: EARLY ACCESS - Model Analyzer can help you find the optimal settings when profiling multiple concurrent models, utilizing our Quick Search alogrithm

• Detailed and summary reports: Model Analyzer is able to generate summarized and detailed reports that can help you better understand the trade-offs between different model configurations that can be used for your model.

• QoS Constraints: Constraints can help you filter out the Model Analyzer results based on your QoS requirements. For example, you can specify a latency budget to filter out model configurations that do not satisfy the specified latency threshold.

Examples and Tutorials

See our Quick Start for a guide of how to use Model Analyzer to profile, analyze and report on a simple PyTorch model.

Documentation

• Installation
• Model Analyzer CLI
• Launch Modes
• Configuring Model Analyzer
• Model Analyzer Metrics
• Model Config Search
• Checkpointing
• Model Analyzer Reports
• Deployment with Kubernetes

Reporting problems, asking questions

We appreciate any feedback, questions or bug reporting regarding this project. When help with code is needed, follow the process outlined in the Stack Overflow (https://stackoverflow.com/help/mcve) document. Ensure posted examples are:

• minimal – use as little code as possible that still produces the same problem

• complete – provide all parts needed to reproduce the problem. Check if you can strip external dependency and still show the problem. The less time we spend on reproducing problems the more time we have to fix it

• verifiable – test the code you're about to provide to make sure it reproduces the problem. Remove all other problems that are not related to your request/question.