How can we optimize the Model Inference time. Single NLQ taking more than a 45seconds. about resdsql

[lihaoyang-ruc]

Hi, here are some suggestions:

1. Keep the model in GPU memory instead of releasing it after each use, as releasing it each time will result in a lot of I/O time.
2. Using GPU to inference instead of CPU.
3. Try to reduce the beam size during decoding.
4. Try using tools that can accelerate model inference, such as TensorRT and FasterTransformer.

However, my suggestions are quite general, so they may not necessarily address your specific issue. If you want to achieve deeper acceleration, I recommend that you try to identify the code blocks that consume the most time.