Why is the calculation result of tensorrt-llm version llava1.5 different from the output of HF? about tensorrt-llm

[byshiue]

Due to different kernel selection and kernel implementation, it often generates different results. Unless there are obvious accuracy regression, we think it is reasonable.

[bleedingfight]

@byshiue I have seen a significant decrease in the accuracy of the output results of TRT on my test set. I would like to know how you have determined that the output of TRT is reasonable.

[byshiue]

We use mmlu and summarization task to evaluate. Could you try reproducing the accuracy on public model and public example, sharing your reproduced steps and let us be easier to reproduce your issue?

[bleedingfight]

@byshiue Thank you very much for your reply. I'm sorry for the delayed reply. In the past few days, I have been trying to provide a Docker and minimum reproduction code. For the convenience of reproduction, I have upgraded to the latest TRT-LLM(0.11.0.dev2024051400). Now my previous code cannot produce results.
look like this #1632

[byshiue]

temperature 0.0 is not a valid number in current TensorRT-LLM. Please use greedy search and don't set temperature directly, or set a very small temperature like 1e-6.

[bleedingfight]

@byshiue oI modified temperature=1e-6 according to your statement, but I found that errors occur at all times except for the first inference that produces output，self.tokenizer.batch_decode(output_ids[0, :, input_lengths[0] :])will produce error:

like this #1299 。but my trt-version=0.11.

[byshiue]

Thank you for the try. Could you share the full end to end steps to reproduce your issue? (how to convert ckpt, building engine and run the example)

[bleedingfight]

@byshiue here

[byshiue]

Could you share the end to end steps to reproduce your issue? It is hard for me to understand how to use the scripts you share.

[bleedingfight]

@byshiue readme.md in zip.I just start webserver for llava-trt to process the result