Llama-2 output from the forward function is nonsense. Output from `.generate()` is okay about transformers

[bhuvanmdev]

@Tai-Mai , shouldn't the code be model_output.logits.squeeze()[-1]? This is because the forward method predicts the last token based on all the preceding ones. To replicate the .generate() method's output, you'd likely need to loop the original tokens with the newly predicted token. Do let me know if I'm wrong. Also, for generic model training, Hugging Face provides a Trainer class .

[Tai-Mai]

@bhuvanmdev Thanks for the reply! Yes, you're right. I was confused because I got multiple tokens (nobody, I are you? I) and I thought that meant that the forward function was already implemented in a way that would take care of the auto-regressive generation for me but that's of course not the case.

Here's the forum post that helped me understand it more as well.

https://discuss.huggingface.co/t/llama-2-output-from-forward-function-is-nonsense-generate-is-okay/88815/2?u=tai-mai

I'll close the issue. Thanks again.