[Bug]: Unexpected Special Tokens in prompt_logprobs Output for Llama3 Prompt about vllm

[leejamesss]

#4688 might solve your issue, can you try that out?

Thank you for your prompt attention to the issue. However, upon careful consideration, it seems that the problem we are facing is distinct from the one addressed in issue #4688. The concern raised in #4688 is related to an additional repetition of the Beginning of Sequence (BOS) token, which is particularly relevant in the context of API interactions. In contrast, our scenario involves offline inference and does not exhibit the same pattern of BOS token repetition as described in the aforementioned issue.
Therefore, the solution or workaround suggested in issue #4688 may not be directly applicable to the problem we are encountering with the prompt_logprobs output and the proliferation of unexpected special tokens.

I appreciate your effort to provide a potential fix, and I will continue to explore alternative solutions to resolve the unexpected token issue we are experiencing. Should there be any further insights or suggestions you can offer, they would be most welcome.

Thank you once again for your assistance with this matter.

[DreamGenX]

@DarkLight1337 This sounds related to #4577 -- something between 0.4.0.post1 and 0.4.1 changed the way tokenization works. I am for whatever reason getting back a sequence of tokens like <, <|, <|im_ etc. instead of the whole <|im_start|> at once.

[DarkLight1337]

#4688 might solve your issue, can you try that out?

[DarkLight1337]

I'm currently investigating a similar issue in #4200. It seems that there is something wrong with the detokenizing logic where new_decoded_token_text gets pre-padded with extra whitespace characters. @Yard1 @njhill do you have any idea about this?

Edit It seems that my particular issue is related to the chat template. @DreamGenX 's issue would be more relevant to this case.

vllm/vllm/transformers_utils/detokenizer.py

Line 115 in 52f8107

(new_tokens, new_decoded_token_text, prefix_offset,

[leejamesss]

Hello @DarkLight1337, @DreamGenX, and everyone involved in this discussion,

Thank you for your ongoing investigation into the tokenization and detokenization logic within the vLLM project. I understand that there may be related issues, such as #4200 and #4577, which are being looked into.

However, for the issue at hand, which is #4772, I would like to clarify that our primary concern is not with the detokenization process or the formatting of the output. We are not using the detokenized text and, therefore, any irregularities in the detokenization are not relevant to our use case.

Our focus is on the correctness of the prompt_logprobs output. We are encountering a significant number of unexpected special tokens in the log probability dictionary, which is causing errors in our downstream processing. The presence of these tokens is not expected and is interfering with the intended functionality of the LLM model.

To reiterate, we need assistance in ensuring that the prompt_logprobs output is accurate and free from these unexpected special tokens for both the Llama3 and Llama2-13b-chat-hf models.

If there are any updates, insights, or suggestions on how to address this specific issue with the log probabilities, we would greatly appreciate the guidance.

Thank you for your attention to this matter, and we look forward to a resolution.

Best regards,
@leejamesss