[Bug] Token IDs not accepted by JSON grammar about mlc-llm

[tqchen]

thanks @dtkettler can you also let us know what is the model and prompt? having an example script would be helpful

[dtkettler]

Sure. I have run into this with every prompt I have tried, but here's a very simple example adapted from the basic usage in the MLC-LLM documentation:

from mlc_llm import LLMEngine

# Create engine
model = "HF://mlc-ai/Llama-3-8B-Instruct-q4f16_1-MLC"
engine = LLMEngine(model)

# Run chat completion in OpenAI API.
response = engine.chat.completions.create(
    messages=[{"role": "user", "content": """
Produce a JSON object with a list of US states and the largest cities in those states
The list should have the following keys:
'state' - Name of the state
'cities' - Another list containing city names"""}],
    model=model,
    response_format={"type": "json_object"},
    stream=False
)

print(response)

engine.terminate()

[Ubospica]

Thanks for the report @dtkettler Currently there are several issues with llama3 because it changes the tokenizer a lot. That will be fixed soon in these days

[FreakTheMighty]

I'm seeing a similar error when using phi-2-q4f16_1-MLC

InternalError: Check failed: (accepted) is false: Token id 0 is not accepted by the grammar state matcher