Getting exception: "llama_decode failed: 'NoKvSlot'" when LLM analyze text (news) about llamasharp

[martindevans]

I'll close this issue since it looks like it's been resolved :)

[MagicMaxxx]

PROBABLY (!!) the reason of that exception is that that every news row started with "Дата:" (which is "Date:" in Russian) and LLM (?) decides that it's a kind of Role... So when I added minus before the Date ("- Дата:") then I can't reproduce the exception... Checking it...

[MagicMaxxx]

I've got this exception again, even if I used "minus" signs before the every news row.

In StatelessExecutor, in the line 81, it comes from LlamaContext:
DecodeResult decodeResult = await Context.DecodeAsync(_batch, cancellationToken);
if (decodeResult != 0)
{
throw new LLamaDecodeError(decodeResult);
}

[martindevans]

The NoKvSlot error means that there's not enough memory to hold all of the tokens you're trying to submit to the system.

You'll probably need to reduce the amount of text you are submitting in the initial prompt.

[MagicMaxxx]

ok, got it, thanks for the info!

I increased ModelParams.ContextSize 1024 -> 4096, checking it now (BTW - will it help? I've just started learning LLMs and LLamaSharp)... The prompt I used before had length of 2009 symbols, and if my assumption is right, it caused these exceptions

UPD - this is for llama-2-7b-chat.Q4_K_M, I can see that Mistral-based LLMs can have 8K as the context size, will check it later.

[martindevans]

There are 2 "contexts" here, which means different things. There's actually a PR in llama.cpp to change the naming, because it's so confusing!

There is a pre-allocated buffer (called the KV cache) which holds information about each token. That's what's set by the ModelParams.ContextSize parameter. If you had this set to 1024 then that will definitely cause an issue with trying to fit in 2009 tokens.

I can see that Mistral-based LLMs can have 8K as the context size

LLMs are trained for a certain number of positions (i.e. how many tokens they can understand, you can feed in more but this isn't something the system is trained for). That's what the 8k "context size" is here.