When using StatelessExecutor, llama_new_context logs to console every time InferAsync is called about llamasharp

[martindevans]

I believe you can do something like this:

NativeApi.llama_log_set((level, message) =>
{
    // This will be called when llama.cpp wants to log a message. Do whatever you like!
    Console.WriteLine($"[{level}]: {message}");
});

[AsakusaRinne]

The reason seems to be that every time calling InferAsync, there's a context loaded. Thank you for reporting us this BUG, we'll fix it soon.

[chatbuildcontact]

Newbie question.. Is there a way to prevent LlamaSharp/llama.ccp from logging these values to Console? I have read the docs but I wasnt able to answer the question myself.

[chatbuildcontact]

I believe you can do something like this:

NativeApi.llama_log_set((level, message) =>
{
    // This will be called when llama.cpp wants to log a message. Do whatever you like!
    Console.WriteLine($"[{level}]: {message}");
});

Thanks for that pointer!

[martindevans]

we'll fix it soon.

What's the planned fix here? As far as I'm aware we want to create a new context every time, so that it's truly stateless. Or should we try to re-use the context, but clean up all the state between inferences?