Multiple sessions on one model. about llamasharp

[martindevans]

At the moment the model weights and the context are all bound together in one object. You need to save and restore "states" to have two contexts in one set of weights. This is due to how llama.cpp itself used to work.

They've since made a change which splits model weights and model contexts into two separate things, so you can make multiple contexts from one set of shared weights. My PR (#64) partially addresses this by adding in support for the new loading system. Future PRs will modify the higher level APIs to use this.

[BruceKristelijn]

Thanks for the response. I am trying it right now but my responses seems to lose some context. I assumed that when a state is loaded / saved it retains the chat and prompt history, or am I mistaken?

[BruceKristelijn]

I tried including the chat history after loading the session again aswell but this seemed to reset the "memory" of the previous conversation aswell,

[martindevans]

Is this testing all being done on top of my PR (#64), with master or with some other version?

[BruceKristelijn]

Not yet, this was my next course of action. Was hoping I understoot the behaviour correctly first.

[martindevans]

I'm not too sure, sorry. I've been contributing PRs on some of the lower level bits of the stack but not the "higher level "stuff yet. I do know there are a few layers, which should all save and reload state together (executor, context etc), so maybe try backtracing some of that to check it all looks reasonable.

[BruceKristelijn]

Thanks, I just build your PR and it seems to work better without changing a lot of code which is great! Might be the wrong place to ask but I couldn't find it in sourcecode. Do you know if LLamaSharp adds things like 'Assistant:', and 'User:' to the chat?

[martindevans]

As far as I know it does not, but that'd be in the higher level parts that I'm not too familiar with so I'm not too sure on that!

[martindevans]

0.4.2 is out now. Does that resolve this issue?