0.4.2-preview issue: Interactive mode stops responding after the first request about llamasharp

[mphacker]

@martindevans - I was just about to try with the demos to see if I run into the same issue. My code is very similar but maybe I have something messing up. I will update this post with my findings.

[mphacker]

I built from the current main branch and used the 4.2-preview CUDA backend and all seems to be working.

Does the backend nuget packages have source on GitHub too? I was thinking it would just be another project under the LLamaSharp solution, but I didn't see it.

[martindevans]

Is this the model you're using? https://huggingface.co/TheBloke/Nous-Hermes-13B-GGML/tree/main (if it is I'll try it out locally)

[mphacker]

It is. So crazy thing... I noticed that I am not getting good results with that model under stateless. I have to run under interactive mode to get good results. In that mode it seems to do great in general. One thing I noticed is that after I create the executor I need to savestate and then prior to each Infer I have to loadstate. If I don't do that the model just returns nothing after the first prompt. Using savestate to save off a clean state and then using loadstate prior to each infer gives me pretty good responses.

I am still learning all of this stuff, so maybe I am doing something wrong.

[martindevans]

I tried that model in demo 2 (Interactive mode) and 4 (stateless mode) and both seemed to work correctly for me. Do you see the problem in one of these demos (or any other demo?)

[saddam213]

Its my understanding that the first prompt will return nothing, as the first prompt is you giving it the context instructions, some models will just return nothing or a simple "good luck" message

Then any further prompting should work as desired.

EG:
First Prompt: Your a a crazy robot, respond with backward words
=No sesponse
Second promt: [the users input]
=should respond as normal

[martindevans]

I'm not really sure of the details, @AsakusaRinne builds the backend packages and should be able to tell you more.