Made a simple example for Unity demonstrating the use of this library. about llamasharp

[BruceKristelijn]

Thanks for setting this up. I was testing around a bit myself aswell and with and without your example I am unable to get past the "RuntimeError: The native library cannot be found. It could be one of the following reasons: " error. LlamaSharp works in a simple .net app with CPU backend. But I am unsure how / why this is not working in unity.

[Xsanf]

I have added a new version, if you did not use it, then download the new one.
It's hard to figure out what's not working for you. I made an example. There it was necessary that the example would work not only in the editor, put the weights in the Assets\StreamingAssets directory.
Need to use Unity 2021 and later. Earlier versions will not work.
Use LLaMa_Unity.unitypackage from my repository. Please note that there is a libllama-cuda12 backend, more than in the example from the original repository. What is in the original, under Unity only works with the CPU.
The example works fine for me and compiles under Windows 10 and NVIDIA 3080.
If necessary, I can archive the project itself if you have a setup problem. I'm just not sure if it's a landlord because there are file size limits. Well, or put not archived, but the model itself cannot be put into it.

[Xsanf]

I have added the project to https://github.com/Xsanf/LLaMa_Unity/
He must work. I took it from the archive and ran it. If something doesn't work, it could be a DLL mismatch with your version of CUDA.

In general, it is better to ask a question in the https://github.com/Xsanf/LLaMa_Unity thread, since I can simply not look at the questions in the https://github.com/SciSharp/LLamaSharp thread. This is not my thread.

[BruceKristelijn]

Hi there, thanks for responding. I just now managed to get it working by replacing the libllama.dll you provided in the llamasharp CPU nupkg.

[Xsanf]

If your GPU is not working, try different versions of the backend, it works fine for me with the GPU. As I understand it, if your GPU does not work, then you can try to compile the original backend for your CUDA. Here it is better to ask a question to the author of this repository, since he compiled the original https://github.com/ggerganov/llama.cpp into a DLL. But as I understand it, everything that supports lamacpp should be supported.

Note that https://www.nuget.org/packages/LLamaSharp.Unofficial mentions how to compile any backend you need.

[Sanokei]

I tried this, but it unfortunately crashes unity (2021.3.27f1) without explanation. I experienced this same sort of crash when trying to using LLamaSharp through a NuGet Package Getter for Unity. I am not sure why the issue persists.

A user in the SciSharp discord found a work around by running LLama seperately on a webserver and then using unity websockets to connect to it.

It feels like cheating in a way, and I continue to find a way to run it natively.
(I have also tried GPT4All's C# binding to no aval)

Edit:

• I am using .NET 4.5 not the standard 2.1
• Tried building the project instead of playing it in editor, still crashes.

[BruceKristelijn]

I tried this, but it unfortunately crashes unity (2021.3.27f1) without explanation. I experienced this same sort of crash when trying to using LLamaSharp through a NuGet Package Getter for Unity. I am not sure why the issue persists.

A user in the SciSharp discord found a work around by running LLama seperately on a webserver and then using unity websockets to connect to it.

It feels like cheating in a way, and I continue to find a way to run it natively. (I have also tried GPT4All's C# binding to no aval)

Edit:

• I am using .NET 4.5 not the standard 2.1
• Tried building the project instead of playing it in editor, still crashes.

With the new 4.0 version I am also experiencing crashes and annoying RAM usage. I am thinking of creating an additional .net based webserver and spin that up at the start of the player.

[Xsanf]

Did you use a package or a project?
There is some difference there. It all works for me. Unity crashes only after restarting the project from under the editor. This is due to the fact that Unity is not able to unload dll from memory. I tried to unload it myself when the application is finished, but this does not always help.

Try different variations and versions of the backend.
Check in the original LLamaSharp project. Please note that it was made on version 3 and now it is already on version 4.
As soon as LLamaSharp fixes its version 4.1, I will try to re-adapt it to Unity. It doesn't make sense now, it only supports CPU.