Comments (7)
end-user of the plugin may not always have an Nvidia gpu
You can add CPU backend by default and add some comments where to get CUDA backend.
from llamasharp.
end-user of the plugin may not always have an Nvidia gpu
@adammikulis Maybe you could include both cpu and cuda native libraries and use LLama.Native.NativeLibraryConfig
to select the backend dynamically at the beginning of the program. Note that installing both cpu and cuda backends is supported in master branch but not v0.10.0. If you want to use v0.10.0, please keep those files manually.
from llamasharp.
@eublefar @Xsanf Maybe you'll be interested in this issue. I hope this mention didn't disturb you. :)
from llamasharp.
I am not familiar with Godot, but maybe you could just download all NuGet dependencies and add them to the plugin manually?
You can just click Download package
on nuget page, rename it's extension to zip and extract the dlls from there.
I am not sure if it will cause assembly conflicts tho.
from llamasharp.
Thanks @eublefar, I was wondering the best way to extract those files (aside from going through an existing build). I've added them to the plugin folder and will try to manually reference them in the project. I'm wondering if PR #603 (SetDllImportResolver based loading) will help with this, as the end-user of the plugin may not always have an Nvidia gpu. It seems like either way the user will need a .csproj file, making this more suitable as a template but unable to be added to the Godot AssetLib as a typical plugin.
from llamasharp.
Thank you for the help! I've made some significant updates to my Godot add-on that bring it closer to being a useful example project. At this point I'm just trying to resolve the issue of properly referring to my backends in my .csproj.
I was able to successfully refer to a locally downloaded LLamaSharp.dll by adding a path reference to my .csproj (within an ItemGroup) but I am unable to get the Assemblies to recognize the Cuda backend I've manually added. Am I using the correct name for it? I also tried LLama and LLama.Native.
<Reference Include="llama"> <HintPath>addons\mind_game\libs\runtimes\win-x64\native\cuda12\llama.dll</HintPath> <Private>True</Private> </Reference>
I am calling NativeLibraryConfig.Instance.WithCuda();
as soon as the project scene loads, so I don't think that's the problem.
from llamasharp.
runtimes\win-x64\native\cuda12\llama.dll
is the correct path, but I'm not sure whether its prefix is correct. Since you've already had an specific path of native library, you can also use NativeLibraryConfig.Instance.WithLibrary
.
BTW, it might be a stupid question but may I ask whether .NET runtime higher than coreapp 3.1 could be used in godot?
from llamasharp.
Related Issues (20)
- CentOS x86_64 Failed Loading 'libllama.so' HOT 4
- System.TypeInitializationException: 'The type initializer for 'LLama.Native.NativeApi' threw an exception.' HOT 12
- How do I continously print the answer word for word when using document ingestion with kernel memory? HOT 1
- How to rebuild LLamaSharp backends HOT 3
- Namespace should be consistent HOT 2
- Mamba HOT 11
- Android Backend HOT 5
- [Feature] Allow async model loading and cancellation
- [CI] Add more unit test to ensure the the outputs are reasonable HOT 3
- Take multiple chat templates into account
- [Feature]: Support for Function Calling or Tools HOT 4
- [BUG]: DefragThreshold default is not matching llama.cpp and probably not intended HOT 6
- [BUG]: Answer stop abruptly after contextsize, even with limiting prompt size HOT 1
- [BUG]: Linux cuda version detection could be incorrect HOT 2
- [BUG]: WSL2 has problem running LLamaSharp with cuda11
- Add unit test about long context HOT 3
- Add debug mode of LLamaSharp
- How to better provide system information for LLMs HOT 3
- LLAVA Configuration HOT 4
- [Feature]: 不同的LLM模型,代码要以怎样的方式融合到项目里 HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from llamasharp.