Support for multiple backends, auto-select the most optimal about llamasharp

[AsakusaRinne]

Could you please take a look of the file structure of your output directory? (x64/debug/net6.0/runtimes for example) Then please use NativeLibraryConfig.Instance.WithLogs() to see the details during backend selection.

[dluc]

Test 1: only CPU nuget

Time taken: 29 secs

<PackageReference Include="LLamaSharp" />

<PackageReference Include="LLamaSharp.Backend.Cpu" />

[LLamaSharp Native] [Info] NativeLibraryConfig Description:
- Path: 
- PreferCuda: True
- PreferredAvxLevel: AVX2
- AllowFallback: True
- SkipCheck: False
- Logging: True
- SearchDirectories and Priorities: { ./ }
[LLamaSharp Native] [Info] Detected OS Platform: WINDOWS
[LLamaSharp Native] [Info] Detected cuda major version 12.
[LLamaSharp Native] [Info] Tried to load runtimes/win-x64/native/cuda12/libllama.dll but failed.
[LLamaSharp Native] [Info] ./runtimes/win-x64/native/avx2/libllama.dll is selected and loaded successfully.

Test 2: only CUDA 12 nuget

Time taken: 22 secs

<PackageReference Include="LLamaSharp" />
<PackageReference Include="LLamaSharp.Backend.Cuda12" />

[LLamaSharp Native] [Info] NativeLibraryConfig Description:
- Path: 
- PreferCuda: True
- PreferredAvxLevel: AVX2
- AllowFallback: True
- SkipCheck: False
- Logging: True
- SearchDirectories and Priorities: { ./ }
[LLamaSharp Native] [Info] Detected OS Platform: WINDOWS
[LLamaSharp Native] [Info] Detected cuda major version 12.
[LLamaSharp Native] [Info] ./runtimes/win-x64/native/cuda12/libllama.dll is selected and loaded successfully.

Under "native" there's only one folder with the DLL inside:

Test 3: both CPU and CUDA 12 nugets

Time taken: 29 secs

<PackageReference Include="LLamaSharp" />

<PackageReference Include="LLamaSharp.Backend.Cpu" />

<PackageReference Include="LLamaSharp.Backend.Cuda12" />

[LLamaSharp Native] [Info] NativeLibraryConfig Description:
- Path: 
- PreferCuda: True
- PreferredAvxLevel: AVX2
- AllowFallback: True
- SkipCheck: False
- Logging: True
- SearchDirectories and Priorities: { ./ }
[LLamaSharp Native] [Info] Detected OS Platform: WINDOWS
[LLamaSharp Native] [Info] Detected cuda major version 12.
[LLamaSharp Native] [Info] ./runtimes/win-x64/native/cuda12/libllama.dll is selected and loaded successfully.

FunctionalTests\bin\Debug\net7.0\runtimes:

FunctionalTests\bin\Debug\net7.0\runtimes\win-x64\native:

[dluc]

@AsakusaRinne so everything looks ok from the logs, but as you can see the time taken doesn't match.

• Test 1: only CPU nuget - Time taken: 29 secs
• Test 2: only CUDA 12 nuget - Time taken: 22 secs
• Test 3: both CPU and CUDA 12 nugets - Time taken: 29 secs

[AsakusaRinne]

If you delete the libllama.dll at the root of output path, would the time taken act as expected? Note that after removing the file, please don't re-compile the project.

[martindevans]

Also you could try adding a breakpoint in here. From there you can see the exact set of binaries it's going to try (libraryTryLoadOrder) and which one it succeeds on.

[dluc]

I've created a new project to run these tests and taken the steps to manually delete the DLL files. Interestingly, in this new project, the performance timings between cuda12 and avx2 are too close to definitively say if there's an issue. For the time being, I'm placing my trust in the logs. If an opportunity arises where I can obtain a machine with a more powerful Nvidia card, I'll conduct further tests where the performance gap would likely be more evident.

[AsakusaRinne]

@dluc If needed, you could set GpuLayerCount a larger number to increase the time gap between cpu version and gpu version.