0.8.1: GGML_ASSERT: D:\a\LLamaSharp\LLamaSharp\llama.cpp:745: data about llamasharp

[JohnGalt1717]

I believe it is.

[AsakusaRinne]

Are you trying to offload layers to gpu as much as possible? Thus you want to catch the error if the current configuration is out-of-memory?

Please let me know if I misunderstand your condition. :)

[JohnGalt1717]

So I got a little more of an error. It appears it's llama.cpp code that is crashing, but instead of being handled, it takes down any process that LlamaSharp is hosted in even if there is a try/catch around it.

CUDA error 2 at D:\a\LLamaSharp\LLamaSharp\ggml-cuda.cu:6016: out of memory

In that error, it should have fallen back to CPU (which had tons of ram) but didn't instead it just crashed to the desktop.

[martindevans]

It looks like this is broadly the same issue as #343, with llama.cpp using assert to terminate the process instead of some kind of recoverable error handling (error return codes, exceptions etc)? If so I'll close this issue and we'll track it over in #343.

[iulix21]

I have thos error: GGML_ASSERT: /tmp/tmp25xydfjh/llama_cpp_python-0.2.53/vendor/llama.cpp/ggml-cuda.cu:8620: ptr == (void *) (g_cuda_pool_addr[device] + g_cuda_pool_used[device])