Sudden shutdown after +1000 tokens. about catai

[ido-pluto]

This is related to the context limit, you can try to extend it.

catai config --edit nano

change the nCtx to something bigger, for example 4096

export const SETTINGS_NODE_LLAMA = {
    enableLogging: false,
    nCtx: 4096,
    nParts: -1,
    seed: 0,
    f16Kv: false,
    logitsAll: false,
    vocabOnly: false,
    useMlock: false,
    embedding: false,
    useMmap: false,
    nGpuLayers: 3,
};

[ceddybi]

@ido-pluto How do you change the config now, i have the same issue but looks like the config command was removed, how to archive the same?

[ido-pluto]

You can change the config via the settings button in the webui

[ceddybi]

@ido-pluto i just editted it to this, but it ignore my changes

{
"bind": "...",
"nCtx": 10000,
"n_ctx": 10000
}

[ido-pluto]

There is a configuration guide in the readme, check it out.

The option you are looking for is here:
https://withcatai.github.io/node-llama-cpp/api/type-aliases/LlamaContextOptions

[ceddybi]

@ido-pluto i just added, but now i get this error

{
"bind": "...",
"contextSize": 10000,
"batchSize": 10000
}

GGML_ASSERT: /home/runner/work/node-llama-cpp/node-llama-cpp/llama/llama.cpp/ggml-backend.c:519: data != NULL && "failed to allocate buffer"

[ido-pluto]

This is too large, to reset the model you can simply delete and reinstall it