Output truncated / error in generating response (using Google Colab free) about llm-search

[snexus]

Hi @vulcano9

The output size can be controlled by max_tokens in the config, you can try to change it in the notebook here -

llm-search/notebooks/llmsearch_google_colab_demo.ipynb

Line 103 in d0f756d

" max_tokens: 512\n",

The config in the notebook is just an example and can be tweaked to fit your use case/model. One limitation though is the amount of GPU memory on the free Google Colab. Pay attention to the GPU memory available, and try to increase the parameter without exceeding the memory.

Another option might be to use a smaller model on the free Google Colab.