pip install -r requirements.txt
pip install torch==1.13.1 --extra-index-url https://download.pytorch.org/whl/cu117 --upgrade
Download the model you want to run from Huggingface: BlinkDL:
- latest 7B with instruct-tuning: 7B-Instruct-test4-20230326
- latest 14B: 14B-20230313-ctx8192-test1050
Make sure you meet the System Requirements
Enter the path to the downloaded file in line 48 of instruct.py
python instruct.py
Make sure you have enough VRAM on your graphics card to run these.
- 7B requires 16GB for fp16 and 8GB for fp16i8 inference
- 14B requires 28GB for fp16 and 14GB for fp16i8 inference (issue 30: low RAM usage)
Make sure you have enough CPU RAM to load these models
- 7B works with 32GB of RAM
- 14B requires 40GB+ of RAM issue 41: not enough memory
for a description of inference strategies see https://pypi.org/project/rwkv/