birch-san / mpt-play

Command-line script for inferencing from models such as MPT-7B-Chat

MPT-Play

Python script to demonstrate how to invoke models such as MPT-7B-Chat from the command-line.

Setup

All instructions are written assuming your command-line shell is bash.

Clone repository:

git clone https://github.com/Birch-san/mpt-play.git
cd mpt-play

Create + activate a new virtual environment

This is to avoid interfering with your current Python environment (other Python scripts on your computer might not appreciate it if you update a bunch of packages they were relying on).

Follow the instructions for virtualenv, or conda, or neither (if you don't care what happens to other Python scripts on your computer).

Using venv

Create environment:

. ./venv/bin/activate
pip install --upgrade pip

Activate environment:

. ./venv/bin/activate

(First-time) update environment's pip:

pip install --upgrade pip

Using conda

Download conda.

Skip this step if you already have conda.

Install conda:

Skip this step if you already have conda.

Assuming you're using a bash shell:

# Linux installs Anaconda via this shell script. Mac installs by running a .pkg installer.
bash Anaconda-latest-Linux-x86_64.sh
# this step probably works on both Linux and Mac.
eval "$(~/anaconda3/bin/conda shell.bash hook)"
conda config --set auto_activate_base false
conda init

Create environment:

conda create -n p311-mpt python=3.11

Activate environment:

conda activate p311-mpt

Install package dependencies

Ensure you have activated the environment you created above.

(Optional) treat yourself to latest nightly of PyTorch, with support for Python 3.11 and CUDA 12.1:

# CUDA
pip install --upgrade --pre torch --extra-index-url https://download.pytorch.org/whl/nightly/cu121
# Mac
pip install --upgrade --pre torch --extra-index-url https://download.pytorch.org/whl/nightly/cpu

Install dependencies:

pip install -r requirements.txt

Run:

From root of repository:

python -m scripts.chat_play --trust_remote_code --bf16

On Mac you'll need to disable bfloat16. PYTORCH_ENABLE_MPS_FALLBACK=1 is not necessary in current PyTorch nightly, but there's no harm keeping it (and it may help on older PyTorch):

PYTORCH_ENABLE_MPS_FALLBACK=1 python -m scripts.chat_play --trust_remote_code

License

This repository is itself MIT-licensed.

Includes MIT-licensed code copied from Artidoro Pagnoni's qlora, and Apache-licensed code copied from MosaicML's mpt-7b-chat Huggingface Space.

mpt-play's Issues

q4

Fantastic! it works like a charm. Is it possible to use a q4_0 .bin ?

Memory usage growing (MacOS)

Thanks for this. Super fantastic.
I notice that memory usage continuously grows. It starts off around 25G -- almost the same as crawling along under 100% CPU -- but with each conversation turn memory increases & swapping ensues. I don't know if you're using the issue tracker here, but just in case I thought I'd mention it. I will look more closely tomorrow, but chances are someone has already beat me to it.

MacOS Ventura 13.4 on a M1 MacBook Pro with 32Gb. Python 3.10 and pytorch 2.1.0.dev20230526