zongyueqin / mjsd

To reproduce our experiments, first install all dependency in requirements.txt. Then download chatalpaca-10k.json from https://github.com/cascip/ChatAlpaca.

Next set your huggingface token to your environment variable with the following command. It is used to download Llama-2 checkpoint from huggingface (you need to request access first).

export HFTOKEN=$YourToken

Run evaluation.py with the following arguments.

python evaluation.py --approx_model_name $small_model_name --target_model_name $large_model_name --max_tokens 128 --max_seconds 1000 --dataset $dataset

the model names used in our experiments are: JackFram/llama-68m and meta-llama/Llama-2-13b-hf, facebook/opt-125m and facebook/opt-13b

The dataset argument can be chatalpaca, ChatGPT, and cnndm.