squeezeailab / llm2llm

LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement

LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement [Paper]

This is the code for the LLM2LLM paper.

Reproducing Main Experiments

We have provided code required to reproduce our main experiments for GSM8K. Instructions for other datasets will be uploaded soon.

1. Download a copy of LLaMA-2-7B, and the appropriate dataset
2. Clone the GSM8K dataset by running

cd GSM8K
git clone https://github.com/openai/grade-school-math.git

3. Run generate_seed_data.py and adjust SUBSAMPLE_SPLIT to get seed data.
4. Ensure that all settings in config.yaml are accurate
5. Run python GSM8K/generator_data.py GSM8K/config.yaml
6. cd into your experiment folder and run ./run_all.sh
7. After all of the iterations have finished, run

python report_results.py --results_file_name test_0.jsonl GSM8K/grade-school-math/grade_school_math/data/test.jsonl $EXP_FOLDER

to get a detailed breakdown of the performance of the model at each iteration.

This will produce an output folder that contains all the data and model checkpoints.

Roadmap

We are planning on adding the code required to reproduce our experiments on other datasets.

Citation

LLM2LLM has been developed as part of the following paper. We would appreciate if you would please cite this paper if you found this library useful for your work:

@article{lee2024llm2llm,
      title={LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement}, 
      author={Lee, Nicholas and Wattanawong, Thanakul and Kim, Sehoon and Mangalam, Karttikeya and Shen, Sheng and Anumanchipali, Gopala and Mahoney, Michael W and Keutzer, Kurt and Gholami, Amir},
      journel={arXiv},
      year={2024},
}

llm2llm's Issues

where is run_all.sh

Thank you for open-sourcing this work . I'd like to try it on my own dataset. But I cannot find the complete running pipeline 'run_all.sh'. Is it missing?

Insightful Connection to My Previous Paper

I recently read your paper and it is a great paper. Your research provides valuable insights into LLM-based data augmentation.

As I was reading your paper, I couldn't help but notice the parallels between your findings and the work AI2 and I published last year in EMNLP, titled "[Let GPT be a Math Tutor: Teaching Math Word Problem Solvers with Customized Exercise Generation]." Our paper delves into the targeted data augmentation for MWP solving, which might complement and extend the discussions in your paper.

Therefore, I was wondering if you might consider acknowledging our work in your paper, as it could provide additional depth to the understanding and implications of your findings for the readers. I would be more than happy to discuss this further or provide any additional information you might need regarding my work.

What is instruction_per_seed_task in your experiment?

It seems that instruction_per_seed_task is not reported in the Experiment Setup section of your paper? I find in your code that it's four by default.

expected dtype float for `end` but got dtype c10::BFloat16

Running the gitcode encountered the above error, the full error is as follows:

transformers == 4.32.1
torch==2.2.0

Other Tasks

There just provide code for GSM8K, Will the code on other datasets be provided？
What is the approximate time if provided, thx~

where is step 6 from the readme

Readme says:
6. cd into your experiment folder and run ./run_all.sh

where is the experiments folder?