craftjarvis / rat

Implementation of "RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Horizon Generation".

RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Horizon Generation

[Website] [Paper] [Demo]

Abstract

We explore how iterative revising a chain of thoughts with the help of information retrieval significantly improves large language models' reasoning and generation ability in long-horizon generation tasks, while hugely mitigating hallucination. In particular, the proposed method — retrieval-augmented thoughts (RAT) — revises each thought step one by one with retrieved information relevant to the task query, the current and the past thought steps, after the initial zero-shot CoT is generated.

Prerequisites

• Python packages can be installed with pip install -r requirements.txt

• OpenAI_API_key is required for the language model. You can get it from OpenAI.

export OPENAI_API_KEY="sk-******"

• You can also use the Huggingface API key for the language model. You can get it from Huggingface.

• You also need to prepare the GOOGLE_API_KEY for Google Search API. You can get it from Google Cloud Platform.

export GOOGLE_API_KEY="********"

Getting Started

You can run the ipython notebook named creative.ipynb to generate open-ended creative text with RAT.

You can also use the following code to start a Gradio interface to interact with RAT:

python app/gradio_app.py

Experimental Results

Applying RAT to various base models substantially improves their performances on various long-horizon generation tasks; on average of relatively increasing rating scores by 13.63% on code generation, 16.96% on mathematical reasoning, 19.2% on creative writing, and 42.78% on embodied task planning.

Check out our paper!

Our paper is available on Arxiv. Please cite our paper if you find RAT useful for your research:

@article{wang2024rat,
    author    = {Zihao, Wang and Anji, Liu and Haowei, Lin and Jiaqi, Li and Xiaojian, Ma and Yitao, Liang},
    title     = {RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Horizon Generation},
    journal   = {arXiv preprint arXiv: 2403.05313},
    year      = {2024},
}

rat's Issues

cannot open https://8c379373eda6987c56.gradio.live/

https://8c379373eda6987c56.gradio.live/, this address lost, and response :No interface is running right now.
Could you have any idea to set with this problem?

Is there any source code that runs directly on mathematical data sets? I would like to repeat your accuracy on GSM8K. Thank you

Is there any source code that runs directly on mathematical data sets? I would like to repeat your accuracy on GSM8K. Thank you

Can you provide a code generation case？

I have a question regarding the task of code generation. Specifically, for a task that ultimately requires generating code, are the "thoughts" generated in RAT textual descriptions or actual code? According to the prompt provided in the paper, the results show that the thoughts are a series of textual descriptions. How are these thoughts ultimately transformed into code?

Is this only a demo page without any experiment source code?

The experiments in the paper show great improvements in many groups, Code Generation, Mathematical Reasoning, Math Reasoning, etc. Are these codes avaible to see the experiments details.

THX.

请问有直接在数学数据集上运行的源代码吗？我想复现一下您在GSM8K上运行之的准确率。谢谢您

请问有直接在数学数据集上运行的源代码吗？我想复现一下您在GSM8K上运行之的准确率。谢谢您

RAT adapt to scenarios with long tokens？

Can the current tokens satisfy RAT when the thinking chain is longer for complex tasks? Looking forward to reply.

what is the retrieval source for GSM8K?

Hello, your work has left a deep impression on me.

It's mentioned in your paper that the codeparrot/github-jupyter dataset are employed as your primary search vector library for code generation and mathematical reasoning tasks. But isn't this dataset only used for the code generation task? So, may I ask what the retrieval knowledge source for GSM8K is? Or is it indeed possible to use codeParrot/gitHub-jupyter for retrieval in the GSM8K task?

Looking forward to your reply.

有实验代码吗？

a long time to execute

I tried the code you propose, but it takes a long time to execute (after 20 minutes I canceled it). I don't know the reason, maybe I am using different versions of the libraries.