Get low accuracy with GPT-3.5.

Luoyang144

Hi, I'm tring to run ReAct with GPT-3.5-Turbo on hotpot dataset with provided jupyter notebook. But only get 0.182 accuracy, is it a reasonable result? I think it is much lower than result showed in paper.

ysymyth

hi can you show your code and example trajectory?

Luoyang144

I'm using this [notebook](https://github.com/ysymyth/ReAct/blob/master/hotpotqa.ipynb), and using API from azure, so I change the llm function(call GPT-3.5).  the final result: 

ysymyth

can you show some trajs

zhiyuanc2001

> Hi, I'm tring to run ReAct with GPT-3.5-Turbo on hotpot dataset with provided jupyter notebook. But only get 0.182 accuracy, is it a reasonable result? I think it is much lower than result showed in paper. Hi, I got similar reults. I think it is the size of GPT-3.5-Turbo and alignment tax result in the low score. :-)

Luoyang144

In fact, the results of ReAct are no longer as good as directly allowing GPT3.5 to reason. Why did this happen?

ysymyth

can you show some trajectories? also, try the original text-davinci-002 and see if scores also become lower?

Jiayi-Pan

It looks like we observed the same phenomenon on at least a subset of tasks on webshop benchmark. We run react/act using the official code on webshop task 2000~2100 with `gpt-3.5-turbo-instruct` The result is - ReAct: 0.5345 avg reward, 0.28 success rate - Act: 0.674 avg reward, 0.38 success rate You can find the raw trajectories [here](https://drive.google.com/file/d/1ugjhYAgYVpGsrlUwFIn543Ku-GMbuwC6/view?usp=sharing)

Jiayi-Pan

Same trend on 2k-3k Method | 2000-2100 | 2000-3000 -- | -- | -- ReAct | 0.5345 / 0.28 | 0.5735 / 0.328 Act | 0.674 / 0.38 | 0.67 / 0.352

Luoyang144

Here is running log of gpt4 ReACT, still get lower result (GPT4 get 0.33). https://github.com/Luoyang144/share/blob/main/gpt4_hotpot_react.log

ysymyth

Interesting. Is it only on HotpotQA or more tasks? Also, maybe check if text-davanci-002 result is reproducible? https://github.com/Luoyang144/share/blob/main/gpt4_hotpot_react.log cannot be opened.

Luoyang144

text-davinci-002 is not available now. This link should be accessible now: https://github.com/Luoyang144/share/blob/main/gpt4_hotpot_react.log

ysymyth

My hypothesis is that later models after text-davinci-002 might be tuned on trajectories similar to Act, plus domains like QA have intuitive tools, and tasks like HotPotQA have intuitive reasoning patterns. On more out-of-distribution domains and tasks (e.g., WebShop, or AlfWorld), reasoning should still improve decision making generalization and transparency. Close it for now but let me know if there's more findings or analysis into this.