Dear Authors,
Thank you for the great work on introducing ReAct.
Since, the original model that you used `text-davinci-002` is deprecated on openai the closest two alternatives are: `gpt-3.5-turbo` and `davinci-002`. The best performance we get on e.g. the first 10 is 0.3, while the reported results on the first 10 envs of Alfworld are 0.7.
Could you share the traces or advice, what your latest scores on this environment is? Or how to reproduce your score of 0.7. @ysymyth @john-b-yang @descrip
Thanks.
@ysymyth Thanks a lot for coming back to me on this one.
Yes, we have run the results several times on the entire test set (i.e. 135 environments). We both used your code and our own implementation.
Did you do modifications beyond the ones present in your code (in this repo?):
- e.g. Correcting common syntax mistakes of the model (e.g. put object x in place y -> put object x **in/on** place y)?
- Sampling outputs again with increasing temperature?
- Anything else?
Yeah I can confirm that newer GPT-3.5 (both latest & instruct) do appear to perform really poorly, around 30%, when running a close clone of the notebook of alfworld.
I would have just said that GPT-3.5 got much worse on this task _or_ needs new prompts, but I see a recent paper that uses gpt-3.5-instruct with decent ReAct results (54%) here: https://arxiv.org/pdf/2405.17402
So @ai-nikolai and I might be missing something.
There is also a discussion this topic here:
https://github.com/noahshinn/reflexion/issues/35
The basic conclusion is the following:
@okhat @noahshinn @ysymyth - It seems there are a couple of issues, which are summarised in this paper **StateAct** (https://arxiv.org/abs/2410.02810), which also perform better than ReAct ;).
Specifically the issues are:
1. Different gpt models have very different performance. (With older models often performing much better).
2. Secondly, StateAct mentions "Corrections" as a way of getting higher scores in an easy fashion. Because the GPT models often produce `put <object> in <place>`, however, the correct alfworld syntax is `put <object> in/on <place>`.