Failure to repeat evaluation results. about db-gpt-hub

[wangzaistone]

I used: train_qlora.py to fine-tuned the model for llama 2-7b, and then used get_predict_qlora.sh ( the checkpoint is 10000) to get the results, but many of the outputs are empty, as shown below:

resulting in poor results when executing: evaluation.py, as follows:

                 easy                 medium               hard                 extra                all                 

count 248 446 174 166 1034 compare etype exec ===================== EXECUTION ACCURACY ===================== execution 0.109 0.052 0.006 0.006 0.050

can you help me to check what went wrong?

I also did an experiment with 10,000 steps with lora, and the effect did not improve. Instead, there was a large drop in effect similar to yours. Now I am also puzzled by the description of a classmate in the issue who trained 10,000 steps based on the default parameters and got a big improvement based on qlora. So,It is not recommended that you use qlora to train 10,000 steps. We are preparing to release the results of recent experiments as soon as possible.In our exp ,the result is better when step smaller than 2500.

[dingtian123]

when i use the chekpoint 2500, the result is :

                 easy                 medium               hard                 extra                all

count 248 446 174 166 1034
compare etype exec
===================== EXECUTION ACCURACY =====================
execution 0.194 0.076 0.029 0.006 0.085

what about yours？ @wangzaistone