Reproduce HumanEval Results about codegen

[boblee22]

Just reproduced the paper results! Thank you very much for your help!

temperature 0.2: {‘pass@1’: 0.2910060975609756, ‘pass@10’: 0.4311547000349374, ‘pass@100’: 0.5379728166388384}
temperature 0.6: {‘pass@1’: 0.26408536585365855, ‘pass@10’: 0.5354897815147874, ‘pass@100’: 0.7727023953428368}
temperature 0.8: {‘pass@1’: 0.23518292682926834, ‘pass@10’: 0.528405455080335, ‘pass@100’: 0.7764527488844969}

[rooa]

Thanks for your comment. The results look oddly uncorrelated with the temperatures.

From the snippet, here are the differences:

• The temperature is specified with temperature, not temp. This may cause ignoring specified temperature values.
• We did not use canonical solutions to determine the maximum length. For HumanEval experiments, we used input_ids_len + 512.

Additionally, please verify that you are using the corresponding tokenizer for the model.

If you could share your sampling script somewhere (e.g. gist), I'd be happy to take a look.

[amd-1221]

If it is possible, please share you evaluation script @boblee22