lavi-lab / cleva Goto Github PK

The comments in the prompt of code synthesis have an extra space on each line, which causes the human_eval to fail the code format check after concatenating the string with completion.

i noticed that there are a few prompt choices in each dataset, i wonder which one is the final using one? maybe i should wait for the code?

Hi, thanks for the great works.

Welcome to update the CLEVA into OpenCompass.

Docs: https://opencompass.readthedocs.io/en/latest/advanced_guides/new_dataset.html
Examples:

• open-compass/opencompass#91
• open-compass/opencompass#75

If you need more help, feel free to raise an issue in OpenCompass

When will the second round of models be updated? I am very interested in the capabilities of Mistral7B and Qwen14B and would like to know if they are really close to 70B.:)

what does this mean in your data?
like the difference between:

        "choices": {
            "item_separator": "\n",
            "item_template": "{idx}. {item}",
            "item_index": "upper"
        },
        "input": "“{translation}”这句话可以用以下哪句古文来表达：\n",
        "answer_context": "答：（请用选项序号回答）",
        "meta": {
            "language": "zh-Hans",
            "mul_as_gen": true,
            "postprocess": ["common", "upper"]
        }
    },

and

        "input": "{translation}\n这句话可以用以下古文来表达：",
        "meta": {
            "language": "zh-Hans",
            "mul_as_gen": false
        }
    },

besides, {idx}. {item}, the idx is like a number(1/2/3/4) or chars(A/B/C/D)?