Problem about calculating the MAP score in CodeXGLUE/Code-Code/Clone-detection-POJ-104/ about codexglue

[guody5]

Hi @deplay, thanks for your pointing problem.
I have modified the metric calculation.

def calculate_scores(answers,predictions):
    scores=[]
    for key in answers:
        if key not in predictions:
            logging.error("Missing prediction for index {}.".format(key))
            sys.exit()

        answer = set(answers[key])   

        Avep = []
        for k, p in enumerate(predictions[key]):
            if p in answer:
                Avep.append((len(Avep)+1)/(k+1))

        scores.append(sum(Avep)/len(answer))

    result={}
    result['MAP']= round(np.mean(scores),4)
    return result

Can you please check whether it's correct?

[guody5]

In fact, we follow this paper in C.4 Evaluation Metrics. According the description of paper, the MAP@R is defined as the mean of average precision scores, each of which is evaluated for retrieving R most similar samples given a query. In our case, the set of queries is the set of all test programs. For a program, R is the number of other programs in the same class, i.e. R=499. Therefore, the calculation is same as the paper and all results are comparable. For distinguishing them, we follow the paper and call MAP@R instead of MAP.

[deplay]

MISIM https://arxiv.org/pdf/2006.05265.pdf refers another paper Musgrave et.al A Metric Learning Reality Check, 2020. https://arxiv.org/pdf/2003.08505.pdf

In Musgrave's paper,they provide following contents:

In CodeXGLUE/Code-Code/Clone-detection-POJ-104/evaluator/evaluator.py,the essential implementation "scores.append(len(set(a&p))/len(a))" seems calculate R-Precision rather than MAP@R

[guody5]

You are right. Thanks. We have corrected the evaluator.py and update the results.

[deplay]

the implementation seems ok, and you may want to update here.