action_evaluation expected file format about simmc

[satwikkottur]

Hello @seo-95 ,

1. That is correct. action_token is indeed the name of the action, for example, SearchDataset.
2. This is a good catch, I'll push a fix to check to account for empty attributes key.

[seo-95]

Thank you @satwikkottur. Can you also please provide an example of attribute label vs attribute value? Thanks

[seo-95]

I debugged the action_evaluation.py and I have noticed that it was a fault of mine. The attributes field in the model output JSON can not be empty, instead, it contains several keys, each one mapping to a list, not a single value as described in the mm_action_prediction/README.md. The right format is then:

{'action': 'SearchDatabase', 'action_log_prob': {'None': -2.091989517211914, 'SearchDatabase': -0.17580336332321167, 'SearchMemory': -3.525150775909424, 'SpecifyInfo': -4.88762903213501, 'AddToCart': -7.144548416137695}, 'attributes': {'attributes': []}}

This issue can be closed after a better description of attribute_label and attribute_val in the README. Thank you.

[satwikkottur]

On taking a closer look, attributes contains all the keys (API attributes) required to be predicted for a given API. The part of the code you're referring to (here) is reached only when the predicted action matches with the ground truth.

I'll add the description of attribute_label and attribute_val in the README.md. Thanks!