Logging losses on the validation / test split about allenact

[Lucaweihs]

Great!

I think if this encompasses loss logging (under the constraints that sensor values are available for the specified loss), that will be quite useful.

Sounds good. In principle we could just allow people to pass in a list of losses that they'd like to have recorded during testing, perhaps that's the most robust solution as the user could also record losses not included during training.

[Lucaweihs]

👍

I've implemented this at some point on an old branch. I think the main roadblock to having this on by default is that we cannot expect that certain sensor readings are available in the validation / test sets. For instance, if we train a model with imitation learning, it is very possible that we do not have expert actions for the test set. Would it be sufficient to be given the option to define new, custom, logable values in the experiment config?

[prithv1]

I think the main roadblock to having this on by default is that we cannot expect that certain sensor readings are available in the validation / test sets. For instance, if we train a model with imitation learning, it is very possible that we do not have expert actions for the test set.

I see. Makes sense.

Would it be sufficient to be given the option to define new, custom, logable values in the experiment config?

Yes. I think this will be useful. However, if it's only restricted to metrics that only utilize task-level information -- path taken, etc. to compute new metrics -- then I guess this would only slightly make things easier for the user in the sense that the user doesn't have to modify the task definition by themselves. I think if this encompasses loss logging (under the constraints that sensor values are available for the specified loss), that will be quite useful. But I do see your point that this might not generalize well to all settings -- might be easier for auxiliary losses that don't necessarily rely on the specific task definition (CPC|A, inverse / forward dynamics) as opposed to others.