Custom callbacks for metrics, saving checkpoints about opennmt-py

[vince62s]

look at this: https://github.com/OpenNMT/OpenNMT-py/blob/master/onmt/utils/scoring_utils.py
and the way it is instanced and called from Trainer.py
This is what we use to compute BLEU score at validation time.

[Garfounkel]

Got it, thanks. This gives me a good idea how I could hack my own custom evaluation mechanism directly into the ScoringPreparator. But just to be clear, you're suggesting that I do that because there's no callback system at the moment ?

Thinking out loud, it seems that ScoringPreparator is indeed the correct place for me to inject my logic, but it feels very hacky. I don't want to compute my metric on the valid set, I have my own test set that I use for this metric. I also need to communicate with a web service to push scores and predictions for further analysis.

Basically my point is: sure I could inject all of that into ScoringPreparator, but what makes more sense to me is a custom callback that I could add in the config file, similar to how custom transforms are implemented.

Does that make sense? Do you think there's need for such a feature?

[vince62s]

I am not sure to understand what exactly you are looking for. You mentioned earlier that you want to compute a metric based on a "saved checkpoint" meaning it has to happen when ever you save a checkpoint, but compute your metric on what data ?
if it is not based on a checkpoint but must happen during training then you can look at where we report the stats (ppl, accuracy, ..)
Again I am not clear about what you are asking.