Evaluating perplexity about nmt-keras

[lvapeab]

Oh, that function was a utility that I quickly wrote to evaluate a model. It shouldn't be there. I've removed it.

To evaluate the perplexity of a model, you can pass 'perplexity' as a KERAS_METRIC in config.py. This is now done by default:

nmt-keras/config.py

Line 32 in 5a29099

KERAS_METRICS = ['perplexity'] # Metrics to be logged by Keras during training (in addition to the loss).

PS: it requires to update Keras.

[philipcori]

Got it, thanks. However, it only computes perplexity during training. Is there a way to compute perplexity on the validation set? I would like to have the metric_name here be 'perplexity' rather than sacrebleu:

callbacks.append(PrintPerformanceMetricOnEpochEndOrEachNUpdates(nmt_model,
                                                                dataset,
                                                                gt_id='target_text',
                                                                metric_name=['sacrebleu'],
                                                                set_name=['val'],
                                                                batch_size=256,
                                                                each_n_epochs=20,
                                                                extra_vars=search_params,
                                                                reload_epoch=0,
                                                                is_text=True,
                                                                input_text_id=input_text_id,
                                                                index2word_y=vocab,
                                                                sampling_type='max_likelihood',
                                                                beam_search=True,
                                                                save_path=nmt_model.model_path,
                                                                start_eval_on_epoch=0,
                                                                write_samples=True,
                                                                write_type='list',
                                                                verbose=True))

Thanks again for the help.

[lvapeab]

Added (MarcBS/multimodal_keras_wrapper@22d10a9).

You can now evaluate with Perpleixty as any other metric, e.g.:

    METRICS = ['sacrebleu', 'perplexity']

You'll need to update multimodal-keras-wrapper and nmt-keras.

[philipcori]

Great, thank you.