Unable to predict gap tags about openkiwi

[captainvera]

Hey @zouharvi,
I've managed to reproduce your issue. Let me take a deeper look into this and I'll get back to you!

[captainvera]

I've found the cause of this (bug). Unfortunately, in the current version of OpenKiwi, we do not support learning gaps + tags at the same time with the NuQE model.

The crash in particular happens because on the forward pass of NuQE you define the model_out as:

        outputs = OrderedDict()

        if self.config.predict_target:
            outputs[const.TARGET_TAGS] = h
        if self.config.predict_gaps:
            outputs[const.GAP_TAGS] = h
        if self.config.predict_source:
            outputs[const.SOURCE_TAGS] = h

        return outputs

We're adding the same hidden state to whatever we assume the output to be. Then, when calculating loss (where your error message comes from) we do the following:

        if self.config.predict_source:
            output_name = const.SOURCE_TAGS
        elif self.config.predict_gaps:
            output_name = const.GAP_TAGS
        else:
            output_name = const.TARGET_TAGS

Again, assuming we only have 1 type of output.
This means if you want to predict word + gap tags you'll have to train 2 separate models to do it.

However, the good news is that we are planning on making public several changes we've made to the OpenKiwi architecture. These changes include this change, amongst many other things, such as multi-threaded dataloading, multi-gpu, etc.
We will open an issue soon™️ to detail these changes and a tentative release date.

[zouharvi]

Hi @captainvera, thank you for your response.

Can you give us a hint at how to train the gap-only model? When we drop every (2n+1)th tag (leaving only tags for gaps), the dimension error is still there (this time the difference is 1024). (the config contains predict-gaps: true predict-target: false.

ValueError: Expected input batch_size (2112) to match target batch_size (1088).

In your 2019 paper (Table 1, WMT18 benchmarks), did you also have two separate models for MT and gaps? We are aiming at replicating your results, including training the models.

[captainvera]

Kiwi automatically knows which tags are gap tags. just settings predict-target: false and wmt18-format: true should be enough!

If you are talking about the Openkiwi paper, then yes. On our submission to the QE-shared task we were already using a slightly different version of Kiwi where we implemented target+gaps training.

[zouharvi]

Thank you, predicting gaps now works as expected. We're looking forward to the upcoming changes.