Can i use openkiwi to do data cleaning? about openkiwi

[captainvera]

Hi @BigBorg,

What you are asking is an open research question :) as such, I can provide no definite answer.
Your logic and your intuition make sense to me. I would say the issue with TER could be alleviated with for example excluding sentences where max(prob_bad_tags) > X but again, this is an open question I do not have the answer for.

I would however, be very interested in your results if you decide to further explore this idea! It looks like a promising application of QE.

[kepler]

Jumping in here.

Have you investigated this further, @BigBorg? My intuition is that dirty sentences will be badly tokenized, causing many words to be encoded as unknown, which in turn will make the QE model predict a high TER. If, however, the dirty sentences are not really that dirty, there's no straightforward way of detecting a "critical" word. It'll all depend on how you trained your model. And in this case it probably makes more sense to do what @captainvera said, to use QE at the word level.

I'm closing this, but feel free to keep the discussion going.