Punctuation and case sensitive about suber

[patrick-wilken]

Yes, thanks for those proposals. As you saw we experimented with casing and punctuation, but also with tokenization, when designing the metric and it is indeed a bit unfortunate that normalized SubER worked best in our experiments. 😅 I excluded other versions from the code mainly to avoid confusion about the metric definition.
But I guess I can add "SubER-cased" as a metric which would be true-cased and with punctuation, in analogy to the "WER-cased" metric. By the way, the default for TER in other tools is also case insensitive...
Regarding tokenization: the default for TER always seems to be to turn it off. Probably for historic reasons? I agree that it is intuitive to enable it, I don't know if somebody has shown rigorously that it improves the TER metric. I can revisit my experiments and see what numbers I get with/without tokenization for SubER.

[sarapapi]

Hi Patrick, it would be great to include the SubER-cased.
Moreover, I saw in the original TER implementation (TERCOM) that the input is not actually tokenized but can be enabled with the "normalized" parameter as it is in sacrebleu. However, in the official paper, the authors wrote "In addition, punctuation tokens are treated as normal words and mis-capitalization is counted as an edit.", thus punctuation is treated as a token (which is true only if we tokenize -- or normalize in the TERCOM library -- the text) and the computation is actually case sensitive. I think that they set as default parameters in the library something different from what they actually used for the official calculation (which I think is the correct one).

[patrick-wilken]

Ok, that all makes sense. :) I implemented it, see #6. Maybe you want to check the details.
Another question is whether we should also change the "TER" metrics to be tokenized and case-sensitive. But I would rather keep it just an interface to sacrebleu with default options. Because it's not really the focus of this repo to provide all the options for the other metrics. But it's easy to set them in suber/metrics/sacrebleu_interface.py if someone needs them.

[sarapapi]

Hi Patrick, sorry for my late reply but I have taken some time to take a look at the implementation and compute the metrics by myself. The cased version seems sound to me and the results are now consistent with that of the other metrics that I am using. Thanks again for your time.

[patrick-wilken]

That sounds good! I will merge then.