Greek encoding about ucto

[proycon]

Very interesting indeed! I suspect it has something to do with unicode normalisation (https://en.wikipedia.org/wiki/Unicode_equivalence#Normal_forms), there are several "unicode normal forms". I doubt this is something ucto deliberately did, but was rather handled in the underlying libicu. We either implicitly or explicitly opt for one of the normal forms I guess (@kosloot)?

In comparions, unicode libraries should treat these two strings as equivalent under the normal forms.

[JessedeDoes]

The problem disappears when I apply unicode normalization to the files after tokenization (but before final validation)

I suppose the Nederlab files will not be harmed by applying option 2. Anyway, it seems only greek text is affected.

1 Normalization Form D (NFD) Canonical Decomposition
2 Normalization Form C (NFC) Canonical Decomposition, followed by Canonical Composition
3 Normalization Form KD (NFKD) Compatibility Decomposition
4 Normalization Form KC (NFKC) Compatibility Decomposition, followed by Canonical Composition

[kosloot]

it seems indeed a normalization 'problem'.
Ucto puts all UTF-8 in the NFC form deliberately. This to weed out the differences per OS (Linux, OSX, M$) and applications even. It is not uncommon for one document to have this intermixed!

==> A quick solution might be to run ucto with the option -N NONE. which should disable normalization.

Background:
This was done to make string comparison in later stages easy. But it look like ucto bites its own tail now.
(since we introduced a more strict text handling)

Internally, ucto and libfolia use Unicode strings, where this problem should be non-existent (folia.py too I assume) Only on conversion too/from UTF8 the problem arises.
But that puzzles me, because libfolia chokes on the file too. Although is uses Unicode String comparison. That needs more research... But is seems that Abstract Characters are handled differently (Or not at all?) by ICU. But converting to a UTF8 form an back to unicode seems help. Odd

Solutions:
There are several solutions I think.

1. use the -N option of ucto.
2. adapt the text-checking in libfolia and folia.py to use normalization too
3. adapt libfolia and folio.py to normalize all incoming/outgoing UTF8 to NFC or NFD

Discussion:

1. Seems unwise, the consequences on other programs like Frog and Tscan are unclear.
2. this could be done quite easy, I thought. But see my puzzle above..
3. this might be a good idea on the long run. But do you want that configurable? etc.
   better let it external, using iconv or such ?

So more questions then answers.

[kosloot]

Well,
I did find the code to implement solution 2 in libfolia. Using the ICU function:
Normalizer::compare( s1, s2, opts, Err )

[kosloot]

That function is part of a deprecated module, so let's not go there.

I think that the best solution is to go with option 3. and adapting libfolia AND follia.py.
I tested this for libfolia, and it is almost a no-brainer.
In this way we assure that all UTF8 produced with the library is in 1 normalization,
all software build upon libfolia will benefit from this.

For Python there is unicodedata.normalize()

[proycon]

Since this is solved in other packages and yet-to-be solved in issue proycon/pynlpl#37, closing this one.