franckbrl / stat_comb_model

Andreev's statistico-combinatorial model for morphology learning

Andreev's statistico-combinatorial model (stat_comb_model)

This program is an implementation of Andreev's statistico-combinatorial model for unsupervised learning of morphology, mainly described in:

Andreev, N. D. (1965). Statistiko-kombinatornoe modelirovanie jazykov. Nauka.
Andreev, N. D. (1967). Statistiko-kombinatornye metody v teoretičeskom i prikladnom jazykovedenii. Nauka.

If you don't know Russian and want to get the idea fast, here is my paper documenting this implementation: Unsupervized learning of morphology in the USSR.

It takes as input a corpus that is tokenized and has one sentence per line, and returns a set of classes, each containing a set of affixes that are associated to a set of stems.

Ex: Class 1

• Affixes: ies, y
• Stems: all, authorit, abilit, ...

USAGE:

stat_comb_model.py -i corpus.txt

stat_comb_model's Issues

End of search for the type (less than n bases left)

Too few base remainders left should not interrupt the search.

The search for the current type ends when accepting the candidate affix leads to a number of base remainders below the threshold. Why not just refuse the affix and check if the next one in the informants list can do better?