aliok / trnltk Goto Github PK

For example, it might make sense to mark the word "bilimsel" as "contains 'Related' suffix"

We don't want to break that word into its subparts, but need to reduce the number of parse results produced for that word.

bilimsel+Adj
bilim+A3sg+Pnon+Nom+Adj+Related

Can be applied to nouns ending with a consonant.

erkeksi
cocuksu
raporsu

but not
masa-m-si
yasli-m-si

-msi is another suffix

Z3's solution : https://code.google.com/p/zemberek3/source/detail?r=321

• Vazgeçtim.
• Vaz mı geçtin?

Seems like a very hard problem.

For example, with brute force, parse results for the word "yapıyordum" are following:

• yap + Prog + Past + P1sg
• yapa + Prog + Past + P1sg
• yapiyo + Aor + Past + P1sg
• yapiyor + Past + P1sg

Nr 2, nr 3 and nr 4 are false positives.

For nr 3 and nr 4 we can have a look at the similarity with other verb roots.
There are a lot of verbs (kapa, ...) in the form of nr 2, so this doesn't solve the problem completely. But it is a start.

1.'yi
2'nci
10000'er
biner
bininci (supported already)

instead of adding words to dictionary, maybe it is better to add the support with the suffix graph

masmavi, simsıcak, yapayalnız, ipince, küpküçük etc.

Doesn't have strict rules, thus need to find syllables for those from a big corpus.

If word ends with a dictionary adverb/adjective, and if it starts with the beginning of the same adjective/adverb, we can suggest that it is intensified.

Try to find as much as possible and add them to a dictionary!

Brute force root extractors already exist. However, the results are too much and it is better to do it statistically

This might be useful for finding roots that doesn't exist in the dictionary (e.g. local words) and proper nouns.

• Save the possible roots for a big corpus (10M words) in a file
• ...

For proper noun recognition

• check if the root has been used with a apostrophe in the corpus
• or check if the word starts with upper case in the middle of a sentence in the corpus

For e.g. verb recognition: for non-dictionary word 'kıvışlıyordu' find the root as 'kıvışlamak'

• Check if there are other surfaces with root candidates as "kıvışlamak", such as 'kıvışladım' 'kıvışla' 'kıvışlarsa'
• Then we would eliminate the some of the candidates : 'kıvışlımak' 'kıvışlıyormak' 'kıvışlıyomak' etc.
• However, it doesn't eliminate the roots such as "kıvış+Noun" 'kıvmak' 'kıvımak' etc.
• For them, check if there is other surfaces such as 'kıvışımı' 'kıvdım' 'kıvıyorum' etc.

That would help a lot.

Another example:
hıllandım -> 'hıllandı(hıllandımak)+Verb+Neg(m[m])+Imp+A2sg',

Duplications:

1. abur cubur both doesn't make sense
2. yemek memek second doesn't make sense
3. iyi kotu opposites
4. zırıl zırıl called "sound reflection" in Turkish
5. sıcak sıcak 2 adjectives turn into an adverb
6. gide gide
7. kırk elli kişi
8. uc bes kurus
9. bata cika
10. enine boyuna
11. ev bark
12. bas basa, daldan dala, ucu ucuna
13. gelir gelmez, yapar yapmaz --> Adverbs

Some of them can be done during tokenization.

Some of them needs to be done after parsing, such as 13

_R+Verb+Pos+Aor+A3sg _R+Verb+Neg+Aor+A3sg %1 %2+Adverb+When

Should accept:

• Surfaces starting with uppercase letter
• Including no apostrophe (since with apostrophe, the root is obvious)

Some suffixes which can be applied to these kind of roots:

• -ler :
  • Turkler,
  • Alilere gidiyorum
• -gil
  • Ahmetgildeyim
• -li
  • Kayserili
• -lik
  • Turkluk

etc.

Check the rules from TDK: http://www.tdk.gov.tr/index.php?option=com_content&view=article&id=187:Noktalama-Isaretleri-Aciklamalar&catid=50:yazm-kurallar&Itemid=132

Some suffixes doesn't make sense to apply proper nouns:

• - len

Really problematic part is, sometimes apostrophe is used, sometimes it is not.

Transitive verbs (verbs that accept an object) could be found from an annotated corpus.
A non-transitive verb can be converted to transitive by adding Causative suffix.
** A very advanced issue. Related with POS tagging.

Finding reciprocal is easy. Rule based (verbs ends with "ş" and no need for POS tagging)

Reflexive (such as giyinmek) is similar to reciprocal

For:

• roots
• lexemes
• suffixes

Think about it, need to investigate. However, it seems it is not necessary/

http://en.wikipedia.org/wiki/Expectation-maximization_algorithm
http://en.wikipedia.org/wiki/Baum%E2%80%93Welch_algorithm

These could give an idea

These could help:

http://www.grsampson.net/AGtf1.html
https://github.com/maxbane/simplegoodturing/blob/master/sgt.py

-vermek
-durmak
.etc

See
http://en.wikipedia.org/wiki/Turkish_grammar

In order to use in tokenizer (sentence to words), we need something like that.
Can be done statistically with some rules, with the support of Issue #25

hafta sonu => hafta_sonu
Turkiye Cumhuriyeti ==> Turkiye_Cumhuriyeti
ilan etmek --> ilan_etmek

Doesn't make sense to parse "ilan" and "etmek" separately.

Zemberek has already a small database about these.

Issue #32 is related

See http://www.tdk.gov.tr/index.php?option=com_content&view=article&id=221:Ayri-Yazilan-Birlesik-Kelimeler&catid=50:yazm-kurallar&Itemid=132

Use statistical parsing + context information(n-gram probability)

Rule based part is already available : https://github.com/aliok/trnltk/blob/master/trnltk/tokenizer/texttokenizer.py

It doesn't work good with:

1. Abbreviations like M.Ö. or ing.
2. Ordinals like 3.
3. Roman numerals like III and III.
4. Paranthesis such as "(abc"
5. Some phrases which are multiple words but should be considered as one : "hafta sonu" => "hafta_sonu"
6. Proper nouns which are multiple words but should be consireded as one : "İç Anadolu" => "İç_Anadolu"
7. Duplications

Ideas:
while tokenization:

1. Check if M.Ö. is used as an abbreviation
2. This is rule based I think. A sentence almost never ends with a cardinal number.
3. Need morphologic support for that first.
4. Seems rule based
5. After tokenization, can have a look if there is a phrase like that. If so, words could be merged
6. Same as 5
7. Issue #32 is related

for 5 and 6, see http://www.tdk.gov.tr/index.php?option=com_content&view=article&id=221:Ayri-Yazilan-Birlesik-Kelimeler&catid=50:yazm-kurallar&Itemid=132

This would be good for deciding what to do when a dot char is seen.
If it makes sense:

• numerals
• roman numerals
• etc.

don't separate it.

Same would go with other ambiguous points.