starlangsoftware / turkishspellchecker-py

Turkish Spell Checker Library

Turkish Spell Checker

This tool is a spelling checker for Modern Turkish. It detects spelling errors and corrects them appropriately, through its list of misspellings and matching to the Turkish dictionary.

Video Lectures

For Developers

You can also see Cython, Java, C++, C, Swift, Js, or C# repository.

Requirements

• Python 3.7 or higher
• Git

Python

To check if you have a compatible version of Python installed, use the following command:

python -V

You can find the latest version of Python here.

Git

Install the latest version of Git.

Pip Install

pip3 install NlpToolkit-SpellChecker

Download Code

In order to work on code, create a fork from GitHub page. Use Git for cloning the code to your local or below line for Ubuntu:

git clone <your-fork-git-link>

A directory called SpellChecker will be created. Or you can use below link for exploring the code:

git clone https://github.com/starlangsoftware/TurkishSpellChecker-Py.git

Open project with Pycharm IDE

Steps for opening the cloned project:

• Start IDE
• Select File | Open from main menu
• Choose DataStructure-PY file
• Select open as project option
• Couple of seconds, project will be downloaded.

Detailed Description

• Creating SpellChecker
• Spell Correction

Creating SpellChecker

SpellChecker finds spelling errors and corrects them in Turkish. There are two types of spell checker available:

• SimpleSpellChecker

  • To instantiate this, a FsmMorphologicalAnalyzer is needed.

      fsm = FsmMorphologicalAnalyzer()
      spellChecker = SimpleSpellChecker(fsm)   
    

• NGramSpellChecker,

  • To create an instance of this, both a FsmMorphologicalAnalyzer and a NGram is required.

  • FsmMorphologicalAnalyzer can be instantiated as follows:

      fsm = FsmMorphologicalAnalyzer()
    

  • NGram can be either trained from scratch or loaded from an existing model.

    • Training from scratch:

        corpus = Corpus("corpus.txt");
        ngram = NGram(corpus.getAllWordsAsArrayList(), 1)
        ngram.calculateNGramProbabilities(LaplaceSmoothing())
      

    There are many smoothing methods available. For other smoothing methods, check here.

    • Loading from an existing model:

        ngram = NGram("ngram.txt")
      

  For further details, please check here.

  • Afterwards, NGramSpellChecker can be created as below:

      spellChecker = NGramSpellChecker(fsm, ngram)
    

Spell Correction

Spell correction can be done as follows:

sentence = Sentence("Dıktor olaç yazdı")
corrected = spellChecker.spellCheck(sentence)
print(corrected)

Output:

Doktor ilaç yazdı

turkishspellchecker-py's Issues

her cümlede spell correction çalışmıyor

merhabalar, çok güzel iş çıkarmışsınız. Fakat;
verdiğiniz test de mesela _"demokratik cumhüriyet rn kımetli varlıgımızdır"_düzeltip veriyor. ama mesela Dıktor olaç yazdi yi düzeltmiyor.

import unittest

from Corpus.Sentence import Sentence
from MorphologicalAnalysis.FsmMorphologicalAnalyzer import FsmMorphologicalAnalyzer
from NGram.NGram import NGram
from NGram.NoSmoothing import NoSmoothing

from SpellChecker.NGramSpellChecker import NGramSpellChecker


class NGramSpellCheckerTest(unittest.TestCase):

    def test_SpellCheck(self):
        """original = [Sentence("demokratik cumhuriyet en kıymetli varlığımızdır"),
               
                Sentence("")]"""
        modified = [Sentence("demokratik cumhüriyet rn kımetli varlıgımızdır"),
                Sentence("")]
        fsm = FsmMorphologicalAnalyzer("TurkishSpellChecker-Py/turkish_dictionary.txt", "TurkishSpellChecker-Py/turkish_misspellings.txt",
                                       "TurkishSpellChecker-Py/turkish_finite_state_machine.xml")
        nGram = NGram("TurkishSpellChecker-Py/ngram.txt")
        nGram.calculateNGramProbabilitiesSimple(NoSmoothing())
        nGramSpellChecker = NGramSpellChecker(fsm, nGram, True)
        for i in range(len(modified)):
            #print("Orjinal:"+original[i].toString())
            print("Pure:"+modified[i].toString())
            print("Modified:"+nGramSpellChecker.spellCheck(modified[i]).toString())
            #self.assertEqual(original[i].toString(), nGramSpellChecker.spellCheck(modified[i]).toString())

    """def test_SpellCheckSurfaceForm(self):
        fsm = FsmMorphologicalAnalyzer("TurkishSpellChecker-Py/turkish_dictionary.txt", "TurkishSpellChecker-Py/turkish_misspellings.txt",
                                       "TurkishSpellChecker-Py/turkish_finite_state_machine.xml")
        nGram = NGram("TurkishSpellChecker-Py/ngram.txt")
        nGram.calculateNGramProbabilitiesSimple(NoSmoothing())
        nGramSpellChecker = NGramSpellChecker(fsm, nGram, False)
        self.assertEqual("noter hakkında", nGramSpellChecker.spellCheck(Sentence("noter hakkınad")).__str__())
        self.assertEqual("arçelik'in çamaşır", nGramSpellChecker.spellCheck(Sentence("arçelik'in çamşaır")).__str__())
        self.assertEqual("ruhsat yanında", nGramSpellChecker.spellCheck(Sentence("ruhset yanında")).__str__())
        """
if __name__ == '__main__':
    unittest.main()

nasıl import ediliyor

merhaba,
colabda yükledim kütüphaneyi. peki nasıl import edeceğim?

test sonucu çıktı cümlesi ile teste giren cümle aynı

from Corpus.Sentence import Sentence
from NGram.NGram import NGram
from NGram.NoSmoothing import NoSmoothing
from MorphologicalAnalysis.FsmMorphologicalAnalyzer import FsmMorphologicalAnalyzer

from SpellChecker.NGramSpellChecker import NGramSpellChecker

fsm = FsmMorphologicalAnalyzer("../turkish_dictionary.txt", "../turkish_misspellings.txt",
"../turkish_finite_state_machine.xml")
nGram = NGram("../ngram.txt")
nGram.calculateNGramProbabilitiesSimple(NoSmoothing())

nGramSpellChecker = NGramSpellChecker(fsm, nGram, False)
sentence = Sentence("dışişleri mütseşarı Öymen'in 1997'nin iljk aylğrında Bağdat'a gitmesi öngörülüyor")
corrected = nGramSpellChecker.spellCheck(sentence)
print(corrected)

D:\Anaconda3\envs\deep-learning-conda-3.7\python.exe C:/Users/BURAK/Desktop/TurkishSpellChecker-Py/test/test-simple.py
dışişleri mütseşarı Öymen'in 1997'nin iljk aylğrında Bağdat'a gitmesi öngörülüyor

Process finished with exit code 0

NGramSpellChecker kullanarak bu şekilde bir cümle ile test etmek istediğimde, çıktı olarak, yine aynı sonucu alıyorum. Bu konuda yardımcı olabilir misiniz?

Platform: Windows 10
Python: Anaconda 3.7

FileNotFoundError: [Errno 2] No such file or directory: 'turkish_finite_state_machine.xml'

Merhaba ellerinize sağlık. Yukarıda bahsettiğim hatayı aldım. Githubınızda bulunan xml'i
site-packages\MorphologicalAnalysis\ ile aynı klasöre ekleyip, FiniteStateMachine.py dosyası 38. satırda root = xml.etree.ElementTree.parse(fileName).getroot() alanında fileName yerine direkt dosya yolu eklediğimde kod çalışmaya başladı sizler ile paylaşayım istedim.

pyhon 3.7.3 te aldığım hata mesajı: AttributeError: 'FsmMorphologicalAnalyzer' object has no attribute 'morphologicalAnalysisSurfaceForm'

Kütüphaneyi denemek için şöyle bir kod bloğu yazdım ancak hata aldım:
`
from Corpus.Sentence import Sentence
from MorphologicalAnalysis.FsmMorphologicalAnalyzer import FsmMorphologicalAnalyzer
from SpellChecker.SimpleSpellChecker import SimpleSpellChecker

text = "Yaızm denetleyci, veriilen mtndki yazım hatlarını bulup düzelten Nlptoolkit bilşenidir."
fsmM = FsmMorphologicalAnalyzer(dictionaryFileName='../data/turkish_dictionary.txt', misspelledFileName=None, fileName='../data/turkish_finite_state_machine.xml')
s = SimpleSpellChecker(fsmM)
sentence = Sentence(fileOrStr=text)
res = s.spellCheck(sentence)
print(res)
`

Hata Mesajı:
Traceback (most recent call last):
File "/snap/pycharm-professional/176/plugins/python/helpers/pydev/pydevd.py", line 1434, in _exec
pydev_imports.execfile(file, globals, locals) # execute the script
File "/snap/pycharm-professional/176/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/home/aerkanc/Workspaces/Python/IssueIntelligence/notebooks/spellchek-test.py", line 9, in
res = s.spellCheck(sentence)
File "/home/aerkanc/.pyenv/versions/3.7.3/lib/python3.7/site-packages/SpellChecker/SimpleSpellChecker.py", line 111, in spellCheck
fsmParseList = self.fsm.morphologicalAnalysisSurfaceForm(word.getName())
AttributeError: 'FsmMorphologicalAnalyzer' object has no attribute 'morphologicalAnalysisSurfaceForm
`

[Feature Request] Twitter, reddit, vs bot versiyonu gelir mi?