Xina Processor is an open source library for cleaning Arabic text. It includes various cleaning functions as well as modules for streaming file and folder cleaning.
If you use pip, you can install xinaprocessor with:
pip install xinaprocessor
You can directly clone this repo and install the library. First clone the repo with:
git clone https://github.com/xina-ai/xinaprocessor.git
Then cd to the directory and install the library with:
pip install -e .
Documentation is still in process here: https://xina-ai.github.io/xinaprocessor/
from xinaprocessor import cleaners
To clean text
Text = "نص عربي!"
Cleaner = cleaners.TextCleaner(text=Text)
Cleaner.keep_arabic_only()
To clean text File
# Creating File MyData.txt
FilePath = "MyData.txt"
with open(FilePath, "w") as f:
f.write("Aالسطر الأول\nالسطر الثاني!")
# Creating FileCleaner object
Cleaner = cleaners.FileCleaner(filepath=FilePath)
Cleaner.remove_english_text().remove_arabic_numbers().remove_punctuations()
# To access the resulted data
CleanedData = Cleaner.lines # the result will look like ['السطر الأول', 'السطر الثاني']
CleanedText = Cleaner.text # the result will look like 'السطر الأول\nالسطر الثاني'
# To save the proccessed/cleaned text to a file
Cleaner.save2file('CleanedData.txt', encoding='utf-8')
To clean large text File
# This Cleaner is used for large text files, the cleaned texts will be saved to CleanedFile.txt file
FilePath = "MyData.txt"
CleanedPath = "CleanedFile.txt"
Cleaner = cleaners.FileStreamCleaner(filepath=FilePath, savepath=CleanedPath)
Cleaner.remove_hashtags().remove_honorific_signs().drop_empty_lines().clean()