Bicleaner AI (bicleaner-ai-classify
) is a tool in Python that aims at detecting noisy sentence pairs in a parallel corpus. It
indicates the likelihood of a pair of sentences being mutual translations (with a value near to 1) or not (with a value near to 0).
Sentence pairs considered very noisy are scored with 0.
Although a training tool (bicleaner-ai-train
) is provided, you may want to use the available ready-to-use language packages.
Please, visit https://github.com/bitextor/bicleaner-ai-data/releases/latest or use ./utils/download-pack.sh
to download the latest language packages.
Visit our Wiki for a detailed example on Bicleaner training.
If you find Bicleaner AI useful, please consider citing the following paper:
J. Zaragoza-Bernabeu, M. Bañón, G. Ramírez-Sánchez, S. Ortiz-Rojas,
"Bicleaner AI: Bicleaner Goes Neural",
in Proceedings of the 13th Language Resources and Evaluation Conference.
Marseille, France: Language Resources and Evaluation Conference, June 2022
@InProceedings{zaragozabernabeu-EtAl:2022:LREC,
author = {Zaragoza-Bernabeu, Jaume and Ramírez-Sánchez, Gema and Bañón, Marta and Ortiz Rojas, Sergio},
title = {Bicleaner AI: Bicleaner Goes Neural},
booktitle = {Proceedings of the Language Resources and Evaluation Conference},
month = {June},
year = {2022},
address = {Marseille, France},
publisher = {European Language Resources Association},
pages = {824--831},
abstract = {This paper describes the experiments carried out during the development of the latest version of Bicleaner, named Bicleaner AI, a tool that aims at detecting noisy sentences in parallel corpora. The tool, which now implements a new neural classifier, uses state-of-the-art techniques based on pre-trained transformer-based language models fine-tuned on a binary classification task. After that, parallel corpus filtering is performed, discarding the sentences that have lower probability of being mutual translations. Our experiments, based on the training of neural machine translation (NMT) with corpora filtered using Bicleaner AI for two different scenarios, show significant improvements in translation quality compared to the previous version of the tool which implemented a classifier based on Extremely Randomized Trees.},
url = {https://aclanthology.org/2022.lrec-1.87}
}
Bicleaner AI is a Bicleaner fork that uses neural networks. It comes with two types of models, lite models for fast scoring and full models for high performance. Lite models use A Decomposable Attention Model for Natural Language Inference (Parikh et al.). Full models use fine-tuned XLMRoberta (Unsupervised Cross-lingual Representation Learning at Scale).
The use of XLMRoberta and 1:10 positive to negative ratio were inspired in the winner of WMT20 Parallel Corpus Filtering Task paper (Filtering noisy parallel corpus using transformers with proxy task learning).
- Python >= 3.7
- TensorFlow >= 2.6.5
- CUDA 11.2 (for training and inference with full models)
Bicleaner AI is written in Python and can be installed using pip
:
pip install bicleaner-ai
Bicleaner AI requires the KenLM Python bindings with support for 7-gram language models. You can easily install it by running the following commands:
git clone https://github.com/kpu/kenlm
cd kenlm
pip install . --install-option="--max_order 7"
mkdir -p build && cd build
cmake .. -DKENLM_MAX_ORDER=7 -DCMAKE_INSTALL_PREFIX:PATH=/your/prefix/path
make -j all install
Hardrules uses FastSpell that requires python-dev
and libhunspell-dev
:
sudo apt install python-dev libhunspell-dev
Also note that Hunspell language packages must be installed by hand if you are going to work with one of languages listed as similar, i.e.:
sudo apt-get install hunspell-es
or downloaded from an external source, such as https://github.com/wooorm/dictionaries/tree/main/dictionaries
You can also provide the path to the Hunspell dictionaries directories by using the dictpath atribute in {/YOUR/INSTALLATION/PATH}/config/hunspell.yaml
(for example, venv/lib/python3.7/site-packages/fastspell/config/hunspell.yaml
) if you are installing from PyPI or with setup.py
, or in /config/hunspell.yaml
if you are running directly the code. Default path is /usr/share/hunspell
.
The remaining extra modules required by Bicleaner AI will be automatically downloaded and installed/upgraded (if required) with the first command.
After installation, three binary files (bicleaner-ai-train
, bicleaner-ai-classify
) will be located in your python/installation/prefix/bin
directory. This is usually $HOME/.local/bin
or /usr/local/bin/
.
TensorFlow 2 will be installed as a dependency and GPU support is required for training.
pip
will install latest TensorFlow but older versions >=2.6.5
are supported and can be installed if your machine does not meet TensorFlow CUDA requirements.
See this table for the CUDA and TensorFlow versions compatibility.
But current allowed versions only supoport CUDA 11.2.
In case you want a different TensorFlow version, you can downgrade using:
pip install tensorflow==2.6.5
TensorFlow logging messages are suppressed by default, in case you want to see them you have to explicitly set TF_CPP_MIN_LOG_LEVEL
environment variable.
For example:
TF_CPP_MIN_LOG_LEVEL=0 bicleaner-ai-classify
WARNING: If you are experiencing slow downs because Bicleaner AI is not running in the GPU, you should check those logs to see if TensorFlow is loading all the libraries correctly.
bicleaner-ai-classify
aims at detecting noisy sentence pairs in a parallel corpus. It
indicates the likelihood of a pair of sentences being mutual translations (with a value near to 1) or not (with a value near to 0). Sentence pairs considered very noisy are scored with 0.
By default, the input file (the parallel corpus to be classified) must contain at least four columns, being:
- col1: URL 1
- col2: URL 2
- col3: Source sentence
- col4: Target sentence
but the source and target sentences column index can be customized by using the --scol
and --tcol
flags.
The generated output file will contain the same lines and columns that the original input file had, adding an extra column containing the Bicleaner AI classifier score.
This tool can be run with
bicleaner-ai-classify [-h]
[-S SOURCE_TOKENIZER_COMMAND]
[-T TARGET_TOKENIZER_COMMAND]
[--header]
[--scol SCOL]
[--tcol TCOL]
[-b BLOCK_SIZE]
[-p PROCESSES]
[--batch_size BATCH_SIZE]
[--tmp_dir TMP_DIR]
[-d DISCARDED_TUS]
[--score_only]
[--calibrated]
[--raw_output]
[--disable_hardrules]
[--disable_lm_filter]
[--disable_porn_removal]
[--disable_minimal_length]
[-q]
[--debug]
[--logfile LOGFILE]
[-v]
input [output] metadata
-
positional arguments:
input
: Tab-separated files to be classified (default line format:URL1 URL2 SOURCE_SENTENCE TARGET_SENTENCE [EXTRA_COLUMNS]
, tab-separated). When input is -, reads standard input.output
: Output of the classification (default: standard output). When output is -, writes standard output.metadata
: Training metadata (YAML file), generated bybicleaner-ai-train
or downloaded as a part of a language pack. You just need tountar
the language pack for the pair of languages of the file you want to clean. The tar file contains the YAML metadata file. There's a script that can download and unpack it for you, use:
$ ./utils/download-pack.sh en cs ./models
to download English-Czech language pack to the ./models directory and unpack it.
-
optional arguments:
-h, --help
: show this help message and exit
-
Optional:
-S SOURCE_TOKENIZER_COMMAND
: Source language tokenizer full command (including flags if needed). If not given, Sacremoses tokenizer is used (withescape=False
option).-T TARGET_TOKENIZER_COMMAND
: Target language tokenizer full command (including flags if needed). If not given, Sacremoses tokenizer is used (withescape=False
option).--scol SCOL
: Source sentence column (starting in 1). If--header
is set, the expected value will be the name of the field (default: 3 if--header
is not set else src_text)--tcol TCOL
: Target sentence column (starting in 1). If--header
is set, the expected value will be the name of the field (default: 4 if--header
is not set else trg_text)--tmp_dir TMP_DIR
: Temporary directory where creating the temporary files of this program (default: default system temp dir, defined by the environment variable TMPDIR in Unix)-b BLOCK_SIZE, --block_size BLOCK_SIZE
: Sentence pairs per block (default: 10000)-p PROCESSES, --processes PROCESSES
: Number of processes to use (default: all CPUs minus one)-d DISCARDED_TUS, --discarded_tus DISCARDED_TUS
: TSV file with discarded TUs. Discarded TUs by the classifier are written in this file in TSV file. (default: None)--lm_threshold LM_THRESHOLD
: Threshold for language model fluency scoring. All sentence pairs whose LM fluency score falls below the threshold are removed (classifier score set to 0), unless the option --keep_lm_result is set. (default: 0.5)--score_only
: Only output one column which is the bicleaner score (default: False)--calibrated
: Output calibrated scores (default: False)--raw_output
: Return raw output without computing positive class probability. (default: False)--disable_hardrules
: Disables the bicleaner_hardrules filtering (only bicleaner_classify is applied) (default: False)--disable_lm_filter
: Disables LM filtering.--disable_porn_removal
: Disables porn removal.--disable_minimal_length
: Don't apply minimal length rule (default: False).
-
Logging:
-q, --quiet
: Silent logging mode (default: False)--debug
: Debug logging mode (default: False)--logfile LOGFILE
: Store log to a file (default: <_io.TextIOWrapper name='' mode='w' encoding='UTF-8'>)-v, --version
: show version of this script and exit
bicleaner-ai-classify \
corpus.en-es.raw \
corpus.en-es.classifed \
model/en-es/metadata.yaml
This will read the corpus.en-es.raw
file,
classify it with the classifier indicated in the models/en-es/metadata.yaml
metadata file,
writing the result of the classification in the corpus.en-es.classified
file.
Each line of the new file will contain the same content as the input file, adding a column with the score given by the Bicleaner classifier.
In case you need to train a new classifier (i.e. because it is not available in the language packs provided at bicleaner-ai-data), you can use bicleaner-ai-train
.
bicleaner-ai-train
is a Python tool that allows you to train a classifier which predicts
whether a pair of sentences are mutual translations or not and discards too noisy sentence pairs. Visit our Wiki for a detailed example on Bicleaner AI training.
In order to train a new classifier, you must provide:
- A clean parallel corpus (500k pairs of sentences is the recommended size).
- Monolingual corpus for the source and the target language (not necessary for
xlmr
classifier). - Gzipped lists of monolingual word frequencies. You can check their format by downloading any of the available language packs.
- The SL list of word frequencies with one entry per line. Each entry must contain the following 2 fields, split by space, in this order: word frequency (number of times a word appears in text), SL word.
- The TL list of word frequencies with one entry per line. Each entry must contain the following 2 fields, split by space, in this order: word frequency (number of times a word appears in text), TL word.
- These lists can easily be obtained from a monolingual corpus and a command line in bash:
$ cat monolingual.SL \
| sacremoses -l SL tokenize -x \
| awk '{print tolower($0)}' \
| tr ' ' '\n' \
| LC_ALL=C sort | uniq -c \
| LC_ALL=C sort -nr \ \
| grep -v '[[:space:]]*1' \
| gzip > wordfreq-SL.gz
$ cat monolingual.TL \
| sacremoses -l TL tokenize -x \
| awk '{print tolower($0)}' \
| tr ' ' '\n' \
| LC_ALL=C sort | uniq -c \
| LC_ALL=C sort -nr \ \
| grep -v '[[:space:]]*1' \
| gzip > wordfreq-TL.gz
Optionally, if you want the classifier to include a porn filter, you must also provide:
- File with training dataset for porn removal classifier. Each sentence must contain at the beginning the
__label__negative
or__label__positive
according to FastText convention. It should be lowercased and tokenized.
It can be used as follows.
bicleaner-ai-train [-h]
-m MODEL_DIR
-s SOURCE_LANG
-t TARGET_LANG
[--mono_train MONO_TRAIN]
--parallel_train PARALLEL_TRAIN
--parallel_dev PARALLEL_DEV
[-S SOURCE_TOKENIZER_COMMAND]
[-T TARGET_TOKENIZER_COMMAND]
[-F TARGET_WORD_FREQS]
[--block_size BLOCK_SIZE]
[-p PROCESSES]
[-g GPU]
[--mixed_precision]
[--save_train_data SAVE_TRAIN_DATA]
[--distilled]
[--seed SEED]
[--classifier_type {dec_attention,transformer,xlmr}]
[--batch_size BATCH_SIZE]
[--steps_per_epoch STEPS_PER_EPOCH]
[--epochs EPOCHS]
[--patience PATIENCE]
[--pos_ratio POS_RATIO]
[--rand_ratio RAND_RATIO]
[--womit_ratio WOMIT_RATIO]
[--freq_ratio FREQ_RATIO]
[--fuzzy_ratio FUZZY_RATIO]
[--neighbour_mix NEIGHBOUR_MIX]
[--porn_removal_train PORN_REMOVAL_TRAIN]
[--porn_removal_test PORN_REMOVAL_TEST]
[--porn_removal_file PORN_REMOVAL_FILE]
[--porn_removal_side {sl,tl}]
[--noisy_examples_file_sl NOISY_EXAMPLES_FILE_SL]
[--noisy_examples_file_tl NOISY_EXAMPLES_FILE_TL]
[--lm_dev_size LM_DEV_SIZE]
[--lm_file_sl LM_FILE_SL]
[--lm_file_tl LM_FILE_TL]
[--lm_training_file_sl LM_TRAINING_FILE_SL]
[--lm_training_file_tl LM_TRAINING_FILE_TL]
[--lm_clean_examples_file_sl LM_CLEAN_EXAMPLES_FILE_SL]
[--lm_clean_examples_file_tl LM_CLEAN_EXAMPLES_FILE_TL]
[-q]
[--debug]
[--logfile LOGFILE]
-
positional arguments:
input
: Tab-separated bilingual input file (default: Standard input)(line format: SOURCE_SENTENCE TARGET_SENTENCE, tab-separated)
-
optional arguments:
-h, --help
: show this help message and exit
-
Mandatory:
-m MODEL_DIR, --model_dir MODEL_DIR
: Model directory, metadata, classifier and SentencePiece models will be saved in the same directory (default: None)-s SOURCE_LANG, --source_lang SOURCE_LANG
: Source language (default: None)-t TARGET_LANG, --target_lang TARGET_LANG
: Target language (default: None)--mono_train MONO_TRAIN
: File containing monolingual sentences of both languages shuffled together, used to train SentencePiece embeddings. Not required for XLMR. (default: None)--parallel_train PARALLEL_TRAIN
: TSV file containing parallel sentences to train the classifier (default: None)--parallel_dev PARALLEL_DEV
: TSV file containing parallel sentences for development (default: None)
-
Options:
-S SOURCE_TOKENIZER_COMMAND, --source_tokenizer_command SOURCE_TOKENIZER_COMMAND
: Source language tokenizer full command (default: None)-T TARGET_TOKENIZER_COMMAND, --target_tokenizer_command TARGET_TOKENIZER_COMMAND
: Target language tokenizer full command (default: None)-F TARGET_WORD_FREQS, --target_word_freqs TARGET_WORD_FREQS
: R language gzipped list of word frequencies (needed for frequence based noise) (default: None)--block_size BLOCK_SIZE
: Sentence pairs per block when apliying multiprocessing in the noise function (default: 10000)-p PROCESSES, --processes PROCESSES
: Number of process to use (default: 71)-g GPU, --gpu GPU
: Which GPU use, starting from 0. Will set the CUDA_VISIBLE_DEVICES. (default: None)--mixed_precision
: Use mixed precision float16 for training (default: False)--save_train_data SAVE_TRAIN_DATA
: Save the generated dataset into a file. If the file already exists the training dataset will be loaded from there. (default: None)--distilled
: Enable Knowledge Distillation training. It needs pre-built training set with raw scores from a teacher model. (default: False)--seed
: SEED Seed for random number generation. By default, no seeed is used. (default: None)--classifier_type {dec_attention,transformer,xlmr}
: Neural network architecture of the classifier (default: dec_attention)--batch_size BATCH_SIZE
: Batch size during classifier training. If None, default architecture value will be used. (default: None)--steps_per_epoch STEPS_PER_EPOCH
: Number of batch updates per epoch during training. If None, default architecture value will be used or the full dataset size. (default: None)--epochs EPOCHS
: Number of epochs for training. If None, default architecture value will be used. (default: None)--patience PATIENCE
: Stop training when validation has stopped improving after PATIENCE number of epochs (default: None)--pos_ratio POS_RATIO
: Ratio of positive samples used to oversample on validation and test sets (default: 1)--rand_ratio RAND_RATIO
: Ratio of negative samples misaligned randomly (default: 3)--womit_ratio WOMIT_RATIO
: Ratio of negative samples misaligned by randomly omitting words (default: 3)--freq_ratio FREQ_RATIO
: Ratio of negative samples misaligned by replacing words by frequence (needs --target_word_freq) (default: 3)--fuzzy_ratio FUZZY_RATIO
: Ratio of negative samples misaligned by fuzzy matching (default: 0)--neighbour_mix NEIGHBOUR_MIX
: If use negative samples misaligned by neighbourhood (default: False)--porn_removal_train PORN_REMOVAL_TRAIN
: File with training dataset for FastText classifier. Each sentence must contain at the beginning the '__label__negative' or '__label__positive' according to FastText convention. It should be lowercased and tokenized. (default: None)--porn_removal_test PORN_REMOVAL_TEST
: Test set to compute precision and accuracy of the porn removal classifier (default: None)--porn_removal_file PORN_REMOVAL_FILE
: Porn removal classifier output file (default: porn_removal.bin)--porn_removal_side {sl,tl}
: Whether the porn removal should be applied at the source or at the target language. (default: sl)--noisy_examples_file_sl NOISY_EXAMPLES_FILE_SL
: File with noisy text in the SL. These are used to estimate the perplexity of noisy text. (default: None)--noisy_examples_file_tl NOISY_EXAMPLES_FILE_TL
: File with noisy text in the TL. These are used to estimate the perplexity of noisy text. (default: None)--lm_dev_size LM_DEV_SIZE
: Number of sentences to be removed from clean text before training LMs. These are used to estimate the perplexity of clean text. (default: 2000)--lm_file_sl LM_FILE_SL
: SL language model output file. (default: None)--lm_file_tl LM_FILE_TL
: TL language model output file. (default: None)--lm_training_file_sl LM_TRAINING_FILE_SL
: SL text from which the SL LM is trained. If this parameter is not specified, SL LM is trained from the SL side of the input file, after removing --lm_dev_size sentences. (default: None)--lm_training_file_tl LM_TRAINING_FILE_TL
: TL text from which the TL LM is trained. If this parameter is not specified, TL LM is trained from the TL side of the input file, after removing --lm_dev_size sentences. (default: None)--lm_clean_examples_file_sl LM_CLEAN_EXAMPLES_FILE_SL
: File with clean text in the SL. Used to estimate the perplexity of clean text. This option must be used together with --lm_training_file_sl and both files must not have common sentences. This option replaces --lm_dev_size. (default: None)--lm_clean_examples_file_tl LM_CLEAN_EXAMPLES_FILE_TL
: File with clean text in the TL. Used to estimate the perplexity of clean text. This option must be used together with --lm_training_file_tl and both files must not have common sentences. This option replaces --lm_dev_size. (default: None)
-
Logging:
-q, --quiet
: Silent logging mode (default: False)--debug
: Debug logging mode (default: False)--logfile LOGFILE
: Store log to a file (default: <_io.TextIOWrapper name='' mode='w' encoding='UTF-8'>)
bicleaner-ai-train \
--parallel_train corpus.en-cs.train \
--parallel_dev corpus.en-cs.dev \
--mono_train mono.en-cs \
-m models/en-cs \
-s en \
-t cs \
-F wordfreqs-cs.gz \
--lm_file_sl models/en-cs/lm.en --lm_file_tl models/en-cs/lm.cs \
--porn_removal_train porn-removal.txt.en --porn_removal_file models/en-cs/porn-model.en \
This will train a lite classifier for English-Czech using the corpus corpus.en-cs.train
, the corpus.en-cs.dev
as development set and the monolingual corpus mono.en-cs
to train the vocabulary embeddings.
All the model files created during training, the language model files, the porn removal file, and the metadata.yaml
will be stored in the model directory models/en-cs
.
To train full models you would need to use --classifier_type xlmr
and --mono_train
is not needed.
By default the training will use rand_ratio
, womit_ratio
and freq_ratio
options with a value of 3.
Both womit_ratio
and freq_ratio
will use Sacremoses tokenizer by default.
So, for languages that are not supported by this tokenizer or are poorly supported, source_tokenizer_command
and/or target_tokenizer_command
should be provided.
Also note that, if a tokenizer command is used, the word frequencies need to be tokenized in the same way to allow noise based on frequency work correctly.
If no tokenization is available for your languages, you can disable these noise option that use tokenization and use fuzzy mathing noise: --womit_ratio 0 --freq_ratio 0 --fuzzy_ratio 6
.
To set the maximum number of threads/processes to be used during training or classifying, --processes
option is no longer available.
You will need to set BICLEANER_AI_THREADS
environment variable to the desired value.
For example:
BICLEANER_AI_THREADS=12 bicleaner-ai-classify ...
If the variable is not set, the program will use all the available CPU cores.
A comparison of the speed in number of sentences per second between different types of models and hardware:
model | speed CPUx1 | speed GPUx1 |
---|---|---|
full | 1.78 rows/sec | 200 rows/sec |
lite | 600 rows/sec | 10,000 rows/sec |
- CPU: Intel Core i9-9960X single core (lite model batch 16, full model batch 1)
- GPU: Nvidia V100 (lite model batch 2048, full model batch 16)
All documents and software contained in this repository reflect only the authors' view. The Innovation and Networks Executive Agency of the European Union is not responsible for any use that may be made of the information it contains.