OpusCleaner is a machine translation/language model data cleaner and training scheduler. The Training scheduler has moved to OpusTrainer.
The cleaner bit takes care of downloading and cleaning multiple different datasets and preparing them for translation.
opuscleaner-clean --parallel 4 data/train-parts/dataset.filter.json | gzip -c > clean.gz
If you just want to use OpusCleaner for cleaning, you can install it from PyPI, and then run it
pip3 install opuscleaner
opuscleaner-server serve
Then you can go to http://127.0.0.1:8000/ to show the interface.
You can also install and run OpusCleaner on a remote machine, and use SSH local forwarding (e.g. ssh -L 8000:localhost:8000 [email protected]
) to access the interface on your local machine.
(Mainly listed as shortcuts to documentation)
- FastAPI as the base for the backend part.
- Pydantic for conversion of untyped JSON to typed objects. And because FastAPI automatically supports it and gives you useful error messages if you mess up things.
- Vue for frontend
List and categorize the datasets you are going to use for training.
Download more datasets right from the interface.
Filter each individual dataset, showing you the results immediately.
Compare the dataset at different stages of filtering to see what the impact is of each filter.
data/train-parts
is scanned for datasets. You can change this by setting theDATA_PATH
environment variable, the default isdata/train-parts/*.*.gz
.filters
should contain filter json files. You can change theFILTER_PATH
environment variable, the default is<PYTHON_PACKAGE>/filters/*.json
.
cd frontend
npm clean-install
npm run build
cd ..
python3 -m venv .env
bash --init-file .env/bin/activate
pip install -e .
Finally you can run opuscleaner-server
as normal. The --reload
option will cause it to restart when any of the python files change.
opuscleaner-server serve --reload
Then go to http://127.0.0.1:8000/ for the "interface" or http://127.0.0.1:8000/docs for the API.
If you're doing frontend development, try also running:
cd frontend
npm run dev
Then go to http://127.0.0.1:5173/ for the "interface".
This will put vite in hot-reloading mode for easier Javascript dev. All API requests will be proxied to the python server running in 8000, which is why you need to run both at the same time.
If you want to use LASER, you will also need to download its assets:
python -m laserembeddings download-models
Run npm build
in the frontend/
directory first, and then run hatch build .
in the project directory to build the wheel and source distribution.
This project has received funding from the European Union’s Horizon Europe research and innovation programme under grant agreement No 101070350 and from UK Research and Innovation (UKRI) under the UK government’s Horizon Europe funding guarantee [grant number 10052546]