This experiment tests whether a large language model can "read past" OCR errors. Specifically, we identify a series of sentences from a corpus that contain OCR errors, which we then correct. Then, we send the erroneous sentence and the fixed sentence to a model and retrieve embeddings for both, with the hope that these embeddings will be interchangeable.
- Test corpus: a collection of 18-19C medical texts compiled by Sarah Bull
- Model used: Llama-2 7B, quantized down to 4-bits with llama.cpp
Process:
-
Run the following:
./src/sample_sents.py --indir path/to/data --outfile path/to/sample.txt
This will yank sentences from the documents using
nltk
'ssent_tokenize()
function. That's by no means a perfect method, but we just need some examples, so it'll do -
Manually read the sentences and identify ones with bad OCR. Place those in a new plaintext file. Each sentence should be on its own line
-
Create a new plaintext file and, on each line, provide a corrected version of the sentences above. Ensure that the lines of each file contain properly aligned sentences.
-
Embed the sentences with the model:
./src/llama_embed.py \ --uncorrected path/to/uncorrected.txt \ --corrected path/to/corrected.txt \ --embeddings path/to/embeddings.jsonl \ --model_path path/to/model
-
Compare the embeddings in
analysis.ipynb
-
Optional: To compare multiple embedding types, run the following:
./src/shuffle_and_measure.py \ --paths path/to/*.jsonl \ --outfile path/to/output.csv
This will use the
shuffle_and_measure
function inanalysis.ipynb
and store the result in a table -
Optional: Compare multiple embeddings in
analysis_aggregate.ipynb