An Unsupervised Model of Orthographic Variation for Historical Document Transcription about awesome-ocr

[wanghaisheng]

摘要
Historical documents frequently exhibit extensive orthographic variation, including archaic spellings and obsolete shorthand. OCR tools typically seek to produce so-called diplomatic transcriptions that preserve these variants, but many end tasks require transcriptions
with normalized orthography. In this paper,we present a novel joint transcription model
that learns, unsupervised, a probabilistic mapping between modern orthography and that
used in the document. Our system thus produces dual diplomatic and normalized tran-
scriptions simultaneously, and achieves a 35% relative error reduction over a state-of-the-art
OCR model on diplomatic transcription, and a 46% reduction on normalized transcription.

Optical Character Recognition (OCR) for historical texts, a challenging problem due to unknown fonts
and deteriorating documents, is made even more difficult by the fact that orthographic conventions in-
cluding spelling, accent usage, and shorthands have not been consistent across the history of printing. For this reason, modern language models (LMs) yield poor performance when trying to recognize characters on the pages of these documents. Furthermore, transcription of the actual printed characters may not always be the most desirable output.

[wanghaisheng]

The Primeros Libros corpus dataset introduced in our previous work consists of multilingual (Spanish/Latin/Nahuatl) books printed in Mexico in the 1500s (Garrette et al., 2015). The original dataset includes gold diplomatic transcriptions of pages from five books of different time periods, regions, languages, and fonts. We additionally include two new monolingual Spanish Primeros Libros
books annotated with both diplomatic and normalized transcriptions. Spanish-only texts were needed in order to find language-competent annotators skilled enough to create the more challenging normalized transcriptions. We used each of the seven books in isolation since they each have a different font. For each book, we used 20 pages for training and 10 for testing. For two of the books, an additional 10 pages were held out for tuning hyperparameters with grid search

To produce the Spanish and Latin LMs, we used texts from Project Gutenberg; these documents were
written during the target historical period, but all follow modernization standards including substitution
for obsolete characters and expansion of shorthand. These texts were chosen because they are a realis-
tic sample set that is freely and publicly available.In the Nahuatl case, scarce online resources made it
necessary to supplement Project Gutenberg text withthat from a private collection.