OCR Error Correction Using Character Correction and Feature-Based Word Classification about awesome-ocr

[wanghaisheng]

摘要

OCR 文本的后处理
weighted confusion matrix and a shallow language model 可以提升准确率

this paper explores the use of a learned classifier for post-OCR text correction. Experiments with the Arabic language show that this approach, which integrates a weighted confusion matrix and a shallow language model, improves the vast majority of segmentation and recognition errors, the most frequent types of error on our dataset.

[wanghaisheng]

方法

We propose an OCR post-correction technique based on a composite machine-learning classification. The method applies a lexical spellchecker and potentially corrects single-error misspellings and a certain class of double-error misspellings, which are the major source of inaccurate recognitions in most
OCR use-cases. The novelty of this method is its ability to take into consideration several valuable word features, each giving additional information for a possible spelling correction. It is
built out of two consecutive stages:

1. word expansion based on a confusion matrix, and
2. word selection by a regression model based on word features.
   The confusion matrix and regression model are built from a transcribed set of images, while the word features rely on a language model built from a large publicly available textual
   dataset.

[wanghaisheng]

OCR 结果校正方法

There has been much research aimed at the automated correction of recognition errors for degraded collections. An early, useful survey is [1]; relevant methods for Arabic OCR are summarized in [2] and in collection [3]. In this work, we use language models on the character and word levels, plus lexicons. We do not apply morphological or syntactical analyses, nor passage-level or topic-based methods.
Three language resources play a rôle:∙

• Dictionary lookup compares OCR-output with the words in a lexicon. When there is a mismatch, one looks for alternatives within a small edit (Levenshtein) distance, under the assumption that OCR errors are often due to character insertions, deletions, and/or substitutions. For this purpose, one commonly uses a noisy-channel model,a probabilistic confusion matrix for character substitutions, and term frequency lists [4], as we do here. One must, however, take into consideration unseen (“out of vocabulary”) words, especially for morphologically-rich languages, like Greek, and even more so for abjads, like Arabic, in which vowels are not represented. The correct reading might not appear in the lexicon (even if it is not a
  named entity), while many mistaken readings will appear,because a large fraction of letter combinations form valid words. Morphological techniques could help here, of course. Dictionary lookup and shallow morphology areused in [2].∙
• We use the term k-mer for the possible contiguous character substrings of words. By collecting statistics on the relative frequency of different k-mers for a particular language, one can often recognize unlikely readings. This technique was employed by BBN’s OCR system for Arabic [5], as well as in [2].
• A language model, based on n-gram frequencies derived from a large corpus, is frequently used to estimate the likelihood of a reading in context [6].