paulchai / line-segmentation-algorithm-to-gcp-vision

Proposed line segmentation algorithm for Google Vision API.

Introduction

Google vision outperforms most of the cloud ocr providers. It provides two options for OCR capabilities.

• TEXT_DETECTION - Words with coordinates
• DOCUMENT_TEXT_DETECTION - OCR on dense text to extract lines and paragraph information

The second option is preferred for data extraction from normal articles (Dense Text eg- News Papers, Books). But for images with sparse text content such as retails invoices the OCR segments the lines in a different order. If the distance of two words in a single line is too far apart then google vision identifies them as two separate paragraphs/lines.

The below images shows the sample output for a typical invoice from google vision.

This behaviour creates a problem in information extraction scenarios. For example when reading a retail invoice and extracting the relevant price for the products. The algorithm proposed below provides line segmentation based on characters polygon coordinates for data extraction.

Proposed Algorithm

The implemented algorithm runs in two stages

• Stage 1 - Groups nearby words to generate a longer strip of line
• Stage 2 - Connects words which are far apart using the bounding polygon approach

Explanation.

Stage 1 should be completed because for price related text like $3.40 is presented as 2 words by Google Vision (word 1: $3. word 2:,40). The first stage helps to concat nearby characters to form a text-block/word. This step helps reduces the computation needed for the second phase.

The stage 2 algorithm draws an imaginary bounding polygon (with a threshold) over the words and computes the words which belongs to each line.

Issues.

The algorithm successfully works for most of the slanted and slightly crumpled images. But it will fail to highly crumpled or folded images.

Usage

Node JS

• cd nodejs
• npm install
• npm test

Future Work

Try to implement the water-flow algorithm for line segmentation and measure accuracies with bounding polygon approach.