ohcnetwork / care_ocr

The Problem

Currently all the patients Electronic Health Record(EHR) are manually written and hence are prone to errors. This manual workflow also gives nurses trouble of making an extra round for recording the data.

The Solution(We are trying to achieve)

With the help of camera installed on smart ICU we can capture the image of 5 Para Monitor at predefined intervals and extract data from this image, making whole process fully automatic and this data gets updated in care module as a part of patient's EHR.

Current Blockers

• Cropping the image so that we can get rid of unecessary noises. Below shows the example:

Original Image:

Cropped Image:

• Detecting whether monitor os off or on.
• Extracting data with correct category.
• ...

Code to start with

Script-1

• care.ipynb gives visualization how image is cropped.

Script-2

• care_script.ipynb uses time displayed on monitor to crop the image and then keras-ocr to extract the data.

care_ocr's Issues

Sanity checker for wrongly parsed characters

Some characters might not be detected correctly in the OCR. A hacky but probable method to correct them can be using a dictionary that stores commonly mistaken characters, with the list of characters they are mistaken for, like this:

{
   1: ["l", "I", "i", "L", 7, 4],
   2: ["Z", "z"],
   3: [8, "e",]
...
}

Crop the image to reduce noise during OCR.

Overview

Image taken from camera installed on smart ICU captures other nearby things as well with the monitors, for example in below image there is a green board captured which makes it harder to categorize the data. Even in the monitor there is a table present which we dont require but will be read by OCR making data categorizing even harder.

What need to be done?

Crop the image so that only relevant data is left. Below shows the example of cropped image.

Current Solution

As in care_script.ipynb we are currently taking time as reference since we know the time at which image is taken from its metadata or system current time+-5 minutes we can get the bounding box of time and can extract crop image using that coordinates as a reference. But this approach is prone to errors if time of monitors is not synced or if time is not clearly visible.

Test different OCR libraries

Overview

Currently keras-ocr is being used to perform OCR on the given image, which is giving satisfactory results but is also reading english letters which are not needed for our purpose, also it misses to read decimal point in a number and reads '/' as '1'.

What needs to be done?

There are many OCR libraries in python, try them and find the best one for our purpose.

CRON Job for enabling OCR with TeleICU Middleware

• Store 5 Para Monitor Preset in Middleware{ x. y, zoom, }
• CRON Job to do:

moveCamera(preset);
getSnapshotUri(cameraOptions, callback)
fetchImageFromURL(uri)
sendImageToOCRapi
sendDataToCARE // mimics updateObservationsToCare in ObservationsController in coronasafe/teleicu_middleware

Reduce image size for a faster data extraction

Overview

Currently image provided by cameras on smart ICU is of high pixel density.

What needs to be done?

Compress image by performing several image pre processing techniques, such that the relevant data is still clearly readable and OCR accuracy on image doesn't reduce.

Clean and Verify extracted data

Overview

Since we are performing OCR on image and image can be blurred or some other noise in image can give unexpected results.

What needs to be done?

Perform verification on extracted data, whether it is in given range of category. For example in blood pressure instead of reading as '/' it is read as 1 which can give false result, for tempetature '.' is not read giving a result of original "29.7" to extracted "297", all these extracted categories require some pre-cleaning.