Adventures in Optical Character Recognition (OCR): Recommendations for Paper Loving Organizations 

<em><strong>Updated</strong>: September 27, 2022.</em> <h2>Optical Character Recognition (OCR)</h2> If your organization uses paper-based forms for operations, then keep reading -- you can go from files and piles of unprocessed paper forms (stuck in a data swamp) towards actionable insights (swimming in a data lake) with computer vision and Optical Character Recognition (OCR). Take paper based-data and transform it into a format that can be analyzed and acted upon. Minimize periods of <b>dark data</b> going forward. With recent advances in ML models and cloud computing, it is easier than ever to reach preliminary results with a Minimum Viable Product (MVP).   Table of contents: <ul><li><a href="#what">What is Optical Character Recognition (OCR)?</a></li><li><a href="#problem">The Problem with Digitizing Huge Piles of Photos and Papers</a></li><li><a href="#machine-learning">Applying Machine Learning and Optical Character Recognition (OCR) to the Data</a></li><li><a href="#use-cases">What Can Users Do with the Data?</a></li><li><a href="#summary">Summing Up Optical Character Recognition (OCR)</a></li></ul> <hr /> <h2 id="what">What is Optical Character Recognition (OCR)?</h2> I get it -- paper looks and feels nice. And you don't have to train a workforce to fill out paper forms. If you want to continue using paper, or if you have paper-based historical data, consider using OCR. <a href="https://codelabs.developers.google.com/codelabs/mobile-vision-ocr/#0">Optical Character Recognition (OCR)</a> gives a computer the ability to read text that appears in an image, letting applications make sense of signs, articles, flyers, pages of text, menus, or any other place where text appears as part of an image.  Naga Kiran of<a href="https://medium.com/datadriveninvestor/4-simple-steps-in-building-ocr-1f41c66099c1"> Data Driven Investor</a> puts it well: “Optical character recognition is the process of classifying optical patterns contained in a digital image. The character recognition is achieved through segmentation, feature extraction, and classification.”  Put simply, OCR is the recognition of printed or written text characters by a computer. This involves photo-scanning of the text character-by-character, analysis of the scanned-in image, and then translation of the character image into character codes, such as ASCII. <a href="https://towardsdatascience.com/@gidishperber?source=post_page-----ee1469a201aa----------------------">Gidi Shperber</a> of<a href="https://towardsdatascience.com/a-gentle-introduction-to-ocr-ee1469a201aa"> Towards Data Science</a> points out that OCR is one of the earliest addressed computer vision tasks; there were different OCR implementations even before the deep learning boom in 2012, and some even dated back to 1914. <img class="size-full wp-image-15863" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b022b5a597077171c06a8b_1-1.webp" alt="Image 1 - Huge piles of paper" width="400" height="300" /> Image 1 - Huge piles of paper Recent advances include cloud computing and improved Machine Learning models. Although the OCR problem is not yet “solved” and many challenges remain, thanks to recent advances we can build a Minimal Viable Product (MVP) faster and cheaper than ever. We encountered some of these challenges in a recent project.   <h2 id="problem">The Problem with Digitizing Huge Piles of Photos and Papers</h2> One of our clients recently requested a Machine Learning model that can read scans and photos of paper forms and convert the data to a CSV file. The company leases and maintains air conditioning and ventilation devices for numerous clients spread over a large area in Central Europe. In most instances, a staff member travels to a customer site and performs one or more service actions (inspects the machines, performs measurements, replaces liquids, cleans the machines). Sometimes, a faulty device is replaced with a functioning one. After the task is completed, they handwrite a paper form detailing the actions taken. The company generates approximately 5,000 forms a month and has a backlog of 60-80,000 forms waiting to be entered into the database. The form contains data about the location, serial numbers of equipment, services provided, cities and addresses, and checkboxes. Most of the fields in the documents are boxes for handwriting. Why would they want to digitize and analyze their huge stack of forms? To glean <b>actionable insights</b>, of course! They currently have staff people who input the forms manually. It’s a repetitive job that is a good candidate for automation.  Here are some of the challenges facing such an automation effort: <ul><li style="font-weight: 400;">The layout of the forms has changed over time. </li><li style="font-weight: 400;">There are many different types of handwriting with varying levels of legibility. </li><li style="font-weight: 400;">Some of the forms were scanned, and others photographed. Quality varies. </li><li style="font-weight: 400;">Some of the forms are held up by staff people for the photos, showing fingers and warped paper surfaces. The forms were held unsteadily resulting in unfocused images.</li><li style="font-weight: 400;">Some fields have been crossed out, and new values written around the field.</li></ul> I imagine that this company’s situation is similar to problems faced by other organizations that capture data via paper and handwriting. <h2 id="machine-learning">Applying Machine Learning and Optical Character Recognition (OCR) to the Data</h2> The Appsilon team’s approach to the problem is as follows:  <ul><li style="font-weight: 400;">Normalize -- clean up the visual “noise”, and remove the background. It’s an advanced type of thresholding. The foreground should be <b>only</b> information, nothing else.  </li><li style="font-weight: 400;">Localize the information. What are the locations of the various pieces of information on the sheet? </li><li style="font-weight: 400;">Find the relevant data fields, then section off. For fields that have bounding boxes, one can leverage them to identify relevant pieces of the form.</li><li style="font-weight: 400;">Use available Application Program Interfaces (API) for text recognition, such as<a href="https://cloud.google.com/vision/docs/ocr"> Google</a>’s.  </li><li style="font-weight: 400;">Collect scraped data in a format useful for later analysis, e.g., a CSV. </li></ul> Unfortunately, the technology will not capture 100% of the hand-written responses at this moment in time. So the machine will still require some help from friendly humans. But with some time and adjustments, it is possible to optimize the results. If your company or organization plans to leverage your paper-based data in the future, here are some recommendations to ensure a smooth transfer. Some of them are non-obvious!   <ol><li style="font-weight: 400;">Use a form that is pale pink or green (to make the fore/background split easiest). Move away from black and white scans. Extra color channels in a picture can play to your advantage!</li><li style="font-weight: 400;">Format as many of the responses as you can into checkboxes</li><li style="font-weight: 400;">Free-response entries should have boxes for characters if possible</li><li style="font-weight: 400;">Standardize formats - for example, ask for dates in dd-mm-yyyy formats</li></ol> <img class="size-full wp-image-15865" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b39d6399485861cb3b91ac_2.webp" alt="Image 2 - Origami paper boats on a Data Lake" width="400" height="267" /> Image 2 - Origami paper boats on a Data Lake <h2 id="use-cases">What Can Users Do with the Data?</h2><ul><li style="font-weight: 400;">Use decision-support dashboards that display updates on the location and condition of their equipment. </li><li style="font-weight: 400;">Filter data for specific types of problems with equipment.</li><li style="font-weight: 400;">Improve deployment of assets and inventory.</li><li style="font-weight: 400;">Trend analysis: Identify which problems happen where and when. Is there a seasonal element to certain purchases and repairs? </li><li style="font-weight: 400;">Minimize wasteful purchases (and storage of excess supplies) </li><li style="font-weight: 400;">Check the efficiency of staff and partners.  </li><li style="font-weight: 400;">Fraud prevention.  </li><li style="font-weight: 400;">Create a data lake with a 360-degree view of the organization. </li></ul> So there are a lot of options for what can be done with the data once it is transferred to an accessible digital format. When the data is available to be put to work, data scientists call it a “data lake.” Once we have a data lake, then we can build decision support systems such as dashboards that can give a company real-time information and even recommendations.   You want to get to the point where there is minimal time between the completion of the paper form and entrance into the database, where the data can be put to work. You want to endure <b>minimal</b> periods of <b>dark data</b>.    <hr /> <h2 id="summary">Summing Up Optical Character Recognition (OCR)</h2> In this post, we discussed different challenges and hacks in the OCR field. As with many problems in deep learning/computer vision, many challenges remain. On the other hand, we’ve seen it is not very hard to reach preliminary results. As with other areas of ML/AI/DS, I recommend consulting an AI team before investing in hardware and building data pipelines. Determining the highest priority business needs and then planning out the data and software requirements from there will save you a great deal of time and money in the long run. Thanks for reading! Follow me on LinkedIn.   Follow Appsilon on Social Media: <ul><li style="font-weight: 400;">Follow<a href="https://twitter.com/appsilon"> @Appsilon</a> on Twitter</li><li style="font-weight: 400;">Follow us on<a href="https://www.linkedin.com/company/appsilon"> LinkedIn</a></li><li style="font-weight: 400;">Sign up for our<a href="https://appsilon.com/blog/"> newsletter</a></li><li style="font-weight: 400;">Try out our R Shiny<a href="https://appsilon.com/opensource/"> open source</a> packages</li></ul>