<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Amannair on Medium]]></title>
        <description><![CDATA[Stories by Amannair on Medium]]></description>
        <link>https://medium.com/@amannair723?source=rss-c1f786da7919------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*TtBH5Zzls--iQsneTpD64w.jpeg</url>
            <title>Stories by Amannair on Medium</title>
            <link>https://medium.com/@amannair723?source=rss-c1f786da7919------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 20:05:52 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@amannair723/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Automating PDF data extraction with AI]]></title>
            <link>https://amannair723.medium.com/automating-pdf-data-extraction-with-ai-4d28ec053420?source=rss-c1f786da7919------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d28ec053420</guid>
            <category><![CDATA[pdf-extraction]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Amannair]]></dc:creator>
            <pubDate>Thu, 04 Apr 2024 21:09:18 GMT</pubDate>
            <atom:updated>2024-04-04T21:09:18.420Z</atom:updated>
            <content:encoded><![CDATA[<h4>Extract PDF data using ML scripting (free ware) in Excel</h4><p>Table of content</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/386/1*zMk7VMoYmL7wpoKIiXu1iQ.jpeg" /></figure><p>Here my aim is to bring in all the techniques/method(along with its code snippet) used in extracting information from the pdf. These snippets can be plugged into the pipeline to increase the overall accuracy of the system(Note these methods are more effective for semi-structured pdf). Some of the code snippets are pretty common yet powerful enough to boost the performance. Interpolation of these codes helps in increasing the accuracy of document ai.</p><h4>What is a semi-structured PDF?</h4><p>A PDF where the template is almost redundant like (invoice, financial report, insurance….etc). These pdf follow the same kind of format and can be tackled via scripting (writing down a set of rules) or using deep-learning models. To tackle this kind of problem one should extrapolate between all the ocr approaches to extract information (feel free to use the methods discussed below and change them according to your needs) and find the most suitable accuracy. <br><strong><em>Notes:</em></strong> <br><em>• Even though a PDF might look alike visually there comes a lot of hidden challenges while scripting hence make sure it is generic to all PDF <br>• Even if a pdf is in readable format convert it into an image this helps in standardizing across all domains of pdf. The image should have a fixed (channel, width, height) through all its transformation (as most of the pdf are monocentric hence keep the default as grayscale).</em></p><h4>What are the different OCR solutions?</h4><p>There are a couple of market-dominant ocr solutions both proprietary and open-source let&#39;s go through each one of them one by one.</p><p><strong><em>1)In the Proprietary Sector:</em></strong></p><p>•<strong>Acrobat X</strong><em>(</em><a href="https://experienceleague.adobe.com/docs/document-services/tutorials/pdfservices/ocr.html?lang=en"><em>source</em></a><em>)</em> — This is an API that takes in the entire image in a document(PDF) format. Using(SEARCHABLE_IMAGE_EXACT) one can convert the image into a text PDF. Mostly used for digital image documents (Its performance decreases on handwritten text). Below is a sample image/code.</p><p>•<strong>Google OCR</strong><em>(</em><a href="https://cloud.google.com/vision/docs/ocr"><em>source</em></a><em>)</em>— This is an API where the contents of the pdf file with base64 encoded are sent in the body of the request. After performing feature detection on the pdf file a JSON is sent back which includes page, block, paragraph, word, and break information. Below is a sample image/code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NlXo9BnguboMAaPWHdIN1g.jpeg" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e7993d1d557c3ddfb713cdfe9f0a13e1/href">https://medium.com/media/e7993d1d557c3ddfb713cdfe9f0a13e1/href</a></iframe><p>•<strong>Azure OCR</strong><em>(</em><a href="https://docs.microsoft.com/en-us/azure/cognitive-services/computer-vision/concept-recognizing-text"><em>source</em></a><em>)</em>- <a href="https://westcentralus.dev.cognitive.microsoft.com/docs/services/computer-vision-v3-ga/operations/5d986960601faab4bf452005"><em>Read API</em></a> that extracts text from images/PDF documents and also can be executed asynchronously. We call the reader operation that takes the input argument as an image and returns the operation ID. Then we call the <a href="https://westcentralus.dev.cognitive.microsoft.com/docs/services/computer-vision-v3-ga/operations/5d9869604be85dee480c8750">GET result operation</a> whose input parameter is the operation ID and returns a JSON consisting of the text extracted from the image along with its bounding box. The ocr can read (English, Spanish, German, French, Italian, Portuguese, and Dutch) language printed text and English handwritten text at different angles, colors, and fonts. Below is a sample image/code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*rWZ_qMvC_CswAOZ3KYfC9w.jpeg" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/eca220f5a1a7a9264f060415fc9477ec/href">https://medium.com/media/eca220f5a1a7a9264f060415fc9477ec/href</a></iframe><p>•<strong>IBM OCR</strong><em>(</em><a href="https://github.com/IBM/MAX-OCR"><em>source</em></a><em>)</em> — This model takes an image as an input and returns the predicted text. This model can recognize 8 different font types for printed text. The API model can be called via cloning the git and building the docker image. Then with a GET request, we can send the image in the body and receive a JSON with text. Below is a sample image/code<em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y4PYiBTsiNyX7cZDVE2rxg.jpeg" /></figure><pre>$ git clone <a href="https://github.com/IBM/MAX-OCR.git">https://github.com/IBM/MAX-OCR.git</a><br>$ cd MAX-OCR<br>$ docker build -t max-ocr .<br>$ docker run -it -p 5000:5000 max-ocr<br>$ docker run -it -e CORS_ENABLE=&#39;true&#39; -p 5000:5000 max-ocr</pre><pre>import requests</pre><pre>payload = {document_image.jpeg} # document image</pre><pre>headers = {&#39;Content-Type&#39;: &#39;multipart/form-data&#39;}<br>try:<br>    url = &#39;<a href="http://localhost:5000/model/predict?&#39;">http://localhost:5000/model/predict?&#39;</a>)<br>    response = requests.request(&quot;POST&quot;, url, headers=headers, data = payload )<br>    data = response.json()<br>    print(data)<br>except Exception as e:<br>    print(&quot;[Errno {0}] {1}&quot;.format(e.errno, e.strerror))</pre><p><strong>2)In the OpenSource Sector:</strong><br>• <strong>Tesseract</strong><em>(</em><a href="https://tesseract-ocr.github.io/tessdoc/Command-Line-Usage.html#simplest-invocation-to-ocr-an-image"><em>source</em></a><em>)</em>- The most popular open-source ocr used is Tesseract(which is a combination of text detection and recognition algorithm) it deals with Word, Line finding Character classification. Line finding is done using row spacing and proportional text size while text lines are split into words using character spacing. It also supports a wide range of languages. Pytesseract is the Tesseract library for Python development. The most commonly used configurations are [-l lang] [ — oem ocrenginemode] [ — psm pagesegmode]. There are four types of ocr engine using the — oem option:<br>0 — Legacy engine only.<br>1 — Neural nets LSTM engine only.<br>2 — Legacy + LSTM engines.<br>3 — Default, based on what is available.<br>There are 14 types of ocr engine using the — psm option:<br>0 — Orientation and script detection (OSD) only. <br>1 — Automatic page segmentation with OSD.<br>2 — Automatic page segmentation, but no OSD, or OCR.<br>3 — Fully automatic page segmentation, but no OSD. (Default)<br>4 — Assume a single column of text of variable sizes.<br>5 — Assume a single uniform block of vertically aligned text.<br>6 — Assume a single uniform block of text.<br>7 — Treat the image as a single text line.<br>8 — Treat the image as a single word.<br>9 — Treat the image as a single word in a circle.<br>10 — Treat the image as a single character.<br>11 — Sparse text. Find as much text as possible in no particular order.<br>12 — Sparse text with OSD.<br>13 — Raw line. Treat the image as a single text line, bypassing hacks that are Tesseract-specific.<br>Below is a sample code.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4e8bf43221ddd172d7d4948ff69fd41f/href">https://medium.com/media/4e8bf43221ddd172d7d4948ff69fd41f/href</a></iframe><p><strong>Custom OCR techniques</strong><br> <strong>Text detection</strong>:</p><p><strong><em>EAST </em></strong>( <a href="https://arxiv.org/pdf/1704.03155.pdf"><em>Efficient accurate scene text detector</em></a>) The EAST network is actually a version of the well-known U-Net, which is good for detecting features of different fonts. It is a simple yet powerful approach for text detection.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ee97a5b99dd348870dc9ae44d786a983/href">https://medium.com/media/ee97a5b99dd348870dc9ae44d786a983/href</a></iframe><p><strong><em>CRAFT </em></strong>(<a href="https://arxiv.org/pdf/1904.01941.pdf"><em>Character Region Awareness for Text Detection</em></a>) — scene text detection method to effectively detect text area by exploring each character and affinity between characters to overcome the lack of individual character level annotations. <a href="https://colab.research.google.com/drive/1Q5OzXGhtxCaiS8n_fd-z2X19kgTwPLml?usp=sharing">Below is a sample code</a>.</p><p><a href="https://colab.research.google.com/drive/1Q5OzXGhtxCaiS8n_fd-z2X19kgTwPLml?usp=sharing">Google Colaboratory</a></p><p><strong><em>Yolo</em></strong>(<a href="https://arxiv.org/pdf/1506.02640.pdf">You Only Look Once</a>) — YOLO is highly generalizable it is less likely to break down when applied to new domains or unexpected inputs (Such as text). While it can quickly identify objects in images it struggles to precisely localize some objects, especially small ones. A single neural network predicts bounding boxes and class probabilities directly from full images in one evaluation. Since the whole detection pipeline is a single network, it can be optimized end-to-end directly on detection performance.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6a7ce323285eed6a9ddaa44603e3a5ad/href">https://medium.com/media/6a7ce323285eed6a9ddaa44603e3a5ad/href</a></iframe><h4>How to Get better OCR results?</h4><p>As ocr works on images we could enhance the quality of the image before sending it to an ocr which could increase the accuracy. <strong>Opencv</strong><em>(</em><a href="https://opencv.org/"><em>source</em></a><em>)</em>- It comes with a lot of image filtering techniques. This comes in handy for document images. Before performing any OCR solutions on the image we should enhance the quality of the image. Some of the commonly used Opencv(techniques) are:<br>•<strong><em>Transform</em></strong><em>(</em><a href="https://docs.opencv.org/2.4/modules/imgproc/doc/geometric_transformations.html"><em>source</em></a><em>)</em>- When an image is clicked at an angle that is not an image top this technique helps in getting the angle right. It is used to set the image to a particular angle. These are also called geometrical transforms most commonly used are affine, and perspective. (Note: if the angle is far-fetched there could be a loss of data). Below is a sample code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fPkyf2MnNwRIuzxsjOPaYQ.jpeg" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/002c41d67e4a4eac91616fcf09e2ecd3/href">https://medium.com/media/002c41d67e4a4eac91616fcf09e2ecd3/href</a></iframe><p>•<strong><em>Thresholding</em></strong><em>(</em><a href="https://docs.opencv.org/master/d7/d4d/tutorial_py_thresholding.html"><em>source</em></a><em>) </em>— Since text can be represented in grayscale, as in document images other dimensions are not required. Scans with poor quality can be enhanced by filling in the partial grey pixel with a black or white pixel. This can also be done relative to the scaling pattern. The below code provides the different types of thresholding.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MuZ2urtu9PtN57--_xVCAA.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/32ca1ea0dedaebfe188f76d2a4d6385f/href">https://medium.com/media/32ca1ea0dedaebfe188f76d2a4d6385f/href</a></iframe><p>•<strong><em>Deskewing</em></strong>- Many times the images come tilted in an x-y plane removing skewness from the text could help in increasing the ocr quality of reading the text. Here the mean angle of all text lines is taken and then the image is rotated accordingly. Below is a sample code.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/145e75c54e59f7a0e77c5787b3459d0f/href">https://medium.com/media/145e75c54e59f7a0e77c5787b3459d0f/href</a></iframe><p>•<strong><em>Sharpening</em></strong> — When thresholding is not able to enhance the quality of the image a usual trick is to sharpen the image. This helps in increasing the clarity of the image after which thresholding can be applied. Also using a noise filter can help. Below is a sample code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4gAI4PnFyFkPEIOMvWIfDA.png" /></figure><pre>import cv2<br>import numpy as np</pre><pre>image = cv2.imread(&#39;document_image.png&#39;,0)<br>kernel = np.array([[-1,-1,-1], <br>                   [-1, 9,-1],<br>                   [-1,-1,-1]])<br>kernel = 1/3 * kernel<br>sharpened = cv2.filter2D(image, -1, kernel)<br>plt.imshow(sharpened , cmap = &#39;gray&#39;)<br>plt.title(&#39;sharp Filter&#39;)<br>plt.show()</pre><p>•<strong>Noise filtering </strong>— Using a noise filter along with the above methods can help at times to increase the ocr accuracy. Some of the noise parameters used in Document images are given below.</p><pre>import cv2<br>from PIL import Image<br>from matplotlib import pyplot as plt</pre><pre>image = cv2.imread(&#39;document_image.png&#39;,0)<br>new_image = cv2.GaussianBlur(image, (figure_size, figure_size),0)<br>image = Image.fromarray(image.astype(&#39;uint8&#39;))<br>new_image = image.filter(ImageFilter.UnsharpMask(radius=2, percent=150))<br>plt.imshow(new_image, cmap = &#39;gray&#39;)<br>plt.title(&#39;Unsharp Filter&#39;)<br>plt.show()</pre><h4>Information Extraction Step by step process</h4><p>Note- As there is no one size fits all solution. Each of the below processes can have multiple pipelines depending on the document process similarity. Create clusters of documents that flow through a different pipeline. For example invoice preprocessing here has 5 pipelines i.e. 5 clusters of documents with similarities to generate an optimum image to OCR. The information extraction can have 3 clusters based on similarity from OCR.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xYKAHh6NGfpmm7WGTdIQ0g.jpeg" /></figure><p><strong>Preprocessing</strong> — The primary step is to take the raw PDF and convert it into an OCR readable format. Most of the image processing happens in these pipelines. Extrapolating on different image processing techniques can help in getting a boosted performance. Some of the most useful ones are thresholding, transformation, sharpening, and noise filtering. At the end of this module, all the images should have the same (h,w,c). Useful techniques like padding can help attain uniform image size.</p><p><strong>OCR</strong> — The images can now be converted to digital character values. The primary focus is to get the correct character in the exact coordinate from the image and convert them into an XML or a PDF format for standardization. OCR Accuracy — To detect the accuracy of an ocr is tough where the test data is not present. There are a couple of ways to handle such cases</p><ol><li>Sampling some of the data with ground truth data — extract a random bunch of documents and their ground truth. Get the F1 score to have a basic idea of accuracy.</li><li>Get a confidence score for each character printed.</li><li>Word level accuracy checks spelling, grammar, and language correctness (Software like Grammarly is helpful).</li><li>Use library lexical or a dict to create some rules specific to the ocr output. It might be a consistent pattern across all the output that needs to be tweaked or it might be some case scenario.</li></ol><p><strong>Information</strong> <strong>Extraction</strong> — This module focuses on using business logic to extract information. It receives a digitized pdf/XML which is parsed to find out the relevant information and put it across a structured format. Usually, key-value pairs are extracted via rules, deep learning (information extraction), and structural extraction (at times information is present at a specific location in the document, by knowing the relative index of such places one can extract the information directly).</p><p><strong>Postprocessing</strong> — This module contains validation in the extracted data, humans in the loop, and business rules structuring the output data to a particular format as it is intended. Validation is usually done via ground truth data and humans in the loop is an avid way to check the key extraction points. Most of the external plugins are used in this module to connect with various third-party vendors to store or progress further.</p><p><strong>SUMMARY<br></strong>All the methods mentioned above need to be worked work with each other for the best possible outcome. Remember data extraction for a PDF does not always look as it works (i.e. sometimes even an easy-looking PDF can have complex XML stacked over one another as PDF crosses through various systems from software to OS to Hardware orders). All of which can impact the way a PDF should work.</p><p>Comment down below if you need any clarification on any of the methods or processes</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d28ec053420" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[FLASK in Colab deploying in Heroku]]></title>
            <link>https://amannair723.medium.com/flask-in-colab-deploying-in-heroku-80a479263b17?source=rss-c1f786da7919------2</link>
            <guid isPermaLink="false">https://medium.com/p/80a479263b17</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Amannair]]></dc:creator>
            <pubDate>Sat, 18 Jul 2020 13:25:35 GMT</pubDate>
            <atom:updated>2020-07-18T13:30:07.104Z</atom:updated>
            <content:encoded><![CDATA[<h3>FLASK in Google Colab deploying through Heroku</h3><h4>A complete guide to make an AI leveraged web app in google colab and cost-efficient depyloment on cloud(leveraging tools to build and increase accuracy of model)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nChvXtRg_1WoZ11Udrez5Q.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/691/1*LPYnvwGZ0oPoQ-yOpMa58g.png" /></figure><p><strong>Aim :</strong>To make an end-to-end AI web app with the power of google colab(GPU/TPU) , cloud web services (scalability , encrypted) with cost-efficient deployment while maximizing the leverage on helper tools/libraries to support building the app</p><p><strong><em>Perquisite</em></strong>:we need to install the below mentioned library and make two folders (static and templates)</p><pre>!pip install flask-ngrok<br>%mkdir templates -p<br>%mkdir static/css -p</pre><h4>How to create html &amp; css file in colab:</h4><p>There are several ways of creating the file creating an html file we can create a blank text file in linux and rename as html file or using python open command to write a file and save it as json but the best way I found was to create a html file is first create a general file with this command %%writefile(creates output code developed in a Notebook via Python module) and save it under templates folder with name “index.html” .We can add as many html pages in the template folder.<strong><em>(Note:The css document can also be created with the same code as below only the desired path changes to “Static”)</em></strong></p><pre>#===============writing html file directly in colab===============</pre><pre>%%writefile templates/index.html<br>&lt;!DOCTYPE html&gt;<br>&lt;html &gt;<br>&lt;!--From <a href="https://codepen.io/frytyler/pen/EGdtg--">https://codepen.io/frytyler/pen/EGdtg--</a>&gt;<br>&lt;head&gt;<br>  &lt;meta charset=&quot;UTF-8&quot;&gt;<br>  &lt;title&gt;SPAM_API&lt;/title&gt;<br>  &lt;link href=&#39;<a href="https://fonts.googleapis.com/css?family=Pacifico&#39;">https://fonts.googleapis.com/css?family=Pacifico&#39;</a> rel=&#39;stylesheet&#39; type=&#39;text/css&#39;&gt;<br>&lt;/head&gt;</pre><pre>&lt;body&gt;<br> &lt;div class=&quot;login&quot;&gt;<br>     &lt;h1&gt;Spam detection&lt;/h1&gt;<br>     &lt;!-- Main Input For Receiving Query to our ML --&gt;<br>     &lt;form action=&quot;{{ url_for(&#39;predict&#39;)}}&quot;method=&quot;post&quot;&gt;<br>     &lt;textarea id=&quot;styled&quot; name=&quot;message&quot; rows=&quot;6&quot; cols=&quot;50&quot; placeholder=&quot;Enter Your Message Here&quot; required=&quot;required&quot; &gt;&lt;/textarea&gt;</pre><pre>     &lt;button type=&quot;submit&quot; class=&quot;btn btn-primary btn-block btn-large&quot;&gt;Predict&lt;/button&gt;<br>    &lt;/form&gt;</pre><pre>   &lt;br&gt;<br>   &lt;br&gt;<br>   {{ prediction_text }}</pre><pre>&lt;/div&gt;</pre><pre>&lt;/body&gt;<br>&lt;/html&gt;</pre><pre>#========here is the code for using linux cmd in colab=============<br>#write the same html file as above<br>!cd templates<br>!cat &gt; index.html<br>#manually open the file and write the html code</pre><pre>#================using sys cmd in python ========================<br>file = open(&quot;/templates/index.html,&quot;w&quot;)<br>html_code = &quot;&quot;&quot;<br>     write your html code here<br>     &quot;&quot;&quot;<br>file.writelines(html_code) <br>file.close()</pre><h4>How to write flask API model:</h4><p>Make sure the helper function to predict the values are also saved using %%writefile, and imported to the app.py along with a h5/bin/pkl file of the trained model.<strong><em>(Note: models run faster in GPU hence a good time to take the advantage of the google colab)</em></strong>.We can make the flask app run over the internet by using the flask_ngrok. This is helpful before putting out the code for deployment. The API created can be test and debug as any other web-app service(i.e loadbalance , frontend-backend connectivity, UI visuality over different devices etc.)<br>• Import with from flask_ngrok import run_with_ngrok<br>• Add run_with_ngrok(app) to make your Flask app available upon running</p><pre>%%writefile app.py<br><em>#importing libraries</em><br>import numpy as np<br>from flask import Flask, request, jsonify, render_template<br>from flask_ngrok import run_with_ngrok<br>import pickle</pre><pre><em>#creating the flask object</em><br>app = Flask(__name__)<br>run_with_ngrok(app)</pre><pre><em>#loading the model weights</em><br>filename = r&quot;/content/new_nlp_model.pkl&quot;<br>clf = pickle.load(open(filename, &#39;rb&#39;))<br>cv=pickle.load(open(r&quot;/content/new_tranform.pkl&quot;,&#39;rb&#39;))</pre><pre><em>#create routes</em><br><a href="http://twitter.com/app">@app</a>.route(&#39;/&#39;)<br>def home():<br>    return render_template(&#39;index.html&#39;)</pre><pre><a href="http://twitter.com/app">@app</a>.route(&#39;/predict&#39;,methods=[&#39;POST&#39;])<br>def predict():<br>    &#39;&#39;&#39;<br>    <em>For rendering results on HTML GUI</em><br>    &#39;&#39;&#39;<br>    if request.method == &#39;POST&#39;:<br>        message = request.form[&#39;message&#39;]<br>        data = [message]<br>        vect = cv.transform(data).toarray()<br>        my_prediction = clf.predict(vect)<br>        if my_prediction == 1:<br>            output = &quot;a Spam&quot;<br>        elif my_prediction == 0:<br>            output = &quot;Not a Spam&quot;<br>    <br>    outputs = &#39;This email is &#39;+output</pre><pre>    return render_template(&#39;index2.html&#39;, prediction_text=outputs , value=message)</pre><pre>if __name__ == &quot;__main__&quot;:<br>    app.run()</pre><h4>How to deploy a model:</h4><p>Once the model is trained and tested along with its web api service,now it is time to deploy the model. Heroku and render are two free services to deploy the model. Create an account in Heroku.First create a Procfile with the below given code make sure that name of the python file is app and the flask object name is app .Below is the code for the same</p><pre>web: gunicorn app:app</pre><p>We also need a file with requirement.txt which contains the name of all the library to be imported in the python complier</p><pre>Flask==1.1.1<br>gunicorn=19.9.0<br>numpy&gt;=1.9.2<br>matplotlib&gt;=1.4.3<br>Pandas&gt;=0.19</pre><p>Upload all the files in git repository i.e. static, templates, procfile.txt, app.py, requirement.txt, model.pkl</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gGVCzjrNtULRYocYUl0ipQ.jpeg" /><figcaption>Sign_in page</figcaption></figure><p>In your heroku account: <br>• Create a New app -&gt; write an app name <br>• Connect to your github account<br>• Enable automatic deployment or deploy Branch as required<br>• In the end of deployment it will give you a link eg (https://app_name.herokuapp.com/) <br>• Click on view to see the app</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d7ZM6fGqldyS0Z50p2YXvw.png" /><figcaption>creating app</figcaption></figure><p>To check the logs download the heroku CLI <br>• Open the cmd/terminal -&gt; type in heroku login<br>• Login via web browser<br>• Type in heroku — app app_name</p><h4><strong>Conclusion:</strong></h4><p>Make use of all the tools which supports building and increasing the accuracy of a DL model in a cost-efficient manner. Here we use the computing power of colab and the cloud deployment service of heroku to build and deploy a web api.</p><h4>References</h4><p>[1] <a href="https://devcenter.heroku.com/articles/heroku-cli">https://devcenter.heroku.com/articles/heroku-cli</a><br>[2] <a href="https://github.com/gstaff/flask-ngrok">https://github.com/gstaff/flask-ngrok</a><br>[3] <a href="https://help.heroku.com/UMAUQ4UF/why-am-i-seeing-application-error">https://help.heroku.com/UMAUQ4UF/why-am-i-seeing-application-error</a><br>[4] <a href="https://trailhead.salesforce.com/en/content/learn/projects/develop-heroku-applications/create-a-heroku-app">https://trailhead.salesforce.com/en/content/learn/projects/develop-heroku-applications/create-a-heroku-app</a><br>[5] <a href="https://readwrite.com/2014/09/23/heroku-for-beginners-app-hosting-101/">https://readwrite.com/2014/09/23/heroku-for-beginners-app-hosting-101/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=80a479263b17" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[PDF to Excel using advance python NLP and Computer Vision aka Document AI]]></title>
            <link>https://amannair723.medium.com/pdf-to-excel-using-advance-python-nlp-and-computer-vision-aka-document-ai-23cc0fb56549?source=rss-c1f786da7919------2</link>
            <guid isPermaLink="false">https://medium.com/p/23cc0fb56549</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[nlu]]></category>
            <category><![CDATA[cv]]></category>
            <dc:creator><![CDATA[Amannair]]></dc:creator>
            <pubDate>Thu, 16 Jul 2020 11:17:15 GMT</pubDate>
            <atom:updated>2020-07-24T12:09:43.071Z</atom:updated>
            <content:encoded><![CDATA[<h3>PDF to Excel using advance python NLP and Computer Vision AKA Document AI</h3><h4>The aim is to convert all kinds of PDF data to a spreadsheet where the PDF data is structured aka Information extraction. This is a general overview of the tools and methodology used in Document AI</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nZggADgxmEQ6_cS6z6JQqA.png" /><figcaption><strong>Document AI</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/1*tqYvyBb7E3tyBaqJFGF3TA.png" /></figure><h3><strong>PDF Types &amp; Structures:</strong></h3><p>Today most of the document are in the form of a PDF either it is a scanned PDF or a text based extracted PDF. These PDF are can be divided into structured / semi-structured / unstructured data. To freed the PDF data into a database in some structured format these documents(PDF) needs to be first extracted there are several ways to extract the data depending on the type of PDF</p><ol><li>Scanned PDF — (structured / semi-structured / unstructured data / text in the wild)</li><li>Readable PDF — (structured / semi-structured / unstructured data)</li></ol><ul><li>The advantage of readable PDF over scanned documents is that the text are to be extracted are already in digital format hence there is no task what so ever to find the text in the image and recognize the text</li><li>While the disadvantage of Readable PDF is that some times it so happens that the PDF are of different file formats or encoding where the best practice work around is <strong>convert PDF to an image file</strong>, then use any recent OCR software (which reconstruct table automatically from the picture) to get data.</li></ul><p><strong>Scanned PDF </strong>: These documents are challenging as there are a number of hidden task attached to it</p><ul><li>Finding the text also get it‘s relative coordinates with respect to the scanned page</li><li>recognizing the text (printed / handwritten)</li></ul><blockquote><em>To make things easy there are tools that do it end to end although none have achieved 100% accuracy in all the task but have divided the documents into various categories such as (Invoice ,ID Card, Purchase Orders , Income Proof, Tax Form , Mortgage forms)</em></blockquote><p><strong>1.Structured PDF</strong>— If it is a tabular data we can use <a href="https://pypi.org/project/camelot-py/"><strong><em>camelot</em></strong></a> , <a href="https://pypi.org/project/tabula-py/"><strong><em>tabula</em></strong></a> or <a href="https://pypi.org/project/pdftotext/"><strong><em>pdftotext</em></strong></a> library to directly convert the data into a dataframe. To extract tabular content either the use of vertical and horizontal lines we can identify a table through this else the libraries mentioned help us in identifying the table . Other convenient hack for readable PDF is to get the table columns , header and footer and deciphering it on the basis of it coordinate with respect to the column area</p><pre>import camelot<br>import pyPdf2 as pyPdf<br>from tabula import read_pdf<br>from matplotlib.pyplot import plt</pre><pre>tables = camelot.read_pdf(&#39;invoice.pdf&#39; ,flavor=&#39;stream&#39;)<br>camelot.plot(tables[0], kind=&#39;text&#39;)<br>plt.show()<br>camelot.plot(tables[0], kind=&#39;grid&#39;)<br>plt.show()<br><br>reader = pyPdf.PdfFileReader(open(&quot;C:\Users\riley\Desktop\Bank Statements\50340.pdf&quot;, mode=&#39;rb&#39; ))<br>n = reader.getNumPages() <br><br>df = []<br>for page in [str(i+1) for i in range(n)]:<br>    if page == &quot;1&quot;:<br>            df.append(read_pdf(r&quot;C:\Users\riley\Desktop\Bank Statements\50340.pdf&quot;, area=(530,12.75,790.5,561), pages=page))<br>    else:<br>            df.append(read_pdf(r&quot;C:\Users\riley\Desktop\Bank Statements\50340.pdf&quot;, pages=page))</pre><p><strong>2.Structured PDF &amp; semi-structured PDF </strong>— If it is a text PDF then <a href="https://pypi.org/project/pdfminer/"><strong><em>PDFMiner</em></strong> </a>, <a href="https://pypi.org/project/PyPDF2/"><strong><em>PyPDF2</em></strong></a>, <a href="https://pypi.org/project/pdfquery/"><strong><em>PDFQuery</em></strong> </a>, <a href="https://pypi.org/project/xpdf-python/"><strong><em>xpdf-python</em></strong> </a>converts the data into textual format as the text are structured then we can use search based information extraction i.e. [keywords surrounding the entity(search for the keyword left to it and a keyword right to it ) or (diagonal to it ) or (above and below to the text needed to be extracted)].While this is good for structured data but in semi-structured data the randomization can be removed on the bases of increasing complexity <strong><em>Using</em></strong> <strong><em>Regex</em></strong>(check image hacks to get the code)-Simple data such as phone number , pincode , email ID , company initials etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/1*pjF5JZ8fRFQvywDh3_m65Q.png" /><figcaption>Spacy(<a href="https://pypi.org/project/spacy/">source</a>)</figcaption></figure><p><strong><em>Using deep learning algorithms-</em></strong>most popularly used are libraries of <a href="https://pypi.org/project/spacy/"><strong><em>spacy </em></strong></a>and <a href="https://pypi.org/project/transformers/"><strong><em>transformer</em></strong></a> .Pattern recognition either by algorithm or with a trained CNN1D model can be used. <em>Relation Extraction</em> — Extracting the relationship between the entity and and nodes within it. <em>Named Entity Recognition</em> (NER)— The task of tagging entities in text with their corresponding type. Typically use BIO notation, (B) the beginning of the entity (I)inside continuation of the token and (O) outside of the entity.</p><pre>import tensorflow as tf<br>import tensorflow_datasets<br>from transformers import *<br><br># Load dataset, tokenizer, model from pretrained model/vocabulary<br>tokenizer = BertTokenizer.from_pretrained(&#39;bert-base-cased&#39;)</pre><pre>model = BertForTokenClassification.from_pretrained(&#39;bert-base-uncased&#39;)<br><br># If you used to have this line in pytorch-pretrained-bert:<br>loss = model(input_ids, labels=labels)<br><br># Now just use this line in transformers to extract the loss from the output tuple:<br>outputs = model(input_ids, labels=labels)<br>loss = outputs[0]<br><br># In transformers you can also have access to the logits:<br>loss, logits = outputs[:2]<br><br># And even the attention weights if you configure the model to output them (and other outputs too, see the docstrings and documentation)<br>model = BertForTokenClassification.from_pretrained(&#39;bert-base-uncased&#39;, output_attentions=True)<br>outputs = model(input_ids, labels=labels)<br>loss, logits, attentions = outputs</pre><p><strong>3.Unstructured PDF</strong>(Thesis in the best of my knowledge) -Can only be solved by a machine learning engine with the gained knowledge on the specific area of interest(i.e. the document domain invoice ,research paper financial papers etc) with the combination off all the models strategies listed below <br> • <strong><em>Reinforcement learning</em></strong> where the environment for the model to learn could be set as an online test environment with video of the test would help in engaging on task such as masking , question answering , match the following <br> • <strong><em>Relation Extraction</em></strong> along with NER could help in predicting Entity Linking (Entity-aware Attention for Single Shot Visual Text Extraction)<br> • <strong><em>Information Retrieval </em></strong>— The task of ranking a list of entity or search results of entity in response to a query or attention focused upon <br> • <strong><em>Language Modelling</em></strong>(Dependency Parsing , Sentence Pair Modeling , Natural Language Inference) — <em>Dependency parsing</em> is a grammatical representation of a sentence to extract relation between “head” word and associate words, which modify those heads. <em>Sentence Pair Modeling</em> is to pair the sentences through its embedding which roots to its similarity. <em>Natural language inference</em> is the task where for a given “premise” the model predicts whether a “hypothesis” is entailment contradiction, or undetermined(i.e. true, false or neutral).</p><h3>The General <strong>Pipeline(Theory)</strong>:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SUGvIl2RXecxw0WwCWd6Zg.png" /><figcaption><strong>General Pipeline</strong></figcaption></figure><p>1. <strong>Preprocessing</strong>: Most scanned Documents are noisy have artefacts and thus for the OCR and information extraction pipeline to work well, it is necessary to pre-process the Documents images. Common pre-processing methods include — Greyscaling, Noise removal and Thresholding (Binarization):<br> • <strong><em>Grayscaling</em></strong>: converting a 3 chanel image into a single channel in simple terms black and white pixles image.<br> • <strong><em>Noise removal</em></strong>: typically involves removing Poisson Noise,Speckle Noise,Salt and Pepper noise or Gaussian noise.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*R7dI-FBOD3EIcsnon9byPA.jpeg" /><figcaption><a href="https://docs.opencv.org/master/d7/d4d/tutorial_py_thresholding.html"><strong>Image filtering</strong></a><strong>(</strong><a href="https://docs.opencv.org/master/d7/d4d/tutorial_py_thresholding.html"><strong>source</strong></a><strong>)</strong></figcaption></figure><p>• <strong><em>Thresholding</em></strong>: Most OCR engines work well on grayscale images. This can be achieved by thresholding, which is the assignment of pixel values in relation to the threshold value provided. Each pixel value is compared with the threshold value. If the pixel value is smaller than the threshold, it is set to 0 a black pixel, otherwise, it is set to a maximum value (generally 255 a white pixel).The most commonly used thresholding technique is Adaptive Thresholding and Simple Thresholding.<br> 2. <strong>Text Detection &amp; recognition (OCR)</strong>: The next step in the pipeline is OCR. It is used to read text from images such as a scanned document or a picture. This technology is used to convert, virtually any kind of images containing written text (typed, handwritten or printed) into machine-readable text data. OCR involves 2 steps — text detection and text recognition. <strong><em>Text detector</em></strong> — the basic openCV countor detection to deeplearing models(craft , East) , <strong><em>Text recognizing</em></strong> — after the text is given out clean cropped image just containing the text to be extracted then we can use all the ocr tools or custom deeplearning models for recognition. There are a number of approaches to OCR.<br> The conventional computer Vision approach is to:<br> • <strong><em>Using filters to separate the characters from the background — </em></strong>Applying filters ,masking and blurring images help in classification of characters better for text recognition.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*wro9WwD_dohYmureSwu_xQ.png" /><figcaption><strong>Countor detection(source)</strong></figcaption></figure><ul><li><strong><em>Apply contour detection to recognize the filtered characters —</em></strong>contour matching is mostly used as a patch code in a semi-structure document while finding countor we have to specify the masking ratio and mostly the text in the document are of varing fonts hence we have to write down all the possible masking ratio, it requires a lot of efforts to generalize for all the document types. They are usually used in template specific problem where the document is static and there is not a lot of movement then ROI is taken into consideration to extract the text.</li><li><strong><em>Use image classification to identify the characters — </em></strong>Deep Learning approaches generalizes very well. The most popular approaches for text detection is EAST and CRAFT .<br> <strong><em>EAST </em></strong>( <a href="https://arxiv.org/pdf/1704.03155.pdf"><em>Efficient accurate scene text detector</em></a>) is a scene text detection which is based on u-net the model can predict arbitrary orientations words or text lines in an image.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*lNGSsrdPmLNYulzhWmX0cQ.jpeg" /><figcaption><a href="https://arxiv.org/pdf/1704.03155.pdf"><strong>EAST ARCHITECTURE</strong></a><strong>(</strong><a href="https://arxiv.org/pdf/1704.03155.pdf"><strong>source</strong></a><strong>)</strong></figcaption></figure><p><em>CRAFT </em>(<a href="https://arxiv.org/pdf/1904.01941.pdf"><em>Character Region Awareness for Text Detection</em></a>) —scene text detection method effectively detect text area via affinity between characters and by focusing on each character,exploiting both the given character level annotations from synthetic images and the estimated character-level (ground-truths) for real images acquired by the learned interim model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/813/1*yLSZYzvN5XGEWSSjpiJi_g.png" /><figcaption><a href="https://arxiv.org/pdf/1904.01941.pdf"><strong>CRAFT ARCHITECTURE</strong></a><strong>(</strong><a href="https://arxiv.org/pdf/1904.01941.pdf"><strong>source</strong></a><strong>)</strong></figcaption></figure><p>3.<strong>Information Extraction: </strong>The traditional approach to extract information was to divide the document into specific template and then write down rules for each particular template type. This used to be exhaustive but were very accurate. While the new techniques does not demolish the older approach but adds on a new pipeline making the documents more generic and not template oriented.Methods like object detection is being used to detect different types of object within the document for example research paper has [formulas, tables, paragraphs, list and figures]. Before performing the ocr the document image is cropped into different labeled images making it easier to extract the relevant information.Commonly used object detection algorithms are YOLO and Faster R-CNN.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UJql_BW4ErrBJnDEDAYzrA.png" /><figcaption><a href="https://arxiv.org/pdf/1506.02640"><strong>YOLO ARCHITECTURE</strong></a><strong>(</strong><a href="https://arxiv.org/pdf/1506.02640"><strong>source</strong></a><strong>)</strong></figcaption></figure><p>4.<strong>Data Dump:</strong> Once the information is extracted the data can be stored in any format required usually it is eithere feeded to a database or a spreedsheet but converting the data into a json is quite handy at times.</p><h3>Image Hacks:</h3><ol><li><strong>Pre_processing :</strong> The general thumb rules of pre_processing image are deskew and cropping the images , increasing sharpness and contrast , adaptive thresholding , affine perspective , gaussian noise removal and resizing of the image use of template matching comes handy when identifying the logo or watermarks.</li></ol><pre>import cv2<br>import numpy as np</pre><pre>img = cv2.imread(&#39;image.jpg&#39;)</pre><pre># get grayscale image<br>def get_grayscale(image):<br>    return cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)</pre><pre># noise removal<br>def remove_noise(image):<br>    return cv2.medianBlur(image,5)<br> <br>#thresholding<br>def thresholding(image):<br>    return cv2.threshold(image, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)[1]</pre><pre>#dilation<br>def dilate(image):<br>    kernel = np.ones((5,5),np.uint8)<br>    return cv2.dilate(image, kernel, iterations = 1)<br>    <br>#erosion<br>def erode(image):<br>    kernel = np.ones((5,5),np.uint8)<br>    return cv2.erode(image, kernel, iterations = 1)</pre><pre>#opening - erosion followed by dilation<br>def opening(image):<br>    kernel = np.ones((5,5),np.uint8)<br>    return cv2.morphologyEx(image, cv2.MORPH_OPEN, kernel)</pre><pre>#canny edge detection<br>def canny(image):<br>    return cv2.Canny(image, 100, 200)</pre><pre>def denoising(image):<br>  # image should have3 channel   <br>  b,g,r = cv2.split(img)<br>  # switch it to rgb         <br>  rgb_img = cv2.merge([r,g,b])<br>  # Denoising<br>  dst = cv2.fastNlMeansDenoisingColored(img,None,10,10,7,21)<br>  # get b,g,r<br>  b,g,r = cv2.split(dst)           <br>  return cv2.merge([r,g,b])</pre><pre>#skew correction<br>def deskew(image):<br>    coords = np.column_stack(np.where(image &gt; 0))<br>    angle = cv2.minAreaRect(coords)[-1]<br>     if angle &lt; -45:<br>        angle = -(90 + angle)<br>    else:<br>        angle = -angle<br>    (h, w) = image.shape[:2]<br>    center = (w // 2, h // 2)<br>    M = cv2.getRotationMatrix2D(center, angle, 1.0)<br>    rotated = cv2.warpAffine(image, M, (w, h), flags=cv2.INTER_CUBIC, borderMode=cv2.BORDER_REPLICATE)<br>    return rotated</pre><pre>#template matching<br>def match_template(image, template):<br>    return cv2.matchTemplate(image, template, cv2.TM_CCOEFF_NORMED)</pre><p>when the text is too noisy and general pre_processing does not work then cropping the desired area works and sending it to a model works</p><pre>import cv2 <br>import pytesseract</pre><pre>img = cv2.imread(&#39;image.jpg&#39;)</pre><pre># Adding custom options<br>custom_config = r&#39;--oem 3 --psm 6&#39;<br>pytesseract.image_to_string(img, config=custom_config)</pre><p>Some more analysis with tesseract with openCV</p><pre>import cv2<br>import pytesseract<br>import numpy as np</pre><pre>image = cv2.imread(&#39;Document_image.jpg&#39;)<br>img = image.copy()<br>mask = np.zeros(image.shape, dtype=np.uint8)<br>gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)<br>thresh = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)[1]</pre><pre># Filter for ROI using contour area and aspect ratio<br>countour = cv2.findContours(thresh, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)<br>countour = countour[0] if len(countour) == 2 else countour[1]<br>for c in countour:<br>    area = cv2.contourArea(c)<br>    peri = cv2.arcLength(c, True)<br>    approx = cv2.approxPolyDP(c, 0.05 * peri, True)<br>    x,y,w,h = cv2.boundingRect(approx)<br>    aspect_ratio = w / float(h)<br>    if area &gt; 2000 and aspect_ratio &gt; .5:<br>        mask[y:y+h, x:x+w] = image[y:y+h, x:x+w]</pre><pre>h, w, c = img.shape<br>boxes = pytesseract.image_to_boxes(img) <br>for b in boxes.splitlines():<br>    b = b.split(&#39; &#39;)<br>    img = cv2.rectangle(img, (int(b[1]), h - int(b[2])), (int(b[3]), h - int(b[4])), (0, 255, 0), 2)</pre><pre>d = pytesseract.image_to_data(img, output_type=Output.DICT)<br>n_boxes = len(d[&#39;text&#39;])<br>for i in range(n_boxes):<br>    if int(d[&#39;conf&#39;][i]) &gt; 60:<br>        (x, y, w, h) = (d[&#39;left&#39;][i], d[&#39;top&#39;][i], d[&#39;width&#39;][i], d[&#39;height&#39;][i])<br>        img2 = cv2.rectangle(img, (x, y), (x + w, y + h), (0, 255, 0), 2)</pre><pre>cv2.imshow(&#39;img&#39;, img)<br>cv2.imshow(&#39;img2&#39;, img2)<br># Perfrom OCR with Pytesseract<br>data = pytesseract.image_to_string(mask, lang=&#39;eng&#39;, config=&#39;--psm 6&#39;)<br>print(data)<br>cv2.imshow(&#39;thresh&#39;, thresh)<br>cv2.imshow(&#39;mask&#39;, mask)</pre><p>2. <strong>Post_processing:</strong> OCR tends to produce erroneous output when the quality of the input image is poor or when the input image has not been pre processed correctly.Extracting the list of OCR errors and their corresponding correctly spelt words is done by making use of 2 similarity measures namely similarity in meaning and similarity on a character level. Following this, a dictionary is used to distinguish the correctly spelt word from the erroneous word.</p><pre>#values in dict<br>df1 = pd.read_excel(r&quot;C:\Users\Testing Custom Dictionaries_v2.xlsx&quot;)[&quot;Diagnosis Codes&quot;].tolist()</pre><pre>def listfromdict(ocr_value):<br>    <br>    #second letter to digit<br>    if not ocr_value[1].isdigit():<br>        ocr_value_list = list(ocr_value)<br>        ocr_value_list[1] = str(letter2number.get(ocr_value[1]))<br>        ocr_value = &#39;&#39;.join(ocr_value_list)</pre><pre>    #first letter to alpha<br>    valalphalist = []<br>    if not ocr_value[0].isalpha():<br>        ocr_value_list = list(ocr_value)<br>        alphalist = number2letter.get(ocr_value[0]) #check for value<br>        for i,let in enumerate(alphalist):<br>            ocr_value_list[0] = let<br>            valalphalist.append(&#39;&#39;.join(ocr_value_list))<br>    else:<br>        ocr_value_list = list(ocr_value)<br>        ocr_value_list[0] = ocr_value[0].upper()<br>        ocr_value = &#39;&#39;.join(ocr_value_list)<br>        valalphalist.append(ocr_value)</pre><pre>return ocr_value</pre><p>Regex — we extract all information related to the number pattern</p><pre>import re<br># regex patter for date asuming format as DD:MM:YYYY</pre><pre>datePattern = r&#39;(0[1-9]|[12][0-9]|3[01])[:](0[1-9]|1[012])[:](19|20)\d\d&#39;<br>date = re.search(datePattern, raw_text).group()<br><br>print(date)</pre><h3>Useful tools/library:</h3><p><strong><em>Note: Here is a general overview of all the advancement and methodology</em></strong></p><p><strong>List of ocr tools/libraries(Note: there are a lot of ocr solution but the best in the market):</strong></p><ol><li><a href="https://github.com/tesseract-ocr/tesseract">Tesseract-ocr</a></li><li>Ocr engine</li></ol><ul><li><a href="https://cloud.google.com/vision/docs/ocr">Google vision api</a></li><li><a href="https://azure.microsoft.com/en-in/services/cognitive-services/computer-vision/">Azure vision</a></li><li><a href="https://developer.ibm.com/technologies/artificial-intelligence/models/max-ocr/">IBM</a></li><li><a href="https://docs.aws.amazon.com/rekognition/latest/dg/text-detection.html">Amazon Rekognition</a></li><li><a href="https://pdf.abbyy.com/">Abbyy fineReader</a></li></ul><p><strong>List of document ai tools that do help in the end to end extraction:</strong></p><ol><li><a href="https://azure.microsoft.com/en-in/services/cognitive-services/form-recognizer/"><em>Form recognizer</em></a></li><li><a href="https://aws.amazon.com/textract/"><em>Amazon Textract</em></a></li><li><a href="https://cloud.google.com/solutions/document-ai">Google document ai</a></li><li><a href="https://www.abbyy.com/flexicapture/"><em>Abbyy flexicapture</em></a></li><li><a href="https://rossum.ai/"><em>Rossum api</em></a></li></ol><p><strong>Deep learning document AI models trying to achieve the end to end modelling or plays a part in the pipeline of document AI:</strong></p><p>1.<strong><em>EATEN( </em></strong><a href="https://arxiv.org/abs/1909.09380"><strong><em>Entity-aware Attention for Single Shot Visual Text Extraction</em></strong></a><strong><em> )</em></strong> -The model proposed in this paper has a series of Entity-aware decoders. After feature extraction from the image it passes through an Entity-aware attention network where each of the Entity-aware decoders is accountable for predicting a certain set of predefined entities/entity.</p><p>The model accuracy in evaluated on the basis of mean entity accuracy which gives an improvement over the existing methods.The accuracy increases if the concept of pattern blurring is used to increase attention which plays huge advantage over traditional methods.</p><p>2. <strong>Object detection and Image segmentation</strong> — Documents can be considered as a live templates consisting of sparse sections i.e. the sections in the live templates mostly similar but they are organized in different formats. For example in case of an invoice document the various components are [Company logo, From address, To address, Tables , Disclaimer , GST , Invoice details, etc.]. If we treat them as different object we can use object detection algorithm to extract these components. Using object detection can come with its own challenges as in the case of table extraction (sometimes graphs , formula and images are a part of table), different font-styles, different languages, template variation, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*w9gwONZH-M3mTNK6qIgzWg.png" /><figcaption>CNN architecture(<a href="https://neurohive.io/wp-content/uploads/2018/10/AlexNet-1.png">source</a>)</figcaption></figure><p>For Document parsing using Object detection, firstly segmentation of the page based of non-text and text area .Later, these are segregated and looked individually while extract different component from the ROI, such as tables, sections from the non-text parts basically object detection and then classification. The use of document images as a regular image opens concepts of object detection , segmentation and classification new ways to tackle parsing over traditional rule-based methods where generalization is difficult and a lot of dependent variables are involved.</p><p>At ICDAR (<a href="https://ieeexplore.ieee.org/document/8269977"><strong><em>International Conference on Document Analysis and Recognition</em></strong></a>) a paper proposed by Xiaohan Yi and his team. The abstract of the paper was to detect region of interest from document images using CNNs. To detect objects in the image, they used rough proposal and pruning strategy. At first Rough Proposal a kind of Breadth-First Search (BFS) is used to find all the eight connected component areas (after filtering the binary image). All these component are replaced by bounding rectangle once generated which apparently removes the irrelevant information from the images. In the second part columns in pages are detected also filter the regions existing in multi-columns by using pruning strategy . This paper was specifically designed to extract information from research papers, they used an architecture of a Spatial Pyramid Pooling (SPP) on top of the VGG-16 Network creates fixed size feature maps with the help of fixed scale down-sampling.The optimizer used was SGD(Stochastic Gradient Descent) to train the network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/733/1*lJh5ooJXkaiEgxpcRqQINg.png" /><figcaption>Object Detection in Document Images(<a href="https://ieeexplore.ieee.org/document/8269977">source</a>)</figcaption></figure><p>• <strong><em>Segmentation:</em></strong> “<a href="https://clgiles.ist.psu.edu/pubs/CVPR2017-connets.pdf">Learning to Extract Semantic Structure from Documents Using Multimodal Fully Convolutional Neural Networks</a>” a collaboration between Adobe and The Pennsylvania State University. They proposed a multi-modal CNN network for extracting semantic structures from PDF. This model takes a PDF image as input and splits it into different segment(regions of interest) then recognize the label of each region and results out the segment labels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1013/1*n7ahr1ma5skRFJhAHY4nVA.png" /><figcaption>Document segmentation(<a href="https://clgiles.ist.psu.edu/pubs/CVPR2017-connets.pdf">source</a>)</figcaption></figure><p>The images are not annotated instead there is a text embedding (a map of text similarity). Text embedding can differentiate between caption , paragraphs and list. This kind of text separation creates a semantics structure analysis which mirrors an image semantics logic where categorizes of each region is semantically-relevant classes like tables, list and Section headings. Below is an image representing the same an image input and a segmented labeled output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ix7jZnEjj6f_imXqKCqjyA.png" /><figcaption>Image segmentation(<a href="https://clgiles.ist.psu.edu/pubs/CVPR2017-connets.pdf">source</a>)</figcaption></figure><h3><strong>Conclusion:</strong></h3><p>All in all, Document AI model is a very broad and challenging topic was quite a difficult topic to tackle. I have tried my best towards generalizing and providing insights over this area of research which might certainly help you go further into more depth of Document AI. Hope that you found this article helpful.</p><h3>Reference:</h3><p><strong><em>(NOTE: All the papers including its image description are linked below)</em></strong>[1]Xinyu Zhou, Cong Yao, He Wen, Yuzhi Wang, Shuchang Zhou, Weiran He, and Jiajun Liang, EAST: <a href="https://arxiv.org/pdf/1704.03155.pdf">An Efficient and Accurate Scene Text Detector</a><br>(2017), Cornell University arxiv.org<br>[2]Youngmin Baek, Bado Lee, Dongyoon Han, Sangdoo Yun, and Hwalsuk Lee,<a href="https://arxiv.org/pdf/1904.01941.pdf">Character Region Awareness for Text Detection</a>(2019), Clova AI Research NAVER Corp Cornell University arxiv.org<br>[3]He guo, Xiameng Qin, Jiaming Liu, Junyu Han, Jingtuo Liu and Errui Ding,EATEN: <a href="https://arxiv.org/pdf/1909.09380.pdf">Entity-aware Attention for Single Shot Visual Text Extraction</a>(2019),Computer Science Computer Vision and Pattern Recognition Cornell University arxiv.org<br>[4]Xiao Yang, Ersin Yumer, Paul Asente, Mike Kraley, Daniel Kifer and C. Lee Giles,<a href="https://clgiles.ist.psu.edu/pubs/CVPR2017-connets.pdf">Learning to Extract Semantic Structure from Documents Using Multimodal Fully Convolutional Neural Networks</a>,The Pennsylvania State University Adobe Research<br>[5]Xiaohan Yi,Liangcai Gao, Yuan Liao, Xiaode Zhang , Runtao Liu and Zhuoren Jiang,<a href="https://ieeexplore.ieee.org/document/8269977">CNN Based Page Object Detection in Document Images</a>(2017),14th IAPR International Conference on Document Analysis and Recognition (ICDAR), Kyoto</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=23cc0fb56549" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Connecting kaggle to Colab]]></title>
            <link>https://amannair723.medium.com/connecting-kaggle-to-colab-28f19c40a0d8?source=rss-c1f786da7919------2</link>
            <guid isPermaLink="false">https://medium.com/p/28f19c40a0d8</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[google]]></category>
            <category><![CDATA[npl]]></category>
            <dc:creator><![CDATA[Amannair]]></dc:creator>
            <pubDate>Mon, 13 Jul 2020 10:41:27 GMT</pubDate>
            <atom:updated>2020-07-13T10:41:27.420Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/473/1*S4lC6UesGxqSCOxXpjRTQg.png" /></figure><h3>Connecting kaggle dataset to Colab an easy fetch</h3><p><strong>MOTIVE :</strong><em> I went through a lot of post which talks the ideology of connecting the dataset loaded to colab but there where many unsolved issues for newbie to data science</em> [“how to run the project in colab with uploading issues where the browser has restrictive access”, “colab run time issue where dataset is deleted after 12 hrs”].</p><blockquote>The aim is to create a colab environment where with the help of an internet connection thereby opening and using kaggle from any computer device with a with a web browser and a web connection.</blockquote><p>Activate colab with kaggle to Explore the power and performance with gpu/tpu</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/353/1*dN2oRsgM9FgidMHaEF_Z9Q.png" /></figure><p><a href="https://github.com/amannairDL/public.git">amannairDL/public</a></p><p>The link will open a colab which goes through all the steps to go through while setting up kaggle dataset within the colab environment itself</p><p>Well!! kaggle dataset on an average carries about 5GB of content including (Train,test and validation) connecting with Google drive is a good option to save the data till the time competition is going On . This can also be used to save the metadata and the training model as it offers upto 15GB of data</p><p><strong>Connect to the Google drive</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pLd-8ux9o8wcBjv_Jty6vA.png" /></figure><p>Install the kaggle api dependencies</p><blockquote>!pip install kaggle</blockquote><ol><li><strong>Getting the api token</strong></li></ol><p>Go to your account, Scroll to API section and Click Expire API Token to remove previous tokens</p><p>read the json file from any of the json viewer also there are many extension it could also be read as a text file and copy it to the api_token parameter</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*66awA9HaLqAD_e2ZPfeltg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qaEbPIAG5JEi7THNwbBkCQ.png" /></figure><p>2. <strong>Import the libraries</strong></p><p>These libraries help in interacting with the files and api. These library are pre-installed in colab hence its a sweet and simple import</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cKw5ozvcdT6SbtHEd1eN1g.png" /><figcaption>importing the libraries</figcaption></figure><p>json is used to create a json file in colab this work around helps in creating files without the dependencies to any uploads and downloads hence can also be used in devices with raspberry pi. Save it as a kaggle.json</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lT9CE3A3GQ39X34P7lE2mQ.png" /></figure><p>3. <strong>Make the directory of kaggle to get the data from the competition</strong></p><p>once the kaggle api is installed it gets loaded in the root directory in lib/python3.7/kaggle. We need to add this json file to the kaggle repo. This is used to authenticate</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QA7zr49KmJzq7vzNk8YY0w.png" /></figure><p>4. <strong>C<em>hange the directory to download the files there</em></strong></p><p>create a repo for the kaggle dataset which contains all the files downloaded with the kaggle api token (The api token can be copied from the competition source)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d3--oGd4k9Uv_0pbrAxBig.png" /></figure><p>5. <strong>Save the downloaded dataset to drive</strong></p><p>Saving the dataset to the drive will make this code run only once as from the next call it can be directly called from the drive. This part help with the issue of the dataset being deleted when the Colab VM is inactive</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rGaEol1649GJ2k7NC0V0qA.png" /></figure><p>6. <strong>Extract the files to the directory</strong></p><p>As the data is a zip file we can unzip it and start using it for the practice performing EDA , ETL on it</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3kccE5PqIzXv7UJItIwf9g.png" /></figure><p>7. <strong>Going to the main content directory to start using colab</strong></p><p>here we can perform all the training to win the competition with the power of colab and also allowing to do all kinds of experimentation within the VM……best of luck!!!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wALA3Uce_Lk6L4_qxKTpBg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/473/1*S4lC6UesGxqSCOxXpjRTQg.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=28f19c40a0d8" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>