<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sankar Bareddy on Medium]]></title>
        <description><![CDATA[Stories by Sankar Bareddy on Medium]]></description>
        <link>https://medium.com/@sankarbareddy?source=rss-2376e9a14671------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*dGKcrtsvfCXgvoGt</url>
            <title>Stories by Sankar Bareddy on Medium</title>
            <link>https://medium.com/@sankarbareddy?source=rss-2376e9a14671------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 07:52:27 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sankarbareddy/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Setting up a robust system for scraping historical data]]></title>
            <link>https://medium.com/@sankarbareddy/setting-up-a-robust-system-for-scraping-historical-data-306fd0277c3e?source=rss-2376e9a14671------2</link>
            <guid isPermaLink="false">https://medium.com/p/306fd0277c3e</guid>
            <dc:creator><![CDATA[Sankar Bareddy]]></dc:creator>
            <pubDate>Sat, 04 Mar 2023 12:33:46 GMT</pubDate>
            <atom:updated>2023-03-16T15:32:43.362Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*Tf8tOJuzNjjwSeD2.jpg" /><figcaption>Image Source -https://rb.gy/omzllu</figcaption></figure><p>Hi, I am Sankar. I have worked as a Junior Python Developer in a legal tech company named <a href="https://www.quickcompany.in/">quickcompany.in</a> . The company offers registration services, GST filing and accountant services online. My role was to scrape all the government websites for trademarks, patents, designs, copyrights, companies, and legal information.</p><p>In this blog, I will walk you through the process of setting a robust system for web scraping. The first question is, how do you decide to go about it. Which tool is most suitable for this particular objective among all the available ones. So let us compare and contrast Selenium and Scrapy frameworks available for web-scraping.</p><h3>SELENIUM vs SCRAPY</h3><p>Selenium is primarily used for web automation and testing. It allows you to automate interactions with a web page, such as filling out forms, clicking buttons, and navigating through pages. It can also be used for web scraping by writing scripts to extract data from web pages. Selenium is typically used for scraping websites that require authentication, JavaScript rendering, or other dynamic content.</p><p>Scrapy, on the other hand, is a dedicated web scraping framework. It provides a set of tools and libraries for extracting data from websites in a structured and organized way. Scrapy can handle large amounts of data, manage the scraping process, and can easily integrate with databases and APIs. It is designed for scraping websites that have a predictable structure and can be scraped without the need for complex interactions.</p><p>Let us consider <a href="http://copyright.gov.in/">copyrigh</a>t website to scrape. The data we are interested in scraping are available in html response, so we don’t have to load all the java scripts or png images for our task. Since selenium acts as a headless browser, it loads all the unnecessary renderings making the scraping process less efficient.</p><p>Where as scrapy is a asynchronous framework which provides the scalability through CONCURRENT_REQUESTS. Since the considered website doesn’t have any session requirements for eg. captcha we can take leverage of this asynchronous framework and speed up the process of scraping.</p><h3>COPYRIGHT ENTITIES</h3><p>For copyrights we have got two different entities to scrape.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/678/1*u4CqNXy6pIh8wbPIDzDE5g.png" /><figcaption>copyright status</figcaption></figure><p>Since we do not have any diary numbers to start with we will use id to populate the form data and get the results. Usually the format of the id will be in a manner of serial_number/year , where serial_number represents the series of copyrights available and year represents the corresponding year when the copyright was filed.</p><p>We will scrape all the columns of this entity i.e. diary_number, class_of_work, title_of_work, applicant_name, communication_address and status.</p><p><strong>NOTE</strong></p><ol><li>For some id’s you will get more than one diary_numbers, i.e. the with copyrights with two different class_of_work.</li><li>For some diary_numbers, you will have multiple applicants.</li><li>For some id’s the result will be “No data found”.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/1*50gnFomCa8-Zer6mgtWYlQ.png" /><figcaption>copyright details</figcaption></figure><p>Once the copyright is registered, you will get its ROC number indicating the certificate of registration. You need diary_number in order to search for ROC number, which in turn returns the copyright details.</p><p><strong>NOTE</strong></p><ol><li>For some diary_number’s you will get the result as “No records Found”.</li><li>For some diary_number’s you can not find out the corresponding ROC number.</li></ol><h3>Establishing a Relational database to store all the data.</h3><p>I have taken a $7 droplet in digital ocean to set up and configure a MySQL database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TricKOVgLW3qx6iYNdJBnQ.png" /><figcaption>Entity Relationship Diagram</figcaption></figure><p>Above ERD diagram indicates the entity relations in the database from which we can infer</p><ol><li>One copyright can have one or many copyright_applicants</li><li>One copyright can have zero or one copyright_details</li></ol><p>Using scrapy framework I have created a scrapy project which comprises of two spiders for two different sites, one is named as <a href="https://copyright.gov.in/frmStatusGenUser.aspx">copyright_status</a> and another is named as <a href="https://copyright.gov.in/SearchRoc.aspx">copyright_details</a>, and alsopipeline class to store the yielded items in the above established database.</p><p><strong>Middlewares and Extensions used in the scrapy project</strong></p><ol><li>RetryMiddleware, to retry the failed requests.</li><li>RandomUserAgentMiddleware, to provide different user agent for each request made.</li><li>We have used ScrapeOpsMonitor extensions to visualize the performance of our spiders.</li></ol><h3>Virtual Machine for clock around scraping</h3><p>You can perform web scraping using your local machine, but it is not ideal for handling thousands of jobs daily to scrape historical data.</p><p>Hence we set up a $28 droplet in digital ocean to set up the scrapyd and scrapydweb services. The reason why I have gone for $28 is that scrapyd offers concurrency i.e. we can run multiple jobs simultaneously, to take full leverage I have gone for decent computing resources.</p><p>By default digital ocean droplets blocks government websites, by not recognizing their domain names in its DNS. So, before setting up these services, you might have to overwrite your nameserver permanently and update your ca-certificates for authentication.</p><p>We will also setup supervisor, a process management system to run scrapyd and scrapydweb process, such that whenever the process fails it will restart immediately.</p><p>Scrapyd by default runs at 6800 port number. We will setup a nginx to tunnel our traffic to another port number (reverse proxy) such that direct access to 6800 port will be denied. We will also set up a basic authentication for protection.</p><p>Before sending a request to your scrapyd, you have to configure your ufw (universal fire wall)setting that allows nginx.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/481/1*fVi2Tg5CAJ9VPjFuKpNGPQ.png" /><figcaption>Flow diagram</figcaption></figure><p>You have to deploy your project which involves eggifying it and uploading the egg to scrapyd server. To achieve this in the easiest way is to use the scrapyd-deploy tool provided by <a href="https://github.com/scrapy/scrapyd-client">scrapyd-client</a> which will do it all for you.</p><p>scrpay.cfg file in the scrapy project file must indicate the details of the virtual machine in order to eggify the project using scrapyd-client.</p><p>Below is the format for scrapy.cfg file</p><pre><br>[deploy:name]<br>url = http://ip:port/<br>project = project_name<br>username=user_name  <br>password=password<br><br>#if any authentication is set then only use username and password<br>#ip means ip address of the virtual machine<br><br>#port means port number where we have redirected our traffic using <br>#nginx to instead of default 6800</pre><h3>SCHEDULING</h3><p>After deploying our project, we have to schedule the jobs that perform scraping. The default way to schedule a job is to make a curl request to the scrapyd server. Scrapyd stores all the jobs in a queue and execute them.</p><pre>curl -u user_name:password curl http://localhost:6800/schedule.json -d project=myproject -d spider=somespider</pre><p>But it is not a efficient way to schedule thousands of jobs, and some of the spiders also need arguments, which are generated iteratively.</p><p>For scheduling <a href="https://copyright.gov.in/frmStatusGenUser.aspx">copyright_status</a> we have to pass id’s as argument.</p><pre>scrap_batch = 500 #no. of ids per job<br>end_batch = 50  # no. of jobs<br>  <br>for batch in range(0, end_batch): #for each batch<br>      diary_numbers=&quot;&quot;<br><br>      for num in range(scrap_batch*batch+1, scrap_batch*(batch+1)+1, 1): <br>      #generate 500 ids<br>          diary_numbers += str(num)+&#39;/&#39;+year+&#39;,&#39;<br><br>          diary_numbers = diary_numbers[:-1] #slicing last comma in the string<br>     <br>          req = requests.post(url=f&quot;http://{ip}:{port}/schedule.json&quot;, <br>                              data={&#39;project&#39;:project_name, <br>                                    &#39;spider&#39;:spider_name, <br>                                    &#39;diary_numbers&#39;:diary_numbers},<br>                              auth=(user_name, password))<br><br>          print(req.text, diary_numbers)<br></pre><p>When all the <a href="https://copyright.gov.in/frmStatusGenUser.aspx">copyright_status</a> jobs are finished, we have stored the data in our MySQL database. Using <a href="https://pypi.org/project/mysql-connector-python/">mysql.connector</a>, we can access our database in python environment.</p><p>As said earlier, for scheduling <a href="https://copyright.gov.in/SearchRoc.aspx">copyright_details</a>, we need all the diary_numbers of registered copyrights. We can fetch these diary_numbures by executing the below query.</p><pre>SELECT diary_number from copyright_status WHERE status=&#39;Registered&#39;;</pre><p>Rather than fetching all the diary_numbers in one go, we will fetch these diary_numbers in batches of 500 which will be suitable for scheduling the <a href="https://copyright.gov.in/SearchRoc.aspx">copyright_details</a> jobs.</p><pre>#establishing a MySQL connection<br>conn = mysql.connector.connect(host = ip, port = port_number, <br>                               user = user_name, password = password, <br>                               database = database)<br>cur = conn.cursor()<br>id_query = &quot;SELECT diary_number from copyright_status WHERE status=&#39;Registered&#39;&quot;<br>cur.execute(id_query)<br><br>batch_size=500<br>loop = True #flag for the batch_size=500<br>job_count = 0 # counter for number of jobs<br>while(loop): #for first iteration, we assume that the batch_size=500<br>    registered=[] #initializing a list to store the fetched diary_numbers<br>    rows = cur.fetchmany(size=batch_size) <br>    #trying to fetch only 500 diary_numbers from the results <br><br>    for row in rows: <br>        registered.append(row[0])<br><br>    diary_numbers = &#39;,&#39;.join(registered) #converting all the diary_numbers into<br>                                         # a string to pass an argument<br>    <br>    req = requests.post(url=f&quot;http://{ip}:{port}/schedule.json&quot;, <br>                              data={&#39;project&#39;:project_name, <br>                                    &#39;spider&#39;:spider_name, <br>                                    &#39;diary_numbers&#39;:diary_numbers},<br>                              auth=(user_name, password))<br><br>    print(req.text, diary_numbers)<br><br>    job_count += 1<br>    logging.info(f&quot;the number of jobs are {job_count}&quot;)<br><br>    if len(rows) != batch_size: #if at all the fetched diary_numbers are less <br>                                #than 500 then it will be our last job <br>        loop=False<br><br>conn.close()</pre><h3>Visualization using Power BI</h3><p>So we got all the historical data from 2000 to 2022 in our database, now let us try to visualize these by plotting them using Power BI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/433/1*lSYGkc9KVXFVn6htaWF49A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wJ-WdwxW3OYCyLIMGUs0ZQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*qUZ4hK-gaqvTJrLwhGpi-w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*07wHkLoB8AhewHHAVtKp0g.png" /></figure><p><a href="https://app.powerbi.com/view?r=eyJrIjoiMWEwMjIzZTktZjBmNi00ZmMyLWIwNTEtOTIxNzliMDQ1NWMyIiwidCI6ImUxNGU3M2ViLTUyNTEtNDM4OC04ZDY3LThmOWYyZTJkNWE0NiIsImMiOjEwfQ%3D%3D">Power BI Report</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/457/1*5Ip4lo7fCCVm4TwaEoM1sA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*5aaYC1COS3zHzAFecRn7VA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/417/1*Nw_6TATJBENP1IkQgTwSwA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/1*-zWpEZDWQ8nZKG_kR_1c8g.png" /></figure><p><a href="https://app.powerbi.com/view?r=eyJrIjoiZDkzNWI0MzMtYzY2My00ZDU5LWE0NWYtYjMyYmQ4NjUyYmZkIiwidCI6ImUxNGU3M2ViLTUyNTEtNDM4OC04ZDY3LThmOWYyZTJkNWE0NiIsImMiOjEwfQ%3D%3D">Power BI Report</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/402/1*bkb1i_9k1AsWawSEqx2mWQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v-y9RsbeAyZgHysNavICRw.png" /></figure><p><a href="https://app.powerbi.com/view?r=eyJrIjoiMjAwNmQzYjAtMTE2Ni00Yjg4LTk0OGUtYjE5YjI2MGRjZDUxIiwidCI6ImUxNGU3M2ViLTUyNTEtNDM4OC04ZDY3LThmOWYyZTJkNWE0NiIsImMiOjEwfQ%3D%3D">Power BI Report</a></p><h3>Updating the copyright database daily</h3><p>This <a href="https://copyright.gov.in/Fresh_Applications.aspx">page</a> provides the information about no. of copyrights filed daily in India. So I have written a copyright_daily spider to fetch all the diary_numbers of the filed copyrights daily.</p><p>In order to fetch them automatically, I have set up a cronjob which runs daily at midnight (0 0 * * * ) to fetch the total number of copyrights filed in the previous day. I have stored this data in a separate entity called copyright_daily in the MySQL database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/346/1*mLQf-Y8ivFpiyTqUe_KvNw.png" /></figure><p>References:</p><ol><li>For getting started with scrapy this will help. <a href="https://www.udemy.com/course/web-scraping-in-python-using-scrapy-and-splash/">https://www.udemy.com/course/web-scraping-in-python-using-scrapy-and-splash/</a></li><li>Overwriting your DNS nameserver in ubuntu <a href="https://youtu.be/NEyXDdBrw2c">https://youtu.be/NEyXDdBrw2c</a></li><li>Install and manage supervisor <a href="https://www.digitalocean.com/community/tutorials/how-to-install-and-manage-supervisor-on-ubuntu-and-debian-vps">https://www.digitalocean.com/community/tutorials/how-to-install-and-manage-supervisor-on-ubuntu-and-debian-vps</a></li><li>Install and manage ngnix <a href="https://www.digitalocean.com/community/tutorials/how-to-install-nginx-on-ubuntu-20-04">https://www.digitalocean.com/community/tutorials/how-to-install-nginx-on-ubuntu-20-04</a></li><li>Install and configure MySQL on ubuntu <a href="https://youtu.be/WltqUaqxBH8">https://youtu.be/WltqUaqxBH8</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=306fd0277c3e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Extracting Structured Data from Images using TableNet]]></title>
            <link>https://medium.com/@sankarbareddy/extracting-structured-data-from-images-using-tablenet-7501903700d1?source=rss-2376e9a14671------2</link>
            <guid isPermaLink="false">https://medium.com/p/7501903700d1</guid>
            <dc:creator><![CDATA[Sankar Bareddy]]></dc:creator>
            <pubDate>Wed, 01 Jun 2022 03:45:25 GMT</pubDate>
            <atom:updated>2022-06-01T15:46:23.442Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*tYSLVDd2o3sfmxnqYMm5-g.jpeg" /><figcaption>sourced from smartengines.com</figcaption></figure><h3>Introduction:</h3><p>This blog is about implementing the <a href="https://arxiv.org/abs/2001.01469">TableNet</a> architecture in Tensorflow. This paper was published in 2019 at the International Conference on Document Analysis and Recognition (ICDAR) by TCS Research.</p><h3>Business Problem:</h3><p>Today most of the documents are scanned through smart phones. Documents may be retail invoices, tax filings, insurance claims and so on. So it has been an imperative to develop an end to end Deep Learning model which extracts the structured data from an document image which is format agnostic. This helps reduce the manual labor required and saves the processing time.</p><h3>Deep Learning Formulation :</h3><p>For a given image we have to detect the region where the table exists. It is more like object detection but in this case the tolerance for noise is very low. To perform this task TableNet uses Encoder-Decoder model. Here Encoder helps us in providing the useful features to detect the table and Decoder part helps to upscale the image to the original size such that table can be detected relative to the input size. At the end we will use classifier to predict whether each pixel belongs to the table or background. Hence it can be treated as binary classification.</p><p>The versatility of TableNet is it offers end-to-end architecture to detect both table region and structure of the table i.e. no. of columns with in the table. The Encoder part is contributed by the ImageNet models like VGG19, DenseNet121, RestNet50 and MobileNet_v2. The classifier layer was removed and fully connected Dense layers are replaced with (1,1) Conv2D layers just like Fully Convolutional Network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CFZYvZl1-2Czutk5SgFqUg.png" /><figcaption>TableNet</figcaption></figure><p>Where as Decoder part is divided into two branches. One is for Table Detection and another is for Structure Detection. The a priori assumption is that knowledge of the structure of the table helps recognizing the table itself. In order to do that there are common (1,1) convolutional filters at the end of the Encoder such that they are reinforced by the structural learning of the table.</p><h3><strong>Data set analysis:</strong></h3><p>The range of data available for this kind of task is very limited. The paper uses <a href="https://www.icst.pku.edu.cn/cpdp/sjzy/">Marmot</a> dataset which contains both Chinese and English documents with table masks. But the novel architecture of the TableNet requires column structure of the table also. Hence the researchers annotated manually providing <a href="https://drive.google.com/drive/folders/1QZiv5RKe3xlOBdTzuTVuYRxixemVIODp">Marmot-Extended</a> Dataset which contains column masks as well. TableNet was trained on at least one table containing English document Images and is validated against ICDAR 2013 table competition dataset. But in this case study only Marmot Dataset was used because <a href="https://roundtrippdf.com/en/downloads/">ICDAR 2013 table competition </a>dataset was not public.</p><h3><strong>Performance Metrics:</strong></h3><p>Since it is Binary classification of each pixel where 0 represents it belongs to the background and 1 represents table/column region of the image respectively we have used micro F1score as performance metric. The loss used to minimize was simple Binary Cross Entropy.</p><h3>Data Preparation:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/1*CbmdRLgUlAc7pVW-86xgvg.png" /><figcaption>Marmot English Dataset Distribution</figcaption></figure><p><em>Table Negative</em> represents no. of English documents containing no table at all and <em>Table Positive </em>represents no. of English documents containing at least one table. The dataset is balanced dataset. But it would not help us cause out of 993 English documents only 509 documents has one at least one table which is very less data available to learn from.</p><p><strong>Extracting Co-ordinates from xml files:</strong></p><p>The labels for table detection and column detection are provided in the format of .xml files. The co-ordinates are encoded in Hexadecimal format to prevent the loss of precision. In extracting the table co-ordinates the key “CropBox” holds the size of the image and key “BBox” holds the location of the each table within the image. For column labels are extracted from Marmot-Extended dataset and keys are denoted individually like object/bndbox/co-ordinate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/267/1*eWXDm7OQslvl6kXQiTCmdw.png" /><figcaption>The co-ordinates are xmin, ymin, xmax, ymax</figcaption></figure><p>The empty image of size (1024, 1024, 1) is filled using the extracted co-ordinates with 255 of dtype np.uint8 thus image masks are created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X-aUGpXEiDMMWtYc_qnZxg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vURH0dcQCWBftwD18SAV_w.png" /><figcaption>Above are examples of masks which are created from .xml files</figcaption></figure><h3>Basic EDA:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*AVSGzCp0liIK5Or38hyixg.png" /></figure><p>Observations:<br>1. Most of the Images heights are in the range(1000, 1200) pixels. The distribution is right skewed.<br>2. Most of the Images widths are centered around 800 pixels. This distribution is some what symmetric.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/444/1*n7YR2T26mOh5-zbOTjLx9A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/891/1*WVol-5SdUJ2bQzMOpYe8lw.png" /></figure><p>Observations:<br>1. As the no. of table per document is high the no. of such documnets are very low.<br>2. The peak of the table height pdf funciton is around 100 indicating that majority of the tables are of 100 pixels height but the distribution is left skewed i.e. most of the tables are below 400 pixels of height.<br>3. The peak of the table width pdf function is around 400 indicating that majority of the tables are of 400 pixels width but the distribution is some what uniform and most of the table width falls between (300, 800) pixels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/444/1*3-B1KWvjC2nYVAh8LFMbaA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*2Lf7nii0HK5wEKv7tLcR9g.png" /></figure><p>Observations:<br>1. Most of the columns in the document are below 14. There are 58 documents have columns greater than 14 out of 993.<br>2. Majority of the column heights falls in the range of (100, 300). The distribution is left skewed.<br>3. Majority of the column widths falls in the range of((0,100). The distribution is more left skewed with minimum variance than column heights distribution.</p><h3>Data Preprocessing:</h3><p>To increase the generalization of the model we have done some image processing over the original image.</p><ol><li>In the task of Table recognition the colors of the Image can be treated as noise. Hence we convert the input image into greyscale and again to RGB to eliminate its color. After processing the output RGB image’s 3 channels are identical to each other (repeated).</li><li>To improve the contrast of the image Histogram Equalization was performed where each pixel’s intensity was altered.</li><li>Finally the image is resized into (1024, 1024, 3) as per the requirement of the Architecture.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z2DNJqXCC9XBC-FmbAL_Sw.png" /><figcaption>Image Preprocessing</figcaption></figure><h3>Data Augmentation:</h3><p>Since the no. of Images actually contribute to the training are very small i.e. 509 and about 20% of them are kept apart as validation. To ensure the variance in each epoch of training we use the following augmentations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1e92Alef0fCg0qLXhEHWXA.png" /><figcaption>Augmentations</figcaption></figure><ol><li>Left-Right Flip</li><li>Up-Down Flip</li><li>Embossing</li><li>Direct Edge Detect</li><li>Sharpening</li></ol><p>All these augmentations are done using<a href="https://imgaug.readthedocs.io/en/latest/"> imgaug </a>module in python.</p><h3>Encoder of the Model:</h3><p>As mentioned above we used Transfer learning from ImageNet dataset trained architectures like VGG19, DenseNet121, RestNet50, MobileNet_v2. All these architectures are downloaded from <a href="https://www.tensorflow.org/api_docs/python/tf/keras/applications">tensorflow.keras.applications.</a></p><p>The fully connected Dense layers are replaced with (1,1) Conv2D layers + ReLU + dropouts and classifier layer is removed for every architecture.</p><p>We have created function for each architecture which takes the input and returns three outputs at different stages of the network having shapes(32, 32) as output, (64, 64) as pool_1 and (128, 128) as pool_2 which are to be concatenated to the upsampled features in the decoder branches.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cfd72ce82f92de7a4ef71faff8e3a453/href">https://medium.com/media/cfd72ce82f92de7a4ef71faff8e3a453/href</a></iframe><h3>Decoder of the Model:</h3><p><strong>Table Branch (subclassed using tf.keras.layers.Layer):</strong></p><p>The output of the common convolutional filters of shape (32, 32) was passed through another (1,1) Conv2D and after upsampled using Conv2Dtranspose to (64,64) and the concatenated with the pool_1 of same shape from Encoder. Later the upsampled is repeated to (128, 128) and again concatenated with pool_2 of the Encoder. Finally we will upsample the image to the (1024, 1024) of the original shape.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f8758719928d2fb050784b42b8234739/href">https://medium.com/media/f8758719928d2fb050784b42b8234739/href</a></iframe><p><strong>Column Branch (subclassed using tf.keras.layers.Layer):</strong></p><p>The output of the common convolutional filters of shape (32, 32) was passed through one more block of Conv2D+ReLU+dropout and repeat the whole table branch as it is.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b21d73b493c5bae5a1ba956f62344f39/href">https://medium.com/media/b21d73b493c5bae5a1ba956f62344f39/href</a></iframe><p><strong>TableNet model (subclassed from tf.keras.Model):</strong></p><p>Using the custom function of Encoder and subclassed Layers like Table Decoder and Column Decoder we formed TableNet model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/05dbdc2819649e5dcd01b0beae955707/href">https://medium.com/media/05dbdc2819649e5dcd01b0beae955707/href</a></iframe><h3>Training:</h3><p><strong>Callbacks:</strong></p><p>We applied following callbacks to oversee the training:</p><ol><li>Model check point which saves the weights of the model for every improvement in validation table binary cross entropy loss.</li><li>Early stopping if the validation combined binary cross entropy loss was not improved for 5 epochs.</li><li>Reduce Learning rate on plateau if the validation combined binary cross entropy loss was not improved for 2 epochs.</li><li>Tensorboard callback to log the training parameters itself.</li></ol><p>We have trained TableNet with all the mentioned Encoders i.e. feature extractors and found that DenseNet121 performs best among them</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fb38050021b58df183b32d0cb9d16244/href">https://medium.com/media/fb38050021b58df183b32d0cb9d16244/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i0mchwOZEm_h316MNSDLBw.png" /><figcaption>Relative Performance</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/823/1*ZHgynB2-H5B6ztpYKrQpBg.png" /><figcaption>Tabulation</figcaption></figure><h3>TableNet Predictions:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nGL_Xd1jBmx6W9sXB9mMfw.png" /><figcaption>Sample 1</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C3c1SCcGRB1oYMbfG1GYvA.png" /><figcaption>Sample 2</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SL1NXplJRtIBvyOhEpmNzg.png" /><figcaption>Sample 3</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sSSJ1epxk19MMZy6FWzNAw.png" /><figcaption>Sample 4</figcaption></figure><h3>Extracting Data using Tesseract-OCR:</h3><ol><li>Before extracting data we must extract the co-ordinates of the table region.</li><li>First we will extract contours from the predicted table mask and try to approximate the contours using cv2.approxPolyDP with some threshold.</li><li>These approximated contours are used to generate bounding co-ordinates for Tables using cv2.boundingRect.</li><li>Later these co-ordinates are used to crop the image and extract the data with in the cropped image using tesseract-ocr.</li><li>We will generate a StrignIO stream for the extracted data to create a pandas DataFrame which will be later stored as .csv files.</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/eaea51253461f810fd6b1cae5f557fd3/href">https://medium.com/media/eaea51253461f810fd6b1cae5f557fd3/href</a></iframe><h3>Results:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/484/1*mhRQS-ANAe21D3zPPplB3w.png" /><figcaption>For this input</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/710/1*QGRpEmaiAKNvQk4ueuEvXQ.png" /><figcaption>Output of the Model</figcaption></figure><p>For more results check out the <a href="https://github.com/sb1440/sb1440-case-study-2">Github</a> repository for this case study.</p><h3>Post Training Analysis:</h3><ol><li>The Model tends to identify pixels containing bold letters as false positives</li><li>It also fails to distinguish between flow chart, circuits and table edges.</li><li>It could distinguish between pictures and tables.</li><li>For some documents the appropriated table boundaries tends to be cutoff outer columns. It effects generalization.</li><li>The approximation also causes merging of two very close tables.</li><li>It also fails to recognize the table without any partitions.</li></ol><h3>Deployment:</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F82SG51siPW4%3Fstart%3D1&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D82SG51siPW4&amp;image=http%3A%2F%2Fi.ytimg.com%2Fvi%2F82SG51siPW4%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/e31f2a8ce915bdf3c4d92815db42c570/href">https://medium.com/media/e31f2a8ce915bdf3c4d92815db42c570/href</a></iframe><h3>Further Work:</h3><ol><li>We can provide the semantic information to the model while training through color coding the column masks based on their datatype i.e. text or number. This improves the performance of the model slightly.</li><li>We can fine tune this model on custom dataset like retail-invoices etc. for better performance.</li></ol><h3>References:</h3><ol><li><a href="https://asagar60.medium.com/tablenet-deep-learning-model-for-end-to-end-table-detection-and-tabular-data-extraction-from-b1547799fe29">https://asagar60.medium.com/tablenet-deep-learning-model-for-end-to-end-table-detection-and-tabular-data-extraction-from-b1547799fe29</a></li><li><a href="https://arxiv.org/abs/2001.01469">https://arxiv.org/abs/2001.01469</a></li><li><a href="https://arxiv.org/abs/1411.4038">https://arxiv.org/abs/1411.4038</a></li><li>appliedaicourse.com</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7501903700d1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Predicting Failure in Trucks using Machine Learning]]></title>
            <link>https://medium.com/@sankarbareddy/using-machine-learning-for-predicting-failure-of-trucks-d3cb9267a024?source=rss-2376e9a14671------2</link>
            <guid isPermaLink="false">https://medium.com/p/d3cb9267a024</guid>
            <dc:creator><![CDATA[Sankar Bareddy]]></dc:creator>
            <pubDate>Fri, 18 Mar 2022 18:15:59 GMT</pubDate>
            <atom:updated>2023-05-02T12:27:32.327Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nXvRxaH300JQEF5FZGYJjg.jpeg" /><figcaption>source: scania.com</figcaption></figure><ol><li><strong>Introduction</strong></li></ol><p>This blog was part of my case study at Applied AI course. The data was sourced from the <a href="https://ida2016.blogs.dsv.su.se/?page_id=1387">competition</a> held by the IDA (Intelligent Data Analysis) 2016 Industrial Challenge. It is about predicting the failure of a specific component of the APS (Air Pressure system) which uses compressed air to signal the actuator to apply the brakes. The high potential energy of the compressed air enables the driver to stop the truck with much less effort. The data was sponsored by SCANIA.</p><p>2.<strong> Business Problem</strong></p><p>The business objective is to reduce the cost of maintenance by replacing manual inspection with automatic inspection system. It finds out whether a truck is prone to imminent failure of the APS system. Data extracted from the trucks themselves are used for the prediction.</p><p>3. <strong>Machine Learning Formulation of the Business Problem</strong></p><p>The outcome of the automated inspection would be either positive indicating apparent failure of the APS or negative indicating the failure other than the APS components. This could be formulated as a simple binary classification.</p><p>4.<strong> Business Constraints</strong></p><p>The misclassification i.e. predicting no failure of APS when actually it is might be an expensive error. So we must prioritize reducing the number of False Negatives. There is no latency problem in this context because anyhow the inspection of a truck is meticulous procedure.</p><p>5.<strong> Performance Metrics</strong></p><p><strong><em>a. Cost metric suggested by the competition</em></strong></p><p>Since it is an imbalanced dataset the performance metric suggested in the competition was to use the metric called “Cost”. It comprises of False Negatives and False Positives such that the penalty for False Negative i.e. predicting no fault in APS when there is an actual failure is 500 much greater than the 10 for the False Positive i.e. predicting a fault in APS when there is no actual failure involved.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/1*y1x8OQ5LVVYMWBGQGA15vQ.png" /></figure><p>False Positives are defined as Type I failure and corresponding cost_1 = 10 is accrued due to unnecessary checks that needs to be done by a mechanic at a workshop.</p><p>False Negatives are defined as Type II failure and corresponding cost_2 = 500 is accrued due to missing checks on faulty trucks which may cause a breakdown.</p><p><strong><em>b. Recall score</em></strong></p><p>In order to compare various models based on the above business constraints recall score i.e. sensitivity of the model which gives us percentage of True positives among the actual positives TP/(TP+FN)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/597/0*WApadrk8RG5jYP2C.jpg" /><figcaption>Confusion Matrix</figcaption></figure><p>6<strong>. EDA (Exploratory Data Analysis)</strong></p><p><strong><em>a. Analyzing data set.</em></strong></p><p>The data set is an entirely numerical data. Features are either single numerical counters or histogram bin counters. It is also highly imbalanced where out of 60,000 train instances 1000 were positive and 59,000 were negative classes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/449/1*VM12nqTrQ7j3exGbNP5aSQ.png" /></figure><p>Large no. of values in the dataset are missing i.e. null values. Only 467 train instances out of 60,000 are completely filled with all the features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/449/1*1o-iqYr0uIDTHG7EK42zZw.png" /></figure><p>We have dropped features which are having more than 50% of their values are missing. There are about eight such features.</p><p><strong><em>b. Imputing data set</em></strong>.</p><p>To further conduct any analysis, we have to impute the missing data. Before Imputing we have to look at the pattern of missing data. There are three mechanisms of missing data MCAR, MAR, MNAR, You can find more about missing mechanisms <a href="https://www.youtube.com/watch?v=YpqUbirqFxQ&amp;t=2s">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GCIlSa4RbUgLRgj8A1G0IQ.png" /><figcaption>The missing pattern in the train data set generated using missingno package.</figcaption></figure><p>The pattern above is arbitrary hence we can Impute using Multi Iterative Imputer i.e. with the help of observed value we will impute missing value. You can find out more about missing pattern <a href="https://www.youtube.com/watch?v=asyJCVLV4LI&amp;t=1s">here</a>.</p><p>The common method is to fill the missing values either by mean or median of the features i.e. Univariate Imputation. Considering the extent of the missing data this may lead to bias cause the variance of the feature is distorted. Hence we would Impute with Multi variate Imputer called KNN Imputer.</p><p>You can find more about the KNN Imputer <a href="https://www.youtube.com/watch?v=AHBHMQyD75U">here</a>.</p><p>To determine n_neighbor in the KNN Imputer we cross validated the imputed data using Random Forest Classifier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/710/1*1bs-Z-Eo_zCJX9hkdPkCnQ.png" /></figure><p>The mean recall score of Random Forest classifier for all the n_neighbors are when cross validated against 15 different test sets are more or less equal. Since we are dealing with imputing values, the approximation must be conservative hence we go with highest standard deviation.</p><p>Hence, we imputed the data set with n_neighbor =15.</p><p><strong><em>c. Univariate Analysis</em></strong></p><p>We have used SelectKBest class from sci-kit learn for feature selection. Since the feature are continuous values we performed using f_classif function.</p><p>How f_classif works under the hood, check <a href="https://youtu.be/NF5_btOaCig">here</a>:<br>1. First we will find out the sum of squared residuals around the mean of the data. TSS (total sum of squares)<br>2. Next, will find out the sum of squared residuals around the fit of the target (here it is binary classification hence, two target classes)<br>3. F_statistic is defined as the ratio of variance explained by the dependent variables in the independent variable to the variance not explained .<br>4. After finding the F_statistic we will measure corresponding p-values.<br>5. With the help of p-values we will findout the features which are significant and which are not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EQbhvYnTJbt77eGwAj0oSg.png" /><figcaption>PDF plots for top 10 features</figcaption></figure><p>Observations:<br>1. For all the features the distribution is right skewed.<br>2. For all the features the maximum no of values can be found around 0.<br>3. For the features cp_000 and ch_000 there is no observable variance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZY_lz0XohLZliXCN-KSiDw.png" /><figcaption>Box Plots for top 10 features</figcaption></figure><p>Both PDF and Box Plot tells us same thing that most of the values are centered around zero and there are more number of outliers. From univariate analysis there aren’t much insights to be derived.</p><p><strong><em>d. Multivariate Analysis.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/476/1*wIUAPDQbi9vm0jWG6X3tkA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/476/1*Y-be0HK8ySrhgxtWBJR8Bg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/476/1*N_XziM80fTMfIeK_9i8BKw.png" /></figure><p>Observations:<br>1. TSNE plots seems to be stable.<br>2. All three contains closely clustered neighborhoods of positive points.(Embedding the distance)<br>3. The outliers of class 1 are far away from the clustered neighborhood. <br>4. There is a great overlap between positive and negative instances<strong>.</strong></p><p>7<strong>. Feature Engineering</strong></p><p><strong><em>a. Oversampling using SMOTE</em></strong></p><p>Before training any models we must deal with the class imbalances. To make the model to predict minority class correctly we must sample more of them such that it starts distinguishing. Here we use SMOTE which generate artificial minority instances based KNN algorithm. It also decreases no. of majority instances.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*jBSW3_fl18q8XoN5tOiy2g.png" /></figure><p>All the 162 features might not be helpful in prediction. Hence to reduce the no. of features required for training we have used standard PCA i.e. Principle Component Analysis which projects the train instances onto most variance planes. This projection causes dimensionality reduction by generating new features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/1*tpCw1GHiCzI7soL0uI0DgQ.png" /></figure><p>Observations:<br>1. When we plot the Number of components and Cumulative variance, we can observe that 70% of the variance can be explained by 8 features and 90% of the variance can be explained by 34 features.<br>2. Reducing features decreases the chance of overfitting the model.</p><p>After trying out 70%, 80% and 90% variance explained PCA features for 80% variance features we got better performance.</p><p>8. <strong>Training of models</strong></p><p>a. <strong><em>K Nearest Neighbor</em></strong></p><p>Being a distance based algorithm it is well suited for reduced PCA features. GridSearchCV cross validation renders k=3 as best parameter. When the best estimator is calibrated using Platt scaling we obtained</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DQkjkt86pLHJpN-fniKOJQ.png" /><figcaption>Results for KNN model</figcaption></figure><p>The cost obtained for the KNN model is 12420 comprising 18 False Negatives and 342 False Positives.</p><p>b. <strong><em>SGD Classifier with log loss (Logistic Regression)</em></strong></p><p>The decision boundary for logistic regression is linear hyper plane, given the data set is highly overlapping it may not be effective. GridSearchCV cross validation renders alpha=0.01 as best parameter. When the best estimator is calibrated using Platt scaling we obtained</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1hMK8GwNXIRJqXhSb8iO3g.png" /><figcaption>Results for Logistic Regression</figcaption></figure><p>The cost obtained for the Logistic Regression is 39750 comprising 78 False Negatives and 75 False Positives.</p><p>c.<strong><em> SGD classifier with hinge loss (SVM-linear)</em></strong></p><p>The decision boundary for SVM-linear is also linear hyper plane, though it uses support vectors to maximize the margin between positive and negative classes it performed poorly relative to logistic regression</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hkfZ_zLyGkfHviXxCP9Zyw.png" /><figcaption>Results for SVM-linear model</figcaption></figure><p>The cost obtained for the SVM-linear is 49430 comprising 98 False Negatives and 43 False Positives.</p><p>d. <strong><em>Random Forest Classifier</em></strong></p><p>Random Forest Classifier is an ensemble model where base estimators are with high variance. Variance is reduced through row sampling with replacement + column sampling without replacement. Sampling process helps in reducing the overall variance without effecting bias.</p><p>GridSearchCV renders n_estimators =350, max_depth =7, min_samples_leaf=3 as best parameters. When the best estimator is calibrated using Platt scaling we obtained</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fxQryjDVzeyFLXsg7ooo2w.png" /><figcaption>Results for Random Forest Classifier</figcaption></figure><p>The cost obtained for the Random Forest Classifier is 9250 comprising 6 False Negatives and 625 False Positives.</p><p>e.<strong><em> XGBoost Implementation of Gradient Boosting Decision Tree</em></strong></p><p>Gradient Boosting Decision Tree is an ensemble model where base estimators are highly biased. Bias is reduced by sequentially training the estimators over the previously obtained bias . The algorithm is non trivial since it can’t be parallelized as Random Forest.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TQcmPWJhXCe1qChS13fyHw.png" /><figcaption>Results for GBDT model</figcaption></figure><p>The cost obtained for the GBDT Classifier is 9960 comprising 12False Negatives and 396 False Positives.</p><p>f. <strong><em>SVM-Radial Bias Function</em></strong></p><p>Radial Bias function allows to form non-linear decision boundary. Kernel function transforms the feature into high dimensional space where they can be linearly separated. GridSearchCV renders gamma=0.06 and C=10 as best parameters. When the best estimator is calibrated on Platt scaling we obtained</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rHFz8pyx9cWDnt1_z0cfjw.png" /><figcaption>Results for SVM-Radial Bias Function</figcaption></figure><p>The cost obtained for the SVM-Radial Bias Function is 10860 comprising 16 False Negatives and 286 False Positives.</p><p>g. <strong><em>Extremely Randomized Tree Classifier</em></strong></p><p>Extremely Randomized Tree is an extension of Random Forest Classifier. For numerical data it is hard to find right value of a feature to split for calculating probabilities of each class to find out Gini Impurity or Information gain of the feature. Hence random sampling is used in Extremely Randomized Tree which makes it faster. GridSearchCV renders max_depth= 12, min_samples_leaf= 3, n_estimators=300 as best parameters. When best estimator is calibrated using Platt scaling we obtained</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B9lBFnKYVcT2Sf0_a1RlUg.png" /><figcaption>Results for Extremely Randomized Tree model</figcaption></figure><p>The cost obtained for the Extremely Randomized Tree is 10040 comprising 7 False Negatives and 654 False Positives.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aoyQgTZ7qsxhgPdshuhD8w.png" /><figcaption>Tabulated Results of all the models</figcaption></figure><p><strong>9. Final Conclusion</strong></p><p>Among all the above models Random Forest Classifier resulted in lowest cost possible. We deployed the model using Flask on local machine.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F_d3y1jSIr6g%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D_d3y1jSIr6g&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F_d3y1jSIr6g%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/27f2279803b5cf048761e1c6ffbf5eb3/href">https://medium.com/media/27f2279803b5cf048761e1c6ffbf5eb3/href</a></iframe><p>10.<strong> Further Work</strong></p><p>In the given data they mentioned it also contains histogram bin counters. We have to know what these features account for and based on that we can do some feature engineering.</p><p>We can also work this data over Neural Networks. Since we are dealing with numeric data Multi layered Perceptron would suffice to work on.</p><p>Code for the case study can be found below.</p><p><a href="https://github.com/sb1440/sb1440-case-study-1">GitHub - sb1440/sb1440-case-study-1: IDA-2016 Prediction of Truck Failures</a></p><p>References:</p><p><a href="https://link.springer.com/chapter/10.1007/978-3-319-46349-0_33">IDA 2016 Industrial Challenge: Using Machine Learning for Predicting Failures</a></p><p><a href="https://www.researchgate.net/publication/330854331_An_Empirical_Comparison_of_Missing_Value_Imputation_Techniques_on_APS_Failure_Prediction">https://www.researchgate.net/publication/330854331_An_Empirical_Comparison_of_Missing_Value_Imputation_Techniques_on_APS_Failure_Prediction</a></p><ul><li><a href="https://www.tandfonline.com/doi/full/10.1080/08839514.2019.1637138">Comparison of Performance of Data Imputation Methods for Numeric Dataset</a></li><li><a href="https://www.sciencedirect.com/science/article/pii/S0020025513005124">An insight into classification with imbalanced data: Empirical results and current trends on using data intrinsic characteristics</a></li><li><a href="https://link.springer.com/chapter/10.1007/978-3-319-46349-0_36">Prediction of Failures in the Air Pressure System of Scania Trucks Using a Random Forest and Feature Engineering</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d3cb9267a024" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>