<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Great Learning Snippets on Medium]]></title>
        <description><![CDATA[Stories by Great Learning Snippets on Medium]]></description>
        <link>https://medium.com/@blogsupport?source=rss-2e2a341c9deb------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*huk-bW24B06fmVJ-9MS08g.png</url>
            <title>Stories by Great Learning Snippets on Medium</title>
            <link>https://medium.com/@blogsupport?source=rss-2e2a341c9deb------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 07:35:08 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@blogsupport/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Content moderation in Social Media with AWS services — Capstone Project]]></title>
            <link>https://medium.com/@blogsupport/content-moderation-in-social-media-with-aws-services-capstone-project-3a10c3f9ac92?source=rss-2e2a341c9deb------2</link>
            <guid isPermaLink="false">https://medium.com/p/3a10c3f9ac92</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[capstone-project]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[social-media]]></category>
            <category><![CDATA[aws-service]]></category>
            <dc:creator><![CDATA[Great Learning Snippets]]></dc:creator>
            <pubDate>Thu, 21 Jan 2021 05:21:57 GMT</pubDate>
            <atom:updated>2021-01-21T05:28:14.318Z</atom:updated>
            <content:encoded><![CDATA[<h3>Content moderation in Social Media with AWS services — Capstone Project</h3><p>Today social media is an integral part of our life, and it’s hard to imagine life without social media. Social media has grown leaps and bounds. As more and more people use social media, there is an increase in the trend of offensive content, which can put the social media platforms into controversy and often in legal issues. Hence, social media platforms have a bigger responsibility for moderating the content that gets published on their platform.</p><p>As part of our PGP-CC capstone project, we came up with an innovative approach to ease the overhead of content moderation for social media platforms. This solution tries to demonstrate how AWS cognitive services can be leveraged to automate content moderation.</p><h3>Proposed solution</h3><p>This solution proposes the usage of following AWS cognitive services for content moderation:</p><ul><li>AWS Comprehends to do the sentimental analysis of the blog texts. If the sentiment of the text is reported as negative with high confidence, then the blog will have a high probability of containing offensive/inappropriate content. Such blogs can be further evaluated for the usage of restrictive words, and if such words are found, the blog can be either auto rejected or put for manual moderation, depending on the requirement.</li><li>AWS Rekognition to moderate the blog photos to identify any inappropriate content. If Rekognition reports any moderation categories with a high confidence level, then there are high chances that blog photos contain inappropriate content. Certain moderation categories (from Rekognition) can be auto rejected (like explicit, violence etc.) and certain categories can be pushed for manual moderation (like suggestive, hate symbols etc.)</li><li>AWS Rekognition to analyze the photos for the presence of celebrities in the blog photos. If any inappropriate content is found with celebrity pictures, then there is a high chance of creating chaos. Moderation rules (text sentiment analysis confidence score &amp; photo moderation analysis confidence score) can be adjusted to have stricter conditions.</li></ul><p>Apart from the above primary objectives, this project demonstrates following aspects:</p><ul><li>AWS Translate to translate the blog text</li><li>AWS Lex to build text chat bot support system in to the web app</li><li>Lambda functions for all backend processing</li><li>API gateway to provide API interface for the Lambda functions</li><li>S3 to store blog photos and host static website</li><li>DynamoDB to store blog text</li><li>Aurora MySQL DB to store blog metadata</li><li>Cognito for user management and authentication</li><li>SNS for sending Email notifications</li><li>Cloud front distribution to enable faster content delivery</li><li>Route 53 to route traffic coming to DNS address to cloud front distribution</li><li>VPC endpoints to enable routing of traffic between Lambda function deployed in VPC with SNS, DynamoDB, S3, Comprehend, Rekognition</li></ul><p>Considering that resilience and fault tolerance are of high importance now a days, cross-region data replication has been implemented for following:</p><ul><li>S3 photo store</li><li>DynamoDB blog data</li><li>Aurora MySQL blog metadata</li></ul><h3>High level architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZsjlyBpfAUK9EvNT" /></figure><h3>Deployment Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*a2v7O_DIj65LSdAU" /></figure><h3>Implementation overview</h3><p>Some of the key use cases are detailed out here to give insight into the implementation details</p><p>Blog creation</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jCCMNEWz6kiYe-FS" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/1*ypvM3iNp551znDTHsmz-7Q.png" /></figure><h3>Blog text moderation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KpTm-Xwc-wdDEjcY" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/755/1*wiW5X343wxqSa2Jn-1BTTQ.png" /></figure><h3>Blog photo moderation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eot2nBpNFEhghmqu" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*asR0xOV7rnavZ7aPHQHweA.png" /></figure><h3>Read Blog</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Faw8IoPQm8HH8VQI" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/822/1*1rdHF4aG1acJ6eMooJgRew.png" /></figure><h3>Admin moderation of blog</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cIaJnHDUE-mTqpuJ" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*HBsuYHw_cwYMf45TY_vbWw.png" /></figure><h3>Key learnings</h3><p>As part of our capstone journey, we came across many challenges which enabled us to get more insight into the AWS cloud. Here are few key takeaways:</p><ul><li>Versioning should be enabled on S3 to enable cross region replication</li><li>Source S3 bucket should be granted access to replicate in destination region S3 bucket via IAM role</li><li>Streaming should be enabled in DynamoDB for enabling global table feature</li><li>Free tier provisioned capacity will not support global tables in DynamoDB</li><li>Cross-region read replicas not supported for MySQL RDS</li><li>RDS proxy not supported for Aurora RDS instances</li><li>Default DB parameter group doesn’t support cross region replication of Aurora RDS</li><li>VPC endpoints enable access to S3, DynamoDB, Comprehend, Rekognition and SNS for Lambda inside VPC</li><li>Custom classification features of Comprehend will support classification of content within predefined sets of labels. Even if a content doesn’t match any label, it will map to one of them. So it is suitable to use when the scope of content is limited to certain context.</li><li>Custom classification endpoint will be charged even when not in use.</li><li>Before enabling RDS cross-region replication, RDS subnet should be created in the destination region.</li><li>RDS instances will be restarted automatically after 7 days if they are not started manually.</li><li>If Lambda doesn’t have necessary rights to access S3, still pre-signed url will be provided by S3 for object access on request. However, the URL access failed.</li><li>S3 will return a pre-signed URL for fetching an object, even if it doesn’t present in the bucket. This needs to be handled at the client side.</li><li>AWS authentication enabled DB users can be used to access Aurora MySQL RDS instances from Lambda.</li><li>CORS should be enabled at the API gateway to enable access of API’s from web apps.</li><li>Lambda will require appropriate rights for accessing Comprehend, Rekognition, Translate, SNS etc. Which can be granted by assigning appropriate policies to the execution role attached to the Lambda.</li><li>Additional dependencies required for Lambda like MySQL connector library can be added via Lambda layers.</li><li>To stop/delete a read replica cluster of RDS, it has to be first promoted as stand-alone.</li><li>To associate alternate domain names to cloud front distribution, you need to get a custom wildcard security certificate and then associate the DNS with cloud front distribution.</li><li>To add routing logic to a cloud front distribution from route53, DNS name should be configured as alternate domain name in cloud front distribution and associate the custom security certificate.</li><li>Custom security certificates for the DNS can be created using Amazon Certificate Manager and the certificate needs to be added as a CNAME record in the DNS.</li></ul><h3>Authored by</h3><p><a href="https://www.linkedin.com/in/ashok-setty-b09169b/">Ashok K A Setty</a> — More than 17+ years of experience in software development in Aerospace, Telecom and Supply Chain domain. Currently working in Boeing India as a Technical Lead for Cabin System software development.</p><p><a href="https://www.linkedin.com/in/vijay-rajagopalan-9b73355a/">Vijay Rajagopalan</a> — Overall 20+ years of experience in the IT industry. Currently working as Senior Manager with Hitachi Vantara managing a global centralized Marketing Center of excellence team to drive initiatives around Delivery excellence , Process adherence and Standardization.</p><p><a href="https://www.linkedin.com/in/maheshkumarraj/">Maheshkumar Rajagopalan</a> — Overall 14+ years of experience in Technical support and Project management. Currently working with Sabre as a Supervisor Service Delivery managing their Asia Pacific operation for End user computing , Unified communication and Office infrastructure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/150/0*1E66T7__s4ej9Kvq.png" /></figure><p>Great Learning’s Blog covers the latest developments and innovations in technology that can be leveraged to build rewarding careers. You’ll find career guides, tech tutorials and industry news to keep yourself updated with the fast-changing world of tech and business.</p><p><em>Originally published at </em><a href="https://www.mygreatlearning.com/blog/content-moderation-in-social-media-with-aws-services/"><em>https://www.mygreatlearning.com</em></a><em> on January 21, 2021.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a10c3f9ac92" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[HealthCare at your Doorstep — Remote Patient Monitoring — IoT and Cloud]]></title>
            <link>https://medium.com/@blogsupport/healthcare-at-your-doorstep-remote-patient-monitoring-iot-and-cloud-5ed76d80680b?source=rss-2e2a341c9deb------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ed76d80680b</guid>
            <category><![CDATA[capstone-project]]></category>
            <category><![CDATA[healthcare]]></category>
            <category><![CDATA[iot]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[cloud]]></category>
            <dc:creator><![CDATA[Great Learning Snippets]]></dc:creator>
            <pubDate>Fri, 18 Dec 2020 07:40:27 GMT</pubDate>
            <atom:updated>2020-12-28T06:53:30.903Z</atom:updated>
            <content:encoded><![CDATA[<h3>HealthCare at your Doorstep — Remote Patient Monitoring — IoT and Cloud</h3><p><strong>Project Mentor:</strong> Mr Nirmallya Mukherjee<br><strong>Contributed by: </strong><a href="https://www.linkedin.com/in/krishnanlnarayan"><strong>Krishnan L Narayan</strong></a>, <a href="https://www.linkedin.com/in/ajit-srinivasan-b459853"><strong>Ajit Srinivasan</strong></a>, <a href="https://www.linkedin.com/in/shirish1"><strong>Shirish Mistry</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/0*fGHcHhQgWkbijh4j" /></figure><p>In traditional hospitals (and hospice situations), patients come into the hospital or doctor’s office when they are not feeling well or, in a worst-case scenario, they need urgent care and are brought directly into Emergency. Digital wearable devices such as smartwatches, digital blood pressure and weight scales have made it possible for patients to self-diagnose problems long before they become urgent cases that require emergency care. However, self-diagnosis can be dangerous, and it is always preferable to have doctors view remote sensor data and perform diagnosis prior to clinic or hospital admittance. The challenge is to make such data available to doctors in real-time. The patients will typically be engaged in their daily tasks in a non-hospital environment (either at home/work or in a senior facility, for example). The future of cardiovascular care is envisioned in the following way by experts in the delivery of such care:</p><ol><li>Incorporating remote monitoring for cardiovascular patients, using devices that record vital signs, heart rhythms like atrial fibrillation, coronary artery disease detecting heart blockages, and congestive heart failure symptoms, without being in the office or hospital.</li><li>Enabled by IoT and Cloud Technology, these remote physiologic monitors in conjunction with best practices at the clinic, have the potential to reduce patient hospitalizations and improve care.</li></ol><p>For this project, we have implemented a cloud-based IoT demo solution on AWS, which includes device transmitter code that simulates heart rates at the patient location, and sends the data to the AWS Cloud, where the data is stored in a NoSQL database , anomalies are detected in real-time via Streams processing, and results can displayed in a dashboard at the doctor’s office. Archival for offline analytics is also made available.</p><h3>Use Case</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/0*8_nBTo7X4EVxQlD6.png" /></figure><h3>Proposed Solution/Key Components</h3><p>The solution comprises the following:</p><ul><li>A front-end simulator program creates data by emulating a ‘sensor’</li><li>The sensor information will comprise the device details (unique identifier for the patient and device) and heart rate readings</li><li>This data will feed into the AWS IoT Core at predefined intervals</li><li>Based on the rules defined, data will be sent downstream to trigger various real-time events such as email notifications where applicable</li><li>Downstream data will be partitioned and sorted into a NoSQL database table, which will be used to enable rapid storage and retrieval of time-series information, and to allow for critical decision-making in near-real time. Goal is to pay attention to scalability, availability, and reliability</li><li>Based on the defined rules, real-time data will be routed to Kinesis Streams, where Kinesis Stream Application will sort data streams based on high and low heart rate and route to a separate Lambda function for real-time SNS and email notifications. Routed real-time sorted streams will be stored in a separate table in the NoSQL database.</li><li>Dashboards will display the patient wise readings and will be designed to be made available to administrators with authorized access, for insights and action</li><li>Downstream data will also be sent via an alternative path for archival and offline analytics with secure access as needed by medical personnel for research and regulatory purposes</li></ul><h3>Data Flow</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/0*fVHixhtFZmIZtEdg.png" /></figure><h3>Technical Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/0*pciJsrENU3vCDJRM.png" /></figure><h3>Role of Cloud</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/1*mtz_lF6gyPjmWLhOwqRrEw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/746/1*yU-HRwFG3_CbkuIHT20ybQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/1*X_fKjhzUido3T2Ug_1UI6g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/725/1*8ILhV8dBZBXsIFJ-HCYHfQ.png" /></figure><p><strong><em>* Only key components are listed; components pertaining to VPC, Subnets, Internet Gateway and Security Groups have been left out to make the document more readable.</em></strong></p><h3>Why AWS?</h3><ul><li><strong>Team Skills</strong>: Major reason for choosing AWS as the preferred cloud platform for the project stems from the fact that this is the technology that all the team members have maximum hands-on experience in AWS compared to any other cloud platform</li><li><strong>Basket of Services</strong>: The breadth and depth of services offered by AWS overwhelms all the other cloud providers by a significant margin (over 175+ services offered by AWS)</li><li><strong>Been Around Longer</strong>: Since AWS was released way before the other cloud service providers, it is touted as being more robust and stable than the later platforms</li><li><strong>Market Leader</strong>: AWS continues to hold the top position among cloud service providers. Reference: Gartner’s latest “Magic Quadrant” report on Cloud Infrastructure and Platform services — <a href="https://www.theregister.com/2020/09/09/gartner_on_cloud_aws_fails/">https://www.theregister.com/2020/09/09/gartner_on_cloud_aws_fails/</a></li><li><strong>Not a Windows Application</strong>: Since the AskUs Capstone project does not use any Windows component, using a platform like Azure did not appeal as there was no need to take advantage of any existing Windows landscape</li></ul><h3>Architecture Principles and Assumptions</h3><p>The analytics requirements for Remote patient monitoring fall into two major categories:</p><ul><li>Real-time, or near-real-time analysis of streaming data from patient sensors</li><li>Long-term research to aid in the analysis of patient data to suit the needs of patient trials or regulatory needs. The goal of this project is to provide capabilities for both categories.</li></ul><h3>Principles and Assumptions</h3><p>Managing the needs of an IoT Cloud Architecture for Medical clients requires balancing the users’ business activity and workflow issues and the associated information needs (effectiveness) with the cost (efficiency) and dynamic, improvement-readiness (dynamic efficiency) issues of the Cloud infrastructure.</p><p>This translates to balancing the (efficiency-based) need for standards with the (effectiveness-based) needs for user applications and user information. Keeping this in mind, the following principles were applied to this Capstone Project:</p><ul><li>Use of managed services — we believe that where there are market leader products available to fulfil a particular function, where these products have been proven to perform, then such products should be leveraged, in preference to embarking on the expensive and error-ridden path of building a custom application from scratch or possibly less expensive but less well designed and supported products. The choice of AWS platform managed services, for example, satisfies this principle</li><li>Use of modular components with granularity and loose coupling — loose coupling refers to the decoupling of the platform and language in which the services are implemented, the communications protocols used to interact between them, and the data formats used to store, and exchange information between the modules.</li><li>Security — The Application uses Sensitive Patient Information (SPI) — device access must be restricted not only to registered devices, but also by IP address so that unauthorized IP addresses are blocked. Administrative access to the dashboards must also be restricted only to authorized IP addresses due to SPI requirements</li></ul><h3>Future Phases</h3><p>The following will be considered enhancements in future versions:</p><ul><li>Edge decision making on the pi-transmitter device or IoT Gateway in case of dire emergencies</li><li>A Machine Learning model for Outcome Predictions</li><li>Dynamic patient profiling — we can define buckets like low risk to very high risk; perhaps any high-risk anomaly data can be processed at edge pi-transmitter device</li><li>The Machine Learning model- point 2 above — can auto shift a patient’s profile for higher alerted monitoring coupled with edge processing</li></ul><h3>CONCLUSION</h3><p>Cloud Computing and IoT have the potential to enable many medical applications such as remote health monitoring, fitness programs, chronic diseases, and elderly care. Various medical devices, sensors, and diagnostic and imaging devices can be viewed as smart devices or objects constituting a core part of IoT. This solution can be envisioned to provide treatment and medication at home by the healthcare providers.</p><h3>THE TEAM</h3><p><strong>Krishnan Narayan</strong>: A former Innovation Leader at IBM for the Telco Industry and a Chief Architect with 20+ years of leading, architecting and implementing Digital Transformation solutions with a variety of clients in the Computer, Telecommunications, and Energy Industries. He has been a keynote presenter at industry events, delivered learning to industry clients, and has a lifelong commitment to continuous learning in Cloud Technologies, Data Science and AI.</p><p><strong>Shirish Mistry: </strong>Shirish is a disciplined and astute professional with more than 14 years of experience. He is currently associated with Larsen and Toubro Infotech Limited as a Specialist, on technologies related to End User Computing, Workplace Modernization and Enterprise Workstation Management. He has expertise in IT Infrastructure Management, solution design, technical support, leading technical teams for transition, transformation and delivery based on various cloud and on-premise infrastructure management technologies. He has a diverse experience working with insurance, manufacturing, legal, federal as well as International Organizations.</p><p><strong>Ajit Srinivasan</strong>: Ajit is a professional with over 30 years of in IT Project Delivery, IT Operations, and IT Strategy primarily in the Finance domain — Banking, Credit Cards and Retail Lending &amp; Microfinance. He has a proven track-record in delivering &amp; implementing mission critical business solutions, Global Stakeholder Management, Program, Project &amp; Product Management, Managing teams delivering Global Production Support SLA driven engagements, and as Profit Centre and Site Head while working with financial giants such as Standard Chartered Bank, GE Capital and Credit Suisse. Ajit is now a reborn, reinvented, rejuvenated technologist with a newfound focus in Cloud Computing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/150/0*vhOQabxRV35mELs3.png" /></figure><p>Great Learning’s Blog covers the latest developments and innovations in technology that can be leveraged to build rewarding careers. You’ll find career guides, tech tutorials and industry news to keep yourself updated with the fast-changing world of tech and business.</p><p><em>Originally published at </em><a href="https://www.mygreatlearning.com/blog/healthcare-at-your-doorstep-remote-patient-monitoring-using-iot-and-cloud/"><em>https://www.mygreatlearning.com</em></a><em> on December 18, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ed76d80680b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[TravoBOT — “Move freely in pandemic” (AWS Serverless Chatbot)]]></title>
            <link>https://medium.com/@blogsupport/travobot-move-freely-in-pandemic-aws-serverless-chatbot-ad39f3fe2301?source=rss-2e2a341c9deb------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad39f3fe2301</guid>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[pandemic]]></category>
            <category><![CDATA[aws-service]]></category>
            <dc:creator><![CDATA[Great Learning Snippets]]></dc:creator>
            <pubDate>Fri, 27 Nov 2020 10:43:43 GMT</pubDate>
            <atom:updated>2020-12-07T07:17:03.876Z</atom:updated>
            <content:encoded><![CDATA[<h3>TravoBOT — “Move freely in pandemic” (AWS Serverless Chatbot)</h3><p><strong>Team Members:</strong><br><strong>1. </strong><a href="https://www.linkedin.com/in/deepmala-agarwal"><strong>Deepmala Agarwal</strong></a><strong><br>2. </strong><a href="http://linkedin.com/in/link2arun"><strong>Arun Mohan</strong></a><strong><br>3. </strong><a href="https://www.linkedin.com/in/ghanshyam-kumar-272a8463"><strong>Ghanshyam Kumar Singh</strong></a><strong><br>4. </strong><a href="https://www.linkedin.com/in/hitaish"><strong>Hitaish Raina</strong></a></p><p>Of all the things that this pandemic has taught us, one of the primary concern that many people are still facing is the uncertainty in planning their travel — either in case of emergency or for leisure. And this resulted in TravoBOT — A chatbot that helps the user by collating information from different data sources and provide user with a travel recommendation to a particular destination.</p><p>This article gives an overview to the serverless chatbot architecture using various AWS services.</p><p>The four sections details the technical build &amp; configuration:</p><ul><li>Purpose: By understanding the intent of our solution, you can enhance the impact of your work. This section explains the use case that our product has been designed to achieve.</li><li>Data Layers: In this section, we will discuss about the conceptual stages of data layers in a data lake in AWS.</li><li>Implementation Flow &amp; Technology Stack: In this section, we will discuss about how to implement chatbot from ingestion to presentation using various AWS services like Lambda, S3, Glue, Athena, etc.</li><li>Presentation: This section details about the integration of facebook with AWS Chatbot service (Amazon LEX)</li></ul><h3>Purpose</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/0*2aL_8Nt6YYXokhj_.png" /></figure><p>TravoBOT takes in one’s current location, travel origin and travel destination and based on<br>these inputs, it provides the following:</p><ol><li>Current pandemic situation in the destination that includes counts of active cases,<br>deaths and recovery</li><li>Travel Advisory &amp; Travel recommendation based on past 14 days trend</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/0*aW2QIEEYMRbQfU7P.png" /></figure><h3>Data Layers</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/0*x1kh9PoCP3Od9B0u.png" /></figure><p>We have categorized the data layers as below:</p><ul><li>Data Ingestion Layer — Data from various sources gets loaded appropriate S3 buckets and the AWS Lambda functions reads the API data and loads the same onto S3 buckets.</li><li>Data Transformation Layer — In this layer, Lambda function reads the data from ingestion layer S3 buckets and loads the same onto transformation layer S3 buckets. As part of transformation, Lambda performs the following actions to ensure accuracy-</li><li><strong>Calculation of moving average of active cases</strong></li><li>Calculates daily growth rate — Daily active cases increase on a given day/Total # of active cases previous day</li><li>Calculates average daily growth rate for past 14 days</li><li><strong>Travel Recommendation</strong></li><li>Predict Risk Zone: Using scatter plot &amp; linear regression, we predict the trend (declining or inclining)</li><li>If the value calculated above is &gt; 0, it determines the location to be in “RED” zone (inclining) and if it is less than 0, it is in “GREEN” zone (declining)</li><li>Data Consumption Layer — In this layer, user logins to Facebook account and adds the bot as their contact. To be able to do this, a Facebook account along with callback URL to AWS was leveraged.</li></ul><h3>Implementation Flow &amp; Technology Stack</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/0*lHJtKlu0sXx7omID.png" /></figure><h4>Consumption Layer</h4><ul><li>User initiates the conversation with BOT through Facebook</li><li>API Gateway has been configured so that this BOT can be leveraged in other channels (if<br>required)</li><li>Amazon LEX has been configured with Intents, Slots, validation &amp; fulfilment to interact<br>with user per the business rules defined in Consumption Lambda</li><li>Consumption Lambda performs the following tasks:<br>a. Validate the user inputs (as appropriate)<br>b. Retrieve data from transformation S3 buckets and provide to LEX in line with the<br>user response based on the defined business rules<br>c. Logging into Cloud Watch</li></ul><h4>Transformation Layer</h4><ul><li>Transformation S3 bucket — Output of Transformation Lambda is stored in this bucket</li><li>Transformation Lambda — Lambda function reads the data from ingestion layer S3<br>buckets and transforms the same onto transformation layer S3 buckets</li></ul><h4>Data Ingestion Layer</h4><ul><li>Ingestion S3 buckets — Stores raw data ingested through APIs into Cloud</li><li>Ingestion Lambda — Initiates the API call to retrieve data on a daily basis</li><li>All the deployment was done using AWS SAM CLI</li></ul><h3>Presentation</h3><p>This bot was implemented using Amazon LEX service provided by AWS for building chatbot. It<br>also integrates with Facebook (only for development purpose) using callback URL.</p><p><strong>Below are the steps to configure Amazon LEX:</strong></p><ul><li>Bot: Amazon Lex bots can understand user input provided with text or speech and converse in natural language. It consists of all the components related to a conversation, which includes:</li><li>Intent: An intent represents an action that the user wants to perform.</li><li>Utterances: An utterance is a text phrase that invokes intent</li><li>Slots: Slots can be defined as a parameter of intent.</li><li>Prompt: A prompt is a question that Lex uses to ask the end user to provide data for a slot that is needed to fulfill an intent</li><li>Fulfillment: Fulfillment provides the business logic that is executed after getting all required slot values. Like AWS Lambda functions for fulfillment of business logic and user input validations.</li></ul><p><strong>Creating &amp; Architecting Bot Conversations:</strong></p><ol><li>Bot Name: TravoBot</li><li>Output: Text</li><li>Set Session Timeout: 5 min</li></ol><p><strong>Create Slots:</strong> We are creating slots as shown below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/677/0*tdDFT8BxayUeOTim.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/963/0*pb4DTOcbiWiw8DCM.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/0*TK3aa-YSaAN8Ud_b.png" /></figure><p><strong>Slots: </strong>Map slots with their types and provide prompt questions that need to be asked to get valid value for the slot. Note the sequence, Lex-bot will ask the questions according to priority.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/0*jw9QUwSfNq0fx7Tk.png" /></figure><p><strong>Confirmation:</strong> This is optional. If required you can provide a confirmation message e.g. Are you sure you want to travel?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/999/0*0LnosycnQvUPGEqD.png" /></figure><p><strong>Fulfillment:</strong> Now we have all necessary data gathered from the chatbot, it can just be passed over in lambda function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/776/0*iSqshnFSj9VNeUi1.png" /></figure><p><strong>Lambda initialization and validation:</strong> Lambda function provided here i.e. TravoBot will be invoked on each user input whose intent is understood by Amazon Lex. It will validate the Source and destination with predefined list of cities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/0*wTKDUoHZVNMpV2fB.png" /></figure><p><strong>Error Handling:</strong> We can customize the error message for bot users. Click on error handling and replace default values with the required ones.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/0*dhzppYPMZLBYZIO_.png" /></figure><p><strong>Bot is Now Ready to Chat:</strong> Build and publish the chatbot and integrate with required channel as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/0*A80hkJNaJI7zqa-1.png" /></figure><h3>Potential Enhancements:</h3><ul><li>TravoBOT can be further enhanced to include questions related to the following:</li><li>Mobility Stats — that includes retail &amp; recreational places, grocery stores, parks, workplace, transit stations &amp; residential places</li><li>Open Businesses</li><li>Nearest hospitals</li><li>As part of data ingestion, AWS Batch can be used if the data load time is more than 15 minutes due to the limitation of AWS Lambda.</li></ul><h3>Summary</h3><p>Key takeaways are:</p><ul><li>Serverless architecture offers effective storage of big data volume &amp; variety using S3 Serverless architecture has “pay as you use” model making it cost efficient</li><li>This architecture provides a way to remove infrastructure responsibilities from your workload including provisioning, scaling &amp; maintenance</li><li>Amazon LEX provides easy integration with Facebook</li><li>Boto3 does not support Pandas library — hence we need to use AWS SAM CLI for importing codes from developer machine to cloud</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/0*YzFp1DFD_MoZS90U.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad39f3fe2301" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Classification of product reviews in e-commerce website and proposing a balanced rating system]]></title>
            <link>https://medium.com/@blogsupport/classification-of-product-reviews-in-e-commerce-website-and-proposing-a-balanced-rating-system-a1ebf6dd192c?source=rss-2e2a341c9deb------2</link>
            <guid isPermaLink="false">https://medium.com/p/a1ebf6dd192c</guid>
            <category><![CDATA[product-reviews]]></category>
            <category><![CDATA[ratings]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[ecommerce]]></category>
            <category><![CDATA[e-commerce-business]]></category>
            <dc:creator><![CDATA[Great Learning Snippets]]></dc:creator>
            <pubDate>Fri, 28 Aug 2020 09:12:41 GMT</pubDate>
            <atom:updated>2020-08-28T09:12:41.782Z</atom:updated>
            <content:encoded><![CDATA[<p>Saikat Bhattacharya, BHARANI SRINIVAS N, Prakash Ramamurthy, Venkataramanan Tharagam, Dr. D Narayana</p><h3>Abstract</h3><p>Online product reviews are of high importance in present world as the e-commerce websites are continuously increasing the volume of business. According to the latest report [2], in 2018, annual retail e-commerce sales growth in India is 25%. Almost 28% of the internet users are digital buyers and this percentage is increasing in a steep manner. So, reviews and ratings of any product in the e-commerce websites are very important to create an impact on buyer’s mind and also it works as a feedback to the product owner. Besides that, when a buyer writes a review about a particular product, it is not necessary that he will review only the product. For example, one user has purchased an iphoneX. Although he is happy with the phone, he might have faced a problem with delivery or the seller took a higher price for that phone. So, when he will go for writing a review, he may rate his experience as 1 star. Currently, in all major e-commerce websites this rating directly affects overall product rating. Hence, even if the product is flawless, due to a bad delivery partner or higher price set by seller the reputation of the product is getting affected. In this work we have proposed a machine learning based approach which will classify the user reviews according to its content. We have targeted three types of reviews — product reviews, delivery reviews and seller reviews. We have implemented a multilayered validation approach (with k-fold). The classification model resulted 78% accuracy for the mobile phones (the product category we chose for this work). Also we have experimented with an attention based neural network and got an improved result with 84% overall accuracy. The accurate classification of these three categories will help both the product owner and the buyer to disambiguate the intent of the past reviews. And as an obvious result a bad seller will not affect the product reputation.</p><h3>Introduction</h3><p>The reviews on e-commerce websites are increasingly playing important role to make a buyer’s decision for the product purchase [5]. The buyers of the product not only give the textual reviews but also provide a star rating. The star rating implicitly defines the product quality. This indication gives both the future buyer and the product owner an overall idea about the performance of the particular item. But in most of the major e-commerce domains the buyer satisfaction does not solely depend on the product quality. The seller and the quality of delivery which includes packaging, delivery charge and speed of delivery also play significant roles. Even if a product is good, the user often provides bad rating due to an unexpected experience with the delivery partner or a high price set by the seller. This affects product rating directly. Websites like amazon.com or flipkart.com provides a scale of 1 to 5 for post-purchase product rating along with the text reviews.</p><p>These websites calculate a simple average to find the rating of the product [6]. But it gives a ambiguous message to the future buyer and product owner. For example, let’s take a case of a particular brand of mobile phone. The past buyers of this phone from a particular seller faced a lot of hassles due to a particular delivery partner. So they have reviewed the bad delivery or seller experience instead of the product. In this case the rating provided by those users will be very low. Although the product quality did not play<br>any role for this, the rating will affect the overall product rating as the product rating is the simple average of all the ratings provided by all the users. Hence, it will draw an incorrect and ambiguous perception about the actual product.</p><p>A study [1] shows that a normal human being is capable of reading 200 words per minute. During the data collection phase we have come across for an average selling mobile phone in amazon.in the number of reviews is more than 2,000 and average word count is 152.3 per review. So, there are more than 30,4600 words to read which needs 1,523 minutes to spend.</p><p>This is evidently a time consuming process for a buyer as the variants and brands of products are increasing everyday. So, reading all the reviews to reach a conclusion is not a plausible solution.<br>In this work we have developed an automated review classification process. Each review of a particular product will be classified as per its content into three categories as it is mentioned above. This approach will ensure that a bad experience with delivery or seller should not affect the product reputation and vice-versa. In this introduction section we have tried to give an overall picture and the importance of the work in present day context. In the following sections we will be presenting the data collection, pre-processing, EDA, model building approaches, experiments and simplistic presentation of<br>product architecture which will be followed by a conclusion what we have reached.</p><p>Also a future plan will be presented at the end.</p><h3>Data Collection</h3><p>Collecting a good number of reviews is the starting point for this project as shown in Fig.1. As we targeted major e-commerce websites in India, the default choice was to be amazon.in. We have chosen four top selling mobile phones to get a good amount of mixed reviews of all three categories we targeted for. As we do not have intention of any kind of sentimental study,<br>the high volume of the reviews is what we looked for. More the number of reviews more the diversity of texts and better the training of the model.</p><p>To avoid fake or deceptive reviews by imposters, sites like amazon provide a tag ’verified user’ or ’verified purchase’. Such reviews are very much available in those websites for popular products. We have taken a technique called web scrapping [3]. An open-source and customized web scrapping tool called Parsehub [4] is used for extracting the reviews from amazon.in website. The reviews collected along with metadata like username, date of review, stars and review title as shown in Table.1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/869/1*1mT7-v_LIE98wvFvqQr74Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/667/1*A9KK6IKxzwqXnfFGGh600A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/1*OADG-9OxGSbb8Avny5K9nw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/1*RluV1BJuaZ8Kp6kcBXOZ5Q.png" /></figure><h3>Exploratory Data Analysis</h3><p>Exploring the collected data is the first step towards any data science project. So, after collecting the reviews, even before proceeding to clean up, we have summarized the observations found in Fig. 2.</p><p>In Fig. 3 we can clearly see that most of these words are articles and prepositions which doesn’t yield any information to us. For solving this issue we have to remove these stopwords using NLTK. Also we need to lemmatize the text to get the original word chopping off the suffixes like ‘-ing’. The next thing was to remove punctuation. The punctuation may help us to add emotions while reading, but it does not add any extra information to understand the intent of it.</p><p>One of the challenges we found during the initial text mining is that we found that certain reviews also had emoticons. We removed those emoticons using ‘emojis’ library. Some of the reviews were in local languages e.g. ‘Hindi’ and we used text blob to identify these and removed. After following these steps the frequency of the words given as per Fig. 4. Few of the important steps are explained in details in the following sub-section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/1*Q9yoZivqMwD2OBFDWgRRsQ.png" /></figure><h3>Pre-processing of Data</h3><p>Generally one review is the collection of multiple sentences. Each sentence may describe different aspects. For example, one of the reviews is like -</p><p>Well, as they claim, it is a flagship killer. Superfast charging, barely takes and hour to charge almost full. Speaker is loud enough. Worth the money. OnePlus also proved their worth over the years by providing timely android updates for a long time. So this phone can be used for atleast 5 years without any worries. Got it delivered in less than 12 hours, all credit goes to Amazon.in for that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/639/1*VcZODlpkCews_ELNcPR5ag.png" /></figure><p>This review has six sentences. First five sentences are describing the product experienced by the user. The last sentence is about the experience with delivery. So, to classify this review we have to scan through each sentence of the whole review text [7].</p><p>In this project it is assumed that the sentences in review will be separable with period ‘.’ and sentences will be written in English language. These assumptions are not valid in all the cases as we have experienced during pre-processing of the dataset. Those challenges will be discussed later.</p><p>The first step of pre-processing was to break each review in its sentences as shown in Fig.1. The sentences are not labelled. So a manual labelling is needed here. Initially we have taken around 300 of sentences to label them in three categories — product, delivery and seller. The sample of labelled dataset is given in Table.2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/628/1*Oz9Wi3p-P0xQQcKemGCSPw.png" /></figure><p>Once the labeling is done, the data is ready for the actual pre-processing steps. As shown in 1, the pre-processing was consist of removal of stopwords, spelling correction, lemmatization, removal of punctuation and removal of most frequesnt words. Three main steps are described in details below.</p><h4>Removal of stopwords</h4><p>Stopwords usually refers to the most common words in a language. These are filtered out before or after processing of natural language text data [9]. For example, in the following review the words like ’is’, ’also’, ’if’ etc. are adding grammatical sense but not adding any merit to the intent of the sentence.<br>fingerprint unlock is also good, if you have face unlock on then all the time it<br>will unlock with face data before the the fingerprint works.</p><p>These are few of the most common words in English vocabulary. So, to improve the performance of the model to be built, it is necessary to remove these stopwords before further processes. In Appendix A, we have given the code snippet to remove stopwords using nltk package.</p><h4>Spelling Correction</h4><p>The reviews are not only having correct spellings always but also people are prone to write in sms languages [12] [11]. So, from those incorrectly spelled words building the actual word is a mandatory step as part of the pre-processing of review texts. In Appendix B the code for spelling correction using textBlob [13] is given for reference.</p><h4>Stemming and lemmatization</h4><p>For grammatical reasons a word changes its form when it is used in a sentence. For example, the word ’run’ can be used as ’ran’ or ’running’ as per the demand of the usage. The goal of both stemming and lemmatization [14] is to reduce inflectional forms and sometimes derivationally related forms of a word to a common base form. In Appendix C the lemmatization and stemming process if given for reference.</p><h3>Model Selection and architecture</h3><p>Algorithms like Latent Dirichlet Allocation (LDA) [25] enables for an unsupervised way of topic modelling. It builds a topic per document model and words per topic model, modeled as Dirichlet distributions. We used the ldaModel provided by Gensim package. The unsupervised process was executed as it provides keywords that helps to determine the topic of a review. It also reveal if there is any category beyond Product, Delivery and Seller. In real-world scenario the product manager can benchmark the results against LDA modelling to identify potential keywords which actually determines multiple annotator of a review.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kPGE_cVCqUQAO0oDPJDMxA.png" /></figure><p>Fig 5 is displaying the outcome of LDA algorithm that found few important keywords like product, quality, display, camera (which all describe the features or forms of a product, mobile phones, in this case), amazon, price, replace (which are very much related to the seller of the product), time (which could be a potential keyword for delivery — although is has ambigu-<br>ous interpretation). So, overall with this unsupervised topic modelling, clearly, we got three categories out of the review texts.</p><p>To classify the reviews as per our desired categories the labelling of the reviews are required. In this work we targeted to label the reviews with a single annotator. But as one person may write about his experience with the product and delivery as well, we split each review into its consisting sentences assuming each sentence will have different topics to convey. As shown in<br>Fig 6, a typical review is written in multiple sentences with periods as the mark of the end of the sentence. So, we split the review based on the occurrence of period in it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*2esjzHJdFzdcPZTEAuqBfA.png" /></figure><p>For a supervised text classification, after splitting the reviews in multiple sentences, each sentence is labelled manually. As it is mentioned earlier single annotator is used for the labelling. We assumed each sentence will have only one intended review.</p><p>In this work we have taken two approaches for model building — TF-IDF and Neural Networks. Both the approaches gave comparable results. In the following subsections we will have a detailed discussion on these two.</p><h3>Implementing TF-IDF</h3><p>TF-IDF, which stands for term frequency — inverse document frequency, is a way to judge the topic of an article. This is done by the kind of words it contains. Here words are given weight so it measures relevance. The intuition behind it is that if a word occurs multiple times in a document, we should boost its relevance as it should be more meaningful than other words that<br>appear fewer times (TF). At the same time, if a word occurs many times in a document but also along many other documents, maybe it is because this word is just a frequent word; not because it was relevant or meaningful (IDF).</p><p>So, TF-IDF produces a score for each word and that score reflects the importance of the words in that corpus. For example, Fig 7 represents how the words of a sentence gets converted to its TF-IDF scores. Arrays of TF-IDF scores being the numerical inputs to the supervised algorithm, we chose</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*82DXB7BIpmYBRI8HX4f7iw.png" /></figure><p>Multinomial Naive Bayes, K-nearest neighbours, Random Forest and SVC with 10-fold cross validation. In k-fold cross-validation, the original sample is randomly partitioned into k equal sized subsamples. Of the k subsamples, a single subsample is retained as the validation data for testing the model, and the remaining k − 1 subsamples are used as training data. The cross-<br>validation process is then repeated k times, with each of the k subsamples used exactly once as the validation data. The k results can then be averaged to produce a single estimation. In training data set it was found that there was an imbalance of data number of ‘product’ reviews being the dominant. So, before training the model we followed a over-sampling technique using SMOTE module from imblearn [22] package. This step helped to adjust the class distribution of the dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/853/1*KsGEmGIelai-YUheUo7MeA.png" /></figure><p>Table 3 displays the validation confusion matrix for Random Forest Algorithm. It is clear that For ‘Delivery’ and ‘Seller’ the scores are lower compared to the same of ‘product’. Table 4 displays the validation confusion matrix for Random Forest Algorithm. It is clear that For ‘Delivery’ and ‘Seller’ the scores are lower compared to the same of ‘product’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*Nca4G6yh6OLVq1ygssNEnA.png" /></figure><h3>Implementation of Neural Networks</h3><p>To improve the accuracy we experimented with Neural Networks. Although Convolutional Neural Networks (CNN) are typically used for computer vision, in [26], the author has described a process for using that in sentence classification. In his work he reported on a series of experiments with convolutional neural networks (CNN) trained on top of pre-trained word<br>vectors for sentence-level classification tasks. Instead of image pixels, the input to most NLP tasks are sentences or documents represented as a matrix. Each row of the matrix corresponds to one token, typically a word, but it could be a character. That is, each row is vector that represents a word. Typically, these vectors are word embeddings (low-dimensional representations) like word2vec or GloVe, but they could also be one-hot vectors that index the word into a vocabulary.</p><p>Based on the discussion done in [26], we have designed a simplified CNN to build a classifier using GloVe vectorizer. As per the official documentation of GloVe: GloVe is an unsupervised learning algorithm for obtaining vector representations for words. Training is performed on aggregated global word-word co-occurrence statistics from a corpus, and the resulting representations showcase interesting linear substructures of the word vector space.</p><p>In Fig 8, the summary of our simplified model is shown. After running this model till 50 epochs we found the best validation accuracy to be 78% which is as good as the result of Random Forest as we discussed earlier. CNN takes a fixed length for all the sentences and does not consider the context of any word in the sentence. So a sequential model could be a better choice for text classification. In [27] we have come across a sequential model named as Hierarchical Attention Network (HAN). HAN is designed to consider the basic insights of the document structure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/1*lcElNW93KUMpDD-wI_n5Iw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*ee9okrv5kAyL7YqqOGZ4Gg.png" /></figure><p>1. Any document consists of a hierarchy (words to sentence, sentences to document),<br>2. Important words and sentences are very much context dependent.</p><p>To adopt these into the model, firstly they have built a hierarchical representation of the document and secondly a sentence level and a word level attention mechanisms are implemented.</p><p>Let us consider a document having L sentences. The ith sentence si is having Ti words. wit represents the tth word in the ith sentence. Each of the words in each of the sentences are converted to vectors through an embedding matrix, We. So, the vectorized form of the word will be — xit = Wewit. We implemented a bidirectional LSTM to summarize the information of the whole sentence centered around wit. Now, not all the words in a sentence has an equal conribution towards the intent of it.</p><p>For example, Ramanujan was a brilliant mathematician.- in this sentence the words brilliant and mathematician are having more sense for expressing the intent than other words. So, an attention mechanism is used to put more weight on the more sensing words in a sentence at the time of extracting the intent of it. Similarly each sentence in a document also needs different<br>attentions to make the content sensible.</p><p>But in our present work, as during pre-processing we already split the document in multiple sentences, we did not follow the exact hierarchical way of attention. Rather we modified the algorithm as a bidirectional LSTM based attention network. The code snippet is given in Appendix K.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/822/1*pwmnaAQ7utpIqAS1t-M_Ow.png" /></figure><p>As depicted in Fig 9, we have constructed a bidirectional LSTM based attention network using Tensorflow keras and after running it for 30 epochs we got maximum validation accuracy as 84.375% which is significantly higher than what we got any of the previous models.</p><h3>Deployment of the Model</h3><p>In Fig. 10, we have presented a simplistic architecture of the model deployment. The deployment consists of the following steps -</p><ul><li>Train the model: As we have shown in previous sections the model training should be the first step of the whole thing. The training could be done in one of the cloud environment (e.g., Google Cloud Platform or Azure or AWS) for a better performance.<br>For training the model, the good amount of data should be supplied from the data lake of the organization where they generally archive the data. These data will be unlabeled. So, even before starting model building process, the labeling is to be done. This can be done manually with the help of the dedicated agencies or a semi-supervised technique could be incorporated. But of course the accuracy of a manual labeling will be much higher than the later one.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/1*IW0GtWmatNoj_Xkg8Wqewg.png" /></figure><ul><li>Build API: Once the model is good enough for building an API over it, it is recommended to build a restful API with Flask or Tornado framework of Python. Also as per the recent release of Tensorflow.JS, it has a compatibility for Node.JS. So, an API could be built with Node.JS to get a better performance than Python based frameworks.<br>• Load Balancer: A load balancer like NGINX is necessary for handling all the incoming requests.<br>• User Interface: An user interface (UI) could be deployed at the frontend for users to interact. Through a form field, user may enter the reviews which will be processed through the API server and after categorization, it may get stored into the database. The database could be in sync with the Datalake with Kafka or similar techniques.</li></ul><h3>Business Value</h3><p>As we have seen in Fig 6, although the buyer is not happy with the seller, his less rating is impacting the product reputation. This may not be a problem for the popular products having thousands of product reviews and when people are more tech-savvy to give product reviews (like mobile phones and all). But it is really a problem for a new product being launched.<br>This product may be of good quality but a particular seller or the delivery partner is causing problem. In that case, the product reputation may get hampered. A overall low rating may turn off buyers.<br>So if an automatic process of intent analysis of a review can be implemented, it will be beneficial for the product managers to get the actual performance of the product and also it will be helpful for the future buyer not to avoid the product but to avoid a seller.</p><h3>Conclusion and Future Plan</h3><p>This project has so far given a good output as per our expectation and the models have shown a high accuracy on unknown data-set. The best accuracy we got is nearly 84%. This work leads to a revised rating system where instead of applying the given rating to the product, it can be weighted and distributed as per the category of the review.</p><p>As we mostly have concentrated on Indian websites, we have encountered some challenges due to language. We have come across many reviews which were although written in roman alphabets, the language used was ’hindi’. In this project so far we have removed those from the experiment dataset. But in future we may take them into consideration and after finding the language we may transliterate to original script which could be followed by a translation using google translation API [21]. Another challenge was to label the dataset. We have collected more than 20,000 reviews. But labeling all of them is a humongous task as that would have been done manually. In [8], a self-<br>training semi supervised learning method is described for labelling a large dataset automatically with the help of a small manually labelled dataset.</p><p>In this work we have considered the word and sentence level attentions. But we ignored user or product information in our analysis or model building. In reviews, it is evident that a user preference and product characteristics play a significant role. For example, most of the reviews of a high end mobile phone or play-station will be purchased by tech-savvy people who are very much aware of the technical details of the product. Hence the reviews will be having more technical and product related information. On the other hand if it is a pressure cooker, most of the buyers will be more about its usability, how fast it got delivered or how the delivery boy behaved or how the seller reacted when the user asked for an after sales service with a small issue. So, obviously the nature of review highly depend on the user and product characteristics. In future we have a plan to incorporate user-product attention (UPA) to improve the prediction of the nature of the review.</p><h3>References</h3><p>[1] N.M.A. Al-Othman, The relationship between online reading rates and performance on proficiency tests, The Reading Matrix 3 (2003).</p><p>[2] <a href="https://www.statista.com/topics/2454/e-commerce-in-india/">https://www.statista.com/topics/2454/e-commerce-in-india/</a></p><p>[3] <a href="https://en.wikipedia.org/wiki/Web">https://en.wikipedia.org/wiki/Web</a></p><p>[4] <a href="https://www.parsehub.com/">https://www.parsehub.com/</a></p><p>[5] Nan Hu, et. al., Manipulation of online reviews: an analysis of ratings, readability, and sentiments, Decis. Support. Syst. 52 (2012).</p><p>[6] Mudambi, S. M., What Makes a Helpful Review? A Study of Customer Reviews on Amazon.com (SSRN Scholarly Paper No. ID 2175066), Social Science Research Network, Rochester, NY, 2010.</p><p>[7] Singh, Abhinav, et.al., A machine learning approach to product review disambiguation based on function, form and behavior classification, Decision Support Systems 97 (2017) 81–91.</p><p>[8] Pavlinek, M., Text classification method based on self-training and LDA topic models, Expert Systems With Applications 80 (2017) 83–93.</p><p>[9] Rajaraman, A., Ullman, J. D. ”Data Mining”. Mining of Massive Datasets. (2011) 1–17.</p><p>[10] Natural Language Toolkit (<a href="https://www.nltk.org/">https://www.nltk.org/</a>)</p><p>[11] Lily Huang, Technology: Textese May Be the Death of English. Newsweek. Retrieved 2011–12–20.</p><p>[12] <a href="https://en.wikipedia.org/wiki/SMS">https://en.wikipedia.org/wiki/SMS</a></p><p>[13] TextBlob: Simplified Text Processing (<a href="https://textblob.readthedocs.io/en/dev/">https://textblob.readthedocs.io/en/dev/</a>)</p><p>[14] Stemming and lemmatization (<a href="https://nlp.stanford.edu/IR-book/html/">https://nlp.stanford.edu/IR-book/html/</a>)</p><p>[15] Vector space model (<a href="https://en.wikipedia.org/wiki/Vector">https://en.wikipedia.org/wiki/Vector</a>)</p><p>[16] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. (2014) GloVe: Global Vectors for Word Representation.</p><p>[17] <a href="https://nlp.stanford.edu/projects/glove/">https://nlp.stanford.edu/projects/glove/</a></p><p>[18] Mikolov, Tomas, et.al., Efficient Estimation of Word Representations in Vector Space, arXiv:1301.3781 [cs.CL]</p><p>[19] <a href="https://www.bbc.com/news/world-44569277">https://www.bbc.com/news/world-44569277</a></p><p>[20] <a href="http://rasbt.github.io/mlxtend/">http://rasbt.github.io/mlxtend/</a></p><p>[21] <a href="https://cloud.google.com/translate/docs/">https://cloud.google.com/translate/docs/</a></p><p>[22] <a href="https://imbalanced-learn.readthedocs.io/en/stable/api.html">https://imbalanced-learn.readthedocs.io/en/stable/api.html</a></p><p>[23] Nitesh V. Chawla, et.al., Special Issue on Learning from Imbalanced Data Sets, Volume 6, Issue 1 -Page 1–6.</p><p>[24] Rushi Longadge et.al., Class Imbalance Problem in Data Mining: Review, International Journal of Computer Science and Network (IJCSN)Volume 2, Issue 1, February 2013.</p><p>[25] David M. Blei, et. al., Latent Dirichlet Allocation, JMLR 3(Jan):993–1022, 2003.</p><p>[26] Kim, Y. Convolutional Neural Networks for Sentence Classification, arXiv:1408.5882v2 [cs.CL] 3 Sep 2014.</p><p>[27] Yang,Z., Hierarchi-cal attention networks for document classification. (InProceedings of the 2016 Conference of the NorthAmerican Chapter of the Association for Computational Linguistics: Human Language Technologies).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/1*3sKoyYnuvW337-9OOYoKcg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/899/1*JZ0YkDcMuhA-H1iJJ3WwRA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/637/1*FJWCNj14svZ6byDXA9fnPQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/627/1*C3xjm_koI-fuFtkJa9zuqA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/1*UYCwA5B7WcwnaqQ7CGd3Qg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/1*IYjl9EEdE4J-7hifUtAx3A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*KFbRq42MFSwGhQoCb7gJ0g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/1*7gNxHBkFNv0Zsf3tZlvsPw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/1*E02siIfQkZd2H6r7gRKuCw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/646/1*SS1s39vUcqf1-xJG25suMA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/1*JT1RZpxMi8EzElsbXvWjDA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/715/1*4ILbVXJfrh3XB5N6sptAzA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/686/1*9VqwZ9H-mUKQ-QB7NoMvlA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a1ebf6dd192c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Gesture Recognition
System using Deep Learning (video)]]></title>
            <link>https://medium.com/@blogsupport/building-a-gesture-recognition-system-using-deep-learning-video-64e4d67459e8?source=rss-2e2a341c9deb------2</link>
            <guid isPermaLink="false">https://medium.com/p/64e4d67459e8</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[videos]]></category>
            <category><![CDATA[gesture-recognition]]></category>
            <category><![CDATA[recognition]]></category>
            <dc:creator><![CDATA[Great Learning Snippets]]></dc:creator>
            <pubDate>Thu, 20 Aug 2020 12:25:30 GMT</pubDate>
            <atom:updated>2020-08-20T12:25:30.020Z</atom:updated>
            <content:encoded><![CDATA[<p>Authors: Swaraj Kumar, Gautam Mishra and Dr. D Narayana</p><p>Gestures are an important aspect of human interaction, both interpersonally and in the context of man-machine interfaces. A gesture is a form of non verbal communication in which visible bodily actions communicate particular messages, either in place of speech or together and in parallel with words. Gestures include movement of the hands, face, or other parts of the body. Where voice cannot reach gesture can reach and with this we can multiple hazardous situations.</p><h3>1. Introduction</h3><p>Now deep learning is at its peak in different field ex: image classification, audio and other data samples. But still there is not much proof of concept on video data. So we are trying to use gestures to perform some activities without any sensor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/646/1*KKNsqhoWvzc8j7F9Q5tCYg.png" /></figure><p>There are a lot of devices available in the market with gesture but all devices need some kind of sensor/device to operate, this is one of the regions for failure of devices operate on gesture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/330/1*P0YZV9mRnppxMsSbLON2hw.png" /></figure><p>So, we planned to try the approach to take video data samples and pass it through the deep learning model to see the outcome of it. This has a very wide range of use in many fields , such as gaming , medical , factories and many others.</p><h3>2. Data Collection</h3><p>Data is the new fuel to create deep learning model. We need minimum 1000 samples per class to start deep learning model. Initially we started with very less amount of data created with normal camera by 10 members doing the same gesture. Initially we just wanted to verify if we are progressing the right way or we need to take different field as our capstone. We converted those images into frames and used data argumentation to significantly increase the diversity of data available for training models, without actually collecting new data. Then we took the data from 20bn dataset , which is available for research prospective. The 20BN-JESTER dataset is a large collection of densely-labeled video clips that show humans performing pre-defined hand gestures in front of a laptop camera or webcam. The dataset was created by a large number of crowd workers.</p><p><a href="https://20bn.com/datasets/jester/v1#download">https://20bn.com/datasets/jester/v1#download</a><br>The video data is provided as one large TGZ archive, split into parts of 1 GB max. The total download size is 22.8 GB. The archive contains directories numbered from 1 to 148092. Each directory corresponds to one video and contains JPG images with height 100px and variable width.</p><p>The JPG images were extracted from the original videos at 12 frames per seconds. The filenames of the JPGs start at 00001.jpg. The number of JPGs varies as the length of the original videos varies.</p><h3>3. Data Processing</h3><p>The 27 key classes are there . After analyzing the dataset we came to know that each folder has different number of images. So, while creating the model we have defined the frame site, if folder contains more than frame size image we are not using it , if it consist of less images we are generating more images. The key gestures are as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/715/1*9bo_uAE-Gz971VgOhmDclA.png" /></figure><h3>4. Different Deep Learning Techniques Followed</h3><p>We have used all 3 frameworks keras, tensorflow ,and py-torch to check the accuracy .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/663/1*eGN_AB7RBGW58_R9Xo4J1g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/1*Gkw5Yim02bSolzieWD_SxA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/715/1*D3zFMJyHG-L-1Ua-nJauEQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/695/1*rdWq4mVPHWsxZnSWPSgFZQ.png" /></figure><h3>5. Model Architecture</h3><p>We now present the functional and technical architecture we derived for our proposed solution. We start with functional architecture.</p><h4>5.1. Functional Architecture</h4><p>”Gesture Recognition System” as we are referring to our prediction model, will serve as a backend processing unit in predicting the gesture from live camera. End user will do the activity in front of the laptop camera and application will read the gesture and perform the action for it. Detailed functional architecture of the system we are proposing is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/410/1*0heYDhCZO2L0jS3m33E5SQ.png" /></figure><h4>5.2 Technical Architecture</h4><p>We started with the initial process of data gathering where we created few manual videos and later we took the dataset of 20bn (<a href="https://20bn.com/datasets/jester/v1#download">https://20bn.com/datasets/jester/v1#download</a>)<br>As we have We have planned to use CNN + LSTM as all the images taken from videos and each images have time series dependencies. But after going through various paper we came to know , now a days most of the image classifications are solved by 3D CNN which is giving better accuracy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/1*B53VyOeUUK7tnafnGRO27A.png" /></figure><p>3D convolutions applies a 3 dimensional filter to the dataset and the filter moves 3-direction (x, y, z) to calculate the low level feature representations. Their output shape is a 3 dimensional volume space such as cube or cuboid. They are helpful in event detection in videos, 3D medical images etc. They are not limited to 3d space but can also be applied to 2d space inputs such as images. The detailed technical architecture of our flow is presented in the following figure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/1*l_LN0WRpmMp8Siwg6p2h0g.png" /></figure><h3>6. Model Implementation and Challenges</h3><h4>Tensorflow /Keras</h4><p>We have used Tensorflow 1.13 and keras framework and 3D CNN model .<br>We are using Pytorch framework for now due to below reasons-</p><p>1. In Tensorflow, you first have to define the entire computation graph of the<br>model and then run your ML model. But in PyTorch, you can define/manipulate your graph on-the-go. This is particularly helpful while using variable length inputs in RNNs.</p><p>2. We have a plan to use LSTM with 3D CNN model so we are initially using it<br>depends on the accuracy either we might go with only 3D CNN.</p><p>3. We have CSV files with Train and Test labels with folder ID / Gesture ID. We have combined both to produce labeled data using Python and we have<br>transferred all the images into RGB format.</p><p><strong>Voluminous Dataset and it’s side effects</strong><br>Analyzing 32 GB of data with limited resources is not easy but in AI/ML data is the petrol for the engine, so do took the opportunity to analyze and build the model with whole dataset. To analyze and create a model on this big data set we have purchased the resource on <a href="http://www.floydhub.com/">www.floydhub.com</a> which provides pay per usages based deep learning environment.</p><p><strong>Hardware Configuration:-</strong></p><p><strong>Intel Xeon Processor (4 Core)<br>61GB RAM<br>Nvidia Tesla K80 GPU with 12GB memory<br>200GB SSD</strong></p><p><em>note:</em> We have taken batch size as 32 which perfectly fits with this hardware however increasing the number tends to OOM error because Keras and PyTorch loads the whole batch in GPU memory which is 12GB only in our case.</p><h3>7. Outcome prediction and accuracy improvement</h3><p>To get proper accuracy, we have to fine tune the model. There are so many layer combinations we tried to get better accuracy. With tensorflow1.13, we used “ADADELTA” optimizer, as using ADAM and SGD we were getting less accuracy. When we used ADADELTA we got accuracy around 95% in first 20 epochs. For Pytorch we used SGD optimizer and we got a 97% accuracy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/1*2JAz2vfy1NwmbSDLxSqILA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/1*rSOwgIwRa3F6olgzLGfYEQ.png" /></figure><h3>8. The Comparative Results</h3><p>The following screenshot shows models performance in predicting case outcome.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/695/1*IG9f59cDF3ORvnpSDgilpg.png" /></figure><h3>9. Business Value</h3><p>Looking at various artificial intelligence (AI) methods and the value it can generate, we are comfortable to record that it can be used in many fields. On factory area we can monitor mishandling by arranging multiple AI enabled cameras at key areas to cover all points and analyse movements/gestures and warn employees through audio based on AI model (signal to cloud) on every mishandling of equipment. We can automate many activities on gesture, which can avoid hazardous situation. For game lovers this will be a new era, where gamers can feel the game without wearing any sensor. While giving presentation where mouse is not accessible or user can not hold any device to change the slide this model can help them. In robotics this can be used to read the gesture and perform efficient actions.</p><h3>10. Limitations</h3><p>To get the output action of this model we need 12 frames per second images which is not possible with VGA camera. And to predict the action we need to convert live videos to images and predict pass through the model which is taking some time.</p><h3>11. Closing Reflection</h3><p>It is really good learning for our team . Project management is the key technique to deliver the project. Data is the key. Once you have labeled data you can easily create your model. Initially we thought to achieve the project easily but cleaning up the data and making it ready took a lot of time. This time we started building the project by taking the huge data, next time, we will first create the data cleanup technique and build the model with minimal data or sample instead of huge dataset.</p><h3>12. References</h3><p>1. <a href="https://medium.com/twentybn/gesture-recognition-using-end-to-end-learning-from-a-large-video-database-2ecbfb4659ff">https://medium.com/twentybn/gesture-recognition-using-end-to-end-learning-from-a-large-video-database-2ecbfb4659ff</a><br>2. <a href="https://medium.com/twentybn/building-a-gesture-recognition-system-using-deep-learning-video-d24f13053a1">https://medium.com/twentybn/building-a-gesture-recognition-system-using-deep-learning-video-d24f13053a1</a><br>3. <a href="https://medium.com/twentybn/watch-and-learn-building-an-ai-that-understands-the-world-through-video-9e2796400176">https://medium.com/twentybn/watch-and-learn-building-an-ai-that understands-the-world-through-video-9e2796400176</a><br>4. <a href="https://arxiv.org/pdf/1412.0767.pdf">https://arxiv.org/pdf/1412.0767.pdf</a><br>5. <a href="https://papers.nips.cc/paper/7465-attention-in-convolutional-lstm-for-gesture-recognition.pdf">https://papers.nips.cc/paper/7465-attention-in-convolutional-lstm-for-gesture-recognition.pdf</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=64e4d67459e8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CONSUMER CASE OUTCOME PREDICTION IN REAL ESTATE]]></title>
            <link>https://medium.com/@blogsupport/consumer-case-outcome-prediction-in-real-estate-eee1304130e0?source=rss-2e2a341c9deb------2</link>
            <guid isPermaLink="false">https://medium.com/p/eee1304130e0</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[predictive-model]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[ai-in-real-estate]]></category>
            <dc:creator><![CDATA[Great Learning Snippets]]></dc:creator>
            <pubDate>Wed, 19 Aug 2020 08:18:05 GMT</pubDate>
            <atom:updated>2020-08-19T08:20:33.449Z</atom:updated>
            <content:encoded><![CDATA[<p>Contributors: PREM PANCHAMI, MADHAN SEDURAMAN,SAI VENKATESHWARAN SRINIVASAN,SHANKAR RAM MOHAN,NAGENDRA<br>DESAI,DR. D NARAYANA,PADMAJA KUNDLA.</p><h3>SCOPE</h3><p>This project represents the starting point in a long series of solutions to help consumers and the legal community with researching consumer court cases, including predicting the probability of a grievance case winning in the Indian Consumer Disputes Redressal Commission (or ‘Consumer Court’ for brevity) at the National and State levels.<br>With little to no prior art to rely upon, this project was scoped down to fit the timelines of the capstone project. The primary scope was to focus on Real Estate consumer cases given that a good amount of digitized data was available for such cases.<br>The secondary scope was to focus on data extraction (manual or otherwise) as well as a minimum viable model to demonstrate the viability of the aforementioned solutions. Cases other than Real Estate as well as Information Extraction are Out of Scope for this capstone. These will be pursued outside the bounds of the capstone project.</p><h3>OVERVIEW</h3><p>Currently, as per National Consumer Disputes Redressal Commission, India (or ‘NCDRC’for brevity), there are a whopping 4.5 Lakh consumer cases pending in the country. The rateof disposal of said cases, however, is much slower and the legal system is constantly bound to be overwhelmed by the case load. There is a serious need to minimize the number of cases<br>entering the system as one way of reducing the aforesaid case load. [7]<br>Machine Learning, with its ability to learn from historical data, has the real potential for automating or augmenting the lifecycle of a grievance in the legal process. While automating the legal process may only be possible in the very long term (or even never fully), it is definitely viable to leverage ML to augment the process. [6]<br>One way to make this augmentation real is to provide a way for a consumer to quickly and reliably evaluate the likelihood of winning a potential Consumer Case with the ease of pushing a few buttons on a smart device before it enters the legal system and associated processes.<br>In this capstone study, we chose to focus on Real Estate consumer cases. The choice was primarily driven by the availability of data for such cases. Nothing in this study precludes the application of the same or similar approaches to other types of cases.<br>Providing an ability for the consumer to evaluate the prospects of winning a case can potentially save effort and money for the complainant, the lawyers and the court system. In the longer run, there could also be broader changes through behavioral changes in the real-estate developer community due to democratization of the knowledge of real-estate case viability.</p><p>[1] gave us the initial direction that even without in-depth understanding of complex legal basis, it is possible to predict case outcome based on historical verbose arguments. It cracks open the door for machine learning practitioners, with little or no legal expertise, to approach the problem of case outcome prediction with some degree of confidence.<br>[2], [3] took this further in a clear direction of utilizing textual information to predict case outcome. “Previous work on predicting judicial decisions, representing disciplinary backgrounds in political science and economics, has largely focused on the analysis and prediction of judges’ votes given non textual information, such as the nature and the gravity of the crime or the<br>preferred policy position of each judge (Kort, 1957; Nagel, 1963; Keown, 1980; Segal,1984; Popple, 1996; Lauderdale &amp; Clark, 2012). More recent research shows that information from texts authored by amici curiae1 improves models for predicting the votes of the US Supreme Court judges (Sim, Routledge &amp; Smith, 2015). Also, a text mining approach utilizes<br>sources of metadata about judge’s votes to estimate the degree to which those votes are about common issues (Lauderdale &amp; Clark, 2014). Accordingly, this paper presents the first systematic study on predicting the decision outcome of cases tried at a major international court by mining the available textual information.” [2], [4], [5] “In developing the method, the team found that judgements by the ECtHR are highly correlated to non-legal facts rather than directly legal arguments, suggesting that judges of the Court are, in the jargon of legal theory, ‘realists’ rather than ‘formalists’. This supports findings from<br>previous studies of the decision-making processes of other high level courts, including the US Supreme Court.” [3]<br>On the basis of the above research, an initial hypothesis was proposed that the Indian Consumer Court judgements are realist rather than purely formalist and to test this on the most readily available data, i.e. Real Estate cases. This study shows the early results of the test that: The early facts of the case as described by a complainant can indeed have reasonable predictive power for the Indian Consumer Court system similar to what [2] did, albeit for<br>simpler cases than the complex cases of Human Rights that [2] focused on.<br>This case study presents the first attempt (to the best of our knowledge) at predicting the outcome of Consumer Court Cases in the Indian legal system by using complaint text as the single and primary feature. This project work goes on to show early promising results of practically usable accuracy and recall for the predictions.</p><h3>APPROACH</h3><p>In order to prove the hypothesis that the description of the facts of the case had predictive power towards the case outcome, we needed to have access to the facts of the case as well as the verdict from the judge.</p><p>Given that there was no prior art in terms of prepped data and that we had no access to the original case documents, the best viable starting point was the judgement order from a judge when a case was closed. [8] These judgement order files contain, 60–70% of the time, the facts of the case as described by the judge. We chose to use this as an approximation of the original complaint summary.<br>Here was our approach:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/180/1*_HbiOJx1j9-IfzNkMKcCeg.png" /></figure><p>The first step was to gather data. This was done using a<br>custom web scraper to pull down the judgement orders for<br>all available Real Estate cases. These files were found to be in<br>PDF format.<br>Once the PDF case files were downloaded, the textual<br>information was extracted using standard tools available in<br>the market.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/216/1*o870rTVuyGAbikPM7GMG6g.png" /></figure><p>The raw text files contain unstructured data including case<br>header, facts of the case (optional), description of legal basis or precedent (optional) and the judge’s final order.<br>We took 20% of the cases and manually tagged the various<br>parts of each document. The primary focus was on getting the<br>‘clean’ facts of the case without any text that might reveal the<br>verdict of the case. Five classes of tags were chosen: Header<br>Text, Complaint Text, Judgement Order Text, Ignored Text and<br>Verdict.<br>The manual tagging task was a time consuming one ridden with<br>human fatigue-induced errors. Hence a thorough review was<br>done of the manual tags.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/211/1*mBeWtCKFSsPe-TgSbmZNeg.png" /></figure><p>With the corpus of manually tagged files as the input, we set<br>out to build auxiliary models that would classify each line of<br>text from the remaining 80% of the case files into one of the following tags.This was done incrementally alternating between generate<br>tags and review/clean-up tags.<br>1. Ignored Text was tagged using simple regular<br>expression based rules.<br>2. Then an ML classifier for Header Text was built.<br>3. Following that an ML classifier for Complaint Text and<br>Judgement Order Text was built.<br>4. Finally a Verdict tag classifier using the Judgement<br>Order Text as input was built</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/209/1*B86coqnZYnz2MxtaVtGkYQ.png" /></figure><p>The tag proposals from the auxiliary models had to, at each<br>stage, be manually reviewed and signed off for the next stage<br>of tag proposals to be effective.<br>However the time taken to review and correct proposed tags was only a fraction of the time it took for manual tagging from scratch. Also the incidence of human fatigue-induced errors reduced dramatically.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/176/1*E5jnZ2Yy1bsmnB8LFO_jjA.png" /></figure><p>Using the Complaint Text as the feature and the Verdict tag as<br>the target label, various models were built, compared and<br>tweaked to find the best performing model.</p><h3>GATHER DATA</h3><p>As part of the early digitization efforts for the Consumer Courts, the National Informatics Centre, Ministry of Communication &amp; Information Technology (Govt. of India) is developing and maintaining the computerization effort and the computer networking of Consumer Courts at the District, State and National levels. One of the outcomes of this effort is the website <a href="https://confonet.nic.in/">https://confonet.nic.in/</a> which provides various services and data including Case History and Judgement Orders of the past.<br>Please note: The status quo of digitization of the Indian legal system does not afford practical access to the original complaint documents. And even if the physical documents were gathered there remains the immediate challenge of extracting the text from physical documents. That exercise is presently out of the scope of this capstone project.<br>The website <a href="https://confonet.nic.in/">https://confonet.nic.in/</a> under the “Judgement” section allows the visitor to download documents in the PDF format for various sectors of consumer cases. The category of interest for this study is ‘Housing’.</p><p>At the time writing this paper, ‘Housing’ data was available for a select set of States as well as NCDRC starting from 2016.</p><h3>TECHNICAL DETAILS</h3><p>A custom web crawler was written in python using Selenium, Splinter and the Chrome web driver. The pseudo-code is given below for reference.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/1*Nc2fKopZzoBUypoH7IRhzg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*13X9BdOfXmz3_jdORIWIpg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/1*q9A_ZwdKFbHdi-c5j0cmRA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/545/1*8KU1b9o6iMTDwuR0GsVbDA.png" /></figure><h3>DOWNLOAD STATS</h3><p>In all, 5136 PDF files were downloaded using the aforementioned crawler. See below for the distribution of disposed cases across the various States/Commissions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/530/1*AL58-jkyiFt60_tlqyVE6w.png" /></figure><p>CONVERTING PDF TO TEXT<br>In order to extract text from the PDF documents we attempted to use various python libraries (like PyPDF2, pdfminer and Slate). However, there were some fidelity issues or unwanted spurious characters. We finally had good conversions using the PDFMate paid tool.<br>Using the tool, we were able to extract text successfully from 4,343 files.<br>Please note: For the remaining 793 files there were predominately two types of challenges.<br>1. The documents were scanned copies of printed versions of the Judgement Orders.<br>These would require OCR extraction and manual review/correction.<br>2. The documents had some PDF encoding corruption that prevented the tool from extracting them. These would require using python libraries for extraction and again manual review/correction.<br>These two challenges were considered as Out of Scope for this capstone project given time constraints.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/1*-gqyff5LnQFhKYDakNKwow.png" /></figure><h3>MANUAL TAGGING</h3><p>We had to make upfront decisions on two things:<br>1. What tools should we use for manual tagging as well as the format in which to store the data?<br>2. Which files should we pick to start manually tagging?</p><p>CHOOSING A TAGGING TOOL<br>Key considerations, for the first decision, were:<br>• Keep the document in as much of its original form as possible with the tagging information. This would help with reducing rework due to resets in thinking or approaches.<br>• The format has to allow for in-place formatting of different tags for visual perusal.<br>• The storage format has to allow for easy distribution to non-technical folks (like legal professionals) for review.<br>• The storage format, being the single source of truth, has to allow for selective extraction of data as well as custom tag definitions.<br>After a brief survey of the tagging tools landscape, it seemed there was no obvious tool that allowed for redistribution without the need for the tool itself. There were other minor limitations as well. Hence, we made a decision with the short-term in mind. We are confident that with a bit of involved research a viable long-term solution is possible with one of the<br>available tools. We treated that as Out of Scope for the capstone project.</p><p>The format we chose to store the document in was HTML. This was primarily because there is a lot of prior art in the form of libraries, tools and browsers that makes HTML a stable choice across platforms. HTML allows for tags and for custom formatting. Any text editor would allow for minor edits thus removing the need for custom tools to be installed.<br>The tool chosen was Visual Studio Code editor. By making use of custom Code Snippets and keyboard shortcuts, the VSCode editor formed a reasonable tagging tool for selecting text and applying HTML tags. The editor has HTML code highlighting which allows for quick visual cues when editing. It also allows for browsing folders, opening multiple files and advanced HTML<br>editing features. Here is a Code Snippet sample for surrounding a selected text with the Complaint Text tag:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/467/1*lyQlQmJlcGLLXfCdHvIh1Q.png" /></figure><h3>CHOOSING FILES FOR MANUAL TAGGING</h3><p>Manually tagging 5000+ files was going to be a huge and unreliable undertaking. Right from the start we realized the need for utilizing the power of ML in tagging these files. This needed some portion of the files to be manually tagged as training data.<br>Based on initial discussions within the team and with the GL mentor, it was decided we needed about 500–600 usable cases. We estimated 1000 files would yield that number. The files were chosen from the top 4 states/commissions — Chandigarh, NCDRC, Delhi and West Bengal. 250chronological cases from each state/commission was chosen.<br>This exercise yielded 532 usable files. The rest either were unverified or did not contain complaint text.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/585/1*YmKLQp6dhkgRxXbhROo8hg.png" /></figure><p>It is important to note a lesson we learnt here. If we were to do this again, we would randomly sample across the states/commissions in proportion to the number of cases in each (Stratified Shuffle Sampling). This would yield a better training set for the auxiliary tag proposal models.</p><h3>AUTO-PROPOSE TAGS</h3><p>At this point we had 532 cases that were tagged, reviewed and usable. We had two choices in front of us:<br>1. Focus only what is needed for the capstone project i.e. Getting the Complaint Text and Verdict tagged.<br>2. For a small incremental cost, get all the tagging done.This would help with the post-capstone work of Feature/Information Extraction. We chose this option.</p><h4>TAGGING ‘IGNORED TEXT’</h4><p>The ‘Ignored Text’ tag was generally used for page numbers or signatory initials. A quick review of the ‘Ignored Text’ tags from the manually tagged corpus yielded a set of regular expressions that could be used to tag the untagged corpus. Here is the list of regular expressions to be applied in order (specialized forms before generalized forms of text). Applying these took care of almost ALL of the ‘Ignored Text’ tags.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/1*Q5SwpPahNFi-t8kcdm8qpQ.png" /></figure><h4>‘HEADER TEXT’ TAG PROPOSAL</h4><p>A typical case file header contains meta information about the complainants, opposite parties,the presiding judge, case number etc. This format varies across state/commissions, number of complainants or opposite parties, number of complaints in one case etc. These are not readily solvable using regular expressions. Rather than build a set of manual heuristics, we chose to<br>train an ML model to classify a line of text as Header or Not Header.<br>A sample of a typical header is below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/397/1*4beD4jZR-WjXvxwL223JDg.png" /></figure><p>From the manually tagged corpus, we created a dataset with each line from the case files as rows of Features and whether it was tagged Header or not, as the target label.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/606/1*Gsza-XrwCXamJrJ0n7Egug.png" /></figure><p>The ratio of Header vs. Not Header was reasonably even, that we did not attempt to down-sample or up-sample. We used all of the data.<br>As can be seen from the sample header above there are a lot of punctuation marks which are characteristic of case headers. In order to leverage the information contained in the punctuations and number formats, we decided to ‘wordify’ them.</p><p>See below for the mapping between punctuations and keywords we introduced.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/530/1*8laGNyws9kkcrkm7OLZ78w.png" /></figure><p>We built models with and without the ‘wordification’. It is clear from the numbers below that the ‘wordified’ set had a 2–3% increase in overall accuracy as well as a 4–6% increase in the Recall for ‘Not Header’ class. We decided to use the ‘wordified’ version of the model. Turns out this model generalized better when it came across files from Telanagana, which was not<br>part of the train/test set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/1*K1x_cNObWX21WjaQEaLPiA.png" /></figure><h4>‘COMPLAINT TEXT’ TAG PROPOSAL</h4><p>Not all case documents contain facts of the case. However, many of them do. In a case document that contains facts of the case, we observed the following:<br>• Complaint text always appears after the Header Text (This is obvious!)<br>• The facts of the case tend to appear earlier in the order portion of the document<br>• The facts of the case ALL appear together in most cases.<br>• If there are multiple cases combined in a single order document, there MAY be separate facts described for each case.<br>• Once the clump of lines that represent the facts of the case have been identified, the rest of the order text (barring Header Text and Ignored Text) can all be confidently marked as Judgement Order Text.<br>Here we faced a challenge: What should we use for training and prediction — Lines or Sentences or the entire blob of Complaint Text from each manually tagged file?<br>• Lines — If we were to do the training and prediction line by line, we would be largely affected by inconsistencies in the way the original PDF documents were formatted or in how the text was extracted. Ex: It was not uncommon to find several lines in the order text with only 1–3 words and often with part words.<br>• Sentence — After brief research on model or tools for predicting sentence boundaries, we were unable to find one that did a reasonable job of extracting sentences.<br>• Blob — With only 532 blobs, we were fairly certain that any model trained on the blob would not generalize well.</p><p>Theoretically it should be possible to develop a custom model + rules-based solution to identify sentence boundaries in the order text. Pressed for time, we chose to keep this Out of Scope for the capstone.<br>We also considered several hybrid methods of using models trained on line information along with models built with the blob information in combination with a custom algorithm to evaluate multiple combinations of sub-texts and picking the best blob as the most likely candidate. Once again, pressed for time, we chose to keep this Out of Scope for the capstone.<br>Instead we came up with an effective workaround that relied upon the fact that a manual review of the tags was anyway needed.<br>The idea was to break up the entire order text into roughly similar lengths. We chose 150–180 characters for any line. Using the TextWrap python library this was easily done on the training/testing dataset. On the production side, we would do the same and classify each line as Complaint Text or Judgement Order Text. This has the obvious issue that some of the lines may contain both but would be classified as one or the other. The idea was:During the manual review, we would look for the likely clumps of Complaint Text and tag them as such. The rest of the order text can then be tagged as Judgement Order Text en masse. From the manually tagged corpus, we created a dataset with the newly split lines as rows of Features and whether it was tagged Complaint Text or Judgement Order Text, as the target label.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/1*AFaC8T_tSvHbPSRY4BEseg.png" /></figure><p>Here we made a rookie mistake of not considering the glaring imbalance in the classes as well as across the states/commission. We proceeded without up/down sampling. This was a BIG lesson learnt on what not to miss!<br>We were attempting to pick the model that gave us the best Recall for class ‘0’ (Complaint Text). So various basic MLs models were built, but not too surprisingly the Recall ‘0’ was quite low.</p><p>As an attempt to converge on models with the best Recall ‘0’, we wrote a custom scorer for use in cross-validation and grid search.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/514/1*bpNDy0YLemO7-wYSX6pcnA.png" /></figure><p>We found that while most models were giving us an overall accuracy of ~85%, we knew that to be mainly due to most of the data being Judgement Order Text. The model that had the best Recall ‘0’ of 92% was KNN, but it had a very poor Recall ‘1’ of 65%.<br>In order to combine the best of each model, we decide to stack all the ML models. We did this by forming the base layer (L1) of models with vanilla ML models that take the TfIdf vectors as input. The L1 output probabilities were then taken as the input for the next layer (L2) model to predict the final outcome.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/689/1*2-a7_ArzevNP0UtjbxkQdQ.png" /></figure><p>The final model was picked for the best Recall ‘0’ with a balanced Recall ‘1’ which was the Stacked L2 KNN model (Recall ‘0’ of 88% and Recall ‘1’ of 85%).</p><p>In production, against the untagged text, the model was found to do a reasonable job of tagging the major part of the Complaint Text as a cluster or sequence of class ‘0’ tags. There were quite a number of false positives distributed across the rest of each file. However, the clump of class ‘0’ tags was enough to provide a visual cue of where the Complaint Text was likely to be. So, fortunately despite making the mistake of not up/down sampling, we ended up with a reasonably practical solution.<br>Post capstone, there is serious work to be done with the larger dataset in building a reliable set of models that can not only tag at the line level, but also classify sentence boundaries as well as blob boundaries for Complaint Text transitioning to Judgement Order Text and vice versa.</p><h4>‘VERDICT’ TAG PROPOSAL</h4><p>Once the text tagging had been done, we needed to generate the target labels — The final verdict for each case. The primary hypothesis of this project is to test whether verdict can be predicted on the basis of Complaint Text. However, while we are in the process of tagging the data, we are free to use the Judgement Order Text to ‘cheat’ and generate Verdict tag proposals for review and correction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/1*hDc02zEnvkZsNwW4aldoPA.png" /></figure><p>As can be seen from the above graph the distribution of verdicts was imbalanced on both fronts: Across states/commissions as well as across Verdict labels. Our original mistake in picking the first set of files for manual tagging without any stratified random sampling had another unintended impact.<br>The text describing the final verdict can be complicated. While there is some consistency in the verdict descriptions, they can vary due to many reasons that makes it challenging to predict without enough balanced data as well as extracted information/features. Each judge has a different way of describing the verdict (and hence a wider and well represented sample is<br>needed). The verdict description may span multiple cases and some part of the text would not be applicable to the current case (and hence information extraction / filtering is needed).<br>Given this, it was clear upfront that any Verdict tag proposal model that we build from this imbalanced data would be very myopic and would not generalize.<br>We chose to train a model anyway to seed the manual review and clean-up process. Sure enough, in training we got Recalls of greater than 95% for ‘Won’ and ‘Dismissed’ cases. However, about 25% of the labels had to be manually corrected.</p><h4>REVIEW / CLEAN-UP TAGS</h4><p>Using VSCode and HTML during the first round of manual tagging exposed some disadvantages. The predominant disadvantage was in tag corrections — It was cumbersome and non-intuitive.</p><p>Given that the current part of the approach was primarily about review and correction, the choice of tagging tool had to be revisited. We needed to retain the advantages of tag formatting for visual cues, the ability to see the entire file in the same sequence as the original file and the ability to look at multiple files contiguously without having to open each one separately. And we needed to come up with a mechanism to make tag corrections low-friction and intuitive.<br>With a bit of VBA magic for Conditional Formatting, Microsoft Excel turned out to be good choice. We decided to store around 12K-13K lines of text in one XLS document, which turned out to be batches of 35–55 case files in each XLS file. Automation scripts did the job of:<br>• Extracting input data sets for the Tag Proposal model described in the previous section<br>• Taking new outputs from each incremental tag proposal step and combining them with<br>the data from the previous review/clean-up step<br>• Automatically formatting the resulting Excel sheet using Conditional Formatting<br>• Automatically hiding/unhiding columns in the resulting Excel sheet for the next<br>review/clean-up step<br>Here is a sample of VBA code for conditional formatting in Excel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/1*7v5IGmkv0xsSsmrN-_zGlA.png" /></figure><p>During the process of manually reviewing, the colors provide a visual cue for the reviewer. The fact that the lines appear in the same order as the original case file allows the reviewer to account for the general structure simply by glancing at the colors. Anything that appears to be incorrect can be reviewed in place and any correction in tags will immediately update the colors of the cells, thus providing the reviewer immediate visual feedback of the action.<br>Being aware of the observations about Complaint Text structure and locations in a case file allows the reviewer to quickly look for the clump of ‘red’ that contains the Complaint Text,make the necessary corrections and mark the rest of the order text as Judgement Order Text by dragging the Excel cell value to all required cells.<br>Despite the generalized model for Header Text, there were some patterns that were unknown to the Header Text tag proposal model. For example: Telangana files has a list of Exhibits used in the legal proceedings which is not recorded in any of the other state/commission case documents. And there were no Telanagana files in the training set.<br>These were manually identified from the review of the very first Excel sheet. These were then easily resolved with a stop-gap measure of tagging using regular expressions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/566/1*xEPt8GOVfQLKljjzJWJdmg.png" /></figure><p>During the first round of manual tagging, each case file took anywhere from 15–20 minutes, of human time, to finalize tagging. The review and clean-up of these manually tagged files could take another 5–15 minutes of human time. With the tag proposal models, auto-formatted Excel files and the stop-gap measures, the per file time was reduced to 75–100 seconds of human<br>time.</p><h3>BUILD MODELS</h3><p>Nearly 90% of the time spent on the capstone project went towards gathering data, prepping, tagging and reviewing tags.At this point, the model building and hypothesis testing can start.<br>Even though we started the entire process with 5136 files, by the time we reached this stage we only had 2077 usable files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/402/1*s5gi0jvA-m4bjr0__MD6Zw.png" /></figure><p>As seen in the table above, we considered verdicts of [ Win | Partial Win | Dismissed ] only. The other verdicts like Withdrawn, Settled etc. are very likely to depend on the complainant circumstances. And hence should be considered as Out of Scope for this capstone project. We went one step further and treated Partial Win as a Win. This is because, for the purposes of<br>testing the primary hypothesis, the model needs to only classify a given Complaint Text as a Win or Dismissible case. In order to figure out if the case will be a Partial Win would require more features to be extracted than just the Complaint Text.<br>As can be seen in the data distribution below, Dismissed cases only form about 20% of the dataset. We will need to use one or more of up/down sampling, adjusting class weights / probability thresholds and using a custom ‘cost’ function to prefer Recall of the minority class.<br>In the long run, there is also an action item to gather the complaint text from the legal system for such files that do not have any complaint text but were Dismissed. These would need to be OCR extracted and cleaned. As such these are presently Out of Scope.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*rqMMZWqkn_xGKIS2wPRWOA.png" /></figure><p>A typical Complaint Text is shown below. It is clear from observation that, unlike Header Text, punctuation does not have any learnable information. However, there are certain specific amounts of money or interest rates etc. These values will change from case to case, but they do have a bearing on the outcome in the form of pecuniary jurisdiction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/565/1*siFZAti3HPEsAc_a8n9kFA.png" /></figure><p>We had scoped this project to focus only on Bag of Words models. Information extraction is Out of Scope presently. In order for the model to leverage some of the monetary information, we decided to ‘wordify’ them in a way that can be offered to the models for learning.<br>We also added the State/Commission name to the word corpus to be used as a feature if useful.<br>The text blobs were prepped so that all English stopwords from nltk.corpus are removed. The nltk.stem WordNetLemmatizer was used to lemmatize words to reduce feature size. Also any single character and 2-character words were removed from the corpus.The resulting blob would look like the sample shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/565/1*DQh2PnUIX2Yl-4um92RGFA.png" /></figure><p>In order to vectorize these text blob for machine learning, we used TfidfVectorizer from sklearn. Here are the settings used for vectorizing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/565/1*lbULdvlx7w5dUUam6bquIw.png" /></figure><p>We considered 3 approaches:<br>1. <strong>Down-sampling</strong> — This would ensure that there is balance in the number of data-points per class. However, the size of the training corpus would reduce as well. Given the variance in the legal writing, it would be challenging to effectively pick a good representative sample in a single down-sampled dataset. Hence, we decided we will build a stacked model with 2 layers.<br><strong> a)</strong> First, from the training data, we will generate ’n’ number of down-sampled<br>datasets such that every ‘majority class’ (Won) case appears at least once in the ’n’ datasets. Then train ’n’ different models using grid search and cross-<br>validation. The best model for each of the ’n’ datasets will be chosen as the first layer. We used ’n’ = 10.<br> <strong>b)</strong>The probability output from the Layer 1 models will then be used as input to train a Layer 2 model, once again using grid search and cross-validation. The entire training set will be used to train the Layer 2 model.</p><p><strong>c)</strong>Use the best Layer 2 model either using custom scoring ‘cost’ function for Recall ‘0’ or using F1 Score.<br>2. <strong>Up-sampling</strong> — Using random replacements from the ‘minority class’ (Dismissed) increase the number of Dismissed data-points to match the ‘majority class’ (Won). Using grid search and cross-validation train and test the best model.<br>3. <strong>AS-IS</strong> — Using the dataset as-is, train and test a model using grid search and cross-validation. We will use this as a reference point to compare the other models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/604/1*U3NLTDxwwwS5TDb3SQPA-w.png" /></figure><p>The final model was picked for the best Recall ‘0’ with a balanced Recall ‘1’ which was the Upsampled Random Forest with class weight ratio of 2:1 (Recall ‘0’ of 82% and Recall ‘1’ of 90% with an overall Accuracy of 88%).<br>The next step post-capstone is to process the 1309 unverified files and use that as production data to evaluate how well the selected model performs.</p><h3>IMPLICATIONS AND CLOSING REFLECTIONS</h3><p>All the models built so far in this capstone project are naïve — i.e. Bag of Words which only considers if a known word occurs in a document or not. Semantics, context and order of words have not be utilized.<br>The implication here is that the Bag of Words based models will NOT find an immediate use in real world scenarios because the text producers will be anyone from the population as opposed to what these models have been built on i.e. Text from a restricted set of judges and para-legal clerks. Information Extraction is the key here. This entire exercise however was arobust introduction to this problem space and allowed for non-legal practitioners, like us, to explore and form a sense of features and information that are most correlated with case outcomes<br>Complaint text used in this project being a synthesized summary and does not contain the volume of text, completeness of information nor richness of the original Complaint. And hence the models are likely to not do well with original Complaint texts. An intermediate summarization model could be experimented with.</p><p>The models described in this report, in themselves, are a stepping stone. Despite being naïve models, these results indicate that it is possible to predict case outcomes using facts. (i.e. Consumer Cases are realist rather than formalist).<br>In the long term, these can take the shape of various features that could be asked as questions to the consumer along with a qualitative description of the complaint (text, voice or video). It is also possible to envision in the shorter term, a para-legal filling in the complaint in legal terms in order to check the probability of the case winning in Consumer Court.<br>Extending on these advanced features, both explicit as well as derived, it is also possible to envision context sensitive searches like</p><ul><li>Finding top-5 cases that have very similar facts as my case (Feature and Document<br>Similarity)<br>• Research cases against a real estate developer before investing in a property (Vanilla<br>queries)<br>• Finding key changes to make to my complaint so the chances of winning goes up<br>(Derived Feature Importance combined with rules/heuristics)<br>• Predicting how long a case would last in the court system (Regression)<br>• Etc.</li></ul><p>There is a strong case to be made for a technology solution that puts the power of these data and models directly in the hands of consumers.<br>For the team, the next steps in the short-term are clear:</p><ul><li>Gather more data — Finish processing the unverified files<br>• Gather more data for imbalanced classes<br>• Define and extract key features and information from the existing corpus (like Value of Property in dispute)</li></ul><h3>REFERENCES</h3><p>[1] Article — <a href="https://fivethirtyeight.com/features/why-the-best-supreme-court-predictor-in-">https://fivethirtyeight.com/features/why-the-best-supreme-court-predictor-in-</a>the-world-is-some-random-guy-in-queens/</p><p>[2] Predicting judicial decisions of the European Court of Human Rights: a Natural Language Processing perspective — Nikolaos Aletras, Dimitrios Tsarapatsanis, Daniel Preoţiuc-Pietro,Vasileios Lampos — <a href="https://peerj.com/articles/cs-93/">https://peerj.com/articles/cs-93/</a></p><p>[3] Article — <a href="https://www.ucl.ac.uk/news/2016/oct/ai-predicts-outcomes-human-rights-trials">https://www.ucl.ac.uk/news/2016/oct/ai-predicts-outcomes-human-rights-trials</a><br>[4] The Utility of Text: The Case of Amicus Briefs and the Supreme Court — Yanchuan Sim, Bryan R. Routledge, Noah A. Smith — <a href="http://www.aaai.org/ocs/index.php/AAAI/AAAI15/paper/download/9361/9558">http://www.aaai.org/ocs/index.php/AAAI/AAAI15/paper/download/9361/9558</a><br>[5] Scaling Politically Meaningful Dimensions Using Texts and Votes — Benjamin E. Lauderdale, Tom S. Clark — <a href="https://onlinelibrary.wiley.com/doi/abs/10.1111/ajps.12085">https://onlinelibrary.wiley.com/doi/abs/10.1111/ajps.12085</a><br>[6] Article — <a href="https://jolt.law.harvard.edu/digest/a-primer-on-using-artificial-intelligence-in-">https://jolt.law.harvard.edu/digest/a-primer-on-using-artificial-intelligence-in-</a> the-legal-profession<br>[7] Article — <a href="http://www.newindianexpress.com/nation/2017/aug/08/over-45-lakh-cases-">http://www.newindianexpress.com/nation/2017/aug/08/over-45-lakh-cases-</a>pending-in-consumer-courts-of-the-country-1639896.html<br>[8] Data Source — <a href="https://confonet.nic.in/">https://confonet.nic.in/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eee1304130e0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Generating automated medical report on radiology images using deep learning approach]]></title>
            <link>https://medium.com/@blogsupport/generating-automated-medical-report-on-radiology-images-using-deep-learning-approach-1e8a1305061d?source=rss-2e2a341c9deb------2</link>
            <guid isPermaLink="false">https://medium.com/p/1e8a1305061d</guid>
            <category><![CDATA[radiology]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[automated-medical-repport]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <dc:creator><![CDATA[Great Learning Snippets]]></dc:creator>
            <pubDate>Fri, 31 Jul 2020 06:50:10 GMT</pubDate>
            <atom:updated>2020-07-31T06:50:10.787Z</atom:updated>
            <content:encoded><![CDATA[<p>Contributed by: Moumita C, Suraj S, Gaurav S, Sivanandhini, Praveen R, Narayana D</p><h3>Abstract</h3><p>In recent years, deep learning algorithms has made tremendous breakthrough at image recognition and is proving to be a boon in medical domain. With a dramatic increase in use of electronic medical reports and diagnostic imaging it has been a lucrative opportunity to implement the success of machine learning algorithms to generate a cost and time saving diagnostic process.<br>The reports discusses the amalgamation of various machine-learning algorithms and harness the huge medical data to develop an automated or computer -aided reporting system for generating inferences from X-Ray images and helping for the diagnosis of any patient who is suspected to be suffering from lung malfunction. This in a way is also intended to reduce the error occurrences when the report analysis is done manually by a radiologist. The discrepancies can come from faulty reasoning, lack of knowledge,<br>staff shortage or excess workload. An automated or computer-aided reporting system can be helpful in generating inferences from X-Ray images for the diagnosis of any patient who is suspected to be suffering from lung malfunction.<br>In this project, we focused on developing a model which generates medical findings from the chest X-ray images automatically. The proposed model incorporates the Convolutional Neural Networks (CNNs) with the Long Short-Term Memory (LSTM) in a recurrent way. The model uses a CNN to extract the informative features from the X-ray images followed by RNN method (LSTM) to generate sequence of words as description which prepares the report. The model has been evaluated using multiple metrics.</p><h3>Problem statement</h3><p>With the increase in aging population, increased pollution, the need for cost-effective medical attention is becoming a big concern in today’s world. Only about 10% of 7 billion populations in the world have access to good healthcare service, and half of the world don’t even access to essential health services. Even among the developed countries, healthcare system is under strain, with rising cost and long wait time.</p><p>Medical imaging is gaining a big ground and becoming central to high quality health care for accurate diagnosis, improve patient outcome and provide a cost effective and low risk approach to the entire diagnosis process. While we see this big leap in technology, the supply of radiologists is not keeping pace with increased demand. Accurate reading of medical images is hence a pending concern as often radiologists have to review and generate hundreds of reports every day. In-accurate readings are the first step of getting into a wrong diagnosis and a wrong treatment for the patients.</p><h3>Proposed solution:</h3><p>To tacking the problem, we explored the feasibility of utilizing deep learning algorithms to automate report generation from radiology X-ray images. A method that combines both Deep Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNN) has shown significant improvements in Image captioning problems.<br>Therefore, we propose a model that uses a CNN to extract the informative features from the X-ray images followed by LSTM to generate sequence of words as description to prepare the report.<br>Our captioning model relies on two main components, a CNN and an RNN. Captioning is all about merging the two to combine their most powerful attributes i.e.<br>1. CNNs excel at preserving spatial information and images, and<br>2. RNNs work well with any kind of sequential data, such as generating a sequence of words. So by merging the two, our model n find patterns and images, and then use that information to help generate a description of those images.</p><h3>Literature survey:</h3><p><strong>AI on Radiology Data:</strong> In recent years, several chest radiography datasets, totaling almost a million X-ray images, have been made publicly available. A summary of these datasets is available in Table 1.Learning effective computational models through leveraging the information in medical images and free-text reports is an emerging field. Such a combination of image and textual data help further improve the model performance in both image annotation and automatic report generation (Litjens et al., 2017).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*3VYHIg1QrKBft2OlG5TwaQ.png" /></figure><p><strong>Image captioning with deep learning:</strong> Image captioning is the process of generating the description of images. It needs to identify the attribute and relationship of the image and generate syntactically correct sentence. Generating well-formed sentences requires both syntactic and semantic understanding of the language. Deep learning algorithms have been capable of handling the complexities of image captioning. Most recent image captioning models are based on a CNN-RNN framework (Vinyals et al., 2015; Fang et al., 2015; Karpathy and Fei-Fei, 2015; Xu et al., 2015; You et al.,2016; Krause et al., 2017).<br>Recently, attention mechanisms have been shown to be useful for image captioning (Xu et al., 2015; You et al., 2016). . You et al. (2016) propose a semantic attention mechanism over tags of given images.To better leverage both visual features and semantic tags, a recent study (Baoyu et al., 2018) proposed a co-attention mechanism for report generation. Instead of only generating one-sentence caption for images, Krause et al. (2017) and Liang et al. (2017) generate paragraph captions using a hierarchical LSTM.</p><p>The problem of learning from images and is solved by training neural networks, specifically Convolutional Neural Networks (CNNs). CNNs are widely used for feature learning and are followed by RNN to generate the caption.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*B_QXg12TSi5LeazZWD6y7w.png" /></figure><p>There are several popular CNN architectures which are available for solving the various business problems of image classification and image captioning.</p><h3>Methods</h3><p>In this work we opt to focus on generating the medical findings section as it is the most direct annotation from the radiological images. We adopt strategy with a CNN-RNN-RNN architecture to generate words in the findings sequentially and compared against the true reports.<br>Model has the following architecture:<br>a) CNN by using the InceptionV3 model — to generate a feature vector of length 2048.<br>b) Every word (or index) is mapped (embedded) to a 200-long vector using a pre-trained GLOVE word embedding model.<br>c) Two dropout layers to reduce overfitting.<br>d) Three LSTM layer each with 256 cells.<br>e) A dense layer.<br>f) An output layer (softmax) which generates the probability distribution across of all words in the vocabulary.</p><p>The model has been further enhanced and evaluated for accuracy by:<br>a) using subset of the data set and also the whole data set,<br>b) tuning the model hyper-parameters to suit our use case</p><ul><li>Learning Rate</li><li>Optimization algorithm</li><li>Number of epochs</li><li>Pre trained model weights</li></ul><p>c) Using Beam Search instead of Greedy Search during Inference</p><h3>Data</h3><p>The Indiana University Chest X-Ray Collection (IU X-Ray) (Demner-Fushman et al., 2015) is used for training the model. IU X-Ray collection is a set of chest x-ray images paired with their corresponding diagnostic reports. The dataset contains 7,470 pairs of images and reports. Each report consists of the<br>following sections: impression and findings. In this study, we used only the frontal images and the contents in Findings section as the target captions to be generated.<br><a href="https://openi.nlm.nih.gov/faq">https://openi.nlm.nih.gov/faq</a><br> Total frontal images with findings in report: 3,258<br> No. of images in training set: 2,638<br> No. of images in validation set: 294<br> No. of images in test set: 326<br>X-ray images: The X-ray images were available in the PNG format. The original size was 512x624 pixels</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/1*Ul9onZ8JjbDPwyEVArIx0A.png" /></figure><p>The descriptions across 3,258 images have been analyzed using a custom python script. The following provides the summary of the description.</p><p>SUMMARY (PER REPORT):<br> Avg. char count: 217.8<br> Avg. word count: 31.2<br> Avg. word length: 7.0<br> Avg. sentence count: 5.6<br> Avg. sentence length: 5.4</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/490/1*rBkEgiy-xFT7s3QwqjRnFQ.png" /></figure><h3>Data Preprocessing</h3><ul><li>Raw data was in XML format which was transformed into tabular format using ‘xml.etree’ library.</li><li>Segregation of data: The Indiana university data set was a combination of frontal and lateral view and a collection of 7,470 images. We used a CNN + K-means approach to segregate the frontal images from the lateral images. This resulted in final data set of 3,258 images.</li></ul><p>All tokens were converted to lowercases, non-alpha tokens were removed resulting in 572 unique tags and 1915 unique words. On average, each image is associated with 2.2 tags, 5.7 sentences per report, and each sentence contains 6.5 words. Our finding was that top 1,000words covered 99.0%- word occurrences in the dataset, therefore we only included top 1,000 words in the dictionary.</p><h3>Data Split</h3><p>The entire bucket of 3258 images was spilt into training set, validation set and test set.<br>a. Training set — the images which were used to develop and train the model, had a total count of 2638.<br>b. Validation set — the images which were used to tune the parameters were 294.<br>c. Test set — the images which we used to test the performance of our model were 326 in count.</p><h3>Building the Model</h3><p>Here came the concept of usage of transfer learning in our model building. Transfer learning is an approach in deep learning where pre-trained models are used as the starting point on computer vision and natural language processing tasks. This helps to leverage on the research done across years and in terms of the vast compute and time resources required to develop neural network models on these problems.<br>We reviewed some of the leading architectures right from a simple single brand CNN to more complex branched out networks such as ResNet , InceptionNet and ChexNet to decide on which CNN to use to solve our business problem. Several tests on the performance of different CNN architectures have been conducted which showed us that Inception Net architectures work best in terms of training time, accuracy and data required.</p><h3>Results</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/1*14rVD-PqCDkOSOmiis96GQ.png" /></figure><p>Inception Net has good Python support and documentation for implementing on customized datasets. And also the accuracy achieved on the sample data of 500 Datasets was considerably good.<br>Hence, we made Inception Net our CNN architecture of choice to tackle our business problem.<br>Inception net was trained on ImageNet dataset to perform image classification on 1000 different classes of images. However, our business problem is not to classify the images but to get fixed-length informative vector for each image. Hence, we tweaked the architecture as per the need and the last SoftMax layer was removed from the model to extract a fixed length vector for every image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ORK_O78BwJr3S0YAKyEfrQ.png" /></figure><p>We used the function preprocess_input to adequate our training / validation and test images to the format the inception V3 model required (299X299). The original size was 512x624 pixels .<br>We have used the process of “Pickling” to save the image names and their corresponding 2048 length feature vector in the disk. Pickle “serializes” the object first before writing it to file. This is a convert a python object into a character stream.<br>Considering the size of data matrix, we have used Data generator — ‘fit_generator’ method, to mitigate the issue of memory usage due to huge data set. Python provides generator functions to help build iterators.<br>Each word/index is mapped to a 200-long vector using a pre-trained GLOVE word embedding model.<br>After this we are merging the two streams of text (Partial captions) and image vector. Functional API is used to create the merge model. We have imported Models from keras.models to achieve this merger</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/1*eYmu9G5N9JiplEeNlDxOAQ.png" /></figure><h3>Evaluation Metrics</h3><p>Quantitative comparison is essential to evaluate the quality of our generated descriptions against the ground truth descriptions. This we have tried to achieve by using the machine translation and summarization metrics such as BLEU, METEOR, ROUGE and CIDEr. The closer a machine translation is to a professional human translation, the better it is.<br> BLEU depends on N-gram precision. It measures word n-gram overlap between the generated and the ground truth caption. The key to BLEU’s success is that all systems are treated similarly and multiple human translators with different styles are used, so this effect cancels out in comparisons between systems. The BLEU metric ranges from 0 to 1. It is important to note that the more reference translations per sentence there are, the higher the score is. BLEU’s strength is that it correlates highly with human judgments by averaging out individual sentence judgment errors over a test corpus rather than attempting to divine the exact human judgment for every sentence.<br> ROUGE, is Recall-Oriented Understudy for Gisting Evaluation. It includes several automatic evaluation methods that measure the similarity between summaries. This is the ratio of the length of the longest common subsequence between the machine-generated description and the reference human description.For this we used ROUGE1 and ROUGE2 score similar to N-gram evaluation method of BLEU.<br> Evaluation Generator , Evaluates the model on a data generator. This is basically inbuilt evaluation method that we used to evaluate our model.</p><h3>References:</h3><p>1. Yuan Xue1, Tao Xu2, et. al .Multimodal Recurrent Model with Attention for Automated Radiology Report Generation. pp. 457–466, 2018<br>2. Baoyu Jingy et al. On the Automatic Generatin of Medical Imaging Reports. pages 2577–2586 2018.<br>3. Yee Liang Thian et. al Convolutional Neural Networks for Automated Fracture Detection and Localization on Wrist Radiographs. Radiology: Artificial Intelligence 2019</p><p>4. <a href="https://towardsdatascience.com/image-captioning-with-keras-teaching-computers-to-">https://towardsdatascience.com/image-captioning-with-keras-teaching-computers-to-</a>describepictures-c88a46a311b8</p><p>5. [BLEU] Papineni, Kishore, et al. “BLEU: a method for automatic evaluation of machine translation.”Proceedings of the 40th annual meeting on association for computational linguistics. Association for Computational Linguistics, 2002.<br>6. [ROUGE] Lin, Chin-Yew. “Rouge: A package for automatic evaluation of summaries.” Text summarization branches out: Proceedings of the ACL-04 workshop. Vol. 8. 2004<br>7. <a href="https://www.researchgate.net/publication/322142760_Deep_Learning_Applications_in_Medical_Im">https://www.researchgate.net/publication/322142760_Deep_Learning_Applications_in_Medical_Im</a><br>age_Analysis</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1e8a1305061d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ELO MERCHANT LOYALTY RECOMMENDATION]]></title>
            <link>https://medium.com/@blogsupport/elo-merchant-loyalty-recommendation-b28096c882b7?source=rss-2e2a341c9deb------2</link>
            <guid isPermaLink="false">https://medium.com/p/b28096c882b7</guid>
            <category><![CDATA[elo]]></category>
            <category><![CDATA[technical]]></category>
            <category><![CDATA[merchant-loyalty]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Great Learning Snippets]]></dc:creator>
            <pubDate>Thu, 30 Jul 2020 08:56:36 GMT</pubDate>
            <atom:updated>2020-07-30T08:56:36.285Z</atom:updated>
            <content:encoded><![CDATA[<p>USING MACHINE LEARNING TECHNIQUES</p><p><strong>Pushparaj BS, Praveen Perumal, A M Sharan Kumar, Sindhu Pushparaj, Dr. D Narayana</strong></p><h3><strong>1. Introduction</strong></h3><p><em>1.1 ELO customer loyalty program</em></p><p>This problem revolves around Customer loyalty of credit cards for consumers in Elo, the largest payment brands in Brazil. Loyalty programs are rewards programs put in place by the owners of an establishment to encourage customers to return.</p><p>These programs are extremely popular in businesses where customers make frequent purchases. Every rewards program is unique to the company who implements it, but they operate on roughly the same premise (The more a customer shops at your establishment, the more rewards they receive, and the more incentive they must come back and continue shopping at your place of business.) Implementing a solid loyalty program is not only an important way for you as a business owner to make sure that your customers keep returning, it is also an important metric for your churn rate. It is important to do your research before adopting any loyalty solutions for your company, but this day in age, adopting some sort of loyalty program is becoming more and more advantageous.</p><h3>2. Problem Statement</h3><p>Elo have built tie-ups with respective merchants to offer their customers with discounts. Those can be any of the following :</p><p>* Cash Back Or Rebate Program : Customers earn money back from their purchases</p><p>* Discount Program: Offers a discount off the original price of any given item</p><p>* Frequency Programs:(Buy 5 subs and the 6th is on us!)</p><p>* Points Programs: Customers are given points, often in exchange for the amount they spend in your store.</p><p>* Tier System: Customers must make a simple purchase. The cost of those purchases will determine the level of rewards.</p><p>Hence, it is important to determine if the above loyalty programs do work or not, if the customer enjoys his/her card for purchases often and on which merchants. This insight helps to fine tune the offerings and maximize the profits.</p><p>Elo picked new merchants to recommend for each card holder. The date when Elo began providing recommendations is called the ‘reference date’. After the reference date, for each card Elo gathered transaction history for all new merchants that appeared on the card. By comparing each card’s new merchant activity and the list of the merchants recommended by Elo, the loyalty score was calculated. Our goal is to evaluate Elo’s recommendation algorithm by trying to predict in which cases it’s going to work well (yielding a high loyalty score) and in which cases — not (yielding a low loyalty score).</p><p>Submissions are scored on the root mean squared error. RMSE is defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/316/0*INaPahFkc9sX_Zxs" /></figure><p>where ​P​i ​is the predicted loyalty score for each ​card_id​, and ​O​i​ ​is the actual loyalty score assigned to a ​card_id​.</p><h3><strong>3. Data Collection</strong></h3><p>The data sets are available in the Kaggle link of the ELO problem</p><p>The Datasets are fictional, and we have five .csv files:</p><ul><li>train.csv (201917 rows, 6 columns)</li><li>test.csv (123623 rows, 5 columns)</li><li>historical_transactions.csv (29112361 rows, 14 columns)</li><li>new_merchant_transactions.csv (1963031 rows, 14 columns)</li><li>merchants.csv (334696 rows, 22 columns)</li></ul><p><strong>3.1 <em>Train.csv/Test.csv:</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*0-wjA5I3GdCHQjIUr3u_wQ.png" /></figure><p><strong>3.2 <em>Historical_transactions.csv/new_merchant_transactions.csv:</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/1*PoVdGF0BH9THbo5yH722fQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*T2PLfi9tGf0CQUHUPv_NqA.png" /></figure><h4><em>3.3 Merchants.csv:</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/579/1*k0jypRTjKb0XunmzMokIIQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/753/1*z2yDKyf2l_QdKS7n-tciHg.png" /></figure><h3><strong>4. Data Preparation</strong></h3><p>Since there are multiple data sets available, each of the data sets is analyzed separately for data inconsistencies.</p><p>Once the EDA is performed on each of the data sets and features are obtained that are consistent across all the data sets are identified.</p><h3>5. <strong>Exploratory Data Analysis</strong></h3><p>5.1 <em>General overview of training data</em></p><p>Record count:</p><ol><li>Total number of observations: 201917</li><li>Number of features: 5</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/677/1*bxowet-dhEqcX6eir0QB4w.png" /></figure><p>Where Loyalty Score on the x-axis represents the target value</p><p>We notice that target values(&lt;-30) are far apart compared to others.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/696/1*difG-7IQ5tk-JprnAw2kLQ.png" /></figure><p>All the 2207 outlier records have the same target value: -33.22. For the time being, we retain these records in the data set</p><p>Next we analyzed the three anonymized feature columns. Based on the violin plot distribution of target value for these feature fields and bar chart drawn on the number of records, we infer that although the count of records across these features are different, but the distribution of target is almost the same across the three features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*57BN387eknqYFVeDVQ1BFw.png" /></figure><p><em>5.2 Overview of transaction data (Historical transactions)</em></p><ul><li>Total number of observations: 29,112,361</li><li>Number of features: 14</li><li>1,963,031</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/621/1*ZHvfqZ_Ns70Kmfv5TcxKsA.png" /></figure><h4>Analysis of Installment feature</h4><p>Count of records against each installment category</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*08PHFD1mFJVw7k5-l3eQJQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*YnSLlRG8QYSffUF9t_elYQ.png" /></figure><ul><li>Largest volume of transactions is with installment 0 and 1</li><li>We see unusual numbers like -1 and 999 as installments. They may be representing missing values</li><li>Only 3% of historical transactions are authorized with installment 999</li><li>Higher the number of installments, lower is the authorization rate</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/467/1*Jiu82cgmdxJ-ThCmPX6wFQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/1*_6LuoKBwWf49_NviPvJu2g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/445/1*MikSTB0_BpOQuwoD7Bigyw.png" /></figure><p>The Historical and new transactions datasets were concatenated as “All_transactions” dataset. We created a train transaction dataset from the all transaction only for the card ids found in the train dataset with the authorized flag set to “Yes”. Similarly, test transaction dataset created based on the card ids and authorized flag found in the test dataset.</p><p>Next, we performed missing value processing on the transaction train and test dataset and label encoded the categorical values in Category_1, Category_2 and Category_3 columns</p><p>We aggregated the historical and new transactions data grouped by card_id. Once the data is aggregated, the same in merged with the training and test dataset based on card_id.</p><h3><strong>6. Data Regularization: Lasso, Ridge and ElasticNet</strong></h3><h4>Lasso Coefficients of training data</h4><p>Using Lasso model, we have identified the features that influence the Target more compared to others.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/1*3UFfmgiiNTCDXcU9rSZz2w.png" /></figure><h3>Ridge Coefficients of training data</h3><p>The following are the features and their coefficients listed based on Ridge regularization</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*7iCj60wtjWCwwBrHFKffGg.png" /></figure><h3>ElasticNet Coefficients of training data</h3><p>The following are the features and their coefficients listed based on ElasticNet regularization</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/1*pypsP4ep7_bvQefM3aSj6A.png" /></figure><h3><strong>7. Building the Loyalty Score model</strong></h3><p>7.1 Linear Regression Model</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/1*eu4RapK0dkVoQdNNtO7rhA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*tJ2sItWxVtj8uOFk4V4rFw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/1*b4j0vOejATmKO_s0LOrxkA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*dDh8jaNc8eWShpZewVi8gA.png" /></figure><p>We used K Means clustering to find out the optimal no of clusters in the data. We standardize the data before applying clustering. We calculate the distortion of the clusters using euclidean distance and plot the average distortion using “Elbow Method”.</p><p>Using this, we see a fairly smooth curve, and it’s unclear what is the best value of k to choose, indicating the data is not very strongly clustered. If we need to cluster, then we need to consider the elbow at k=4 &amp; 6.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/1*fWbpEbcL-OKQp56kcT37_w.png" /></figure><p>7.6 AdaBoosting Regressor</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/1*TDEc0XssY-h8Y9NAic8ljw.png" /></figure><p>7.7 Gradient Boosting Regressor</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/672/1*mm9kSCpU1Fjtbv3-IwuwUg.png" /></figure><p>7.8 XG Boosting Regressor</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*PLgyfVQsY7YplBtRVk4T0A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/674/1*XInS4u5VerrD7qTmSSZ7Ug.png" /></figure><p>We built a k-fold cross validation module with 5-folds and ran for the below algorithms</p><ol><li>Linear Regression</li><li>ElasticNet, Lasso and Ridge</li><li>SVR</li><li>LGBM Regression</li><li>Gradient Boosting Regressor</li><li>XG Booster Regressor</li><li>Decision Tree Regressor</li><li>KNN Regressor</li><li>Random Forest Regressor</li><li>AdaBoost Regression</li><li>Bagging and Stacking Regressors</li></ol><p>Below shown are the rmse scores against each of the algorithms from the cross validation model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/279/1*gByVgKpqKfYcl6xp3CHlkg.png" /></figure><p>We see that the Light GBM model provided the lowest rmse value of 3.74.</p><p>We used the high-level programming language Keras for creating the model. Tensorflow acted as the backend for Keras. We were able to improve the rmse value substantially 3.04</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/639/1*iDJx3FZHTuV2E4ltGBB4vQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/452/1*jouXYf5H4jSYeZJPWfBCjA.png" /></figure><h3>8. <strong>Challenges</strong></h3><ul><li>Data volume is very high</li></ul><p><em>Mitigation — We used Kaggle kernel to run the code</em></p><ul><li>Identifying the characteristics of some of the columns like Feature_1,2,3 or Category_1,2,3 as the data description provided by Elo was minimal</li><li>Finding the relationship between the various data sets provided and building a common training set for modelling</li></ul><h3><strong>9. Conclusion</strong></h3><p>We made use of the various machine learning techniques learning during our course to solve a business problem. We analyzed both supervised as well as unsupervised learning models during the project. We will continue to improve the accuracy of the model by working on the various features identified so far and optimizing them for use in the model.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b28096c882b7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Capstone: Investment strategy by predicting stock prices based on quarterly and annual reports of…]]></title>
            <link>https://medium.com/@blogsupport/capstone-investment-strategy-by-predicting-stock-prices-based-on-quarterly-and-annual-reports-of-e0c5c39fdbb2?source=rss-2e2a341c9deb------2</link>
            <guid isPermaLink="false">https://medium.com/p/e0c5c39fdbb2</guid>
            <dc:creator><![CDATA[Great Learning Snippets]]></dc:creator>
            <pubDate>Thu, 30 Jul 2020 05:36:16 GMT</pubDate>
            <atom:updated>2020-07-30T05:36:16.077Z</atom:updated>
            <content:encoded><![CDATA[<h3>Capstone: Investment strategy by predicting stock prices based on quarterly and annual reports of the firm.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*3r3UIHcM6GHi6CSmomxTSg.jpeg" /></figure><p>Project Submitted by:</p><p>Yogesh Kumar Agarwal <br>Dinesh Bartwal <br>Ankit Kapoor <br>Barinder Singh <br>Dr. D Narayana</p><p><strong>Abstract</strong></p><p>An important aspect of portfolio management is to know when to enter or exit a stock. This is the key to make maximum profit from the investment we make in stock market. A lot of work has been done in this direction and there are different approaches to build this strategy. Recent work is focused on predicting the stock price using sentiment analysis of news, articles and twitter post. Or, predicting the price based on historic trends and patterns. However, impact on stock price by financial reports filled by companies has not been studied at length.</p><p>As a retail trader, we believe that financial filing by firms has a strong impact on their stock price. By building predictive model/s on financial papers is not only feasible, but also a practical approach. Our project focuses on a class of AI methods known as “machine learning” techniques to come up with accurate stock prediction model. In order to limit the scope, we are focusing on companies which are part of Nifty50. To build models, we are using their quarterly reports, cash flow statements and financial ratios of last 10 years.</p><p>In this document, we will focus on providing a background of financial market. Later we will cover the problem statement and proposed solution. We are also going to touch upon the exploratory data analysis, exploratory visualization, evaluation metrics, initial findings, challenges and next steps.</p><p><strong>Table of Contents</strong></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.gjdgxs"><strong>Capstone Project Final Submission 1</strong></a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.1fob9te"><strong>1.1</strong> <strong>Quarterly Report 3</strong></a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.3znysh7"><strong>1.2</strong> <strong>Quarterly Report and Stock Price 3</strong></a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.2et92p0"><strong>1.3</strong> <strong>Cash Flow 3</strong></a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.tyjcwt"><strong>1.4</strong> <strong>Financial Ratios 3</strong></a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.3dy6vkm"><strong>2.</strong> <strong>Summary of Problem Statement, Data and Findings 4</strong></a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.1t3h5sf"><strong>2.1</strong> <strong>Summary of Problem Statement 4</strong></a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.4d34og8"><strong>2.2</strong> <strong>Data Sources, Collection, Processing &amp; Data Split 4</strong></a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.2s8eyo1">2.2.1 Data Sources 4</a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.17dp8vu">2.2.2 Collection 4</a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.3rdcrjn">2.2.3 Data Processing 5</a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.26in1rg">2.2.4 Data Split (Walk Forward Analysis) 5</a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.lnxbz9">2.2.5 Findings 6</a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.35nkun2"><strong>3.</strong> <strong>Overview of the final process 6</strong></a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.1ksv4uv"><strong>3.1</strong> <strong>Problem Solving Methodology 6</strong></a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.44sinio"><strong>3.2</strong> <strong>Salient Features of Data 6</strong></a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.2jxsxqh"><strong>3.3</strong> <strong>Data Pre-Processing 7</strong></a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.z337ya"><strong>3.4</strong> <strong>Algorithms used 7</strong></a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.3j2qqm3"><strong>4.</strong> <strong>Step-by-step walk through of the solution 7</strong></a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.1y810tw"><strong>4.1</strong> <strong>First Step 8</strong></a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.4i7ojhp"><strong>4.2</strong> <strong>Second Step 8</strong></a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.2xcytpi"><strong>4.3</strong> <strong>Third Step 8</strong></a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.1ci93xb">4.3.1 Descriptive Data Analysis 8</a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.3whwml4">4.3.2 Box plot 11</a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.2bn6wsx">4.3.3 Correlation 12</a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.qsh70q">4.3.4 Data Analysis and handing NULL values 13</a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.3as4poj">4.3.4 Merging all the dataset 21</a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.1pxezwc">4.3.5 Train Test Split (Walk Forward Analysis) 21</a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.49x2ik5">4.3.6 Model building 22</a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.2p2csry"><strong>5.</strong> <strong>Model Evaluation and Model Evaluation Metrics 22</strong></a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.147n2zr"><strong>5.1</strong> <strong>Precision and Recall 22</strong></a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.3o7alnk"><strong>5.2</strong> <strong>F1 Score 23</strong></a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.23ckvvd"><strong>5.3</strong> <strong>Models 23</strong></a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.ihv636"><strong>5.4</strong> <strong>AdaBoost 26</strong></a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.32hioqz"><strong>6.</strong> <strong>Comparison to Benchmark 27</strong></a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.1hmsyys"><strong>7.</strong> <strong>Visualization 27</strong></a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.41mghml"><strong>8.</strong> <strong>Implications 27</strong></a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.2grqrue"><strong>9.</strong> <strong>Limitations 28</strong></a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.vx1227"><strong>10.</strong> <strong>Closing Reflections 28</strong></a></p><p><a href="https://docs.google.com/document/d/1A-wviklH6_BABFPS8W88msHmBz9S9Lxy/edit#heading=h.3fwokq0"><strong>11.</strong> <strong>Reference 29</strong></a></p><p>1. Financial Market</p><p>The stock market is a financial market that enables investors to buy and sell shares of publicly traded companies. The primary stock market is where new issues of stocks are first offered. Any subsequent trading of stock securities occurs in the secondary market.</p><ul><li>Quarterly Report</li></ul><p>A quarterly report is a combination of <a href="https://investinganswers.com/dictionary/c/consolidated-financial-statements">financial statements</a> issued by a company every three months. It typically includes an <a href="https://investinganswers.com/dictionary/i/income-statement">income statement</a>, <a href="https://investinganswers.com/node/1083">balance sheet</a>, and <a href="https://investinganswers.com/dictionary/c/cash-flow-statement">cash flow statement</a> for the quarter and the year-to-date (<a href="https://investinganswers.com/dictionary/y/ytd">YTD</a>), as well as comparative results for the prior <a href="https://investinganswers.com/dictionary/y/year">year</a>. Quarterly reports also include a discussion and analysis of the company’s financial condition, disclosures about risk <a href="https://investinganswers.com/dictionary/f/factor">factors</a> that may affect the value of the company, a discussion of matters submitted to a vote by shareholders during the quarter; and any other pertinent information related to the company and its business.</p><p>Generally, <a href="https://investinganswers.com/dictionary/q/quarter-q1-q2-q3-q4">quarters</a> end in March, June, September, and December, and quarterly reports are filed a few weeks later.</p><ul><li>Quarterly Report and Stock Price</li></ul><p>Quarterly reports help investors take the pulse of public companies. By comparing the quarterly information to the previous <a href="https://investinganswers.com/dictionary/y/year">year</a>’s information for the same quarter, investors can get rich insight into a business’s performance and growth. Furthermore, quarterly reports help investors predict future <a href="https://investinganswers.com/dictionary/e/earnings">earnings</a> potential, which is highly correlated to a company’s share price.</p><ul><li>Cash Flow</li></ul><p>Annual and Quarterly reports provide good insights of a company. However, to understand the overall picture it is important to club them with Cash Flow statements. Cash Flow statements vividly explain the amount of money coming in and going out of a company. Thus, to understand the overall health of a firm it is important to pay attention to these statements.</p><ul><li>Financial Ratios</li></ul><p>Financial Rations is a combination of metrics, calculated by taking different components of companies’ financial statement. Though, components of financial statement give a numerical understanding of each, but it is the ratios which brings out the hidden trends. Each of these metrics is critical in analyzing the trend of firm over time, forecasting business activity, estimating budgets and others.</p><ul><li>Summary of Problem Statement, Data and Findings</li><li>Summary of Problem Statement</li></ul><p>As a retail trader, we believe that financial filing by firms has a strong impact on their stock price. By building predictive model/s on financial papers is not only feasible, but also a practical approach. Our project focuses on a class of AI methods known as “machine learning” techniques to come up with accurate stock prediction model. In order to limit the scope, we are focusing on companies which are part of Nifty50. To build models, we are using their quarterly reports, cash flow statements and financial ratios of last 10 years.</p><p>The problem statement is that as an investor, we want to build a system that empowers us to make intelligent decisions. We envision a system, that can analyze the stocks fed into it and come up with predictions, whether they will return X% in next Y months. These predictions are based on the quarterly and annual reports of the firms.</p><p>We believe a framework based on Machine Learning and Artificial Intelligence can empower investors to make profitable decisions no matter the market conditions. We intent to start from basic, where our model should predict a set of stock which can generate 5% profit in next quarter. And, set of stocks which can go 5% down in next quarter, so that we can short them in time.</p><ul><li>Data Sources, Collection, Processing &amp; Data Split</li></ul><h3>2.2.1 Data Sources</h3><p>Quarterly data of listed firms is publically available on NSE site; however, there are other sites like screener.in, moneycontrol.com, yahoo finance, etc where these reports are readily available. For our project, we need quarterly reports of the firms in Nifty50 and their stock price at the end of that quarter.</p><p>While hunting for data, we understood that fetching data from moneycontrol.com is much easily, as compared with other sites.</p><h3>2.2.2 Collection</h3><p>After fetching data, our approach is to create an excel file for every stock/company. Every file has three sheets, sheet one contains quarterly data of the firm from year 2000 onwards. Whereas, sheet two contains stock price at the end of every month from year 2000 onwards. And the third sheet has cash flow and ratio information of the stock. However, after doing analysis of data, we decided to take data from 2007 onwards. The reasons for the same has been stated ahead.</p><p>Here, we are listing all the columns we obtained from quarterly report.</p><ul><li>Sales</li><li>Other Income</li><li>Expenses</li><li>Raw Material Cost</li><li>Change in Inventory</li><li>Power and Fuel</li><li>Employee Cost</li><li>Selling and admin</li><li>Any Other Expenses</li><li>Gross/Operating Profit</li><li>EBIDTA (Earnings Before Interest Depreciation Tax &amp; Amortization)</li><li>Interest</li><li>Depreciation</li><li>Profit Before Tax (PBT)</li><li>Tax</li><li>Net profit</li><li>Details from Balance Sheet</li><li>Equity Share Capital</li><li>Promoters Equity</li><li>Public Equity</li><li>Reserves</li><li>Debt/Borrowings</li><li>Secured</li><li>Unsecured</li><li>Other Liabilities</li><li>Total Liabilities</li><li>Assets</li><li>Capital Work in Progress</li><li>Investments</li><li>Other Assets</li><li>Receivables</li><li>Inventory</li><li>Cash &amp; Cash Equivalents</li><li>Gross Block</li><li>Net Block</li><li>Details from Cash Flow Statement</li><li>Cash Flows From Operations</li><li>Cash Flows From Investing</li><li>Cash Flows From Financing</li></ul><h3>2.2.3 Data Processing</h3><p>Our target variable is categorical in nature, stating whether a stock generates 5% profit in next quarter, goes 5% down in next quarter or remains the same. To train our model we generate target variable by using the formula below: -</p><p>Percentage change (this month) = Price (next month) — Price (this month) / Price (this month)</p><p>Target variable (this month) is:-</p><p>1 🡪 if Percentage change (this month) &gt;5</p><p>0 🡪 if Percentage change (this month) &lt;5</p><h3>2.2.4 Data Split (Walk Forward Analysis)</h3><p>Traditional methods of validation and cross-validation are problematic for time series prediction problems. The solution is to use a “walk-forward-analysis” approach which incorporates new information as it becomes available. This approach gives us a more realistic view of how effective our model would truly have been in the past and helps to avoid the overfitting trap.</p><h3>2.2.5 Findings</h3><p>We created multiple models on our data, ranging from Logistic Regression, Decision Tree, Random Forest, AdaBoost, Gradient Boost, Gaussian Naive Bayes, Support Vector Machine, K-Nearest Neighbors, Deep Neural Networks and others. What we found is that Random Forest provides the best F1 score. The reason we are using F1 instead of Accuracy is because we have uneven class distribution. Moreover, F1 score is weightage average of precision and recall. Therefore, it works better for biased datasets.</p><ul><li>Overview of the final process</li><li>Problem Solving Methodology</li></ul><p>For our Capstone, the decision was to use Nifty 50 data starting from 1990’s onward. It was not only difficult to collect old data, but also to get it in proper format. Template to file quarterly report and other information has changed over the year. Also, initially there were lesser fields, when all data was compile together. Then we found high presence of NANs in our data set. Thus, we decided to stick to data post 2008.</p><p>Our problem is a time series problem, within time series we used sliding window methodology. We train on n-data points and validate the prediction on the next n-data points, sliding the 2n training/validation window in time for the next step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/707/0*FGUgeMJ-fx2D8M8e" /></figure><ul><li>Salient Features of Data</li></ul><p>In the beginning of the project we started with Quarterly Data only, there were a lot of features which drives the stock value. Like net sales, net expenditure, net profit/loss, number of share, etc. These further had sub-sections to explain each component granularly. Half way through the project we realized that cash flow statement is very crucial to understand the overall health of the firm. Healthier the firm stronger is the movement of their stock. Thus, we decided to include cashflow statement and rations to our dataset.</p><p>After merging quarterly statement, cashflow statement and ratio, there were close to 100 features in the dataset. Not all features were equally important, a lot has null values, some were highly correlated, hence data clean up took a lot of time.</p><ul><li>Data Pre-Processing</li></ul><p>As stated earlier, raw data had close to 100 features. But not all were used for building the model. Multiple pre-processing activities were performed, that too multiple time whenever the composition of dataset changed. Under step by step walk through we will discuss these in depth, in this section will just touch upon them.</p><p>Descriptive Data Analysis</p><p>We used descriptive data analysis in understanding the characteristics of the data. Moreover, it provides a quick summary of the complete data. Mean, median, standard deviation, etc help to statistically visualize the data by providing insides to data skewness, outliers among others.</p><p>Box Plot</p><p>Box plot helps us identify extreme values or outlies in the data. It is a pictorial representation of data with extreme values, median and quartiles.</p><p>Correlation</p><p>Finding correlation between fields is very important. Having highly correlated columns is detrimental for the model. After identifying highly correlated columns, we can choose some and ignore others.</p><p>Null Values</p><p>Initially we wanted to consume data of all companies under Nifty50 since 2000. However, later we found that data before 2008 wasn’t consistent, as it has a lot of null values and other format issues. Even after that there were null values. Certain values could be filed using interpolation because fields of financial data can be mathematically computed. For other columns, we used backword fill approach. Though, we preferred to backword fill missing values after due discussion within us and with mentor.</p><ul><li>Algorithms used</li></ul><p>We created multiple models on our data, ranging from Logistic Regression, Decision Tree, Random Forest, AdaBoost, Gradient Boost, Gaussian Naive Bayes, Support Vector Machine, K-Nearest Neighbors, Deep Neural Networks and others. What we found is that Random Forest provides the best F1 score. The reason we are using F1 instead of Accuracy is because we have uneven class distribution. Moreover, F1 score is weightage average of precision and recall. Therefore, it works better for biased datasets.</p><ul><li>Step-by-step walk through of the solution</li></ul><p>We feel that, we can divide the solution in two phases. The first phase was when we used just the Quarterly Data, there were a lot of features which drives the stock value. Like net sales, net expenditure, net profit/loss, number of share, etc. These further had sub-sections to explain each component granularly. Half way through the project we realized that cash flow statement is very crucial to understand the overall health of the firm. Healthier the firm stronger is the movement of their stock. Thus, we decided to include cashflow statement and rations to our dataset. The operational steps performed in each phase remains more or less same. In here, will we will talk about the final solution.</p><ul><li>First Step</li></ul><p>The very first operation was to collect data of listed firms is publically available on NSE site; however, there are other sites like screener.in, moneycontrol.com, yahoo finance, etc where these reports are readily available. While hunting for data, we understood that fetching data from moneycontrol.com is much easily, as compared with other sites. One excel file was created for every company.</p><ul><li>Second Step</li></ul><p>Combining all the data in a single dataset was a cumbersome task. We were primarily interested in quarterly data, thus we had to process raw data for quarters and ignore the rest. While building the data, we also had to calculate the target variable, for the we subtracted the stock price of [T+1] month from [T] month. Thus to calculate for March 2018 we used:-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/0*U7J49bOsMyA4hn-j" /></figure><p>Target variable (this month) is:-</p><p>1 🡪 if Percentage change (this month) &gt;5</p><p>0 🡪 if Percentage change (this month) &lt;5</p><ul><li>Third Step</li></ul><h3>4.3.1 Descriptive Data Analysis</h3><p>Descriptive Data Analysis is used in understanding the characteristics of the data. Moreover, it provides a quick summary of the complete data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/993/0*lXAjMKxVevVn5hhm" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/987/0*dxnDIpw85WINTXki" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/0*mrf6zvptGmb5pGjP" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Bz3peiC4YQrgXrW_" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/895/0*uglmZpQXYE9hzigG" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/916/0*N8fJL6uFknBEg2l2" /></figure><h3>4.3.2 Box plot</h3><p>Box plot helps us identify extreme values or outlies in the data. It is a pictorial representation of data with extreme values, median and quartiles.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1MVAZEHi9kEw_QFY" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IdbfFUOetII_CgSt" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*M_weT1lZxFR4KHTu" /></figure><h3>4.3.3 Correlation</h3><p>Finding correlation between fields is very important. Having highly correlated columns is detrimental for the model. After identifying highly correlated columns, we can choose some and ignore others.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*X_d__eEjluHydg4C" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GLCDOU1BA3vh6DQJ" /></figure><ul><li>“Net Sales/Income from operations” has high correlation with “Total Income From Operations” and “Consumption of Raw Materials”</li><li>“P/L Before Int., Excpt. Items &amp; Tax” has high correlation with “P/L Before Other Inc. , Int., Excpt. Items &amp; Tax”, “P/L Before Exceptional Items &amp; Tax”, “P/L Before Tax” and “P/L After Tax from Ordinary Activities”</li><li>“Net Profit/(Loss) For the Period” has high correlation with “P/L Before Exceptional Items &amp; Tax”, “P/L Before Tax” and “P/L After Tax from Ordinary Activities”</li><li>“DilutedEPS_BeforeExtraOrdinary” has high correlation with “BasicEPS_AfterExtraOrdinary” and “DilutedEPS_AfterExtraOrdinary”.</li><li>“PerShare_TotalShofPromoter_NonEncumbered” and “PerShare_TotalShCapitalComp_NonEncumbered” are highly correlated.</li></ul><h3>4.3.4 Data Analysis and handing NULL values</h3><p>Initially we wanted to consume data of all companies under Nifty50 since 2000. However, later we found that data before 2008 wasn’t consistent, as it has a lot of null values and other format issues. Even after that there were null values. Certain values could be filed using interpolation because fields of financial data can be mathematically computed. For other columns, we used backword fill approach. Though, we preferred to backword fill missing values after due discussion within us and with mentor.</p><h4>Analysis on Quarterly Report Data</h4><p><strong>Analysis 1</strong> — We felt the need of having a column showing consolidated income from operations. Thus, we created a feature ‘Total Income From Operations’ as: -</p><p>‘Total Income From Operations’ = ‘Net Sales/Income from operations’ + ‘Other Operating Income’</p><p>All the three features had kind of same information, hence we dropped features: -</p><ul><li>Net Sales/Income from operations</li><li>Other Operating Income</li></ul><p><strong>Analysis 2</strong> — Here, we tried to identify features of companies having null values, whereas, other firms have value for those features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*okWOZJNpkVvr3ybC" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/0*FDjYP8-QGu_VUZzD" /></figure><p>This analysis suggested us that following columns had null values for more than 30 companies. Therefore, these columns were dropped: -</p><ul><li>Power &amp; Fuel</li><li>Admin. And Selling Expenses</li><li>R &amp; D Expenses</li><li>Provisions And Contingencies</li><li>Exp. Capitalised</li><li>Equity Dividend Rate (%)</li></ul><p><strong>Analysis 3</strong> — Here, we tried to identify all the columns in dataset with maximum values as NULL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3SBRCMO0P94H37sx" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/0*EOwMhPmMMZ2aNrF3" /></figure><p>Along with this, we plotted the columns which we thought have high correlation. So that, we can pick the best and neglect the ones not needed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/0*27kR1pUNHR45mJSW" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/755/0*LVn-8MpwCY47Irwy" /></figure><p>After doing analysis of this section, we decided to drop following features: -</p><ul><li>Excise Duty</li><li>Prior Year Adjustments</li><li>Extra Ordinary Items</li><li>Reserves Excluding Revaluation Reserves</li><li>PerShare_TotalShCapitalComp_PledgedEncumbered</li><li>No_ofShare_PledgedEncumbered</li><li>PerShare_PledgedEncumbered</li><li>Exceptional Items</li><li>PerShare_TotalShCapitalComp_NonEncumbered ( Choosing no. of share )</li><li>PerShare_TotalShofPromoter_NonEncumbered ( Choosing no. of share )</li><li>DilutedEPS_BeforeExtraOrdinary</li><li>BasicEPS_AfterExtraOrdinary</li><li>DilutedEPS_AfterExtraOrdinary</li></ul><p><strong>Analysis 4</strong> — Even after this we had a lot of NULL values. To negate their effect, certain values were filled using backword fill approach.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tuRmTgQkrz9YaGgm" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/357/0*GmvADvWVITdNptPM" /></figure><p>Following columns were populated backword fill: -</p><ul><li>No_ofShare_NonEncumbered</li><li>Purchase of Traded Goods</li><li>No Of Shares (Crores)</li><li>Consumption of Raw Materials</li><li>Increase/Decrease in Stocks</li><li>BasicEPS_BeforeExtraOrdinary</li><li>Tax</li><li>Other Income</li><li>Interest</li><li>Share Holding (%)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*o3oDaB1CMzUoslm7" /></figure><p><strong>Analysis 5</strong> — Still there were some NULL values, in the end we decided to drop those features which had over 500 rows of NULL values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*n5JssIroSy-gD8iJ" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/345/0*eTGVOV0IDFdEZY1z" /></figure><p>Ended up dropping features: -</p><ul><li>No_ofShare_NonEncumbered</li><li>No Of Shares (Crores)</li><li>Consumption of Raw Materials</li><li>Increase/Decrease in Stocks</li><li>Purchase of Traded Goods</li></ul><p><strong>Analysis 6</strong> — later we found that data before 2008 wasn’t consistent, as it has a lot of null values and other format issues. The format issues were because template for quarterly report filing changed. After multiple discussion, we decided to still with data post 2007.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/392/0*TgFhA_HuNitb8o43" /></figure><h4>Analysis on Annual Balancesheet Data</h4><p><strong>Analysis 1 </strong>— Here, we tried to identify all the columns in dataset with maximum values as NULL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xrRk-X913KULVvq5" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/376/0*wY0MqWNwRbmvR4Be" /></figure><p>This analysis suggested us that following columns had null values, therefore, they were dropped: -</p><ul><li>Deferred Tax Assets [Net]</li><li>Other Non-Current Assets</li><li>Current Investments</li><li>Long Term Provisions</li><li>Long Term Loans And Advances</li><li>Short Term Borrowings</li><li>OtherCurrentAssets</li><li>Deferred Tax Liabilities [Net]</li><li>Long Term Borrowings</li><li>Inventories</li></ul><h3>4.3.4 Merging all the dataset</h3><p>We have used Quaterly Report data, Annual Report data, Cashflow statements and Ratio statements. They were merged together to form a dataset over which model building is to happen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0mBmqddx9ApgZNMl" /></figure><h3>4.3.5 Train Test Split (Walk Forward Analysis)</h3><p>Traditional methods of validation and cross-validation are problematic for time series prediction problems. The solution is to use a “walk-forward-analysis” approach which incorporates new information as it becomes available. This approach gives us a more realistic view of how effective our model would truly have been in the past and helps to avoid the overfitting trap.</p><p>We trained our models on T to T+4 years of data and tested them on T+5 year’s data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/0*PPwCTw2xOQw2fp3a" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/545/0*Y_idqRMXNUSbzIuD" /></figure><h3>4.3.6 Model building</h3><p>We created multiple models on our data, ranging from Logistic Regression, Decision Tree, Random Forest, AdaBoost, Gradient Boost, Gaussian Naive Bayes, Support Vector Machine, K-Nearest Neighbors, Deep Neural Networks and others. What we found is that Random Forest provides the best F1 score. The reason we are using F1 instead of Accuracy is because we have uneven class distribution. Moreover, F1 score is weightage average of precision and recall. Therefore, it works better for biased datasets.</p><ul><li>Model Evaluation and Model Evaluation Metrics</li></ul><p>Accuracy is an important metrics, but it is not always the be-all and end-all model metric that we should use to choose best model. If we consider the following confusion metrics: -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/0*ZcuwL-IiwgjxuLcD" /></figure><p>We could see that the accuracy of the model is 99.99%, however, one out of two positive cases has been classified incorrectly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/0*7lgJTdEHL5X5mfvU" /></figure><ul><li>Precision and Recall</li></ul><p>Precision talks about how accurate your model is out of those predicted positive. It tells how many of the classificatios are actual positive.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/459/0*R2YXOKQACDKYq70B" /></figure><p>Whereas, recall actually calculates how many of the Actual Positives in our model capture through labelling it as Positive (True Positive).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/436/0*sx6eQJKiSJmU8U6E" /></figure><ul><li>F1 Score</li></ul><p>F1 Score is needed when we want to have a balance between Precision and Recall. We know that accuracy can be largely contributed by a large number of True Negatives which in most business circumstances, we do not focus on much whereas False Negative and False Positive usually has business costs. Therefore, F1 Score is a better measure to use if we need to seek a balance between Precision and Recall, also if there is an uneven class distribution (large number of Actual Negatives).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/349/0*hYlXnDpxMnTSXEbO" /></figure><ul><li>Models</li></ul><p>As stated above in this document, we have used walk forward approach for building our model because of time series based data. The data belongs to Nifty fifty stock from year 2007 to 2019. We decided to train our model for 4 years of data and then test it on the 5th year. As a result, we got 8 set of data to train and test. Till now we have used following models: -</p><ol><li>LogisticRegression</li><li>DecisionTreeClassifier</li><li>KNeighborsClassifier</li><li>RandomForestClassifier</li><li>AdaBoostClassifier</li><li>GradientBoostingClassifier</li><li>GaussianNB</li><li>Deep Neural Networks</li></ol><p>We followed two approaches to build models on our datasets. The first approach is used test each of the above model against a dataset and use the one, which gives the best F1 score. Second approach is to use PCA, to come up with best combination of features for the above models against a dataset and use the one, which gives the best F1 score. Results from both the approaches have been shared here.</p><p>Grid Search was also used to come up with best set of parameters for all the models. And it was part of both set of approaches, which is using PCA and without using PCA.</p><p>Grid Search Pipeline Code</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZzdnQOaFfejhXu97" /></figure><p>Results without PCA</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1023/0*YtK2kkaCK10mhxA3" /></figure><p>Results with PCA</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/0*S7QY2qf9g_mDK0nI" /></figure><p>PCA suggested that number of components to be taken up for model building is 27 features. These 27 features were able to cover 99% of the variance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/0*exBmv8mPEZ1K_h6F" /></figure><p>Results from DNN</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/0*k1R3LwnpDdaOIoNL" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/899/0*fi4eWaOQ5nxUt61V" /></figure><ul><li>AdaBoost</li></ul><p>From the results, we could see that AdaBoost has performed better than other models. Therefore, we took it as our final model. Next step was to refine the model further by hyper tuning the parameters using grid search.</p><p>GridSearch based pipeline</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YW-OXrr5n2KtjF1R" /></figure><p>Result comparison with and without Hyper Parameter Turing</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/0*A2-Smn9z6VkjJimw" /></figure><p>Final Score 49% F1 accuracy score</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/0*ZLfAKjRisik4ujxp" /></figure><ul><li>Comparison to Benchmark</li></ul><p>Our benchmark was to come up with a model, which can predict a stock which with 5% upward movement in a quarter. While, working on it we came across multiple challenges, though we were able to build machine learning based model using AdaBoost with 49% F1 accuracy. We seriously feel that by moving past Nifty 50 and having data of more firms will improve the accuracy.</p><p>At the same time we see, big potential in Deep Neural Network, where we were able to achieve 55% F1 accuracy. However, to use DNN based model as our base model we would like to train it on more data, to avoid overfitting.</p><ul><li>Visualization</li></ul><p>Throughout the document we have shared multiple screenshots of code and the results we obtained after running them. All the data visualization techniques mentioned in the document were part of our capstone working. We could add these images again under visualization, but that will make the document lengthy and repetitive.</p><ul><li>Implications</li></ul><p>Majority of the work in Global Markets Operations/Capital Markets/Equity Trading and Investments is done on trading standpoint. Following are the details of few business solutions that companies have created in United States. Though, there are solutions which are existing for similar opportunity/problem statement but the market exists for more players to exist.</p><p>Hence, there is a merit in terms of enhancing the solution further with 2 possible approaches. Either it to be an off the shelf solution for a retail customer or provide it to a portfolio management company which utilizes the same towards managing the portfolios of its end consumers.</p><p>We certainly feel that the potential of Indian market hasn’t really been explored. Still only 2% of the people in the country invest in equity market, which is way less than global average. It means there is a great growth potential, as more and more retails investors join equity market. They need for Machine Learning and Artificial Intelligence based predictive model will increase.</p><p>At present, our model stands at model 49% F1 accuracy score using AdaBoost. And preliminary results using Deep Neural Network are very promising over 55% F1 accuracy score. We envision to capture more data in months to come so that we can scale up our model, making it robust with higher accuracy.</p><ul><li>Limitations</li></ul><p>There are multiple challenges we have faced while working on this capstone. Not to deny that there have been numerous learnings too. and are facing right at this moment.</p><ol><li>Data — Data is always a challenge for data scientists. In absence of funds, we decided to obtain our data from open source free websites. We had decent success, however, to take our model to production readiness we need to have more data. In months to come, post our capstone submission, we have decided to keep this initiative alive. Our next step would be to use scrapping techniques to collect data of all companies listed on NSE. This way we will be able to scale up our model, making it robust with higher accuracy.</li><li>Deep Neural Network — At present, our model stands at model 52% F1 accuracy score using AdaBoost. And preliminary results using Deep Neural Network are very promising over 55% F1 accuracy score. With more data in hand we can overcome DNN’s overfitting issue and explore its true potential.</li><li>Scale up — At present, our model does predictions for the next quarter. We can tweak the model to make it predict on yearly or half-yearly basis. This is another area where we need to work before being industry ready.</li></ol><ul><li>Closing Reflections</li></ul><p>One of the biggest reason for doing capstone is learning. Learning is not only limited to Machine Learning and Artificial Intelligence, but also extended to domain which is security market in this case. All four of our team mates, had different level of understand of the domains, some were equity investors while others didn’t even have a DMAT account. We definitely learned domain while working on the capstone.</p><p>Other important learning was on data collection and problem statement. As part of the projects under our curriculum, we always had close to perfect dataset and a well-defined problem statement. However, while running this capstone we first hand experienced the challenges if these are not available. We defined out own problem statement, collected our own data, cleaned it, compiled it before we could think of creating any models.</p><p>Other important learning was on handing Time Series data. With the help of our mentor, who introduced us to the concept of Walk Forward Analysis. Implementing it against our data was an interesting job.</p><p>Also, we practically implemented all our learning from AIML course. Using Python, we crated functions to read, clean data. We did exploratory data analysis in depth, by using almost all charts we learned during our course. While building model, we created all sort of models ranging from independent models to PCA based models which were fined tuned using grid search. Moreover, we made use of deep neural network and came up with decent model with good accuracy.</p><p>Moreover, we got the confidence that we can truly lead projects from start till the end. That is the most important learning or take away for us from this capstone.</p><ul><li>Reference</li></ul><ol><li>Quarterly report <a href="https://www.moneycontrol.com/">https://www.moneycontrol.com/</a></li><li>Stock price details <a href="https://in.finance.yahoo.com/recent-quotes/">https://in.finance.yahoo.com/recent-quotes/</a> , History period from April 2000 to March 2019</li><li>Building predictive models <a href="https://medium.com/sfu-big-data/predicting-stable-portfolios-using-machine-learning-f2e27d6dbbec">https://medium.com/sfu-big-data/predicting-stable-portfolios-using-machine-learning-f2e27d6dbbec</a></li><li>Application of machine learning techniques <a href="https://medium.com/auquan/https-medium-com-auquan-machine-learning-techniques-trading-b7120cee4f05">https://medium.com/auquan/https-medium-com-auquan-machine-learning-techniques-trading-b7120cee4f05</a></li><li>Fundamental analysis <a href="https://medium.com/@amrut.patil88/stock-investing-fundamental-analysis-using-free-cash-flow-statement-454beaa7f00">https://medium.com/@amrut.patil88/stock-investing-fundamental-analysis-using-free-cash-flow-statement-454beaa7f00</a></li><li>Fundamental analysis balancesheet <a href="https://medium.com/@amrut.patil88/stock-investing-fundamental-analysis-using-companys-balance-sheet-153531d98f8b">https://medium.com/@amrut.patil88/stock-investing-fundamental-analysis-using-companys-balance-sheet-153531d98f8b</a></li><li>Fundamental analysis using ratios <a href="https://medium.com/@amrut.patil88/stock-investing-fundamental-analysis-using-key-ratios-ed02f32e88ae">https://medium.com/@amrut.patil88/stock-investing-fundamental-analysis-using-key-ratios-ed02f32e88ae</a></li><li>Fundamental analysis using company income <a href="https://medium.com/@amrut.patil88/stock-investing-fundamental-analysis-using-companys-income-statement-37b0d80531f1">https://medium.com/@amrut.patil88/stock-investing-fundamental-analysis-using-companys-income-statement-37b0d80531f1</a></li><li>F1 metrics <a href="https://towardsdatascience.com/accuracy-precision-recall-or-f1-331fb37c5cb9">https://towardsdatascience.com/accuracy-precision-recall-or-f1-331fb37c5cb9</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0c5c39fdbb2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[“Cancer Diagnosis with Machine Learning”]]></title>
            <link>https://medium.com/@blogsupport/cancer-diagnosis-with-machine-learning-7663730797b9?source=rss-2e2a341c9deb------2</link>
            <guid isPermaLink="false">https://medium.com/p/7663730797b9</guid>
            <dc:creator><![CDATA[Great Learning Snippets]]></dc:creator>
            <pubDate>Tue, 28 Jul 2020 04:38:14 GMT</pubDate>
            <atom:updated>2020-07-28T04:38:14.976Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*7mpi-EVWFvbduzneNa9ebQ.jpeg" /></figure><p><strong>Project by: </strong>Agila Sangar &amp; Meena Naganathan</p><p><strong>INTRODUCTION</strong></p><p>Healthcare Analytics is the branch of analysis that focuses on offering insights into hospital management, patient records, costs and diagnoses.</p><p>Genetic testing is going to disrupt the way diseases like cancer are treated. However, there is a long way to go on this since there is a lot of manual work currently being done in this area.</p><p>A cancer tumour can have thousands of genetic mutations. The challenge is distinguishing the mutations that contribute to growth, from the ones that are neutral.</p><p><strong>PROBLEM STATEMENT</strong></p><p>Classifying Clinically Actionable Genetic Mutations</p><p>Currently, these classifications of genetic mutations are done manually. However, this is a very time</p><p>consuming process where a clinical pathologist has to manually classify the gene mutations based on</p><p>evidence from text-based clinical literature.</p><p>This project is aimed at developing a machine learning algorithm that automatically classified gene</p><p>mutations.</p><p><strong>OVERVIEW OF THE FINAL PROCESS</strong></p><p><em>Classifying Clinically Actionable Genetic Mutations</em></p><p>Currently, these classifications of genetic mutations are done manually. However, this is a very time-consuming process where a clinical pathologist has to manually classify the gene mutations based on evidence from text-based clinical literature.</p><p>This project is aimed at developing a machine learning algorithm that automatically classified gene mutations. We propose to use Naïve-Bayes and Logistics Regression models as they would be appropriate for this text-based classification and is expected to provide good results.</p><p>The steps we would take in implementing the solution is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/405/0*k1u-rDhSN-hSIuFt" /></figure><p><strong>Modelling techniques proposed to be used:</strong></p><p>We propose to use the following models:</p><ol><li>Naïve Bayes</li><li>Logistic Regression</li></ol><p>The reason we used Naïve-Bayes model is because it gives good results for text-based classification. Logistics regression is highly interpretable, and we would get a minimum mis-classification error with this model.</p><p><strong>Evaluation Metrics</strong></p><p>The Evaluation Metrics that we propose to use are:</p><ul><li>Logarithmic- loss between the predicted probabilities.</li></ul><p><strong>SOLUTION WALKTHROUGH</strong></p><p><strong>READ DATA</strong></p><ul><li>Read Gene and Variation Data</li><li>Read text data</li></ul><p><strong>PREPROCESS TEXT</strong></p><ul><li>Replace special char with space</li><li>Replace multiple spaces with single space</li><li>Convert all chars to lower case</li><li>Retain words that are NOT stopwords</li><li>NLP preprocessing of text</li><li>Merge ‘gene variations’ and text data files based on ID</li><li>Where text is missing, update with gene+variation</li></ul><p><strong>TRAIN, TEST &amp; CROSS VALIDATION SPLIT</strong></p><p>Split data into Train, Test and Cross validation in the ration 64:20:16</p><p><strong>EXPLORATORY DATA ANALYSIS</strong></p><p>We have performed EDA for the following variables:</p><p><strong>Frequency of Classes</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/739/0*Zw9Wg6lHYk2Ztoxc" /></figure><p><strong>Number of times Gene appeared</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/693/0*ldkLqmx2BnWtF7wN" /></figure><p><strong>No of times Variation appeared:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/0*pBqVGtwNz1rPgDRi" /></figure><p><strong>Frequency of no of words</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/674/0*I46QTvcGRQTGsYu_" /></figure><p><strong>Frequency of no of Characters</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/693/0*lT8JRCYV3TruyGyt" /></figure><p><strong>Y’s distribution vs Text — Number of words</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/679/0*49rSGX4eNr0siyQ3" /></figure><ul><li><strong>After Train, Test &amp; Cross Validation split, we have plotted Bar graph for Y’s (Class variables)distribution in Train, Test and CV.</strong></li></ul><p><strong>Distribution of Y in Train data:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/367/0*VpHxQqYh__TsPPkj" /></figure><p><strong>Distribution of Y in Test data:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/0*d-7sdmlA-FFs3fUo" /></figure><p><strong>Distribution of Y in Cross validation data:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/370/0*i32SD7KSLeHWABn3" /></figure><p><strong>Summary of Findings</strong></p><ul><li>Number of data points in class 7 &amp; 4 is the highest (accounts for total 28%), and lowest in class 8 in all the data sets — train, test and cross validation.</li><li>Number of words of text is highest in class 8 &amp; 9.</li><li>Gene has appeared ~5 times for the most part</li></ul><p><strong>PREDICTION USING A ‘RANDOM’ MODEL</strong></p><p>Generate 9 numbers such that the sum equals 1</p><p>Create an output array that has exactly the same size as cross validation data</p><p>Create an output array that has the same size as test data</p><p>Log loss on Cross Validation data using Random Model = 2.5365</p><p>Log loss on Test data using Random Model = 2.5015</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/988/0*SvXmDkfof5_5cl5u" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/0*VmkgF2BGyvRJ4brC" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/985/0*in_OX5V9arc2r4-0" /></figure><p><strong>UNIVARIATE ANALYSIS</strong></p><p>Univariate Analysis on Gene Feature</p><p>Number of unique genes in the train data = 229</p><p>Response coding of gene feature</p><p>One hot coding of gene feature</p><p>Univariate Analysis on Variation Feature</p><p>Univariate Analysis on text feature</p><p><strong>MACHINE LEARNING MODELS</strong></p><p>NAÏVE BAYES MODEL</p><p>Hyperparameter tuning</p><p>Best value of Alpha = 0.1</p><p>Train Log loss = 0.903</p><p>Cross validation log loss = 1.35</p><p>Test log loss = 1.278</p><p>Testing the model with best hyper parameters</p><p>Log loss = 1.3509</p><p>Number of misclassified points = 0.4436</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1007/0*YgzO_GCE9l2nOhS9" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/0*3nAn75yCdsvU-EiP" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/993/0*qwCn8Z-fLCMpYUzz" /></figure><p>Feature Importance, Correctly classified point</p><p>Predicted Class = 7</p><p>Predicted Class Probabilities = [[0.0911 0.1284 0.0141 0.1119 0.0334 0.0365 0.5764 0.0057 0.0025]]</p><p>Actual Class = 7</p><p>Feature Importance, Incorrectly classified point</p><p>Predicted Class = 7</p><p>Predicted Class Probabilities = [[0.0876 0.0895 0.0136 0.1069 0.0322 0.0356 0.6267 0.0055 0.0024]]</p><p>Actual Class = 7</p><p><strong>Summary: </strong>Naïve Bayes is very good for text modelling especially for high dimension data.</p><p><strong>Interpretation: </strong>Misclassification error rate is 39.84%. This can be further reduced to improve the prediction.</p><p>LOGISTIC REGRESSION</p><p>With Class Balancing</p><p>Hyper parameter tuning</p><p>Best value of Alpha = 0.001</p><p>Train log loss = 0.6153</p><p>Cross validation log loss = 1.2173</p><p>Test log loss = 1.1047</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/490/0*6DMgC9Qradymb8RN" /></figure><p>Testing the model with best hyper parameters</p><p>Log loss = 1.2173</p><p>Number of mis-classified points = 0.3834</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1005/0*xuYZvhgUb9S4d12M" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/998/0*0Pz9BjT92FdI2YJc" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/0*aul5wur5URFuOtN9" /></figure><p>Feature Importance</p><p>Correctly Classified point</p><p>Predicted Class : 7</p><p>Predicted Class Probabilities: [[0.0045 0.1905 0.0012 0.0012 0.0047 0.0014 0.7872 0.0076 0.0017]]</p><p>Actual Class : 7</p><p>Incorrectly classified point</p><p>Predicted Class : 7</p><p>Predicted Class Probabilities: [[0.0482 0.2032 0.0108 0.0446 0.071 0.0164 0.5932 0.0078 0.0046]]</p><p>Actual Class : 7</p><p>Without Class Balancing</p><p>Hyper parameter tuning</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/446/0*x73KKDa5YX5bO04-" /></figure><p>Best value of Alpha = 0.001</p><p>Testing model with best hyper parameters</p><p>Log loss = 1.2521</p><p>Number of mis classified points = 0.3721</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1013/0*o9rHycM-p9r0OhRe" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/987/0*Iny7d50YxvEDFOgT" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/996/0*8QYNBuMvSRWHFRc4" /></figure><p>Feature Importance, Correctly Classified point</p><p>Predicted Class : 7</p><p>Predicted Class Probabilities: [[5.100e-03 1.255e-01 2.000e-04 1.300e-03 2.300e-03 1.400e-03 8.556e-01</p><p>8.500e-03 1.000e-04]]</p><p>Actual Class : 7</p><p>Feature Importance, Incorrectly classified point</p><p>Predicted Class : 7</p><p>Predicted Class Probabilities: [[0.0485 0.1851 0.0052 0.0442 0.0617 0.0143 0.6317 0.0072 0.0022]]</p><p>Actual Class : 7</p><p><strong>Summary:</strong> Logistic Regression is highly interpretable which is essential for healthcare analytics and we have used Logistic Regression with balancing as we have unbalanced data</p><p><strong>Interpretation:</strong> Misclassification error rate is very small at 34.77%. Approx. for 68% of the cases, correct results are obtained. This prediction is better than that of Naïve Bayes.</p><p><strong>MODEL EVALUATION</strong></p><p>Incorporated in solution walkthrough above</p><p><strong>COMPARISON TO BENCHMARK</strong></p><p>Incorporated in solution walkthrough above</p><p><strong>VISUALIZATION</strong></p><p>Incorporated in solution walkthrough above</p><p><strong>IMPLICATIONS</strong></p><p>Genetic testing is going to disrupt the way diseases like cancer are treated. However, there is a long way to go on this since there is a lot of manual work currently being done in this area.</p><p>Since currently, these classifications are done manually, it is a very time consuming process. Leveraging machine learning techniques would significantly reduce the time involved in doing this classification.</p><p><strong>LIMITATIONS</strong></p><p>Currently, only 2 models have been developed.</p><p><strong>CLOSING REFLECTIONS</strong></p><p>While we have incorporated 2 models viz. Naïve Bayes and Logistic Regression, we could explore with additional models and compare for best performance. Other models that were considered are KNN and Random Forest. However, they are not interpretable for this scenario and hence we have gone with only Naïve Bayes and Logistic Regression models. Additionally, Logistic Regression with balancing gives minimum misclassification error.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7663730797b9" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>