<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ganesh Prasad on Medium]]></title>
        <description><![CDATA[Stories by Ganesh Prasad on Medium]]></description>
        <link>https://medium.com/@ganeshn1411?source=rss-541b3eabee6f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*TLuTMcTkG0Ir0k0w2QZpBg.jpeg</url>
            <title>Stories by Ganesh Prasad on Medium</title>
            <link>https://medium.com/@ganeshn1411?source=rss-541b3eabee6f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 02:25:08 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ganeshn1411/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The role of AI Product Management]]></title>
            <link>https://medium.com/@ganeshn1411/the-role-of-ai-product-management-33c381635689?source=rss-541b3eabee6f------2</link>
            <guid isPermaLink="false">https://medium.com/p/33c381635689</guid>
            <category><![CDATA[ai-product-management]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Ganesh Prasad]]></dc:creator>
            <pubDate>Mon, 29 Nov 2021 23:05:40 GMT</pubDate>
            <atom:updated>2021-11-29T23:05:40.576Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pmnmxVPicVk5680C" /></figure><p>The often-cited figure when people talk about AI is that <a href="https://www.techrepublic.com/article/why-85-of-ai-projects-fail/">more than 85% of AI projects fail to make it to production</a>. I have heard this stat used by most AI companies to justify the need for better tools to make Data Scientists efficient. While having better tools is good for the Data Science community, it is not just the lack of tools but the lack of a clear product mindset that leads most AI projects to fail. That’s where AI product management takes a front seat.</p><p>While AI product management plays a very critical role in the success of a company’s AI strategy, there are very few resources that talk about the role of AI product management. Having spent a few years driving multiple AI projects, here is my take on the role of AI product management.</p><h3><strong>AI Product Management — Primary roles</strong></h3><p>The primary roles in AI product management involve 4 main items:</p><ol><li>Defining the right problem statement</li><li>Gathering the right dataset</li><li>Defining the right success metrics</li><li>Taking an AI prototype to production</li></ol><p>Let’s look at each of these 4 items in detail through an example of building a recommendation engine on a multinational retailer’s website to recommend similar products to buy when a customer selects a product.</p><p><strong>Defining the right problem statement: </strong>The first and foremost role in AI product management is to define the right problem statement. It is the product manager’s role to understand the customer&#39;s needs and translate them into a crisp, quantifiable problem that can be solved using Machine Learning. The key thing here is to determine that machine learning is the right technology to use rather than traditional rule-based software systems.</p><p>In our example, only a machine learning solution can recommend the most relevant product based on users’ past behavior and scale it across millions of users. Additionally, a recommendation engine is a useful feature to have as it can help in up-sell and cross-sell of the retailer’s products and improve the bottom line, making it the right problem for AI.</p><p><strong>Gathering the right dataset: </strong>Once the problem statement has been identified, the next step is to identify that the right datasets needed for the AI solution. There might be gaps in existing data sources and might require the product manager to acquire external data sources to fill these gaps. One important consideration while thinking about data for an AI product is to ensure that it follows the virtuous cycle of AI as shown below; the product automatically generates data and more data, in turn, improves the product.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/0*NMHHaUnalMF1Gg4N.jpeg" /><figcaption>The Virtuous cycle of AI [Andrew Ng]</figcaption></figure><p>In our example, the primary dataset needed for the recommendation engine is the set of products that each user has bought at the retailer. However, this data alone might not be sufficient, and additional data such as user profiles, product characteristics, etc. might be needed to enhance the recommendations. Additionally, the recommendation engine needs to be designed to take in feedback as users continue to buy more products and provide recommendations derived from the most recent data.</p><p><strong>Defining the right success metrics:</strong> As with any other product management role, one of the most important roles in AI product management is to define the success metrics of the product. Like software products, every AI product involves selecting the appropriate business metric such as revenue, # of users, growth rate, etc. to optimize. In addition, it also involves a Data Science metric such as accuracy, precision, recall, etc. that needs to be optimized to ensure that the product meets the customer&#39;s needs.</p><p>In our example, the primary business metric for a recommendation engine would be the revenue generated from products purchased through the recommendations. In addition to tracking the revenue, it is also important to track the correctness of the recommendation through Data Science metrics such as precision and recall.</p><p><strong>Taking an AI prototype to production:</strong> The next step in AI product management is to manage the build of an AI solution that addresses the problem and meets the success metrics benchmark. Taking an AI product from ideation to production is slightly different from that of traditional software projects. AI projects start as an experiment and progress as iterations to constantly improve performance. It is the product manager’s responsibility to balance expectations between customers and the data science team to ensure that product meets minimum performance thresholds and improves quality over time.</p><p>Coming back to the recommendation engine example, the product manager should work with the Data Science team and the customers to determine a minimum threshold for precision and recall that makes the product usable and release the product once it meets the threshold rather than aiming for the best possible performance. The feedback loop built into the engine needs to be monitored to ensure that the quality of the recommendations constantly improves over time with more data.</p><h3>AI Product Management — Peripheral roles</h3><p>In addition to the primary roles mentioned above, depending on the maturity of the company, AI product management could involve several peripheral roles to gain traction for the AI product:</p><ol><li><strong>Evangelism of AI:</strong> AI product managers are the conduit between customers, the Data Science teams, and the company leadership and it often falls on them to bring in thought leadership and trust with the AI technology across all their stakeholders.</li><li><strong>Driving the AI toolset:</strong> While the Data Science teams primarily drive the AI toolsets used to build the product, the product team might need to get involved in the toolset decision if it is the product’s differentiating factor.</li><li><strong>Driving Ethics and Governance in AI:</strong> Since AI governance is very loosely defined today, AI product managers are responsible to ensure that the product is built with the right ethical and governance principles.</li></ol><h3>Final Thoughts</h3><p>AI product management is an evolving role and is still in its infancy in most companies. As companies mature in the field of AI/ML, the role will evolve to become more specific and get split into multiple specializations. However, product managers today are expected to do a little more than traditional product management to be successful.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=33c381635689" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is an AI/ML Product?]]></title>
            <link>https://medium.com/@ganeshn1411/what-is-an-ai-ml-product-63e9657edb6a?source=rss-541b3eabee6f------2</link>
            <guid isPermaLink="false">https://medium.com/p/63e9657edb6a</guid>
            <category><![CDATA[artificalintelligence]]></category>
            <category><![CDATA[product]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Ganesh Prasad]]></dc:creator>
            <pubDate>Thu, 07 Oct 2021 01:39:32 GMT</pubDate>
            <atom:updated>2021-10-07T01:39:32.069Z</atom:updated>
            <content:encoded><![CDATA[<p>Artificial Intelligence and Machine Learning have become hot buzzwords and every sector across the globe is in a race to adopt AI/ML into their workstream. The number of AI startups has increased 14 times in the last 2 decades <em>(Source: Forbes)</em>.</p><p>As AI products become mainstream, it is critical to clearly define the AI product offering for both people building these products and the consumers of these products. A good way to think about the various AI/ML products &amp; services is their fit across various layers of the software stack.</p><p>In this article, I would like to go in-depth about each of these layers and some examples of products in the market that fall in each of the layers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WyCtRW1qch7_4eXuutvNoA.png" /></figure><ol><li><strong>AI/ML infra-as-a-service</strong>: Companies that offer the hardware infrastructure needed to run Machine Learning applications fall in the infra-as-a-service bucket. As more and more companies move towards Deep learning-based models, regular CPU machines are not well equipped to meet the computing needs and require powerful GPU machines tuned to run high-performance computing for AI applications. <br>In addition to GPUs, the quantum computer is a new entrant in this space trying to leapfrog the processing speed needed for AI applications. <br><em>Examples: Nvidia’s GPU machines, AWS/Azure/GCP GPU services</em></li><li><strong>AI/ML platform-as-a-service:</strong> The next layer that has reached significant maturity in the last few years in the AI/ML platform-as-a-service. These platforms offer a collection of Machine Learning pre-built algorithms like classification, clustering, forecasting, etc. bundled together to help data scientists and ML engineers to build Machine Learning applications. <br><em>Examples: Scikit-learn, H2O.ai, Tensorflow, Pytorch, Amazon Sagemaker, etc.</em></li><li><strong>AI/ML data(prediction)-as-a-service</strong>: Data-as-a-service/Prediction-as-a-service is the most common product in companies that own vast amounts of data and provide the outputs from their AI engines as a service. <br><em>Examples: Experian’s credit score, Fraud scores by credit card companies, etc.</em></li><li><strong>ML-driven recommendation-as-a-service:</strong> Companies that have tons of products and services offer product recommendations as a core product to help customers navigate across their range of products.<br><em>Examples: Amazon’s product recommendations, LinkedIn’s user recommendations, etc.</em></li><li><strong>Decision-as-a-service:</strong> Decision services are full-fledged software applications that combine rule-based systems along with ML-driven predictions and recommendations to automate the decision-making process either partially or completely. <br><em>Examples: Examples involve human-in-the-loop decision engines such as lending systems and fully automated systems like Uber, Google Maps, Google Search, and Self Driving cars</em></li></ol><p>If you can think about any AI/ML products that do not fit into any of these layers, please provide your thoughts in the comments below.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=63e9657edb6a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Good coding practices for Data Science]]></title>
            <link>https://medium.com/data-science/good-coding-practices-for-data-science-e9237783784c?source=rss-541b3eabee6f------2</link>
            <guid isPermaLink="false">https://medium.com/p/e9237783784c</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[good-coding-practice]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Ganesh Prasad]]></dc:creator>
            <pubDate>Sun, 21 Apr 2019 18:57:15 GMT</pubDate>
            <atom:updated>2019-06-11T13:02:25.568Z</atom:updated>
            <content:encoded><![CDATA[<p>While Data Science is not a field that directly stems out of computer science, the rise of Machine Learning and AI has made coding an integral part of Data Science.</p><p>While Data Science projects are more experimental and not as well-defined as software projects, following some good coding principles help increase efficiency and scale up data science projects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/276/1*JuenG1TIgLpTwD-pSisKTw@2x.jpeg" /></figure><p><strong>Here are some good coding practices that I try to follow:</strong></p><ul><li><strong>Code Organization</strong></li></ul><p>I ensure that all my code is not put in a single file but split across multiple files. I usually separate my code into the following 4 major parts.</p><p><em>a. Specification Files</em></p><p>In most of my projects, I tend to have specification file either as a YAML or a JSON where I can specify various parameters to be set to run the code. Having these specifications allow me to use the code in different ways with no code changes. Eg, Let’s say I built a model for one country. Instead of hardcoding the country variable in my code, if I specify a country variable in the spec file, I can use this code in a different country by just changing the specification.</p><p><em>b. Utilities</em></p><p>If there are parts of my code that is generic enough to be used across multiple projects, I put them in separate files called ‘utilities’ so that I can use them across multiple projects and minimize re-work.</p><p><em>c. Core Functionality</em></p><p>When it comes to the code for the core logic, I again try not to put them all into a single file but instead spread them across multiple files. For Eg., Every Data Science project will have a data extraction piece, data exploration piece, modeling piece and so on. I ensure that these pieces are separated out across multiple files.</p><p>d. Main Executable</p><p>Finally, I have a separate file(usually called main.py) running which should execute the entire code. I try to have minimal logic in this file. The objective of this file is for someone to understand the interdependencies of the different parts of the code and code flow and not the detailed logic of the code.</p><ul><li><strong>Documentation</strong></li></ul><p>In all my projects, I try to maintain a <strong>Readme page </strong>which is regularly updated when my code changes<strong>.</strong> The Readme page provides the objective of the code, instructions to install and use it, and code architecture and high-level file structure.</p><p>While the Readme is documentation focused on the code, I maintain a separate documentation explaining the statistics and machine learning logic. The objective of this is to help other Data Scientists understand my logic and algorithm.</p><p>While documentation seems like a mundane task, more than helping others, it helps me gain better clarity of my code.</p><ul><li><strong>Commenting</strong></li></ul><p>I usually add high-level comments about the code at the top of every file. In addition to giving the reader an overview of the file, this helps me organize my files better.</p><p>For every method that I have, I write <em>comments about the objective of the method, the arguments it takes </em>and<em> what it returns. </em>This again helps me split my code into appropriate methods.</p><p>In addition to these high-level comments, if I have any complicated logic in my code, I try to write some high-level comments about it.</p><ul><li><strong>Naming convention</strong></li></ul><p>Majority of the Data Science code that I have seen have variables and functions named as x, y, z etc. This probably stems from the fact that most Data Scientists come from math and statistics backgrounds. These naming conventions make the code very abstruse for anyone trying to understand it.</p><p>When I write my code, I take some time to think of the most intuitive names to give to my methods, classes, variables etc. I also ensure that there is consistency in part of speech and the letter case that I use for these code elements. For Eg, I try to use lowercase verbs for methods, camel-case nouns for classes etc.</p><p>In fact, good naming conventions reduce the need for comments and documentation.</p><ul><li><strong>Version control</strong></li></ul><p>One of the best practices of good coding is to maintain version control. There are tons of benefits from maintaining a version control system. You can seamlessly collaborate across multiple people, switch back to an older version of the code, add new changes/features to the code without affecting an older version etc. If you have not used <a href="https://git-scm.com/">Git</a> before, create a <a href="https://github.com/">Github</a> account and start by uploading your projects there. It’s free and will give you a good understanding of version control systems.</p><p>Given the fact that Data Science projects involve continuous experimentation and edits, version control becomes all the more important. Let’s say you developed a model and later added a new feature into the model. If you save these as two versions of the code, this will easily allow you to compare the performance of these two models.</p><ul><li><strong>Automated Testing</strong></li></ul><p>While Data Science may not demand exhaustive automated test cases to validate the sanity of code like the software world, having test cases to validate nuances of data is a very good practice.</p><p>In my projects, I use the <a href="https://docs.python.org/2/library/unittest.html">unittest</a> package to have automated test-cases to validate the functionality of the different parts of the code and more importantly check for the handling of potential data anomalies like null values, missing values, outliers etc.</p><p>While good coding practices ensure robustness of the module, writing good code takes a lot of time and effort. Not all projects justify this. If you are doing an analysis to be delivered in a span of a week, you do not have the luxury to beautify your code.</p><p>For this reason, I tend to think of every project into 3 phases : POC, MVP and Production as explained below:</p><ul><li>POC is where I want to get a solution to prove the feasibility of modeling</li><li>MVP is where the solution is robust enough to be used( additional features and model tuning) and</li><li>Production is where the solution is fully automated and deployed.</li></ul><p>If I’m in POC phase, I don’t tend to spend too much time on cleaning up my code but as soon as I go to the MVP stage, I make sure that my code follows good coding practices.</p><p>If you feel that there are any other practices that you follow in your data science practices, feel free to comment below! Cheers.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e9237783784c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/good-coding-practices-for-data-science-e9237783784c">Good coding practices for Data Science</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Machine Learning: How is it different than any other software program?]]></title>
            <link>https://medium.com/@ganeshn1411/machine-learning-how-is-it-different-than-any-other-software-program-a8140a75150e?source=rss-541b3eabee6f------2</link>
            <guid isPermaLink="false">https://medium.com/p/a8140a75150e</guid>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Ganesh Prasad]]></dc:creator>
            <pubDate>Fri, 16 Nov 2018 22:18:55 GMT</pubDate>
            <atom:updated>2018-11-17T02:24:24.547Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently there has been much discussion within the tech industry about machine learning. Machine Learning is termed as the brains behind the rise of Artificial Intelligence.</p><p><strong>So, what is Machine Learning?</strong></p><p>Machine Learning, as the name suggests is a technique for machines to learn. One can think of it analogous to teaching a child. Although the teaching process is different, (Humans understand concepts through words and images; Machines understand it through numbers and mathematical equations), the outcome in either scenario is the same: learning to make decisions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*yjVIv6g8-pHK-G3io83u6Q.gif" /></figure><p>You would have probably seen a machine take decision for you when you book a flight ticket, book a cab on Uber, search for an image of a dog and so on. Does that mean all of these machine learning?? Not necessarily.</p><p><strong>So, what is the difference between machine learning and any other software development?</strong></p><p>To understand this difference, let’s look at how traditional software applications work. Let’s say you want to book a flight. As you go to the website of an airline carrier, you would see a form with a set of fields like date, time, start location, end location etc. Once you fill these, the machine processes your request, checks for availability and books your flight if available. Internally, the machine has a set of predefined rules [Eg. if the location is x to y, the cost is 500$]and based on your inputs, it will take a decision [like give you cost for your flight].</p><p>Machine Learning tackles the same problem with a different approach. Imagine that you have data about a million historical flight bookings. A machine learning algorithm learns from this data to determine the price of a ticket given certain parameters like location, date, time etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/224/1*6W0gSDdgN8Z435JTN3LcIA.jpeg" /></figure><p>OK, so you may start wondering that this is just a different approach to solve a software problem.</p><p><strong>What is so different about machine learning?</strong></p><ol><li><em>Machine Learning eliminates personal bias</em></li></ol><p>The primary distinction between a machine learning approach and traditional software approach is “who makes the rules”. In traditional software, the application developer makes the set of rules. In machine learning, you let data tell you the rules. This process eliminates the personal bias when making these rules.</p><p>2.<em> Machine learning can tackle stochastic problems</em></p><p>Since software systems are rule-based, for a given input, there will be a defined output. Eg. Given flight details, it will give an exact cost. On the other hand, machine learning always gives output in terms of probabilities. Eg. Given your flight details, it might say the cost is 500$ with a probability of 0.9 and 5000$ with a probability of 0.1. I’m sure you like the exact output here. But the world is inherently stochastic. There is no set of rules that govern the way the world operates. There is inherent randomness in everything. Hence, it makes more sense to think in terms of probabilities than well-defined rules. This distinction of Machine Learning allows it to open itself to a wide set of real-world problems.</p><p><strong>What are some of the problems that machine learning can solve?</strong></p><ul><li><a href="https://en.wikipedia.org/wiki/Statistical_classification"><em>Classification</em></a><em>: </em>Identifying which category a new observation belongs to. <br>For Eg., Classifying an image as a cat or a dog.</li><li><a href="https://en.wikipedia.org/wiki/Regression_analysis"><em>Regression</em></a><em>:</em> Identifying the relationships between variables to formulate how a change in one could affect the other.<br>For Eg., What’s is the expected salary in New York for someone aged 50</li><li><a href="https://en.wikipedia.org/wiki/Forecasting"><em>Forecasting</em></a><em>:</em> Predicting the future state based on historical data<br>For Eg., Forecasting tomorrow’s weather</li><li><a href="https://en.wikipedia.org/wiki/Mathematical_optimization"><em>Optimization</em></a><em>:</em> Identifying the optimal values of entity/entities to optimize an objective.<br>For Eg., Optimizing the inventory of a restaurant to optimize revenue</li><li><a href="https://en.wikipedia.org/wiki/Cluster_analysis"><em>Clustering/ Segmentation</em></a><em>:</em> Grouping similar data into buckets<br>For Eg., Bucketing shoppers into categories based on their spend</li><li><a href="https://en.wikipedia.org/wiki/Recommender_system"><em>Recommender System</em></a><em>:</em> Customizing an offering for specific needs/interests<br>For Eg., Personalized restaurant recommendations</li><li><a href="https://en.wikipedia.org/wiki/Anomaly_detection"><em>Anomaly Detection</em></a><em>:</em> Identifying rare events in data<br>For Eg., Detecting a fraud transaction</li></ul><p>In short, Machine Learning is a different way to look at a problem and systematically solve it using data and statistics.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a8140a75150e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Let It Go]]></title>
            <link>https://medium.com/@ganeshn1411/let-it-go-64f0528096a1?source=rss-541b3eabee6f------2</link>
            <guid isPermaLink="false">https://medium.com/p/64f0528096a1</guid>
            <category><![CDATA[control]]></category>
            <category><![CDATA[life]]></category>
            <dc:creator><![CDATA[Ganesh Prasad]]></dc:creator>
            <pubDate>Sun, 23 Sep 2018 16:03:00 GMT</pubDate>
            <atom:updated>2018-09-23T16:03:00.124Z</atom:updated>
            <content:encoded><![CDATA[<p>Let me start by saying I am a control freak. I love when things are well thought of, well planned and organized. While at work, I tend to make daily, weekly and monthly to-do lists. Whenever I go grocery shopping, I plan for the meals I want to cook for a week. I always need to do my research when I travel to any new place so that I don’t end up missing anything. Anyone who has lived with me knows about my obsession with cleanliness and organizing stuff. I have probably inherited this quality of mine from my Dad. He is a fanatic when it comes to keeping things under control :P</p><p>Being in control and having a plan is not a bad thing. It helps me be more efficient in life. Whenever I set a target, I can come up with a game plan to achieve it and have timely checks for self-introspection. It helps me prioritize the different tasks to be done to ensure things are completed in a timely manner. It also helps me achieve the best quality of work I can possibly deliver. I would definitely attribute all of my success so far to my ability to plan and organize.</p><p>But for people like me who like to have things under control, life gets daunting when things do not go according to plan. When I was in grad school, I had assignments to do day and night, hunt for jobs, do my daily chores, cook my meals (which I had never done before)….The list just goes on. There were so many variables that suddenly came into my life that things started getting out of control. Whenever I tried to put up a plan, it would always fall apart. And that is when I decided that if I had to stay sane and move on in life, I had to let it go. I had to accept life as it comes and move on. And guess what, it wasn’t as bad as I thought it would be. I probably didn’t achieve everything I wanted to accomplish by the end of my grad school. But I was still happy with where I had reached. One of the biggest learnings I had from this experience is that it’s ok to make mistakes; it’s ok to fail. It’s just part of being human. In retrospective, just the fact that I gave my best in a tough scenario gives me more happiness all the things that went according to plan.</p><p>As I have moved on in my life, I am slowly starting to let go of some of the things that I controlled so much in life. I should admit, there is a thrill when things are not well planned. What is the fun when you plan everything and things go according to plan, right?</p><p>So, to all my fellow “control freak” colleagues out there, give it a try to let it go. Sometimes, life is more fun this way.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=64f0528096a1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I approach a Data Science problem]]></title>
            <link>https://medium.com/data-science/how-i-approach-a-data-science-problem-714bfbc5c68c?source=rss-541b3eabee6f------2</link>
            <guid isPermaLink="false">https://medium.com/p/714bfbc5c68c</guid>
            <category><![CDATA[approach]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Ganesh Prasad]]></dc:creator>
            <pubDate>Thu, 06 Oct 2016 16:28:07 GMT</pubDate>
            <atom:updated>2019-06-11T13:02:06.415Z</atom:updated>
            <content:encoded><![CDATA[<p>For someone who has just entered the field of data science, things can definitely get daunting. Data Science has so many techniques and procedures that can confuse anyone.</p><p>Having personally experienced this problem and having seen most of my classmates struggle with the same, I have decided to write this blog about how I typically approach a data science problem.</p><p><strong>Step 1: Translate your business problem statement into technical one</strong></p><p>Analogous to any other software problem, data science aims at solving a business problem. Most of the times, business problem statements are vague and can be interpreted in multiple ways. This occurs mostly because we generally use qualitative words in our language which cannot be directly translated into a machine readable code.</p><p>Eg. Let’s say we need to develop a solution to reduce crime rate of a city. The term “reduce” can be interpreted as:</p><ul><li>Decreasing crime rate of areas with high crime rate</li><li>Decreasing crime rate of the most common type of crime</li></ul><p>It is a good practice to circle back with the client or the business team who define the problem to decide on the right interpretation.</p><p><strong>Step 2: Decide on the supervised learning technique</strong></p><p>The end goal of almost every data science problem is usually classification or regression. Deciding the supervised technique for the problem will help you get more clarity on the business statement.</p><p>Eg. Let’s look at our problem of reducing crime rate. While the problem of reducing crime rate is more of a policy decision, depending on the choice above, we would have to decide if we need to do classification or regression.</p><ul><li>If we need to decrease crime rate of areas with high crime rate, we would need to determine the crime rate rate of an area. This is a regression problem.</li><li>If we need to decrease crime rate of most common type of crime, we would need to determine the most common type of crime in an area. This is a classification problem.</li></ul><p>Again it is a good practice to circle back with the client or the business team who define the problem requirements to clarify on the exact requirement.</p><p><strong>Step 3: Literature Survey</strong></p><p>Literature Survey is one of the most important step (and often most ignored step) to approach any problem. If you read any article about components of Data Science, you will find computer science, statistics / math and domain knowledge. As it is quite inhuman for someone to have subject expertise in all possible fields, literature survey can often help in bridging the gaps of inadequate subject expertise.</p><p>After going through existing literature related to a problem, I usually try to come up with a set of hypotheses that could form my potential set of features. Going through existing literature helps you understand existing proofs in the domain serving as a guide to take the right direction in your problem. It also helps in interpretation of the results obtained from the prediction models.</p><p>Eg. Going back to our problem of reducing crime rate, if you want to predict crime rate of an area, you would consider factors from general knowledge like demographics, neighboring areas, law enforcement rules etc. Literature survey will help you consider additional variables like climate, mode of transportation, divorce rate etc.</p><p><strong>Step 4: Data Cleaning</strong></p><p>If you speak with anyone who has spent some time in data science, they will always say that most of their time is spent on cleaning the data. Real world data is always messy. Here are a few common discrepancies in most data-sets and some techniques of how to clean them:</p><ul><li><em>Missing values</em></li></ul><p>Missing values are values that are blank in the data-set. This can be due to various reasons like value being unknown, unrecorded, confidential etc. Since the reason for a value being missing is not clear, it is hard to guess the value.</p><p>You could try different techniques to impute missing values starting with simple methods like column mean, median etc. and complex methods like using machine leaning models to estimate missing values.</p><ul><li><em>Duplicate records</em></li></ul><p>The challenge with duplicate records is identifying a record being duplicate. Duplicate records often occur while merging data from multiple sources. It could also occur due to human error. To identify duplicates, you could approximate a numeric values to certain decimal places and for text values, fuzzy matching could be a good start. Identification of duplicates could help the data engineering team to improve collection of data to prevent such errors.</p><ul><li><em>Incorrect values</em></li></ul><p>Incorrect values are mostly due to human error. For Eg. If there is a field called age and the value is 500, it is clearly wrong. Having domain knowledge of the data will help identify such values. A good technique to identify incorrect values for numerical columns could be to manually look at values beyond 3 standard deviations from the mean to check for correctness.</p><p><strong>Step 5: Feature Engineering</strong></p><p>Feature Engineering is one of the most important step in any data science problem. Good set of features might make simple models work for your data. If features are not good enough, you might need to go for complex models. Feature Engineering mostly involves:</p><ul><li><em>Removing redundant features</em></li></ul><p>If a feature is not contributing a lot to the output value or is a function of other features, you can remove the feature. There are various metrics like AIC and BIC to identify redundant features. There are built in packages to perform operations like forward selection, backward selection etc. to remove redundant features.</p><ul><li><em>Transforming a feature</em></li></ul><p>A feature might have a non linear relationship with the output column. While complex models can capture this with enough data, simple models might not be able to capture this. I usually try to visualize different functions of each column like log, inverse, quadratic, cubic etc. and choose the transformation that looks closest to a normal curve.</p><p><strong>Step 6: Data Modification</strong></p><p>Once the data is cleaned, there are a few modifications that might be needed before applying machine learning models. One of the most common modification would be scaling every column to the same range in order to give same weight to all columns. Some of the other required modifications might be data specific Eg. If output column is skewed, you might need to up-sample or down-sample.</p><p>Steps 7 through 9 are iterative.</p><p><strong>Step 7: Modelling</strong></p><p>Once I have the data ready, I usually start with trying all the standard machine learning models. If it is a classification problem, a good start will be <a href="https://en.wikipedia.org/wiki/Logistic_regression">Logistic Regression</a>, <a href="https://en.wikipedia.org/wiki/Naive_Bayes_classifier">Naive Bayes</a>, <a href="https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm">k-Nearest Neighbors</a>, <a href="https://en.wikipedia.org/wiki/Decision_tree">Decision Tree</a> etc. If it is a regression problem, you could try <a href="https://en.wikipedia.org/wiki/Linear_regression">linear regression</a>, <a href="https://en.wikipedia.org/wiki/Decision_tree">regression tree</a> etc. The reason for starting with simple models is that simple models have lesser parameters to alter. If we start with a complex model like <a href="https://en.wikipedia.org/wiki/Artificial_neural_network">Neural Network</a> or <a href="https://en.wikipedia.org/wiki/Support_vector_machine">Support Vector Machines</a>, there are so many parameters that you could change that trying all options exhaustively might be time consuming.</p><p>Each of the machine learning models make some underlying assumptions about the data. For Eg. Linear Regression / Logistic Regression assumes that the data comes from a linear combination of input parameters. Naive Bayes makes an assumption that the input parameters are independent of each other. Having the knowledge of these assumptions can help you judge the results of the different models. It is often helpful to visualize the actual vs predicted values to see these differences.</p><p><strong>Step 8: Model Comparison</strong></p><p>One of the most standard technique to evaluate different machine learning models would be through the process of <a href="https://en.wikipedia.org/wiki/Cross-validation_(statistics)">cross validation</a>. I usually choose 10-fold cross validation but you may choose the right cross validation split based on the size of the data. Cross validation basically brings out an average performance of a model. This can help eliminate choosing a model that performs good specific to the data or in other words avoid <a href="https://en.wikipedia.org/wiki/Overfitting">over-fitting</a>. It is often a good practice to randomize data before cross validation.</p><p>A good technique to compare performance of different models is <a href="https://en.wikipedia.org/wiki/Receiver_operating_characteristic">ROC curves</a>. ROC curves help you visualize performance of different models across different thresholds. While ROC curves give a holistic sense of model performance, based on the business decision, you must choose the performance metric like <a href="https://en.wikipedia.org/wiki/Accuracy_and_precision">Accuracy</a>, <a href="https://en.wikipedia.org/wiki/Sensitivity_and_specificity">True Positive Rate, False Positive Rate</a>, <a href="https://en.wikipedia.org/wiki/F1_score">F1-Score</a> etc.</p><p><strong>Step 9: Error Analysis</strong></p><p>At this point, you have tried a bunch of machine learning models and got the results. It is a good usage of time to not just look at the results like accuracy or True Positive Rate but to look at the set of data points that failed in some of the models. This will help you understand the data better and improve the models faster than trying all possible combinations of models. This is the time to try ensemble models like <a href="https://en.wikipedia.org/wiki/Random_forest">Random Forest</a>, <a href="https://en.wikipedia.org/wiki/Gradient_boosting">Gradient Boosting</a> or a meta model of your own [Eg. Decision tree + Logistic Regression]. Ensemble models are almost always guaranteed to perform better than any standard model.</p><p><strong>Step 10: Improving your best model</strong></p><p>Once I have the best model, I usually plot training vs testing accuracy [or the right metric] against the number of parameters. Usually, it is easy to check training and testing accuracy against number of data points. Basically this plot will tell you whether your model is over-fitting or under-fitting. This article <a href="http://digitheadslabnotebook.blogspot.com/2011/12/practical-advice-for-applying-machine.html">Detecting over-fitting vs under-fitting</a> explains this concept clearly.</p><p>Understanding if your model is over-fitting or under-fitting will tell you how to proceed with the next steps. If the model is over-fitting, you might consider collecting more data. If the model is under-fitting, you might consider making the models more complex. [Eg. Adding higher order terms to a linear / logistic regression]</p><p><strong>Step 11: Deploying the model</strong></p><p>Once you have your final model, you would want the model to be deployed so that it automatically predicts output for new data point without retraining. While you can derive a formula for simple models like Linear Regression, Logistic Regression, Decision Tree etc. , it is not so straight forward for complex models like SVM, Neural Networks, Random Forest etc. I’m not very familiar with other languages but Python has a library called <a href="http://scikit-learn.org/stable/modules/model_persistence.html">pickle</a> which allows you to save models and use it to predict output for new data.</p><p><strong>Step 12: Adding Feedback</strong></p><p>Usually, data for any data science problem is historical data. While this might be similar to the current data up-to a certain degree, it might not be able to capture the current trends or changes. For Eg. If you are using population as an input parameter, while population from 2015–2016 might vary slightly, if you use the model after 5 years, it might give incorrect results.</p><p>One way to deal with this problem is to keep retraining your model with additional data. This might be a good option but retraining a model might be time consuming. Also, if you have applications in which data inflow is huge, this might need to be done at regular intervals. An alternative and a better option would be to use <a href="https://en.wikipedia.org/wiki/Active_learning">active learning</a>. Active learning basically tries to use real time data as feedback and automatically update the model. The most common approaches to do this are <a href="http://sebastianruder.com/optimizing-gradient-descent/index.html#batchgradientdescent">Batch Gradient Descent</a> and <a href="https://en.wikipedia.org/wiki/Stochastic_gradient_descent">Stochastic Gradient Descent</a>. It might be appropriate to use the right approach based on the application.</p><p><strong>Concluding remarks</strong></p><p>The field of data science is really vast. People spend their lifetime researching on individual topics discussed above. As a data scientist, you would mostly have to solve business problems than researching on individual subtopics. Additionally, you will have to explain the technical process and results to business teams who might not have enough technical knowledge. Thus, while you might not need a very in-depth knowledge of every technique, you need to have enough clarity to abstract the technical process and results and explain it in business terms.</p><p>Link on LinkedIn : <a href="https://www.linkedin.com/pulse/how-i-approach-data-science-problem-ganesh-n-prasad?trk=prof-post">https://www.linkedin.com/pulse/how-i-approach-data-science-problem-ganesh-n-prasad?trk=prof-post</a></p><p><em>if you have any suggestions, please leave a comment or reach out to me at </em><a href="/service/mailto:ganeshn1411@gmail.com"><em>ganeshn1411@gmail.com</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=714bfbc5c68c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-i-approach-a-data-science-problem-714bfbc5c68c">How I approach a Data Science problem</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Is Digital India sufficient for Healthcare to use the power of Data?]]></title>
            <link>https://medium.com/@ganeshn1411/is-digital-india-sufficient-for-healthcare-to-use-the-power-of-data-31c24f2437d1?source=rss-541b3eabee6f------2</link>
            <guid isPermaLink="false">https://medium.com/p/31c24f2437d1</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[healthcare]]></category>
            <category><![CDATA[digital-india]]></category>
            <dc:creator><![CDATA[Ganesh Prasad]]></dc:creator>
            <pubDate>Thu, 02 Jun 2016 05:35:44 GMT</pubDate>
            <atom:updated>2016-06-02T05:35:44.450Z</atom:updated>
            <content:encoded><![CDATA[<p>India till date has very little data documented. The very small data that is actually documented is mostly handwritten or on paper. Keeping this in mind, the government has come up with a new initiative called “Digital India”. In this program, the existing paper records are being converted into digital form. While this is a really good initiative, the time it might take to get all the data in a structured format would take decades. Additionally, we are just talking about those documents that are in paper format. There is still a huge untapped segment where data is hardly collected.</p><p>Data can be categorized into 2 formats based on its organization:</p><ul><li>Structured data : Data is stored in tabular format</li><li>Unstructured data : Raw data Eg. Speech recording, blog etc.</li></ul><p>The focus of “Digital India” is to collect structured data. While structured data might ease storage, access and analysis, it might be too late to utilize the data if we have to wait to get it in the right format.</p><p>In this blog, I would like to talk about a few ways that could be used to complement the “Digital India” initiative, especially in the healthcare sector, in supporting collection of unstructured data.</p><h4><strong>Why does healthcare need data science??</strong></h4><p>Healthcare is one of those industries that can get a leap with the power of data. In a continuously changing world, intuition and prior knowledge alone will not be sufficient to provide quality healthcare. This is where data science comes into play, trying to understand patterns in the data and providing valuable insights.</p><p>Data science can prove valuable to almost every segment of the healthcare industry:</p><ul><li>It can help government allocate resources to combat situations like an epidemic in a particular geographic.</li><li>It can assist doctors understand similarities and differences among their patients helping them discover patterns that can assist in their diagnosis.</li><li>It can support the pharmaceutical industries in enhancing their drug discoveries.</li><li>It can aid the healthcare insurance institutes to better predict the risk of its insurers.</li></ul><p>While the pharmaceutical and the insurance industry have been trying to use data for quite sometime, the other segments of healthcare industry hardly use data to drive their decisions. And surprisingly it is the other segments (doctors and the government) that can make a real impact on the society.</p><h4><strong>What other ways can we collect data?</strong></h4><ol><li><strong>Medical diagnosis data</strong></li></ol><p>Medical diagnosis data involves the conversation of a doctor and a patient during a patient’s visit to a hospital or a clinic.</p><p><em>Why do we need this?</em></p><p>An aggregated data of the diagnoses of patients in a demographic can help identify epidemic breakouts.</p><p><em>How to collect this data?</em></p><ul><li>If a doctor has time to record the patient details and symptoms or has an assistant that can do it, that could be ideal input. But mostly in India, doctors have little time to sit and make notes of each such patient or have assistants that can do this.</li><li>An alternative is to use a speech to text converter can be used in a doctor’s clinic or a hospital that records the conversation between the doctor and the patient and convert that speech into text.</li><li>Once a textual format of the speech is obtained, a machine learning algorithm could be used to extract parts of the text and convert it into structured format. Eg. name, age, symptoms etc.</li><li>The output can be sent to the doctor on a daily basis to validate the data.</li></ul><p><em>Challenges:</em></p><ul><li>One of the biggest challenge will be Security. I order to accommodate confidentiality of data, patient de-identification can be done to associate patients with IDs.</li><li>Unlike US where every citizen has SSN which can be linked to his/her records, India does not have a unique identifier. We need to link multiple visits of the same patient in order to get correct analyses. A combination of name, age and other basic details might help or a mention of previous visit in the conversation can be used to handle this.</li></ul><p><strong>2. Medical prescription data</strong></p><p>Medical prescription data refers to the drugs that were prescribed by the doctor to the patient.</p><p><em>Why do we need this?</em></p><p>It can help the entire healthcare industry understand the drugs that work and the patient and geographic characteristics of patient population that drug worked for.</p><p><em>How can this be done?</em></p><p>An Optical character recognition(OCR) is a machine trained to understand handwriting of different people. These OCR’s can be used to learn handwriting of doctors to digitize the prescriptions.</p><p><em>Challenges:</em></p><ul><li>One of the primary challenge might be for the OCR to recognize handwriting of doctors as they are known to have worst of handwriting.</li><li>The same challenges of Security and linkage mentioned above needs to be addressed.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=31c24f2437d1" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>