<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Deena Gergis on Medium]]></title>
        <description><![CDATA[Stories by Deena Gergis on Medium]]></description>
        <link>https://medium.com/@deena-gergis?source=rss-92696979df5a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*R7HwISj6RWtM1WmN0rQ9Vw.jpeg</url>
            <title>Stories by Deena Gergis on Medium</title>
            <link>https://medium.com/@deena-gergis?source=rss-92696979df5a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 04:48:46 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@deena-gergis/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The 7 steps to choose the best topic for your #DataScience graduation project]]></title>
            <link>https://deena-gergis.medium.com/the-7-steps-to-choose-the-best-topic-for-your-datascience-graduation-project-208c1881ce14?source=rss-92696979df5a------2</link>
            <guid isPermaLink="false">https://medium.com/p/208c1881ce14</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[university]]></category>
            <category><![CDATA[study]]></category>
            <category><![CDATA[projects]]></category>
            <category><![CDATA[thesis]]></category>
            <dc:creator><![CDATA[Deena Gergis]]></dc:creator>
            <pubDate>Thu, 13 Oct 2022 19:36:15 GMT</pubDate>
            <atom:updated>2022-10-13T19:36:15.574Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XGUpzGMtZmrSE0aBa11YKQ.jpeg" /><figcaption>Photo by Jasmine Coro on Unsplash</figcaption></figure><p>So, you’ve made the very wise decision of studying Data Science a while back. You have studied a bunch of various difficult topics. And those challenges further sparked your interest to continue. And after all of the long-hours studying, shone the light at the end of this very long tunnel: Your graduation project / degree thesis</p><p>And with this bright light, comes the very common question:</p><blockquote><strong><em>“Do you have any ideas for the final thesis or project?</em></strong></blockquote><p>“ Not-so-spoiler alert: This article is your very, very detailed answer to this question .</p><p>But first things first, let’s start with the ultimate rule for this phase:</p><p><strong><em>** Attitude first, details next **</em></strong></p><p>This phase will feel somehow like the time you went shopping for your prom dress. <strong>It should have been an exciting experience, but the overwhelming number of choices for that one-time event has turned the experience into a stressful one.</strong> When deciding on a topic for your thesis or project, you will probably feel stressed. There is a multitude of topics and you might not be sure of what and how to choose, or what to consider when deciding. You might find researching all of the different topics overwhelming, and this is completely normal and expected. What I would strongly advice is to try your best to enjoy the exploration phase in itself, regardless of your final decision. <strong>Dress shopping should always be fun, so is choosing your thesis / project’s topic!</strong></p><p>And speaking about a stress-free experience, let’s start by highlighting why it is important to do enough background research before deciding on your final topic:</p><p><strong>Your graduation project / thesis will be a major opportunity to:</strong></p><ul><li><strong>A chance to discover and dig deeper in your topics of your interest:</strong> By that point, you would have studied a bunch of courses and implemented some methods here and there. Some you liked, and others not as much. This is your chance to dig deeper and gain more depth of knowledge and practical experience in the topics that you have liked.</li><li><strong>A chance to stand out from a crowd of identical CVs:</strong> For us as interviewers and hiring managers, choosing good fresh graduates is a more difficult task than choosing experienced professionals. Most of the CVs for fresh grads look very similar. And one of the main differentiating factors between one CV and another is the relevance of the graduation project / thesis. So use this chance to stand-out from the pool of identical CVs.</li></ul><p>If the decision is that important, what is the best way to decide on your topic?</p><p><strong>There are two ways to decide on your graduation project or thesis:</strong></p><ul><li><strong>A bottom-up approach: </strong>Search for “Ideas for a Data Science project or thesis” online or check <a href="https://www.kaggle.com/competitions">Kaggle</a> , and you would get numerous ideas. Those lists are very useful as an initial inspiration. However, you shouldn’t decide exclusively based on those lists, as the results tend to be generic and biased towards trending topics.</li><li><strong>A top-down approach: </strong>To get the best topic for your project, you will need to approach the problem the other way around. First, refine the paths and methodologies that interest you first, and then use the bottom-up approach to get ideas about the possible topics in those areas.</li></ul><p>So how to do this exactly? The tactics are concrete, here are 7 questions that will guide you through your decision process:</p><h3>The 7 guiding questions:</h3><p><strong><em>Part I. PATH:</em></strong></p><p>1. Which path would you like to take, a <strong>research path or an applied one</strong>?</p><p><strong><em>Part II: PILLARS:</em></strong></p><p>2. Which <strong>data type(s)</strong> are you interested in?</p><p>3. Which <strong>algorithm(s) and method(s)</strong> are you interested in?</p><p>4. Which <strong>domain(s) or function(s)</strong> are you interested in?</p><p><strong><em>Part III. PROJECT:</em></strong></p><p>5. What are the <strong>concrete topic(s) </strong>that will cover your previous preferences?</p><p>6. Which <strong>data sources(s)</strong> will you use for your project / thesis?</p><p>7. Who will <strong>supervise</strong> your project / thesis?</p><p><em>Continue reading to understand how to answer each of the previous questions and what to consider during the final decision.</em></p><h3>Part I: Choose your path: Research thesis or Applied project</h3><p>Would you like to answer a novel research question that has never been addressed before, or would you like to apply this fascinating science to develop and AI program that would address a present challenge?</p><blockquote><em>Q1. Which path: research or applied?</em></blockquote><p>If you need more details to decide, here is a comparison that will guide you through:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*oS84x21LCnzR98CM2gd92Q.png" /></figure><h3>Part II: Your Pillars — Data type, Algorithm, and Domain</h3><p>By now you have studied numerous topics in Machine Learning and Artificial Intelligence and their applications used on different kinds of data. But probably a subset has caught your deeper interest. Now its your chance to dig deeper.</p><p>In this section you will be guided to <strong>answer 3 questions</strong>:</p><ol><li>Which <strong>data type(s)</strong> are you interested in?</li><li>Which <strong>algorithm(s) and method(s) </strong>are you interested in?</li><li>Which <strong>domain(s)</strong> are you interested in?</li></ol><p>Use the following guide to refine the data type, algorithms, and domain that you are interested in. For each of the following questions, <strong>you are not restricted only by one answer</strong>, but also you can choose multiple answers. But also it is OK if you have no specific preference. And you can also have an exclusion-answer — which is anything but those methods.</p><h3>A. Data Type</h3><p>There is the standard structured data, and there are special data types that are have their own properties, techniques and challenges.</p><blockquote>Q2. Which data type(s) are you interested in?</blockquote><p><strong>Some examples of those data types are:</strong></p><ul><li><strong>Time Series</strong>: e.g. Stock data, Sales data, Heights of ocean tides, …</li><li><strong>Geographical / Spatial</strong>: e.g. Locations of earthquakes, Cell phone data, …</li><li><strong>Images</strong>: e.g. Satellite images, X-rays, Scanned documents, …</li><li><strong>Text</strong>: e.g. News articles, Twitter feeds, Product reviews, …</li><li><strong>Sound</strong>: e.g. Music, Historical speeches, Audio with different accents, …</li><li>…</li></ul><p>What if you are interested in <strong>multiple data types</strong>? You have 2 options:</p><ol><li><strong>Single data source that has both types. </strong>For example, if you are interested in images and geographical data, you might look for a topic that uses Satellite images. Or if you are interested Text and Time Series, news articles might be a good fit.</li><li><strong>Different data sources of different types. </strong>For example: If you will predict the optimal budget allocation for the marketing campaigns, you can use time-series sales data as the first data source, and text data from news articles as a second data source.</li></ol><h3>B. Algorithms &amp; Methods</h3><p>You’ve studied, studied, and studied. And now it’s your time to choose your favourites!</p><blockquote>Q3. Which algorithm(s) and method(s) are you interested in?</blockquote><p>Here is a broad overview of the major topics that you should think about when choosing your topic. Does any or multiple of those ignite your interest? Does any of those belong to your exclusion list? It is also fine if you do not have any preference.</p><ul><li><strong>Supervised learning: </strong>Classification, Regression, …</li><li><strong>Semi-supervised learning:</strong> Reinforcement learning, …</li><li><strong>Unsupervised learning:</strong> Clustering, Dimensionality reduction, Anomaly detection, …</li><li><strong>Other:</strong> Bayesian methods, Evolutionary algorithms, Optimisation algorithms, Deep learning, Generative models, Graphical models, Active learning, Transfer learning, Interpretable ML, Auto ML, Ensemble methods, …</li></ul><p>If you have chosen a research path, this is your playground! In addition to the option of digging deeper into a single method or algorithm, think also about <strong>combining multiple ones.</strong> Example? Bayesian methods for interpretable ML. That’s cool!</p><h3>C. Domain</h3><p>Data Science and Machine Learning can be applied in almost any field you can think of. And now it’s your time to pick one.</p><blockquote>Q4. Which domain(s) or function(s) are you interested in?</blockquote><p>Here are<strong> 3 sources</strong> that will help you to decide which domain to choose:</p><ol><li><strong>Prior preference:</strong> If you are <strong>passionate</strong> about a specific domain, then that’s the one.</li><li><strong>Local job boards:</strong> Understanding your local job market is very important, specifically if you are interested in landing a job directly after your studies. Start by understanding your local job market through searching for the <strong>Data Science jobs on your local job boards</strong>. Get a good understanding of who is hiring and what they are interested in. Use this information to choose your domain.</li><li><strong>Global AI in industry reviews:</strong> There are general “Data/AI maturity levels” for the various industries. Generally, industries that are digital-natives or early adopters of the digital transformation have higher adoption of AI solutions. This topic has been a research area for most of the big tech and consulting firms. <strong>Search for “state of AI in industry”</strong>, and you will get numerous reports. Get started with <a href="https://www.mckinsey.com/business-functions/quantumblack/our-insights/global-survey-the-state-of-ai-in-2021">McKinsey’s report here</a> to understand the overall maturity of the different industries, use cases and domains.</li></ol><h3>Part III. Concretise your project</h3><p>By now you should have decided on the first 2 Ps: Path and Pillars. That is, you should have chosen if you will proceed with a research or an industrial topic. You should have had an idea about your preferred data type and/or methodology and/or domain. Now it is time to <strong>concretise the details</strong> of your project / thesis.</p><p>**</p><h3>A. Finalise your topic</h3><p>Now It’s your time to research the possibilities of your preferred data type and/or methodology and/or domain.</p><blockquote>Q5. Which concrete topic will you work on?</blockquote><p>In addition to the <strong>regular search engines</strong> that will enable you to find ideas from blogs and other forums, use <a href="https://scholar.google.com/">https://scholar.google.com/</a> and look for the latest topics and developments of the 3 pillars you are interested in.</p><p><strong>Example: </strong>If you are interested in using Audio in Reinforcement learning:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zmn-it_uCtpl4NgqS0a8RA.png" /></figure><p>Spend time to read and understand what has been done, and based on those insights concretise what exactly you will be working on in your project / thesis. <strong>For example,</strong> in this case you might decide to work on an <strong>audio-based language training</strong> (speaking / listening) app that dynamically generates and assigns lessons based on the users inferred need and progress.</p><h3>B. Find your data source(s)</h3><blockquote>Q6. Which data sources(s) will you use for your project?</blockquote><p>After deciding on the concrete topic of your project, find your data sets online. You could also use <a href="https://datasetsearch.research.google.com/">https://datasetsearch.research.google.com</a> or any other specialised portal that you trust. If you are planning to use standard datasets, you might find <a href="https://paperswithcode.com/datasets">https://paperswithcode.com/datasets</a> to be very useful.</p><p>Regardless of the source of your, <strong>make sure that your data has:</strong></p><ol><li><strong>Open license: </strong>Not all of the online data-sets are free to use. Some are subject to charge, and others are subject to conditions. Before using the data, make sure that your data source has a fitting license for your usage.</li><li><strong>High quality:</strong> Before deciding on the data source that you will use, triple check the actual quality of the content. You know the rule: Garbage in — Garbage out.</li><li><strong>(Optional) Data updates:</strong> If you are developing an applied solution that is using the current weather data in a certain location, you will need as a first step batch data to train your model. Afterwards, when you proceed to deployment, you will need the same dataset in real time. As to avoid such unpleasant surprise during your deployment, make sure to check the updates frames beforehand.</li></ol><h3>C. Agree with your supervisor(s)</h3><blockquote>Q7. Who will supervise your project?</blockquote><p>Last but not least, after finalising your topic(s), you will need to find a supervisor and a mentor who will guide you through this journey. Your supervisor will play a crucial role in guiding and mentoring you through success. Make sure to choose a supervisor who is experienced and<strong> interested in your topic</strong>, and has <strong>enough time</strong> to guide you through your thesis. You might also need <strong>multiple supervisors, </strong>specifically if you have chosen a topic that is interdisciplinary. For example, if you have chosen a topic of Boosting techniques in Drug Design, you might need a supervisor who is expert in Machine Learning and another who is an expert in Pharma or Chemistry.</p><h3>Here you are, that was your guide with 7 simple questions to guide you through finding the best graduation project / degree thesis for you.</h3><h3>Don’t forget to ENJOY this process of exploration. Our field is full with incredibly exciting opportunities, so make sure you are not missing the fun!</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=208c1881ce14" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MLflow: a better way to track your models]]></title>
            <link>https://medium.com/data-science/mlflow-a-better-way-to-track-your-models-756fc03254ee?source=rss-92696979df5a------2</link>
            <guid isPermaLink="false">https://medium.com/p/756fc03254ee</guid>
            <category><![CDATA[mlflow]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Deena Gergis]]></dc:creator>
            <pubDate>Mon, 27 Sep 2021 20:15:43 GMT</pubDate>
            <atom:updated>2021-09-29T14:11:07.127Z</atom:updated>
            <content:encoded><![CDATA[<p>In a perfect world, you would get clean data that you will feed into a single machine learning model and voila, done. But fortunately, reality is much more interesting than a perfect world: a well-preforming ML model will only be ready after a bunch of trials. You will have to try different algorithms along with a bunch of hyper parameters, different variants of your training data, and maybe even different variants of your target variable.</p><p>This complex mix and match makes the challenge a fun one, but also a messy one to track. Solution? Not so spoiler alert: Comes MLflow to the rescue</p><p><strong><em>MLflow</em></strong> is a platform that helps us manage the life cycle of machine learning. It has four different modules: tracking, projects, models and registry.</p><p>In this article, we will get started with the tracking module which you can use to keep track of your different models.</p><p>The source code of this tutorial is pushed to GitHub:</p><p><a href="https://github.com/Deena-Gergis/mlflow_tracking">GitHub - Deena-Gergis/mlflow_tracking: Illustarting basic functionality of MLflow tracking</a></p><h3>Getting started in 3 minutes:</h3><p>So how could you manage your different trials in 3 steps?</p><p>After installing Mlflow on your system using `pip install mlflow` or `conda install -c conda-forge mlflow`, go ahead and follow those 3 simple steps:</p><h4>1. Train your model</h4><p>Our toy model will be very simple logistic regression that was preceded with standard scaling and a PCA. We will play around with the variance cutoff of the PCA to see which will be the best.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0e509b141f5cfb50d5d4ff00536060ad/href">https://medium.com/media/0e509b141f5cfb50d5d4ff00536060ad/href</a></iframe><h4>2. Track your model:</h4><p>Set up a new experiment for your project. For each experiment, you can track multiple models, each referred to as a run.</p><p>For each run, you can keep track of the following:</p><ul><li><strong><em>Parameters</em></strong>: What are the parameters that you have set for your model? For example the depth of you tree, your learning rate, number of estimators, etc. For our toy model, we will track the cutoff <strong>variance of the PCA</strong> as our parameter.</li><li><strong><em>Metrics</em></strong>: How well your model is preforming? You are free to choose any metric of your choice. You can also keep track of multiple metrics. For our toy model, we will track the <strong>MSE</strong> as our metric.</li><li><strong><em>Artifacts</em></strong>: This can be any file that you can export to a file. For example: Pickle files for models, CSVs for data, or even JPEGs for plots. For our toy model, we will store the <strong>sklearn pipeline as a pickle file</strong> and track it as an artifact</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7628855143abba4e7cd7a22ff03dc33b/href">https://medium.com/media/7628855143abba4e7cd7a22ff03dc33b/href</a></iframe><p>And now, <strong>change the PCA variance</strong> for each new run, and MLflow will take care of the rest.</p><h4>3. Retrieve the best:</h4><p>After training and tracking your different models, it is time to see what you have and retrieve them for further work. Simply get the ID of your experiment and query the runs.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/beb3fe6df55a3c7ff4c60fb68a7d29f3/href">https://medium.com/media/beb3fe6df55a3c7ff4c60fb68a7d29f3/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/921/1*oBQHKfvwA677gPTzdrCbGA.png" /></figure><p>Note how each run is assigned a unique identifier that you can save and use for your next run. We can also automatically select the model with the least MSE and load it back.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0f7aee1b670724c2260fb49f49f25afb/href">https://medium.com/media/0f7aee1b670724c2260fb49f49f25afb/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*eG3LsJNGv-IDobQ1Je3dhg.png" /></figure><p>And congratulations, you are now officially qualified to untangle the tracking mess if your models!</p><h3>So what’s happening behind the scenes?</h3><p>The tracking mechanism of MLflow is actually quite simple: <strong>YAML + Directory Structure.</strong></p><p>You will find a new directory created called `<a href="https://github.com/Deena-Gergis/mlflow_tracking/tree/master/mlruns">mlruns</a>`, which contains a sub-directory for each of your experiments. Each of the experiments will also contain a <a href="https://github.com/Deena-Gergis/mlflow_tracking/blob/master/mlruns/1/meta.yaml">YAML</a> file with the configuration and <a href="https://github.com/Deena-Gergis/mlflow_tracking/tree/master/mlruns/1/41e1628508fc4a2f83651ceceede6d8a">subdirectories</a>, each containing a run. And in the run’s sub directory you will find another <a href="https://github.com/Deena-Gergis/mlflow_tracking/blob/master/mlruns/1/77c97ff7b2fe46d9becba4c230dd3193/meta.yaml">YAML </a>file, along with other <a href="https://github.com/Deena-Gergis/mlflow_tracking/tree/master/mlruns/1/77c97ff7b2fe46d9becba4c230dd3193">subdirectories </a>where the parameters, metrics and artifacts are tracked.</p><h4>Other backends:</h4><p>Even though the local storage back-end that we have been using is very simple to setup and use, it is also limited in terms of multiple users and functionality. And this is why MLflow also offers multiple options for the backend, in case that you decide to adopt the platform on a wider scale. MLflow also supports <a href="https://www.mlflow.org/docs/latest/tracking.html#scenario-4-mlflow-with-remote-tracking-server-backend-and-artifact-stores">distributed architectures</a>, where the tracking server, backend store, and artifact store reside on remote hosts. And here is a sneak peak of their fancy GUI that you could also get.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*22LYLUK7ZrC7p75cjHFaiA.png" /></figure><p><strong><em>And, happy tracking!</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=756fc03254ee" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/mlflow-a-better-way-to-track-your-models-756fc03254ee">MLflow: a better way to track your models</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your 7 references to master Data Science]]></title>
            <link>https://deena-gergis.medium.com/if-you-are-starting-your-career-in-data-science-chances-are-you-are-feeling-lost-in-all-of-the-61fdf313c95d?source=rss-92696979df5a------2</link>
            <guid isPermaLink="false">https://medium.com/p/61fdf313c95d</guid>
            <category><![CDATA[resources]]></category>
            <category><![CDATA[beginners-guide]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Deena Gergis]]></dc:creator>
            <pubDate>Sun, 30 May 2021 21:45:37 GMT</pubDate>
            <atom:updated>2021-05-30T21:53:14.985Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xykuw3L4Ukyis1GkPK2oYA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@fredmarriage?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">freddie marriage</a> on <a href="https://unsplash.com/s/photos/study?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>If you are starting your career in Data Science, chances are you are feeling lost in all of the different resources that you can use. If this is the case, this article will guide you through the different resources that you will get to use through out your career in Data Science.</p><h3>1. Courses: Where it all begins</h3><p>To start your path in Data Science, you will need to study the science and skills needed through courses. And thankfully, there are currently a wide range of educational platforms that offer a wide range of courses from abstract mathematics to the analytics and engineering.</p><ul><li>For Data Science &amp; Analytics, here are the top platforms: https://www.coursera.org/, https://www.udacity.com/, https://www.datacamp.com/</li><li>For Mathematics, Probability and statistics https://www.khanacademy.org/</li></ul><p><strong><em>My recommendations:</em></strong></p><ul><li>Andrew Ng’s brilliant Machine &amp; Deep Learning courses: https://www.coursera.org/courses?query=machine%20learning%20andrew%20ng</li><li>3B1B YouTube channel for the basics of Linear Algebra and Calculus: https://www.youtube.com/channel/UCYO_jab_esuFRV4b17AJtAw</li><li>David Silver’s lecture series for Reinforcement Learning: https://deepmind.com/learning-resources/-introduction-reinforcement-learning-david-silver</li></ul><h3>2. Practical projects: Kaggle is your safe-to-try</h3><p>You can keep studying how do bikes move according to the law of physics. But this will not teach you how to ride a bike.</p><p>Similarly, our career is not only based on the science in the books and courses, but that it also has a practical &quot;craft&quot; that we have to master. And this is its beauty. Similar to learning how to ride a bike, you will need to practice the skills you have learned in the courses practically.</p><p>https://www.kaggle.com is the best place to start practicing your skills in a safe environment. There you will find projects to get you started, discussions where you can ask for help, and the code developed by professional experts that solve the same problem that you are working on.</p><p><strong><em>My advice:</em></strong> Start practicing on Kaggle as early as possible. It is never too early.</p><h3>3. Code &amp; Syntax: StackOverflow to the rescue</h3><p>Once you start building projects and writing code yourself, you will face programming challenges. Those challenges might include, how can I best preform this task, or what on earth does that cryptic error message mean. And here, https://stackoverflow.com/ is your rescuer and best friend.</p><p><strong><em>My advice: </em></strong>Extend your use for StackOverflow. Spend 30 minutes a day to browse and answer questions. This will also sharpen your programming skills and help you better understand the challenges that others face. When you answer questions, you will also get points and badges. And you can add your StackOverflow profile to your CV.</p><h3>4. Literature: Don’t reinvent the wheel</h3><p>Once you are faced with a real business problem that you need to solve, it will be important to understand what have the others done to solve similar challenges. And here you will need to refer to the literature. https://scholar.google.com/ is the go-to tool for literature browsing. Give it a try.</p><p><strong><em>My advice:</em></strong> Start your research by looking for good and comprehensive review papers and dig deeper into the relevant references afterwards.</p><h3>5. Datasets search: the high quality fuel for your Porsche</h3><p>As value-driven data scientists, we often start with the business problem and find afterwards the data that will enable us to solve this problem. Use https://datasetsearch.research.google.com/ to find publicly available data sources that you can use for your project.</p><p><strong><em>My advice:</em></strong> Make sure to conduct proper quality checks before using the data in your project. And if you are planning to put your project to production, make sure that the data will get updated and that you are able to access it via an API/ETL pipeline.</p><h3>6. Blogs: the modern newspapers</h3><p>Blogs to us are like newspapers to 20 years ago. Readers used newspapers to get informed, stay up-to-date, and also get entertained.</p><p>And here are 3 of the best data science blogs: https://towardsdatascience.com/, https://www.kdnuggets.com/, https://www.datasciencecentral.com/</p><p><strong><em>My advice:</em></strong> Make it a habit to browse through those blogs, as your parents had their habit to read newspaper with their cup of morning coffee</p><h3>7. Your colleagues: the extra eyes and brain power</h3><p>Regardless of how experienced you will get, each real-life project that you will work on will be uniquely complicated. And to best solve the business problems, you will always need a sparring partner who you can think and brainstorm with.</p><p><strong><em>My advice</em></strong>: Never underestimate the power of diverse perspectives. Look for colleagues who are different from you. The more diverse your colleagues are, the better solution you will get to reach.</p><h3>And now it is your turn: What are the other Data Science references that you recommend?</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=61fdf313c95d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 advanced Scikit-learn features that will transform the way you code]]></title>
            <link>https://medium.com/data-science/5-advanced-scikit-learn-features-that-will-transform-the-way-you-code-48262282ef0d?source=rss-92696979df5a------2</link>
            <guid isPermaLink="false">https://medium.com/p/48262282ef0d</guid>
            <category><![CDATA[sklearn]]></category>
            <category><![CDATA[predictive-analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Deena Gergis]]></dc:creator>
            <pubDate>Wed, 02 Sep 2020 18:38:19 GMT</pubDate>
            <atom:updated>2020-09-04T13:25:42.459Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4WPidStTrafB3XRRq6hXMA.jpeg" /><figcaption><em>Photo by </em><a href="https://unsplash.com/@dallimonti?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText"><em>Andrés Dallimonti</em></a><em> on </em><a href="https://unsplash.com/s/photos/cockpit?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText"><em>Unsplash</em></a></figcaption></figure><p>Very few packages have been successful in achieving what sklearn reached. It is not only that they provide almost all of the commonly used <strong>ML algorithms</strong>, it is also how they provide those algorithms.The core code of Sklearn is written in <strong>Cython </strong>granting optimized performance. Their <strong>API</strong> has been designed to provide consistency, readability and extensibility. And on top of the core ML algorithms, sklearn provides you additional functionalities that for creating <strong>end-to-end pipelines</strong>. If there is a single adjective that could describe this package, it should be “<strong><em>Beautiful</em></strong><em>”.</em></p><p>If you have ever worked with sklearn, you will probably be familiar with the common methods such as fit, predict and transform. Maybe you will be familiar with a couple of other preprocessing methods as well. But the power of this package goes way beyond the commonly used functionalities.</p><p><strong>The goal of this article is to</strong> <strong>highlight some of the very powerful and less known features of sklearn</strong>. Those features will enable you to unleash the maximum potential available in sklearn. You will get a quick glimpse of what those features are and how you can use them. A very short code snippet will be provided, followed by a reference for more details. The purpose of the code snippet is to illustrate the <strong>functionality and syntax only</strong>. The snippets do not represent the complete workflow. And finally, <em>0.22.1 </em>was used.</p><h3>1. Pipelines</h3><p>Your models will always consist of <strong>multiple sequential phases</strong>, where the output of one phase will be the input of the next. For example, a classifier for high dimensional input would typically include normalisation, dimensionality reduction and the classification model.</p><p>Sklearn’s pipelines provide an <strong>elegant wrapper</strong> for chaining those sequential steps. When you will use pipelines, you will not have to worry about managing the intermediate objects. All you will need to do is to specify the steps and call the fitmethod. When persisting your model, you only need to pickle one object, which is the pipeline. Using the pipeline will improve your code’s readability, decrease bugs and ease the persistence of your trained model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/48721d5b00fbda96f024c0ce2d983cd5/href">https://medium.com/media/48721d5b00fbda96f024c0ce2d983cd5/href</a></iframe><p><a href="https://scikit-learn.org/stable/modules/compose.html">Read more</a></p><h3>2. Inline target transformers</h3><p>For some cases, you could strongly benefit from a <strong>non-linear transformation</strong> of your target before training your model. For example, a log transformation for a heavy-tailed target is usually a very wise step. When using the model to predict new data, you need to make sure that you <strong>inverse this transformation </strong>for the predictions.</p><p>Here is some good new for you: you do not need to use Pandas or Numpy to create those transformations. You can <strong>use Sklearn</strong> to apply target transformations directly as illustrated below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dd4d1bf278762d1ad3bbd73712072046/href">https://medium.com/media/dd4d1bf278762d1ad3bbd73712072046/href</a></iframe><p><em>The following will automatically happen under the hood:</em><strong><em> </em></strong><em>While training: </em>regressor.fit(X,func(y))<em>and while predicting:</em> inverse_func(regressor.predict(X))</p><p><a href="https://scikit-learn.org/stable/modules/generated/sklearn.compose.TransformedTargetRegressor.html">Read more</a></p><h3>3. Feature Union</h3><p>Even with the sequential steps sklearn offers, you will not be limited by only one transformer per step. You can use <strong>multiple transformers</strong>, and <strong>concatenate</strong> the results in a single step.</p><p>In the Pipeline example above, we have used a single PCA to transform the normalized data before training. Let’s take an example where we want to use a kernel PCA, in addition to the Linear PCA:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9b7279d9238a9761676d39c1edd13850/href">https://medium.com/media/9b7279d9238a9761676d39c1edd13850/href</a></iframe><p>You can also plug-in the feature unions in the pipeline. And of course, you can also write your own custom transformers.</p><p><a href="https://scikit-learn.org/0.18/auto_examples/hetero_feature_union.html">Read more</a></p><h3>4. Chaining models for rolling predictions</h3><p>Sometimes you will face a situation where you will need to chain multiple models, such that the <strong>output of the first model is the input of the second model</strong>. A very common use case for such chaining is in time series models: if we need to predict two timesteps, the prediction of y(t+1)will be an input for predicting at y(t+2) .</p><p>With sklearn, you have the option to create that chaining automatically. The functionality is eitherRegressorChain or ClassifierChain. Your y will not be an array, but rather a matrix that contains the multiple dependent targets. The RegressorChain will automatically include the <strong>previous target to predict the next one</strong>. And during predictions, the chain will predict the next target based on the predictions of the previous one. All you need to do is to use the fit and predict methods as usual.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b12a698de73f6ca0c1e278ca430a81a2/href">https://medium.com/media/b12a698de73f6ca0c1e278ca430a81a2/href</a></iframe><p><a href="https://machinelearningmastery.com/multi-output-regression-models-with-python/">Read more</a></p><h3>5. Feature importance using permutation</h3><p>Feature importance is usually one of the most important modelling insights we can have and present to the end user. But, depending on the algorithm that you are using, it is not always straight-forward to get them.</p><p>Permutation can be used to <strong>infer the importance</strong> of each of the features, regardless of the modelling method. The core idea behind is very intuitive: a single feature is randomly shuffled, and the decrease in the model score is quantified. The higher the change, the more important the feature.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c56ed6ce243f6110f192df8a36c4e850/href">https://medium.com/media/c56ed6ce243f6110f192df8a36c4e850/href</a></iframe><p><a href="https://scikit-learn.org/stable/modules/permutation_importance.html">Read more</a></p><blockquote><strong>Now it’s your turn: Share one other advanced method in Sklearn that has made your code better. Write this method in the comments below</strong></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=48262282ef0d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/5-advanced-scikit-learn-features-that-will-transform-the-way-you-code-48262282ef0d">5 advanced Scikit-learn features that will transform the way you code</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Real-life AI: Your guide to defining the requirements]]></title>
            <link>https://deena-gergis.medium.com/real-life-ai-your-guide-to-defining-the-requirements-3f583c67225b?source=rss-92696979df5a------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f583c67225b</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[product]]></category>
            <dc:creator><![CDATA[Deena Gergis]]></dc:creator>
            <pubDate>Sat, 22 Aug 2020 10:27:48 GMT</pubDate>
            <atom:updated>2020-08-23T07:10:30.636Z</atom:updated>
            <content:encoded><![CDATA[<h3>Real-life AI: Your guide to building a useable product</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*KnsE5JonKtUVockGnRneTA.png" /></figure><p>What is the difference between <strong>academic and commercial</strong> Data Science? Academic Data Science is like competing in a Formula 1 race. Your driving skills has to be top notch and your car must be flawless. But also you do not have to think about where your <strong>destination</strong> should be, which <strong>route</strong> should you choose or pedestrians <strong>jaywalking</strong>.</p><p><strong><em>So, welcome to commercial Data Science: the everyday driving in the crowded city.</em></strong></p><p>It was at an incredible startup where I kick-started my industrial career in Data Science 6 years ago. That was my first industrial job after 4 years in academia and research. When I decided to switch career paths, I simply thought that it will only be a domain shift. After all <strong>both are data science</strong>, right? Boy, was I naive!</p><p>After reflecting on the “<a href="https://www.linkedin.com/pulse/5-things-i-wish-knew-real-life-ai-deena-gergis/">5 things I wish I have known about real-life AI</a>”, one of the most important point that I had to learn was that “doing the right thing != doing something right”. Given the feedback and comments that I have received, here is a dedicated article to to provide you with a practical guide of<strong> “doing the right things” in commercial AI projects.</strong></p><h3>The leaking kettle</h3><p><strong>Here is a question for you: If you would be creating a brand new kettle, what are the features that you will include?</strong></p><p>Before moving to Düsseldorf, I decided to sell the furniture to the next tenant and start fresh in the new city. When I moved in to a new flat with basically nothing but my clothes and an air mattress, I knew that furnishing the place will take sometime. But who cares, fresh starts have their own beauty. After a good first night sleep, I went to the kitchen to prepare my coffee. I searched in the boxes for the kettle. Unfortunately, the next tenant was enjoying her morning coffee using it. I left the kettle back in Bonn. So I went to one of those hypermarkets, grabbed my groceries and a random kettle and went back home.</p><p>After using the new kettle for a couple of time, I noticed that the countertop is wet. A couple of days later, I realised that the new kettle leaks each time the water inside exceeds a certain limit. But living in a completely empty flat and having to use my phone as my light source did not exactly prioritise the kettle to the top of my todos.</p><p>So, as a temporary plan B, I just have to be careful how much water I put inside. Well, no, I also have to be careful not to place the kettle on its electric base after using it. Well, maybe be on the safe side and unplug the kettle when not in use. Oh, did I unplug the kettle today before I leave or would I to back to find my place burned down?</p><p><strong>Now back to the original question: If you would be creating a brand new kettle, what are the features that you will include? </strong>If you are like most Data Scientists, speed and energy efficiency of the kettle would have been your list. And that is it. The problem here is, even though the speed and efficiency are indeed very important, your customer would much rather choose a mediocre kettle that doesn’t leak, rather than a very efficient kettle that leaks.</p><h3>Commercial AI: the 3 things that you need to get right</h3><p>Here is a fact that we all know, a high percentage of commercial Data Science projects fail. And most of the time the reason behind the failure is not that the model’s accuracy is too low. Despite that the accuracy is crucially important for the success of your product, <strong>the accuracy is not the only factor</strong> that determines your projects success.</p><p>So what are the properties that you need to create to have a successful AI project?</p><blockquote><strong>1. The core features:</strong> Your purpose and final destination</blockquote><blockquote><strong>2. The common-sense features: </strong>The unspoken, no-brainer and trust-breakers</blockquote><blockquote><strong>3. The relatable features:</strong> From good to great</blockquote><h3>1. The core features: your purpose and final destination</h3><p><em>What should a kettle do? It should heat the water up.</em></p><h4><strong><em>What are the core features?</em></strong></h4><p>Those are the main goal of your project. Out of the 3 sets of features, the core features are the most obvious, and the easiest one to define.</p><h4><strong><em>How to define your core features?</em></strong></h4><p>A key to a thorough definition of the core features is asking your customer <strong>detailed questions</strong> about what they need from the system. Break down their requirements to a set of detailed list. <strong>For example</strong>, your customer has requested you to build an Email classifier system. Understand how would the actions offer from one class to another. Ask them about if there is a hierarchy to the classification. Enquire about the different departments handling the various classes. Clarify if they expect that one email could belong to multiple classes. Understand the importance and sensitivity of some labels over the others.</p><h4><strong><em>Why to granularly define your core features?</em></strong></h4><p>Each of those questions would change how you translate the business problem to a machine learning model. For example: Would it a single or a multi-class problem? Should you implement a custom loss function to balance the different accuracy vs precision expectations for the different classes? How should the threshold be set, below which you would not assign any predictions?</p><p><em>You see, the more you understand the </em><strong><em>details of the system</em></strong><em>, the better you would be able to design your model. So make sure you ask as many detailed questions as you can.</em></p><h3>2. The common-sense features: the unspoken, no-brainer, trust-breakers</h3><p><em>If someone would ask you: how would your dream kettle look like? Would you ever answer with: “A kettle that doesn’t leak.”? On the other hand, would you use a kettle that leaks?</em></p><h4><strong>What are the common-sense features?</strong></h4><p>Those are the features that ensure that your system is working in a <strong>controlled logical manner</strong>.</p><p>For example: you are building a dynamic pricing system for an e-commerce website. Your goal is to build a system that assigns the optimal prices for the items as to maximise the final revenue. Would your system assign a negative price for an item? Could the total price of 3 items be less than a single one? Are the assigned prices stable enough, or would they change by a factor of 3 when the user hits refresh? Do you see how those features are not directly related to the accuracy of your model?</p><h4><strong>Why should you include those features?</strong></h4><p><strong>Your customers assess what they don’t understand based on what they do understand</strong>. If the basic logic of your system is flawed, why should they trust that your complex model isn’t flawed as well? As a customer yourself, would you every buy an oven from the same brand that created that leaky kettle?</p><h4><strong>How to define those features?</strong></h4><p>As intuitive and common-sense those features are, as <strong>challenging</strong> as they are to define. Most of the time, your customer will perceive those features as too obvious to be discussed or explicitly mention. And let’s face it: it is boring to properly define and implement your ‘anti-leakage’ features. Indeed, if there is an Emmy’s prize for the most <strong>boring</strong> task that could be ever be done, it will go to the anti leakage features. But remember, if there is a Grammy award for the <strong>trust-breakers</strong>, it will also go to those features. So, here are two practical tips that will help you in defining the common-sense features :</p><p><strong>1. Shadow your user:</strong> If possible, grab a chair and observe your user in their day-to-day job for some time. Focus on what they do in their “auto-pilot” mode; the things they even won’t explain because it is too obvious. Write those down and aim for the most extensive, comprehensive and boring list you can come up. It will pay off at the end.</p><p><strong>2. Use your system as your user:</strong> Use the system as your customer would do, before your use does. If their auto-pilot is indeed a no-brainer, you will also see if the basic logic is broken.</p><p><strong><em>Spoiler-alert:</em></strong><em> if your customer opts-in for a very primitive solution ratherthan your cutting-edge AI system, check if your system needs some basic “anti-leakage” functionality. And do not underestimate the effort it would take to build back the trust.</em></p><p>_______</p><h3>3. The relatable features: from good to great</h3><p><em>How does my dream kettle look like? It is a kettle that can listen to my alarm clock and knows my non-optimal snoozing habit, so it will start heating the water early on, for me to wake up to find the water boiling and ready for my coffee.</em></p><p><strong>Objectively speaking</strong>: this feature will save me 1 minute a day. And given my non-optimal snoozing habit, this feature will also reduce the energy efficiency of the kettle. So — objectively speaking — my dream kettle is not better, if not worse, than a regular kettle. But <strong>subjectively speaking</strong>: I am not after saving 1 minute of my time, but rather a <strong>feeling of connection</strong> with the smart appliance I own.</p><h4><strong>What are the relatable features?</strong></h4><p>The relatable features are the non-core features that extend your main goal to create a more relatable and likeable system.</p><p>Taking the example of the dynamic pricing for an e-commerce: An attachment feature could be an ad hoc analysis that shows which products that have the highest price changes from the manual process to the AI one. Maybe you would find out that there is a supplier providing forgeries rather than original brands. Or maybe you would see items that were super trendy a couple of years ago, converse shoes for example, that are still priced in the same range as it was, despite the lower demand. Maybe the black dresses are assigned a higher price, as those are more popular with the end-users more than the pricing experts expected? Do you see how those questions would create that feeling of connection between your customer and your model?</p><h4><strong>Why should you include relatable features?</strong></h4><p>Presenting such results to your customer will <strong>transform the mere Euros </strong>that your system is assigns to the items, to a relatable, understandable and likeable system that your customer trusts. And when your customer likes and trusts your system, they will much more likely use it and integrate it into their daily business.</p><h4><strong>How to define those features?</strong></h4><p>Similar to the common-sense features, the relatable features are usually not discussed. And most of the times neither you nor your customer would be aware what those would look like. You need to be creative. And here are <strong>2 guidelines</strong> that might inspire you:</p><p><strong>1. Think mirrors:</strong><em> </em>Your customers have assumptions how their business and processes work. And often times those assumptions are different from the data. So, use the data at hand as a mirror: <strong>validate and extend those assumptions</strong>. Do their expectations truly meet the reality? Extend and correct their knowledge. Not only would such analysis deepen your customer’s understanding of their own business, but they will also <strong>build trust </strong>in your black-box models. Remember: customers assess what they don’t understand based on what they do understand.</p><p><strong>2. Think emotions:</strong><em> </em>Working with data is a job that is filled with emotions and surprises. How many times did finding out a hidden pattern excite you? We all do know the frustrations with the data quality. That anticipation that we feel when the system goes live? Or the happiness when the results exceed the acceptance criteria? Don’t keep those emotions to yourself, but <strong>carry them over to your customer.</strong> As analytical people, we usually miss out on utilising those emotions. But emotions are the<strong> root of connection</strong>. So share those emotions with your customer, it will deepen their connection to your AI system, a level deeper the pragmatism and intellect.</p><h3>To conclude:</h3><blockquote>It is completely natural for us as data scientist to focus on the model’s accuracy. But remember, even though the accuracy is crucial, it is not the only factor determining the success of your AI project. You need to design a system that accurately addresses the core task, behaves in a logicallycontrolled manner, and a product that your customer could relate to.</blockquote><p><strong>And now it is your turn: which one of the previous sets of features do you feel that they are the most underrated? Leave your thoughts in the comments below</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f583c67225b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 Things I Wish I Knew About Real-Life AI]]></title>
            <link>https://medium.com/swlh/5-things-i-wish-i-knew-about-real-life-ai-84eecb369f4d?source=rss-92696979df5a------2</link>
            <guid isPermaLink="false">https://medium.com/p/84eecb369f4d</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[product]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Deena Gergis]]></dc:creator>
            <pubDate>Wed, 12 Aug 2020 19:01:19 GMT</pubDate>
            <atom:updated>2021-01-31T15:59:25.268Z</atom:updated>
            <content:encoded><![CDATA[<p>It was at an incredible startup where I kick-started my industrial career in Data Science 6 years ago. That was my first industrial job after 4 years in academia and research. When I decided to switch career paths, I simply thought that it will only be a domain shift. During academia, my focus was on biomedical data science and in the startup my focus will be on using data science for building automation. So I thought that the shift from academia to industry would only change the domain —<strong> after all both are data science</strong>, right? Boy, was I naive!</p><p>Looking back, it is true that the technical skillset in academia and industry is very similar, but the mindset is vastly different. Yes, both academia and industry careers are data science focused, but the goals are drastically different. The goal of academia is to create a <strong>state-of-the-art solution</strong> to address novel problems, with the ultimate hope of getting a paper published, but the goal of industry is to <strong>generate revenue </strong>and have happy customers, with the ultimate hope of having a functioning software.</p><p>Data Science is like a knife that could be used to <strong>cut cake and steak</strong>. If you’ve been cutting steak your whole life, probably you will end up breaking the plate the first time you cut a cake. And if you have been cutting cake your whole life, cutting your first piece of steak will be a quite frustrating task. So, if you are in this transitioning state, <strong>the following five points</strong> might prepare you to what you could expect and save you a significant amount of frustration.</p><h3>1. It is all about the delivered value, not the method</h3><p>Let me break the news: no one cares about how deep your neural network is or if you are using the latest transformer, no one cares if you are using decision tree or a gradient boosting machine. <strong>Customers care about the accuracy, stability and usability</strong> of the AI system you will create, regardless of the method you are using for your machine learning model.</p><p>This might be surprising for people with research focus, since the main focus in academia is creating the state-of-the-art methods to solve novel problems. No paper will get published for using a vanilla logistic regression. But remember, business ultimately hopes for a <strong>functional smart system</strong>, and not a sophisticated scientific paper.</p><p><em>Take a minute to remember the most delicious dish you’ve had in a restaurant. Does it really matter if the chef used 30 ingredients or 5?</em></p><h3>2. Packaging does matter</h3><p>As analytical people, it is our job to <strong>unwrap the bright and shiny sales pitches</strong> and get our hands dirty in the data and facts, asking the right questions and finding out the true answers from the data. And excelling in this ability — to challenge the presented pitch and dig up the real data — is what makes the data science superstars.</p><p>It is very unnatural for most of us to <strong>reverse that pattern</strong>: to rewrap the data into a story line, to visualise the data, the model and the predictions in a user-appealing way and wrap all of that up with a smooth narrative. As technical people, we care about the “how”, and often times it is unnatural to wrap our technical how, with the user-relevant “what” and “why”. But this unnatural reverse of the skill is — ironically — a key to users’ adoption of your system.</p><p>See, <strong>your product is like a Porsche </strong>that you go to check out. You walk into the store, and to your surprise, you were given the keys by an uninterested sales agent who directly walked away, only after a couple of seconds you find the interiors stained. Even though that Porsche is functioning as well as a bright and shiny one, the lack of sales interest and the stained interiors will significantly take its value down. <strong>You would not say that the motor is working just as fine</strong>, would you?</p><p>We are creating <strong>innovative and high-end expensive tech</strong> and the packaging should present our work as such. <em>Don’t throw the keys to the user and walk away, expecting them to see that the “motor is working just fine.”</em></p><h3>3. Doing the right things != doing things right</h3><p>As Data Scientists, we are wired to minimise a loss function and maximise the area under the ROC curve. And that is it. The problem here is: you might be perfectly predicting a KPI, that no one cares about. Your model could be indeed <strong>pinpoint accurate</strong>, and your deployment is rock stable, but your output is completely <strong>irrelevant to the user</strong>, and that would be doing the wrong thing right.</p><p>To succeed in industrial AI, you need to address the pain points of your customer, and address it well. You need to be dead sure that you are doing the right thing: that the scope of the problem you want to solve will address an actual need of your customer, and that you are <strong>solving that exact user-relevant need well</strong>.</p><p>Let’s quantify this principle: your end <strong>success metric is the product of two percentages</strong>: how relevant the problem is, and how well you solve a problem. A solution that perfectly solves a completely irrelevant problem would have a score of <strong>0% x 100%,</strong> which is equal to 0%. And a solution that doesn’t solve a very critical problem would score <strong>100% x 0%</strong>, which is still equal to 0%. Aim for an excellent solution that addresses the actual user-relevant problem to score the final 100%.</p><p><strong>Just remember:</strong> don’t bake that uniquely delicious walnut cheesecake, that no one can bake as well as you do, for your friend who is allergic to nuts. You would be doing the wrong thing, regardless of how “right” you do it.</p><p><em>If you are interested in digging deeper on what are the right things to do, check out this break-out article: </em><a href="https://www.linkedin.com/pulse/real-life-data-science-your-guide-doing-right-things-deena-gergis/"><em>Real-life Data Science: Your guide to doing the right things</em></a></p><h3>4. Set realistic expectations</h3><p>Here is a fun experiment for you: ask random people around you <strong>what does artificial intelligence mean</strong>. Chances are that you would be getting answers like “a robot that will bring my coffee”, “killer robots that will take over the world”, to “A what?”. Here is the follow-up experiment: ask the same people if they can name 3 apps on their smart phonethat use AI. Chances are that a good ratio would be shocked that “there is AI on their smart phone.”</p><p>Fact is: lots of people still <strong>associate AI with science fiction movies</strong>, and as a result unrealistic expectations are always on the table. And those unrealistic expectations usually cause major disappointment for everyone who is involved. But where to start and how to debunk those unrealistic expectations? Here are 3 points to get you started:</p><ol><li><strong><em>Ask your customer</em></strong> about their expectations and acceptance criteria before the project starts. <strong>Concretise, quantify and minimise</strong> those as much as possible. This will help you and your customer reach a common concise of how a realistic successful AI system should look like.</li><li><strong><em>Explain the challenges</em></strong> that you are facing. “The market dynamics is very unstable due to COVID-19, and we doubt that the forecast would be accurate enough.” This is what we have <strong>communicated transparently</strong> to our customers. And guess what, our beloved customers were beyond understanding and supportive.</li><li><strong><em>Work in smaller iterations </em></strong>by setting small goals and test phase in between. Why? it is inevitable that we might fail. It is not a question if <strong>we might fail</strong>, but rather how to fail. Avoid the deadly mistake of keeping your customer waiting for months, only to surprise them at the end with a “sorry, it didn’t work out.”</li></ol><p><strong>What will happen if you ignore the unrealistic expectations and jump directly to hyper-tuning?</strong> Your customer will be disappointed that you did not deliver the crystal ball that they were expecting. The business will be disappointed that they are not getting any return of investment on the significant amount of money they are putting in data science. And you will get disappointed that your efforts are not appreciated and your well functioning system is ending up gathering dust on Git.</p><h3>5. Keep humans in the loop</h3><p>Ending with the point is the closest to my heart: I am here to help my customers succeed and reach their true potential by automating repetitive tasks and supporting them in their daily job. My job is much more about those people rather than modelling. And my <strong>customers are my partners</strong>, not just end-users. So for me, this point became more than just a learning, it became my professional purpose.</p><p>But even if you have a different purpose, it will still benefit you to keep humans in the loop to <strong>increase your system’s performance</strong> and the final value you will deliver. How and why?</p><ol><li><strong><em>Use their domain knowledge: </em></strong>Your customers know their day-to-day job much better than you will ever do. Avoid the mistake of directly jumping to “how can I connect to your database”, but rather take the time and use their extensive domain knowledge and work experience to guide you through the development. What you might spend days trying to understand on your own, your customers and domain experts could explain it to you in minutes.</li><li><strong><em>Exploit their feedback:</em></strong> Do you know those thumbs up and down icons that you have in Netflix? That is your valuable user feedback which Netflix uses to continue improving their recommendations. Create a feedback mechanism in your system, that will generate even more data to retrain and fine-tune your model. You will come to discover interesting and solvable edge cases that you could work on for better performance.</li></ol><p><strong><em>Now it’s your turn: how do those 5 learnings resonate with your daily job? What are other learnings that you have gathered while working in industrial AI? Write those in the comments below</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=84eecb369f4d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/5-things-i-wish-i-knew-about-real-life-ai-84eecb369f4d">5 Things I Wish I Knew About Real-Life AI</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>