<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kemal Tugrul on Medium]]></title>
        <description><![CDATA[Stories by Kemal Tugrul on Medium]]></description>
        <link>https://medium.com/@kemal.tugrul?source=rss-8fd17f72334a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*j1Wdg759OTNBWJU49Z8vBA.jpeg</url>
            <title>Stories by Kemal Tugrul on Medium</title>
            <link>https://medium.com/@kemal.tugrul?source=rss-8fd17f72334a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 23:37:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kemal.tugrul/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Static Machine Learning Models in a Dynamic World]]></title>
            <link>https://medium.com/data-science/static-machine-learning-models-in-a-dynamic-world-ff1ea1b0892c?source=rss-8fd17f72334a------2</link>
            <guid isPermaLink="false">https://medium.com/p/ff1ea1b0892c</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[tds-narrated]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Kemal Tugrul]]></dc:creator>
            <pubDate>Fri, 06 Mar 2020 04:13:05 GMT</pubDate>
            <atom:updated>2020-03-09T17:36:44.430Z</atom:updated>
            <content:encoded><![CDATA[<h4>Historic data is being used for the models that will work in the future. Our models stay static while the world changes constantly.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZkLtfFSKi53lM3fOV82idA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@chrislawton?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Chris Lawton</a> on <a href="https://unsplash.com/s/photos/change?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>If you work in academia, or in industry, you work on real-life problems. The data we work on are not synthetic, they are from the readings from the real world. While working in our offices, abstracted from the real-life events happening related to our projects, businesses, or challenges; we tend to lose the connection with the problems we are solving. It is easy to see the projects as numbers. Static numeric values, strings of characters, and meaningless IDs. We store them in binary, disconnected from the real world. In this setting, it is easy to forget that the real world is dynamic, and so are the data. The problem starts to arise when we neglect, or naively forget about this reality, and focus on our beloved KPIs. This story is about how our machine learning models stay static, while the data change day by day with the dynamic world.</p><p>After I had an incident in one of the previous companies I worked for, I started to become a little bit obsessive about the continuity of my models’ performance. In short, one of our models that was live and directly responsible for the output of a critical engine started to predict absurd results. The worst thing was that we did not discover something was wrong until we got angry phone calls from the client and spent long days to debug the entire pipeline before finding the root cause. The model was outputting extremely high or extremely low predictions. In retrospect, we found out that there were no faulty components in our stack. No bugs, no errors. The problem caused by the upstream data. The data that were consumed from upstream, the real-world readings, had changed.</p><p><strong>Models are Built by the Snapshot of the World</strong></p><p>If you are not employing some kind of online learning where models update themselves at regular intervals, you work on historic data. You consume some data from your data store and train your models. You constantly do this. You fetch the historic data, move it through the pipeline, get your model. The keyword here is historic. You work on data that are already old. You are building your solution for the future while looking at the past.</p><p>With your data, you actually develop your solution on the snapshot of the world. The raw data, the real world readings represent the snapshot. This snapshot can be weeks, months, or years. The longer the snapshot the better. Still, this gives you no guarantee that the future will follow the same behavior that was in your snapshot.</p><p><strong>Models Fail When the World Changes</strong></p><p>When you work with historic data, you sign a contract with your model. You promise your model that the properties of the data will not change. In exchange, the model gives you an accurate description of history. You rely on your KPIs. Celebrate when you exceed a threshold. Push your solution into production. While the same model is still up there, your contract is still binding.</p><p>The real-world changes constantly, and so do our upstream data. If you are not taking an additional step, your models will be ignorant of the changes. Your contract is still there. Following a change, when a major event happens, or slowly over time, your upstream data start to change. It may change so drastically that the upstream data may not resemble the historic data your model was trained on anymore at some point. At this point, you violate your contract, and your model starts to fail silently.</p><p><strong>Continuity of the Performance</strong></p><p>As the world change, and so does our data, we need to be sure that our model is still doing a good job. Even though we may not do anything to put the model on the right path, we need to know the performance. To make sure my models’ performances are still satisfactory, I tend to put the following checks. Mind that these checks are in the service of the model/solution you are serving.</p><p><strong>1. Upstream Data Checks</strong></p><p>For this one, you define which stage is upstream depending on your project. The idea is to observe the data at any stage before entering the model to make a prediction. While this is the case, probably the last stage of the data before they are fed into the model is a good stage.</p><p>The idea in this check is to make sure the upstream data still resembles the distribution, or follow the same pattern, of the historic data used while training the model. A simple but powerful way could be calculating the distribution of the upstream data at every fixed time interval, and comparing it with the historic data’s distribution. If the similarity between the two distributions is close, you are green. Otherwise, you raise an alert, and let the owner of the solution know.</p><p><strong>2. Downstream Output Checks</strong></p><p>The idea behind the downstream check is very similar to the upstream data checks. Instead of checking the upstream, in this one, you check the output of your model or your service. Like the upstream check, you monitor the similarity between the distribution of the outputs of your model in production and the output of your model when fed with historic data. Monitor the similarity between two distributions, raise an alert if it starts to drift away.</p><p>These two checks are simple to implement. You can put them in a real-time monitoring dashboard, or just send a mail to the owner of the repository when a threshold is exceeded. Like with any checks, it is good to know that they are there.</p><p>If you are curious about the end of the incident I mentioned at the start of this story, let me continue. The change came from the client-side. Our readings were dependent on a complex and long list of configurations of frameworks that were running on the client-side. It was later apparent that the client changed some major configurations. This change was so impactful to our readings, it resulted in major shifts in the upstream data. As a result, the new data points fed into the model were completely different than the data we used for training. Only if we had some checks on the data, we could be aware of that change and could take some actions to put things in the right direction. Luckily this incident did not impact the business much and everything was back to normal in a couple of days. This valuable lesson made me a little more paranoid about the static nature of our models, and if my solutions still work as I expected in the production.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ff1ea1b0892c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/static-machine-learning-models-in-a-dynamic-world-ff1ea1b0892c">Static Machine Learning Models in a Dynamic World</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Crash Your Data Science Code Before It is Too Late]]></title>
            <link>https://medium.com/data-science/crash-your-data-science-code-before-it-is-too-late-d4697c836a1d?source=rss-8fd17f72334a------2</link>
            <guid isPermaLink="false">https://medium.com/p/d4697c836a1d</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Kemal Tugrul]]></dc:creator>
            <pubDate>Wed, 07 Nov 2018 18:35:54 GMT</pubDate>
            <atom:updated>2018-11-07T21:44:35.650Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*94hSQw1yT0coBlpnEGP-LQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/fzjWFym9jjE?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Dlanor S</a> on <a href="https://unsplash.com/search/photos/warning?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>How do you feel when your script crashes? If the script is in production, you have a problem. Even if it is not in production, it is not the best feeling on earth. However it may make you feel, a crashing script can be a great thing for you. A script crashed in correct place and time can save you tremendous time and effort.</p><p>As we code, we constantly make assumptions. Those assumptions are our expectations from the data, how the code will be used, and more. Sometimes we make assumptions willingly. Other times, we do not even realize when we make them. Assumptions are everywhere in code. Even dividing a number with a variable contains assumptions. Assumptions can save your code, or break it in the most unexpected place and time. Assumptions are inevitable, but can be tested against.</p><p>Let me present you how a simplest division can be disastrous. In this example, we will divide 1 with a variable (1/var). If the variable’s value is zero, you will get an error. While it is true for most cases, some libraries like numpy allow it. In those cases, the expression will result with infinite value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fKiQ0m9b32AKzB9xWe_mCQ.png" /><figcaption>While python (std) gives an error, numpy results a division by 0 with infinity.</figcaption></figure><p>It is better if code throws an error than keep going with erroneous data. It is because if your code stops there, you will immediately know that there is a problem. Moreover, you will know the exact place of the issue. If it does not stop, your code will go on and continue executing with corrupted/erroneous data. Depending on the code, the issue may spread like a disease. It may update data storage with corrupted data or start a training sequence that would take days. In any case, as the issue has propagated itself to later states, it will be a bigger hassle detecting and debugging the issue.</p><p>The program that did not crash on the correct spot can cause a big hassle. If program finishes without any error, it may also be difficult even spotting that there is a problem. This is an even more concrete issue in the field of data science. We often work with uncertainty and complex machine learning systems. If your system is stochastic and/or has machine learning components, weird results become harder to justify. It is common to blame the machine learning components upon unexpected results while the code is problematic, and vice-versa.</p><p>After discussing the importance of the assumptions in code, let me present my approach. Assumptions can be dangerous, but luckily they can be easily tested. The easiest way I learned is to put checks for as many assumptions you can find in your code. The checks are to make sure the assumptions still hold in runtime, every time it runs. In python, we can check the assumptions (along other things) with assert. The expression is very simple. The syntax is as follows:</p><p>assert &lt;&lt;the expression that must be always true&gt;&gt; , or:</p><p>assert &lt;&lt;the expression that must be always true&gt;&gt;, &lt;&lt;string that will be printed if not&gt;&gt;</p><p>It is as simple is that. Assert does one thing. In run-time, it checks if the expression is true. If it is not, it will crash the program. If you provided a string, as it crash, it will print that string. If the expression is true, it will not do a single thing. The program flow will simply continue.</p><p>For example, let us put a check on the length of the list. Let us assume that we never expect the list to be empty. Testing against this expectation (assumption) is as simple as this:</p><p>assert len(mylist) &gt; 0 , or assert mylist</p><p>This single liner will crash your script at that line if mylistis empty. When it crashes, you will see that the assert has failed and you will know exactly what caused the crash.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P07WA9KDEKazyrnCASBpLg.png" /></figure><p>Sometimes the assert itself may not be descriptive. It is a better practice to put a descriptive message why the assert has failed.</p><p>assert mylist, &quot;Expected non-empty list, got an empty one&quot;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NtdMyxQ257__3HKiABgfmQ.png" /></figure><p>As an another example, let us go back to the division example. In this example we will have a function that takes a numpy array and divides 1 with the array. For the sake of demonstration, let us assume that we do not expect any value zeros in the array. That is a quite valid assumption. It is not common working with infinities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C8s7YQETTCch6sIB6tTUgA.png" /></figure><p>As you can see, the array contains a zero number that caused one of the elements of the resulting array to be infinite. There is a warning, but it can be easily overlooked. Instead let it go on like this, we can put an assert to check our assumption:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hsmXrgdtSNFd3Onh10mLOw.png" /></figure><p>Like writing tests, it is best that you put asserts as you code. Do not wait to finish a big chunk of code to put asserts. As you code, you constantly think about the problem. Thus, putting checks as you change or add code is the best approach. You may not have the same mindset if you wait.</p><p>Even though the examples in this post are simple, you can use asserts for various occasions. One example might be checking the data distribution. For instance you can use KL-Divergence to check if a feature’s distribution is still similar to training in runtime. Another one that I find very useful is to check if the feature ordering is still the same after each data processing step. Especially in the field of data science, we cannot escape assuming. But we can check them to deliver high quality and valid results/code.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d4697c836a1d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/crash-your-data-science-code-before-it-is-too-late-d4697c836a1d">Crash Your Data Science Code Before It is Too Late</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA["Can I Train my Model on Your Computer?"]]></title>
            <link>https://medium.com/data-science/can-i-train-my-model-on-your-computer-f59329eacd0?source=rss-8fd17f72334a------2</link>
            <guid isPermaLink="false">https://medium.com/p/f59329eacd0</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-science]]></category>
            <dc:creator><![CDATA[Kemal Tugrul]]></dc:creator>
            <pubDate>Mon, 06 Aug 2018 13:14:40 GMT</pubDate>
            <atom:updated>2018-08-06T13:14:40.176Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AuDhaX0NXKCtIorada-1Mw@2x.jpeg" /><figcaption>(Photo by Hal Gatewood on Unsplash)</figcaption></figure><h3>“Can I Train my Model on Your Computer?”</h3><h4>The Need for Massive Computational Power</h4><p>Computers are good at one thing: they are fast. They are fast enough to do calculations in lighting speed. Fast enough to send information across the globe in split-second. However it is still not fast enough. The constant race between the software and hardware still continues. As hardware get faster and faster, we keep demanding more and more. If it is not fast enough, we demand more of them to compute in parallel. Gaining access to faster processors allowed us to push the frontiers in data science. Thanks to the advancements and availability in hardware, AI can beat humans in complex video games (<a href="https://blog.openai.com/openai-five/">here</a>), synthesize music, and let us not forget about detecting cats in youtube videos!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/614/1*X5mr9DX-Lpu7MyWBhtxbGA@2x.jpeg" /><figcaption>(Hardware used in OpenAI Five)</figcaption></figure><p>Computational power is correlated to success in AI. You can see that in the past, research in machine learning was limited by the available computational power. Even though they had good amount of data and access to the modern methods we use today, they were not able to exploit their full potentials. As a result, the applications and impact of the machine learning and data science were limited. As researchers and practitioners gained access to more computational power, the field bloomed with advanced methods and interesting applications.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/1*g-FM14UlKGXk7wPGY__BwA@2x.jpeg" /><figcaption>(Hardware used in a study in 2006)</figcaption></figure><h4>Frustration over Limits</h4><p>If you are a practitioner of machine learning or data science there is a good chance that you hit the annoying wall of lack of computational resources. How many times you waited for midnight to run your experiments? How many times you reduced the hyper-parameter search volume so that it will end quicker? This kind of limitation is a great source of frustration for me. It was just this month that I had to quit a personal project because it would require a lot of resources. I either had to wait for weeks for a single experiment, or spend a good amount of money in cloud services. I also remember the facial expression of my manager when I told each single run of an experiment for the project would cost 200$. It also happened multiple times that I saw people, or was one of them, asking for favors from friends or colleagues to run some experiments on their machines when they were not using them.</p><h4>The Sleeping Devices</h4><p>I was spoiled by the high performance cluster that I had access when I was in academia, where I could run thousands of jobs in parallel. But when I left, the resources that I had access got scarce. When I was searching for more computational power, I realized that there are computers lying around, doing nothing for a good amount of time! My co-workers are shutting down their computers every day after work. All the devices that I have, just sleep in the night doing nothing. The devices we have, just “sleep” doing nothing for a good amount of time every day.</p><p>We constantly waste resources we carry in our pockets and bags. Your phones, computers, smart TVs and basically anything that has a processor is a computational device. They can compute and that is what we are in need of. Ask yourself, how long do you use your devices actively in the cycle of the day? The most basic example is night time. Every night I connect my phone to charge and lay it in my night stand for long hours, where it sleeps while I also do.</p><p>The idea of using “sleeping devices” is not new. CERN has a network for people to donate their computers for scientist to use (<a href="http://lhcathome.web.cern.ch/join-us">here</a>). Cryptocurrencies depends on people attaching their computers to the network to be alive. Even though the concept is not new and there are working examples, the problem is not solved yet. The problem would be solved when it is a very logical thing to do for everybody who has access to a computational device, electricity, and internet to attach all of their devices to a computing network.</p><h4>We Need the Resources Now</h4><p>The resources that we need as data scientists are available now. They are in people’s pockets, on their night desk, and in locked cabinets in offices. However we do not have access to them. We are willing to pay for the resources, but they are not accessible. The large portion of individuals who hold the computational devices are not aware of their values. They can convert those resources in to money, however there is a missing link.</p><p>The missing link is between the suppliers and the consumers. Data scientists wish to consume the resource in exchange of money, and I believe suppliers want to earn credit by lending their devices when they do not use. I assume the link between two parties will be established when the imbalance between the suppliers and the consumers become high enough so it cannot be neglected. Till then, I will keep paying cloud services and asking for favors to use the computers that sleep after working hours.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f59329eacd0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/can-i-train-my-model-on-your-computer-f59329eacd0">&quot;Can I Train my Model on Your Computer?&quot;</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When Even a Human is Not Good Enough as Artificial Intelligence]]></title>
            <link>https://medium.com/data-science/when-even-a-human-is-not-good-enough-as-artificial-intelligence-c39c9fda4644?source=rss-8fd17f72334a------2</link>
            <guid isPermaLink="false">https://medium.com/p/c39c9fda4644</guid>
            <category><![CDATA[machine-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[intelligence]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Kemal Tugrul]]></dc:creator>
            <pubDate>Sun, 06 May 2018 12:54:40 GMT</pubDate>
            <atom:updated>2018-05-06T12:54:40.778Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D3GCvWM8-9JLUZpmyNrEkQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/xMObPS6V_gY?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Gertrūda Valasevičiūtė</a> on <a href="https://unsplash.com/?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>In 2012, I collected and compiled a dataset for my machine learning and applications research. During the study I faked an AI myself. During the post-interviews I heard the most bizarre statement from multiple attendees: <strong>“The AI did a lot of mistakes.”</strong>. This post is about people’s bias against AI’s performance and how realistic our expectations are from AI systems.</p><h3><strong>Human Intelligence vs. Artificial Intelligence</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/344/1*78EsQl4SoX2FwSu-MRQeEA.png" /></figure><p>I love measuring and explaining things in numbers. The most common and quick measurement I do is with scores and sub-scores. If I need to compare two things, I list the variables and give them weights. With that list, I give scores to variables, multiply them with weights and sum up. This gives me a single number. For many things I find it easy to use. Picking which TV to buy, which task to work on next, or testing if a method is better than other.</p><p>While some things are easy to measure, intelligence is not one of them. Intelligence is a very abstract and complex thing to measure. Even the definition of intelligence is complex. What is intelligence and how to measure it are not the kind of questions I am after to answer. The curiosity that I have for this post is how do we, people, perceive artificial intelligence and what are our expectations from it. It seems to me that when people judge AI’s ability, we are harsh. We want AI to be perfect. We do not show the flexibility we provide to human mistakes proportionally to AIs. We want AI to be “like a human”. Maybe it is because of the recent “works like brain” advertisements (google “<em>acts like brain”</em>).</p><p>Whatever the reason is, it is a common pattern. We expect artificial intelligence to be comparable to human intelligence.</p><h3>Wizard with an Artificial Intelligence in Electronic Oz</h3><p>I conducted a series of data collection experiments that got me writing this post in 2012. I was conducting a research about sketch recognition (i.e. classifying hand written symbols and drawings) using machine learning. If you are curious about sketch recognizers or would like to have a more concrete idea, here is a great one you can try: <a href="https://quickdraw.withgoogle.com/">https://quickdraw.withgoogle.com/ </a>. One of the things I wanted to achieve was to convert a real-time strategy video game’s interaction model from mouse/keyboard to sketching. With a sketching interface, people would play the game via sketching instead of mouse clicks and keystrokes. Imagine when you want to construct a building, instead of picking it from a menu and placing it onto map with point-and-click, you would sketch a simple house onto the part of the map where you want to construct it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*qVCo77JAAlv9TvKZ0aWGzg.png" /><figcaption>A Snapshot from Experiment Records Where of a User Plays the Strategy Game via Sketching</figcaption></figure><p>The plan was to develop a sketch recognizer with machine learning. In order to achieve it, the first step was to collect many hand drawn sketches from multiple people so I could use them to train a model. Instead of asking people to sketch what they see on the screen, I wanted to put them into a real environment. The environment that they would play the game with sketch like the AI was already in place. I thought that with this setting people would sketch more “realistically” than they would do in a relaxed, no-time constrained, and no-stress “copy the picture” environment. In order to collect data in that environment, I created a “wizard of oz” environment.</p><p>The idea is simple. You prepare a setting where there are two rooms/spaces. In one space you put the computer where the attendee will use. In the second room you put another computer that you will use. The idea is to connect those two computers so that you can see what the attendee is up to, and take actions accordingly from your own computer. In my case it was for sketch recognition. So if attendee draws a house, I was able to see it from my computer and give the command for the house drawing. The goal was to replace an AI system that was not in place, so it would look like there is an actual AI, which is actually a human in a different room.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*3p5BetDO4anKEuB3uA3eAQ.jpeg" /><figcaption>“Mechanical Turk” was a chess playing machine from late 18th century that seems autonomous, but in reality is operated by a human. In that sense, it was a Wizard of Oz setting. <a href="https://en.wikipedia.org/wiki/The_Turk">https://en.wikipedia.org/wiki/The_Turk</a> .</figcaption></figure><p>When my “Oz” was ready, I invited people to join my experiment (which is not an easy task to ask people give up their time for free). As explained, there was a room with a computer and a drawing screen where attendee sits, and another room where attendee cannot see where I mimic the AI. I instructed attendees that they would play a game using sketching as the only input medium and there is an AI that will recognize their sketches and perform actions accordingly.</p><p>After the experiment, I asked them to fill a questionnaire followed by informal small talks as some people were interested in this way of playing a game. This is the part where I was confused. Many people told me and wrote down in the questionnaire the same thing:</p><blockquote>“The AI that was recognizing my sketches did mistakes”</blockquote><p>You can see why I was confused. I was the “AI” that was recognizing the sketches and they tell me that AI was underperforming! How come is this possible? Have I done mistakes recognizing the symbols? It might be the case but the symbols were simple enough so I did not think that I did a lot of mistakes.</p><p>These anecdotal evidences never went into the research results but got me puzzled for a long time. If I, as a human, failed the expectations of people as a recognizer how can we come up with an AI that satisfies the users? Would it be different if I would disclose I was the AI? Were they pre-judged that an AI would always do mistakes? What do we expect from an AI?</p><h3>What is Your Expectation from the Machine?</h3><p>90% accuracy is a milestone for most of the recognizers. It feels like a great number. “Your model got 90% accuracy”. I would expect myself that in most cases I would declare the problem solved and present it to stakeholders. I would also expect them to be happy with the result. Imagine you scored 90% in an exam.</p><p>However there is one thing. It actually may not be that good in real-life use. Imagine that you are sketching a circle and 9 times out of 10, the AI recognizes it correctly and takes action accordingly. But that 1 time out of 10, it recognizes 1 as another thing (e.g. triangle) and still takes an action. And there you are trying to stop the action that AI took and redo it. If you are using a voice assistant you may probably already had that: “NO!, STOP!, DO NOT PLAY THAT SONG!, THAT IS NOT EVEN CLOSE OF WHAT I ASKED! ABORT! ABORT!”. Do you think that speech recognition performance is not good enough? Microsoft announced 5.1% error rate (95.9% accurate) in its blog in 2017 (<a href="https://www.microsoft.com/en-us/research/blog/microsoft-researchers-achieve-new-conversational-speech-recognition-milestone/">https://www.microsoft.com/en-us/research/blog/microsoft-researchers-achieve-new-conversational-speech-recognition-milestone/</a>). They also measured the human’s error rate with the same metric as 5.1%. So even if you think that the performance of your speech assistant is bad, it is actually very close to a human’s.</p><p>When you start an AI project you need to determine the expectations correctly from the stakeholders. If you manage expectations, both parties will be happier. For instance if your stakeholder had a perfect image recognizer in her mind, you have already failed at start. Even if you develop a great recognizer it will not be as perfect as she have thought. So manage expectations, and think about the accuracy in terms of real life setting. If it is not acceptable to shout “ABORT” to your voice assistant every once a while, maybe you should think twice before you buy or start developing one.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c39c9fda4644" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/when-even-a-human-is-not-good-enough-as-artificial-intelligence-c39c9fda4644">When Even a Human is Not Good Enough as Artificial Intelligence</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Effect of Naming in Data Science Code]]></title>
            <link>https://medium.com/data-science/effective-naming-in-data-science-ea847c04f51b?source=rss-8fd17f72334a------2</link>
            <guid isPermaLink="false">https://medium.com/p/ea847c04f51b</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <dc:creator><![CDATA[Kemal Tugrul]]></dc:creator>
            <pubDate>Tue, 13 Mar 2018 01:15:26 GMT</pubDate>
            <atom:updated>2018-03-13T15:48:35.245Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oD0Kb93iePyHdefQJx31lQ.jpeg" /></figure><p>Even though there are tools allowing to practice data science without coding, they are far from sufficient. Data scientists will be writing and reading code. Reading code that has poor readability is a horrible experience. This post focuses on the importance of naming entities (e.g. variables, functions) and how easily it improves the quality of your code.</p><h3>“There will be code”</h3><blockquote>“… I also expect that the number of domain-specific languages will continue to grow. This will be a good thing. But it will not eliminate code.”</blockquote><p>wrote <a href="https://en.wikipedia.org/wiki/Robert_C._Martin">Robert C. Martin</a> in the first page of his book <a href="https://www.amazon.com/Clean-Code-Handbook-Software-Craftsmanship/dp/0132350882">Clean Code: A Handbook of Agile Software Craftsmanship</a>, in the first chapter called “there will be code”. We, data scientists, write and read code. Even though there are helping tools, we will be still writing and reading code. It is one of our core practices. This is how we analyze the data, train models, predict outcomes and many more. I strongly believe that there is no escape from code for a data scientist.</p><p>We can still practice data science without coding, at some level. From early days to today there were graphical tools that allow analyzing data or practicing machine learning. One example of such tools is <a href="https://www.cs.waikato.ac.nz/ml/weka/">WEKA</a>. WEKA is a bundle of machine learning tools with a graphical user interface. According to Wikipedia, its development started in 1993<a href="https://en.wikipedia.org/wiki/Weka_(machine_learning)">*</a>. It allows users to conduct machine learning experiments and more without writing a single line of code. Then why I insist on saying writing and reading code is fundamental to data scientists? Because there will be custom operations.</p><p>Graphical tools have only so many operations. If you are not doing same tasks everyday, there will be a time when tools will not have the operation you need. This can be an analysis, a machine learning model, or some other operation. You need control, customization, and expansion over your operations at some level. The level depends on the task, libraries, or tools. As many problems require new or custom approaches, it is quite soon that you will overgrow those graphical tools.</p><p>It is seldom that we work alone. Often organizations do not have a single data scientist, they have data science teams. Even if it is not the case, data scientists work with other disciplines. This collaboration requires good data science code quality.</p><p>Moreover, if you are working for a company, your colleagues from other teams will need your code. In order to embed your models into backend, frontend or another system, you will need code. A model that cannot be deployed or integrated would be useless for your company.</p><p>If you agree that there will be code in data science, let us talk about how to write good code for data science. Writing good code is a hard task. There are well written texts explaining why it is necessary and how to achieve it. In this post, my goal is to focus on a tiny bit of that. The bit that requires no training or education, but will improve your code quality significantly. That bit I will focus on is <strong>naming, </strong>and it will improve the <a href="https://en.wikipedia.org/wiki/Computer_programming#Readability_of_source_code">readability of your code</a>.</p><h3>One Easy Trick: Renaming</h3><p>It is a horrible experience reading a piece of code that has poor readability. Let us look at this simple example below:</p><pre>import pandas as pd</pre><pre>df = pd.DataFrame({&quot;f1&quot;: [42, 12, 5, 8, 15, 65], <br>                   &quot;f2&quot;: [172, 155, 110, 120, 158, 168]})<br>df2 = df[df.f1 &gt;= 18]<br>out = df2.f2.mean()<br># out: 170.0</pre><p>Try to guess what this code does. Author of this code knew the goal of each line, and the goal of entire code while she was writing it. However, as a reader you see a code snippet you need to decipher. As the author did not pay attention to readability, you will spend more time and energy trying to understand this code. Furthermore, you will be more prone to make mistakes. Let us dissect this example to see why it is bad:</p><ul><li>import pandas as pdI know importing pandas as pd is very standard these days. All the data scientists who work with pandas will understand this shorthand. It is not the biggest problem in the code, but I believe that this can be improved. Addition to that, short (e.g. 2 characters) variable names are troublesome in autocompletion (e.g. pd vs pdb).</li><li>df = ... Again, probably because of the pandas tutorials, it is a wide applied practice to name a DataFrame df. However it is hiding what the data is in this context. What kind of data is that?</li><li>&quot;f1&quot;:... &quot;f2&quot;:... These are the columns of our data frame. However it is not informative. What kind of data those columns have? Why they are enumerated as 1 and 2? Does enumerating have a purpose (e.g. first and second of something), or not?</li><li>df[df.f1 &gt;= 18] What is the meaning of 18 in this line? Is it some kind of magic number? Why are we filtering larger than or equal to 18, and why are we filtering on f1 column?</li><li>out = df2.f2.mean() Same problem all over. What is the significance of column f2? Why are we taking the mean of it?</li></ul><p>Now let me rewrite it in a more readable way instead of explaining the purpose of it. I will just rename the variables and keep the rest of the code same.</p><pre>import pandas</pre><pre>physical_data = \<br>        pandas.DataFrame({&quot;age&quot;: [42, 12, 5, 8, 15, 65], <br>                          &quot;height&quot;: [172, 155, 110, 120, 158, 168]})</pre><pre>adult_physical_data = physical_data[physical_data.age &gt;= 18]<br>mean_height_of_adults = adult_physical_data.height.mean()<br># mean_height_of_adults: 170.0</pre><p>With just renaming, you understand the goal of the snippet and the operation in each line only with a glimpse. I believe that explanation of the code is not necessary now as it is very clear.</p><h3>data = …, But Which Data?</h3><p>I believe that naming in data science code can be harder than a generic software code. We have less concepts that fit into object design, more abstract entities, and diverse collections. This makes naming harder in data science. As an example, think about working with table that has lots of columns. For instance a table that has columns about customer as person (e.g. name, age), customer behavior (e.g. purchases), geographic information (e.g. address of the purchase), and temporal information (e.g. time of the purchase). How would you name this table?</p><p>You can come up with different names for this table but there are common bad ones. For instance do not name it dt or df. Even though you will see df in every place in pandas documentation, you must understand that they are short example codes that do not belong to a project. Also do not name it data . Yes, it is data, but would you name a variable that holds the age “integer”? “Data” as a name is very vague. Which data is that? What kind of information does it hold?</p><p>Do not be afraid to use long names. Longer names usually carry more information about the entity. Having longer names is not a burden. There are many decent IDEs with auto-complete features, like <a href="https://www.jetbrains.com/pycharm/download/">PyCharm</a>, so you do not have to write the full name.</p><p>When coding, think about the next person who will read your code. Will she understand the main goal of the script and how each line serves to that goal? More specific to naming: will she capture the meaning of that entity quickly by just the name?</p><h3><strong>No Excuse for Bad Code</strong></h3><p>Using better names is not only for your teammates or only yourself. Both your teammates and yourself benefit from this habit. From your teammate’s perspective, she will read the code you wrote easier and faster. From your personal perspective, you will write your code easier as you will have less cognitive load memorizing the meaning of your entities. As a result, your team and organization will benefit from this habit.</p><p>The habit of naming better might seem hard to build first. You may not want to spend your time on finding better names. However this is a habit that pays back. You should practice it even if the code you write is a prototype, or part of a tiny project. Without knowing, such code may turn into a bigger project. There are many cases where projects that supposed to be very small, but development kept going for years; projects that designed as “fire and forget” ended up being very important for the organization. This is why even your shortest code should have good naming.</p><p>If you never thought about better naming, I hope after reading this you will try naming your entities better, and see how it improves the quality of your code.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ea847c04f51b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/effective-naming-in-data-science-ea847c04f51b">The Effect of Naming in Data Science Code</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Great Pitfall: Neglecting Validation]]></title>
            <link>https://medium.com/data-science/a-great-pitfall-neglecting-validation-9b8621dc5a87?source=rss-8fd17f72334a------2</link>
            <guid isPermaLink="false">https://medium.com/p/9b8621dc5a87</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Kemal Tugrul]]></dc:creator>
            <pubDate>Mon, 26 Feb 2018 13:54:44 GMT</pubDate>
            <atom:updated>2018-02-26T13:54:44.000Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7e5_hADcH6D3HB1DHdZmJQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/w2DsS-ZAP4U?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Paul</a> on <a href="https://unsplash.com/search/photos/score?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>It was in the first chapter in your book, it was in the first lecture you attended, or it was mentioned in the first tutorial you watched. It seems simple: you do not measure your predictor’s performance based on the predictions you made on the data that you used for training. However, as management keeps putting on more pressure, or your stack takes longer to run, you start to sacrifice on validation. This whole story is about this one very basic thing: validation.</p><h3>What is Validation?</h3><p>You can skip this section if you already know the answer. If you are not sure, let me explain it roughly. Let us say you want to have a predictor that performs prediction at a specific task. After you get your predictor, you will want to know how well does it perform? Does it perform very poorly that it is useless, or does it perform excellently so you can declare the problem solved? In order to measure the performance, you need validation.</p><p>In order to perform validation, you need data. More specifically, you need data with the information that you want to predict. We call this information the ground truth. According to Wikipedia, ground truth is defined as: <em>“information provided by direct observation”</em><a href="https://en.wikipedia.org/wiki/Ground_truth"><em>*</em></a><em>. </em>Ground truth is usually provided by humans. In our processes, we believe that the ground truth is the actual value that we want to predict for our data.</p><p>The adventure of validation begins once you have both your predictor and the data with ground truth. If the data with ground truth was not present to your development process, it is easy. You just compare the predictions with the ground truth. However, this situation is rare. You rarely lock up your whole data and never peak into it till the very end. Very often you use this data while developing your predictor. You use some of your data for training, check the data for adjusting the thresholds, or you peak into the data to get some idea how to achieve a good predictor. In this case, you need to think about how to establish a solid and valid validation mechanism.</p><h3><strong>Why Are We Doing Validation?</strong></h3><p>It seems simple: we are doing validation because we want to measure the performance of our predictor. But let us not fooled by this off-the-shelf definition. We do not just want to measure the performance. We want to know if our predictor will do a good job in a real-life setting.</p><p>Let us establish a prediction task example. In this example, we want to know if an image has a cat in it. In this task, you feed an image to the predictor, and it tells you whether there is a cat in the image or not. So by definition, it is a binary classification problem.</p><p>For this task, you can find a dataset or prepare your own. After that, you can go ahead and start developing your predictor. But one should not forget why you are developing a predictor. You do not want to detect cats in your dataset. You want a predictor that detects cats when you pick up your camera and take a picture of a cat in your local park. For validation, the dataset in your hard-disk is to help you so you do not have to chase cats in your local park all the time. But with this comfort, we often forget about the real goal and shoot for the grant price: a higher KPI (key performance indicator) value!</p><p>We are doing validation so that it will give us a good idea how our predictors will work in the real-life setting. Your KPI is just to help you, it is not your actual goal. We need to keep this in mind when developing our predictors.</p><h3><strong>When You Forget About Validation</strong></h3><p>What happens in a scenario where you forget or fail to establish a valid validation mechanism? Let me paint you a possible outcome of this scenario.</p><p>You work on your project. You focus on getting your KPI better and better. You work hard on it. You get more data, you do your research on different methods, you run a lot of experiments. In the end, you get a performance that looks great. Mission accomplished! You are now ready to present your work and deploy it.</p><p>Everybody is happy at this point. You present it to your team, to management or who you work with. If you are lucky, at this point somebody will catch the mistake you made. If you are lucky somebody will tell you that you failed to establish a valid validation mechanism. This is not the worst case scenario. You still did not deploy your predictor. Still, nobody is affected by this mistake. You just need to go back to sketching board and introduce a valid validation mechanism. But if you are not lucky you will march into a bad situation.</p><p>In the worst case situation, you deploy your predictor, it goes live. After a while, people start to realize that the predictor you developed is not performing well. Users will start to complain. Everybody becomes confused. The KPI suggest that the predictor performs very well. But why users are unhappy about the performance? After a while, you are convinced that there is something wrong. You go back to sketching board, trying to debug your model. Finally, you realize that what your KPI shows is not right. Your validation mechanism is broken. You were so focused on increasing the KPI, you sacrificed on validation. Now your reputation is damaged, and you need to come up with a new model quickly before churns start.</p><h3><strong>Why Do We Give Up or Forget Validation</strong></h3><p>There are a lot of reasons why we sacrifice on validation or forget to establish it completely. I believe the list goes very long. My list below is not complete, but these are the ones that I came across often.</p><h4><em>Management Puts Pressure</em></h4><p>This issue is not specific to data science or validation. Management or business side always want to have results fast. They put pressure on the team to come up with a predictor fast. As pressure builds up, you start to give up on certain aspects of your method. One aspect you choose to give up may be validation. You know that validation is not fun. You need to have a different workflow just because of that. Addition to that it takes longer time. Because of these, it may seem like a good place prune.</p><h4><em>You Think You Do Not Need Validation</em></h4><p>I believe this is caused by forgetting why we are doing validation. Whatever the reason is, you may think you do not need validation. Let me list a couple of reasons that I came across with:</p><p><em>Because you do not use machine learning.</em></p><p>The need for validation is not specific to machine learning. The complexity of your model does not determine the need for validation. Even if your method is a single if statement, or a complicated machine learning model, it does not matter. You need validation.</p><p><em>Because you think your method is not overfitting.</em></p><p>Even if you are using complicated regularization methods, it does not mean it works perfectly. Overfitting preventions are not perfect. Means that you cannot rely on them and measure the performance on the training set.</p><p><em>Because your method has nothing to do with validation.</em></p><p>Maybe you think you do not need validation because you have not used any data in training. However, are you sure that you have not introduced any bias into your method by yourself? This is the part where the rabbit hole goes deeper. Even if you are not using any data in training process, you as a person may have developed a method that is biased to the data you use. Let me use the cat detection example to clarify this: if your dataset only consists of dark-colored cats and you use this information while developing your algorithm, it will perform better on your dataset than performing in real life setting where there are cats of different colors.</p><h4><em>Your Runs/Experiments Are Taking a Long Time</em></h4><p>It is not a secret that usually machine learning operations take a long time. Even if a single run does not, you want a lot of them because of various reasons like trying different hyper-parameters, methods, etc. When your patience starts to wear off, cloud server bills start to pile up, and the deadline approaches; you try to cut some operations off. At this point, you tend to cut from validation. You may start to use fewer folds, switch to subsampling, or worst: cancel out the validation.</p><h3><strong>No Excuses. You Need Validation</strong></h3><p>Whatever the reason is, whether listed above or not, you cannot effort not having a valid validation mechanism. Validation is basic for a prediction work. You may choose one from different methods (e.g. cross-validation, subsampling, etc.) but you must have one. It is the best to one from the very early stage. It may evolve as the project goes, but even in your first runs, you must employ validation. Failing to have a valid validation mechanism will cause bigger problems in the later stages of the project.</p><h3><strong>Rabbit Hole Goes Way Deeper: Valid Validation</strong></h3><p>When we think again about why we want validation, the situation goes deeper than just employing off-the-shelf validation methods like K-fold cross-validation. If we want to make sure that we get a similar performance in real-life settings, we need to think deeper about validation.</p><p>This topic exceeds the boundaries of this story, but here are some foods for thought:</p><ul><li>Is your data leaking the label of the instance in an inexplicit way?</li><li>Does the dataset contain a range of variation?</li><li>Is the data collected in synthetic (in-lab) environment, or collected realistically?</li><li>Did you introduce bias in your method through your yourself?</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9b8621dc5a87" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/a-great-pitfall-neglecting-validation-9b8621dc5a87">A Great Pitfall: Neglecting Validation</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>