<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Donat Shaher on Medium]]></title>
        <description><![CDATA[Stories by Donat Shaher on Medium]]></description>
        <link>https://medium.com/@donshaker?source=rss-86f9f3bf542c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*l1tdt_XqwImUzwpj.</url>
            <title>Stories by Donat Shaher on Medium</title>
            <link>https://medium.com/@donshaker?source=rss-86f9f3bf542c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 12:59:59 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@donshaker/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Lifetime Value Model we deployed for a big tech client]]></title>
            <link>https://medium.com/data-science/lifetime-value-model-we-deployed-for-a-big-tech-client-fc6be23e685?source=rss-86f9f3bf542c------2</link>
            <guid isPermaLink="false">https://medium.com/p/fc6be23e685</guid>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Donat Shaher]]></dc:creator>
            <pubDate>Thu, 27 Feb 2020 01:15:01 GMT</pubDate>
            <atom:updated>2020-02-27T13:27:24.688Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Lifetime Value Model We Deployed for a Big Tech Client</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0mg5n5avZqsT_Mql" /><figcaption>Photo by <a href="https://unsplash.com/@charlesdeluvio?utm_source=medium&amp;utm_medium=referral">Charles Deluvio</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Let me just dive into it. I was working with a tech firm whose business was hosting digitized fliers and coupons for retailers. How did the company make money? Essentially, off clicks on the coupons.</p><p>One day, the top brass of the company swung by our office and asked — “Can we know what value users accumulate at the end of the year?” And just like that, we jumped into a machine learning adventure.</p><p>We set out to build a model that predicted customer lifetime value. And, we productionized it and had it automatically run end-to-end on the servers to predict lifetime value for all the users.</p><p>I want to use that experience to highlight how different the deployment of an actual model is from how machine learning is taught in school.</p><p><strong>Model objective and use case</strong></p><p>We wanted to see what the customer’s value is at the end of the year. The use case was to use predicted customer lifetime value to tailor marketing strategies for users and also to cap acquisition spend per given acquired user cohort.</p><p><strong>Tech Stack</strong></p><p>The model was created on Spark and we used Scala MLlib to for model and transformer classes. Don’t worry too much about MLlib if you aren’t familiar with it — it’s very similar to scikit learn, except it’s made to work on distributed dataset rather than local pandas dataframes.</p><p><strong>How we set up the data</strong></p><p>The data we had was users and some engagement metrics that these users accumulated over time. We could use those engagement metrics to calculate revenue accumulated by any given user on any given day. I won’t go too deep into the structure of the data, because the model was done for an actual client and I want to ensure nothing too specific is disclosed. But with their permission, here is an abstraction of how the data looked.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3ctzTAkOwOWyhRHDy5unLw.png" /><figcaption>user age defined as the number of days the user has been on our platform</figcaption></figure><p>As you remember, this is a 1 year lifetime value calculation. So, to build a model that predicts revenue one year away from a given date, we need to make sure we have at least 365 days of data for the users.</p><p>The creation of the base tables from the raw unstructured data required quite a bit of aggregation. Also, scalability became quite important because aggregation of huge amounts of unstructured data can be quite costly.</p><p>Also, a note about building features. We built many, many features. And then, we would use feature selection, dimensionality reduction and feature importance criteria to cut the number of features down.</p><p>The next step was splitting data into train and test data.</p><p><strong>Setting up train and test</strong></p><p>The dataset was then split into train and test subsets. The train dataset was used to train the model and test dataset was used to track how the model is performing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xu1ZS1KdMsjCABJ0UUTsxQ.png" /></figure><p>In production, how did we use the test dataset? We used it firstly to select the best model. Also, we use it to track error of the trained models over time. So, we would train a model on day one. And then we would want to test the model on day 1, 2, 3, etc. And, then we want to see by which day the error begins to increase and that determines the time interval at which we want to retrain the model.</p><p><strong>Models</strong></p><p>We did some exploratory data analysis and dove into the models. We had a bunch of predictive variables like a user’s revenue on day 1, etc. And the variable we wanted to predict was the user’s revenue on day 365.</p><p>This was clearly a regression problem, so we threw a couple of regression algorithms at it and ones that worked best given fairly high dimensionality of data were regression trees and ensembles of regression trees.</p><p>We wanted to try different algorithms. We wanted to try algorithms with and without PCA to reduce dimensionality. We wanted to try using different sets of columns features. The list went on and on. We wanted to try many different permutations. How did we do it? We set up an interface function that allowed us to try many different parameters for each model iteration without rewriting the whole script.</p><p><strong>Error metrics and monitoring error</strong></p><p>The error function we picked was RMSE. So, every model iteration was aimed at minimizing the RMSE cost function.</p><p><strong>The pipeline in terms of structure:</strong></p><p>So, let’s think about it, how does the model work in production?</p><p>First, we create our test and train datasets. These run daily, and we collect the error metrics on the test datasets. This lets us know how much time it takes roughly for model performance to degrade significantly. Then if model performance degrades after x days, we retrain models separately on the latest data.</p><p>Then, the retrained models are saved to S3. From S3, the models, on a daily basis, are used to score the users. So, on any given day we pull all the users we have. Every user is going to be a different age. Each user, and the metrics they have accumulated by a certain age, are going to be fed into the machine learning models. Once the model finishes predicting for all the users, for each user we generate a prediction. So, the final table is provided below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3ctzTAkOwOWyhRHDy5unLw.png" /></figure><p><strong>Conclusion</strong></p><p>I had barely touched on the complex logistics of how we deployed machine learning models. So, what I wanted to do in this article is to get away from the paradigm of how machine learning is taught in most courses and how it is actually deployed. Firstly, there is no clean dataset for you to do machine learning on. Realistically, most of the work you will do will be building features and the base tables for machine learning. Also, when you fit actual machine learning algorithms on data, you will want to fit many permutations of parameters. So, build your functions in a way that allows you to try different combinations.</p><p>Finally, it’s good to think about how everything will tie together. When will the new base tables be generated and when will machine learning models run? How will you use error metrics from the trained models to inform how often model gets retrained? And, how will you score your users? All of these nuances are the things that make machine learning so complex and fascinating.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fc6be23e685" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/lifetime-value-model-we-deployed-for-a-big-tech-client-fc6be23e685">Lifetime Value Model we deployed for a big tech client</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hardest part of machine learning? Communication.]]></title>
            <link>https://medium.com/data-science/hardest-part-of-machine-learning-communication-1b2a3bd68a7e?source=rss-86f9f3bf542c------2</link>
            <guid isPermaLink="false">https://medium.com/p/1b2a3bd68a7e</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ds-in-the-real-world]]></category>
            <dc:creator><![CDATA[Donat Shaher]]></dc:creator>
            <pubDate>Fri, 07 Feb 2020 15:57:24 GMT</pubDate>
            <atom:updated>2020-02-11T03:51:23.484Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>The hardest part of machine learning? Communication.</strong></h3><h4>To ensure your machine learning project survives business scrutiny — make sure you communicate it well.</h4><p>Back in the day, for one of my clients, we have built out a random forest model that predicted lifetime value of any user. And, as with most models, we realized communication was key.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OgUzgeDT2lpBJ9fTzRCyNQ.png" /><figcaption>Photo by <a href="https://unsplash.com/@youxventures?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">You X Ventures</a> on <a href="https://unsplash.com/s/photos/meeting?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h4><strong>Communication from the get-go</strong></h4><p>Ok, so first of all, communication is never an afterthought. In reality, you want to be communicating your model to the stakeholders from the start. The thing is, building productionized machine learning models is a very time-intensive process. And, you need 100% buy-in from the stakeholders. Otherwise, believe me, your model idea, akin to baby turtle devoured by pelican, will die a quick death.</p><h4><strong>“Is your model done yet?” (Sincerely, the stakeholders)</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*v3Ikq_QV5Y5ThN4QWBzwMw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@youxventures?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">s</a>tock123 on <a href="https://unsplash.com/s/photos/meeting?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>It takes a couple of months to build a strong model with acceptable error metrics. Guess what? — That is a mighty long time to ask for. The stakeholders will grill you on progress and wonder if your time could be put to better use.</p><p>First things first — make sure to first have a very solid pitch and use case for the model. There is this concept of MVP — it stands for “minimum value proposition.” In terms of models, an MVP is a basic model that is trained and spits out some predictions in some data format.</p><p>So, make sure to deliver your model in increments. Please, don’t build out a complicated automated pipeline for an algorithm until you have delivered something more basic. Don’t seek perfection. Put tangible data in front of stakeholders and thus give them a taste of all the things they can do with that data. This gets you more and more buy-in for the next steps.</p><p>Alternatively, the stakeholders might realize the data is useless to them and they rightfully pull the plug on the project. This would ensure that you don’t work on a model that no one cares about — it might be fun to do, but it will tank your career at a company.</p><h4><strong>Do I need a model?</strong></h4><p>Please, please, please — make sure to do cost-benefit analysis of making a model. I know that as data scientists, we are excited by the complexities and learning involved in executing a machine learning project. But, do realize that the gains of the machine<strong> </strong>learning model must be huge enough to justify the long amount of time it takes to build a full machine learning model. So, do seek the simplest solution. Avoid machine learning solutions at first and see if a much simpler solution would suffice. Only if the problem is important and can’t be solved without machine learning, do you embark on a machine learning project.</p><h4><strong>Communication barrier</strong></h4><p>As stakeholders listen to you — they have one thing on their mind “how does this affect or concern my business?” So, tailor your information accordingly. Also, business stakeholders don’t have the technical background that you have, so here is what your detailed explanation of a random forest algorithm sounds like — “bla bla lalala bla bla bla blaa lala bla”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*ZPRjXUlQpk-8hK-O2X9qtA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@helloquence?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Helloquence</a> on <a href="https://unsplash.com/s/photos/meeting?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>So, does this mean not talking about the model at all and just stating the error of the model? Well, no. The goal is to abstract away from the model to the point that it is understandable by anyone. There are 2 ways of explaining gravity, for example — you can write out all the physics equations or say “it’s a force that makes things fall when we drop them”. Choose the easier one.</p><h4><strong>How we did it</strong></h4><p>Ok, so enough of me preaching what to do or not to do. Here is what we did.</p><p>We realized that stakeholders did not quite care all that much about nitty gritty of the algorithms we were using. So, we spared the details and talked about the model in general abstracted terms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/274/1*hHtLc2T1yhcMgKUSSnwVAQ.png" /><figcaption>The metric we used internally</figcaption></figure><p>The error metric we used internally as a team was RMSE. However, the RMSE idea was a bit complex and intangible for stakeholders. Also, we built the model on user level but then we rolled it up into really granular cohorts. So, we picked a different, more fitting metric for stakeholders.</p><p>The metric we picked was the percentage difference of rolled up actual vs predicted values. Then, we would look at distribution of that error across cohorts. Internally, as a team, however, when we were tuning the model, we were using RMSE as a metric we were trying to minimize. But, when we talked to stakeholders we showed a much less intimidating, and more fitting percentage difference metric.</p><p>And, because RMSE is a much more sensitive error measure, whatever we did to decrease RMSE would also decrease the more downstream metric of cohort percentage difference.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1b2a3bd68a7e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/hardest-part-of-machine-learning-communication-1b2a3bd68a7e">Hardest part of machine learning? Communication.</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to cure data trust issues]]></title>
            <link>https://medium.com/data-science/how-to-cure-data-trust-issues-49d15b9149ac?source=rss-86f9f3bf542c------2</link>
            <guid isPermaLink="false">https://medium.com/p/49d15b9149ac</guid>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[ds-in-the-real-world]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Donat Shaher]]></dc:creator>
            <pubDate>Thu, 06 Feb 2020 16:50:43 GMT</pubDate>
            <atom:updated>2020-02-10T20:04:56.322Z</atom:updated>
            <content:encoded><![CDATA[<p>So, if you’ve worked for even a year in data, you’ve probably been in a meeting that goes something like this:</p><p>Analyst: “The number is X”</p><p>Business person: “But my other dashboard says this number is Y”</p><p>Analyst: *hopes for an earthquake to end her misery*</p><p>So, what went wrong? Many sources of information that all have different numbers. How does this happen?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8RGPD8iWbBZZmTc_" /><figcaption>Photo by <a href="https://unsplash.com/@helloquence?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">s</a>tockRand on <a href="https://unsplash.com/s/photos/meeting?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Here is an example of evolution of data needs at an organization. A stakeholder needs some information about app users. So, she chats with the data team. Then, data engineers create some pipelines and analysts write code to generate dashboards that visualize the information.</p><p>Every single time a data product like a dashboard is created, a whole set of assumptions and business logic rules are used. For example, do we define churned users as “users inactive for more than 70 days?” Or “users inactive for 50 days?” Do we use table 1 which doesn’t count users who are not registered? Or do we use the bigger table? Many decisions are made in the process of creating a data product.</p><p>So, say we created some dashboard that tracks user churn. A month passes by, and a different stakeholder comes in and asks for another dashboard. Now business leadership has decided that churned users are defined as “users inactive for 50 days.” However, the previously created dashboard defines churned users as “users inactive for 70 days.” Unless everyone pays close attention to what is going on, there will be 2 conflicting numbers circulating around the organization.</p><p>What are the issues associated with conflicting information? Well, for one, it becomes hard for business owners to make the right decision. As a result of bad decisions, the business might suffer. Also, data analysts begin to seem incompetent and business leaders lose trust in what they do.</p><h3><strong>Solution: The Single Source of Truth</strong></h3><p>There is this heaven-like ideal my team always strived towards. This nirvana is called “The single source of truth.” This is a state where all of your data products (Ex: dashboards, models) use the same pipelines, same business logic and all numbers line up. Of course, that is easier said than done. And, it costs time and money. But, it is well worth it. So, how do we execute it?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MutZraiHKP5hWTMd" /><figcaption>Photo by <a href="https://unsplash.com/@headwayio?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Headway</a> on <a href="https://unsplash.com/s/photos/meeting?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>First of all, stop the proliferation of dashboards and pipelines. Sometimes, it might seem that the more information we provide to the users, the more value we create. No. This is incorrect. Less is more. It is much better to have a fewer dashboards that are all integrated with each other and use the same business logic. Put an actual effort into reviewing all dashboards you have created and mercilessly delete anything that is out of sync with current state of business rules and reporting standards. Do the same thing with data engineering pipelines.</p><p>Also, make sure everyone across analytics and business sides of the business are always talking to each other. I know that we all dread having more meetings. They can seem like a colossal waste of time. But, when done right, meetings and slack updates can save you lots of confusion.</p><p>Maintaining single source of truth is an ongoing, painful but very worthwhile process. Remember that there is no state of perfection that you are striving towards. You are simply trying to run fast and keep up with the changes to the business and business reporting. I will leave you with a quote from ‘Alice in Wonderland,’ where the Red Queen character says how you have to run to even stay in one place. It’s up to you whether you think this quote is inspiring or the opposite. I think it’s both.</p><p><em>“Now, here, you see, it takes all the running you can do, to keep in the same place. If you want to get somewhere else, you must run at least twice as fast as that!”</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=49d15b9149ac" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-cure-data-trust-issues-49d15b9149ac">How to cure data trust issues</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>