<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Yuiti Ara on Medium]]></title>
        <description><![CDATA[Stories by Yuiti Ara on Medium]]></description>
        <link>https://medium.com/@yuiti?source=rss-2d89b4c7a2d0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Faf_Lj02dipqMlA6pVUdhA.jpeg</url>
            <title>Stories by Yuiti Ara on Medium</title>
            <link>https://medium.com/@yuiti?source=rss-2d89b4c7a2d0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 10:13:29 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@yuiti/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Feature Importances: Which features really matter to my model?]]></title>
            <link>https://medium.com/legiti/feature-importances-which-features-really-matter-to-my-model-5dd6bae59a06?source=rss-2d89b4c7a2d0------2</link>
            <guid isPermaLink="false">https://medium.com/p/5dd6bae59a06</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[feature-importance]]></category>
            <category><![CDATA[model-interpretability]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[startup]]></category>
            <dc:creator><![CDATA[Yuiti Ara]]></dc:creator>
            <pubDate>Mon, 19 Oct 2020 18:52:39 GMT</pubDate>
            <atom:updated>2020-10-19T18:52:39.103Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*2sSiErbcNP7-KskN" /></figure><p>In this post, we will present a little bit about the overall intuition behind Permutation Importance, a simple but very efficient technique that we have been using here at Legiti. This approach allows us to evaluate the impact of each feature on the performance of our models. It has been an invaluable tool to understand which features are helping the most in our fight against fraud.</p><h3>Why you should care</h3><p>Getting the first trained model that achieves good performance on historical data is a very important step, however, it is far from being the end of our work. In many cases, ours included, after deploying the initial model to production, multiple model iterations will still be needed. At Legiti, it is a continuous process that never really ends. To help in the iterations it is very useful to know how each feature is contributing to the model performance. Also, especially for us, those insights are critical when we consider the development and computation costs of using new features in the production models. We are an anti-fraud solution, thus our model inferences are expected to happen in an online setting under tight restrictions in response time. As a consequence, we need to be very careful about each new feature we decide to add, not only regarding its impact on the model performance but also its potential influence on our general response time on inference.</p><h3>How you can do it</h3><p>A very common approach to evaluating feature importance is to rely on the coefficients of a linear model, a very straightforward method where you simply interpret their absolute values as importances. However, models based on ensembles of trees have become ubiquitous and it is common for data scientists to experiment with different classes of models. Because of that, a model agnostic method would be highly preferred, so we could apply the same procedure regardless of the specific model we decide to use.</p><p>Permutation Importance is a model agnostic technique that ends up solving the problem for us. But to understand the intuition behind it, it might be helpful to first look at another simpler but very similar approach, the Leave One Feature Out.</p><h3>Leave One Feature Out</h3><p>Intuitively, the technique just tries to answer the following question:</p><blockquote>How much worse would the model be if a given feature was not present?</blockquote><p>From that, we interpret that the contribution of a feature to the model will be inversely proportional to how much worse the model will be without it. Notice that, answering this question could also inform the opposite, the absence of the feature may improve the model performance, which we could interpret as a negative contribution. More concretely, the Leave One Feature Out will answer that question with the following algorithm:</p><blockquote>1. With all features, train and evaluate the model performance, this performance value will be our “baseline”</blockquote><blockquote>2. For each feature we:</blockquote><blockquote>- Exclude only this feature from the dataset, train and evaluate the model</blockquote><blockquote>- Then take the difference between the baseline and the new performance</blockquote><blockquote>- Multiply the resulting value by minus one</blockquote><blockquote>- Now we have the importance value for that feature</blockquote><blockquote>3. At the end we just sort the features by its importance values, so we can rank their relative importance</blockquote><p>At first, it seemed that this algorithm would already provide what we needed.</p><p>However, Its computation costs make it an impractical alternative for us. A single backtest run that would train &amp; evaluate a model on all historical data takes in our case several minutes to complete. Given that our models usually use a couple of hundreds of features, to loop through all the features would be very time-consuming.</p><p>Another limitation of this method is the case in which we will have two or more very highly correlated features, they may just end up replacing each other in the model and would yield very low importances even if they are in fact very important. In an extreme case, we could imagine that if we had two identical features, both could yield importance near to 0.</p><p>Fortunately, these are both constraints that the Permutation Importance can solve for us.</p><h3>Permutation Importance</h3><p>Permutation Importance will still use the same general approach from Leave One Feature Out. It will try to estimate a feature’s importance relative to how much worse the model would be without it. However, it differs in how it handles feature exclusion. To avoid the taxing computation costs, instead of excluding the feature and re-train the whole model, it just makes the feature column non-informative by randomizing its values. Instead of eliminating, the idea is to break the existing relationship between the feature and the target variable, so it would cease to provide useful information that would help in the prediction task. We are rephrasing the question a little bit as:</p><blockquote>How much worse would the model be if a given feature became non-informative?</blockquote><p>The only additional issue that still needs to be taken care of is the randomization. The model was trained assuming a very specific distribution of values for each feature, which means that values will be expected to be within a specific range of domain values (e.g. a label encoded categorical feature with integer values from 0 to 4 should not be assigned a value of 42). To achieve that, given that a dataset will have multiple observation rows, we just randomly permute the values on that feature column. Then we will have a new pseudo-random value for each row, while at the same time will still be keeping the domain values correct. Thus our general algorithm becomes:</p><blockquote>1. With all features, train and evaluate the model performance, this performance value will be our “baseline”</blockquote><blockquote>2. For each feature we:</blockquote><blockquote><strong>- Randomly permute the feature values on that column, make a new prediction using the new values of features, and evaluate the model (notice that no model re-training will be needed here)</strong></blockquote><blockquote>- Then take the difference between the baseline and the new performance</blockquote><blockquote>- Multiply the resulting value by minus one</blockquote><blockquote>- Now we have the importance value for that feature</blockquote><blockquote>3. At the end we just sort the features by its importance values, so we can rank their relative importance</blockquote><p>Now we can still compute feature importance estimates, but with a cost of a single backtest run for the whole feature set. Also, for highly correlated features, its importances won’t be nullified by each other. In an extreme case, if we have two identical features, the total importance will be distributed between the two of them.</p><h3>Practical considerations</h3><p>Because of the stochastic nature of this technique, the feature importances will have some level of variance between each different execution (or between each different seed value, if you use them when generating random numbers). To have better confidence in the estimates we may want to have a more stable measure, we can do that by running this algorithm multiple times, (with different random seeds, if you use them) and then take the average of the importances. However, the computation time will increase a bit, so it will be a trade-off we will need to make between the metric stability and the additional computation cost.</p><p>Another important thing to remember is to use separate training and validation sets for this procedure, and to evaluate the feature importances only on the validation set. Otherwise, we would not be generating estimates that generalize to unseen data in production, which is usually the goal for this whole method.</p><p>Finally, if you happen to be using only linear models, it might be worth it relying on the linear coefficients instead, as it would incur zero computation costs and their relationship with the outputs could be somewhat simpler to understand.</p><h3>Existing implementations</h3><p>For concrete usage in Python, there are good open-source implementations for the Permutation Importance, which are well tested and supported. <a href="https://eli5.readthedocs.io/en/latest/index.html">ELI5</a> is a package focused on model interpretation techniques, which includes a module for Permutation Importance. More recently, <a href="https://scikit-learn.org/stable/">scikit-learn</a> has also added a module for Permutation Importance, this is the actual implementation we chose to use given that we already use a lot from this package. Also, because of its simplicity, implementing the algorithm from scratch could be another reasonable option, especially if you would want to customize some aspects of the algorithm.</p><h3>How we use it at Legiti</h3><p>Currently, the permutation feature importances are the main feedback mechanism we use at Legiti for decisions regarding features. Both to evaluate which features would be most beneficial to add to our production models, and to validate our hypotheses regarding our intuitions on new features we are exploring.</p><p>Another interesting usage we have been considering is to integrate it into our feature selection process with Optimus. Optimus is our in-house Auto-ML module for feature selection and hyper-parameter optimization. If you are interested to know a bit more, you are welcome to also check the <a href="https://medium.com/legiti/our-data-science-robot-intern-cea29894d40">article</a> we wrote about it.</p><h3>Summary</h3><p>In this post, we gave an overview of the Permutation Importance technique. We have mostly focused on the overall intuition behind the algorithm, but if you are still interested you can find the complete details from the <a href="https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-9-307">paper</a>. Checking both the code and documentation in <a href="https://eli5.readthedocs.io/en/latest/index.html">ELI5</a> and <a href="https://scikit-learn.org/stable/">scikit-learn</a> packages might also help bring a more concrete understanding of the mechanisms.</p><p>This has been an exceptionally useful tool to help in fighting fraud here at Legiti, but we believe it would also be as useful for any other predictive challenge.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5dd6bae59a06" width="1" height="1" alt=""><hr><p><a href="https://medium.com/legiti/feature-importances-which-features-really-matter-to-my-model-5dd6bae59a06">Feature Importances: Which features really matter to my model?</a> was originally published in <a href="https://medium.com/legiti">Legiti</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Looking at retention & lifetime value with data science]]></title>
            <link>https://medium.com/liv-up-inside-the-kitchen/looking-at-retention-lifetime-value-with-data-science-1fc884e6a4ad?source=rss-2d89b4c7a2d0------2</link>
            <guid isPermaLink="false">https://medium.com/p/1fc884e6a4ad</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[foodtech]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[startup]]></category>
            <dc:creator><![CDATA[Yuiti Ara]]></dc:creator>
            <pubDate>Tue, 30 Jul 2019 15:35:15 GMT</pubDate>
            <atom:updated>2019-07-30T18:02:35.387Z</atom:updated>
            <content:encoded><![CDATA[<p>In this post, we will present a little bit about the Beta-Geometric/Negative Binomial (BG/NBD) model and the Gamma-Gamma model, simple but very useful Bayesian models that we have used here at Liv Up to evaluate retention and lifetime value of our customers.</p><h3>Why is it useful?</h3><p>Customer retention and customer lifetime value are both very important metrics for any e-commerce to evaluate the past relationship with its customers. With these, we can answer important questions such as:</p><ul><li>How frequently our customers have returned to our e-commerce?</li><li>How much revenue have our customers generated?</li></ul><p>However, in some cases it would be very useful if we could look at these metrics in the future, then we would be able to answer a bit more interesting questions such as:</p><ul><li>How much will this new feature improve/deteriorate retention in our e-commerce?</li><li>How much revenue will this customer acquisition campaign bring in the long run?</li></ul><h3>How does it work?</h3><p>So the context in which we are working is the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/562/1*hVvt5duowg4_c9NdHO0kHw.png" /></figure><p>We will have customers with histories of purchases and would like to know how the future timeline would look like in terms of purchases. From this scenario we are going to focus on the following historic information from each customer:</p><ul><li>How long ago a customer did their last purchase (recency)</li><li>how many purchases they made so far (frequency)</li><li>how much they spent so far (monetary value)</li></ul><p>Now we break the modeling problem in three questions:</p><ul><li>How likely it is that a customer will buy again?</li><li>How many purchases we would expect a customer to make in the future?</li><li>How much can we expect him to spend in the future?</li></ul><p>For the first question, the model will try to answer the opposite, how likely it is for a customer to never buy again. It will assume that after every purchase, a customer will have a probability p to not return and that this event will happen following a Geometric distribution on each day following the purchase. In order to model the variance between different customers, the model uses a Beta distribution. With that we will have a Beta-Geometric distribution, that is part of the BG/NBD model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*WdRbpuU-pwk-eUtuXE41aA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*saqcoxIU4_RcGa11LcaKNA.png" /><figcaption>Geometric &amp; Beta distributions</figcaption></figure><p>Now for the second question, what we could say about the number of purchases a customer would make in a given period? For that, the model assumes that each user will have a natural purchase rate, and it uses a Poisson distribution to model this behavior. For a given time period, some customers will have a higher purchase rate and some will have a lower one. To capture this variance between different purchase rates, a Gamma distribution is used. The combination of these distributions, a Gamma mixture of Poisson distributions, is also known as a “Negative Binomial distribution”, which makes the second half of the BG/NBD model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*caPx00-qwsQ42FTWcGCN0g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*h-Pz-qYrIQzqxWhA4TQ94w.png" /><figcaption>Poisson &amp; Gamma distributions</figcaption></figure><p>Finally for the third question, how much can we expect him to spend? For that, we will have a Gamma distribution to model the purchase average value for each customer. To capture the variance in the distribution of value spending among each customer we will have another Gamma distribution. As a result, we have our Gamma-Gamma model. One caveat is that this model will assume independence between the number of purchases and their values, which may or may not be the case depending on the context.</p><h3>How can I use it?</h3><p>For practical use here at Liv Up we use the <a href="https://github.com/CamDavidsonPilon/lifetimes">lifetimes’</a> package, an awesome python implementation for both models by Cameron Davidson-Pilon. Below you can check a few interesting ways you can use the results from the models.</p><h4><strong>Probability to Buy Again</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*8MDjTeawKj_aYx7NPEUxOg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*csoF49QM3_5Kyh8Vk4yySg.png" /></figure><p>The blue line is the probability to be “alive” (buy again) for an individual customer for each day, while the dashed red lines are actual purchases made. We can see how the model adapts differently based on the customer history, for a slow buyer his probability takes more time to really decrease, while for a fast buyer it decreases really fast.</p><h4><strong>Customer segmentation by lifetime value</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*d-_LUh1tKhQCnyudCwbqmA.png" /></figure><p>Here we sorted all the customers by their predicted lifetime value (from lower to higher), then we break into chunks of percentile groups. Now we can check how much each group contributes to our overall revenue. This is interesting information when we want to understand more about our different groups of customers.</p><h3>How do we use at Liv Up</h3><p>Today we use the predicted probability to rebuy and lifetime value in several Liv Up areas.</p><h4><strong>Measuring product performance on retaining and resurrecting customers</strong></h4><p>Our business growth is driven by retention. We need to know whether our new products and new features are helping us achieve the objective of retaining customers. However, it is hard to define when a customer is active or inactive and the model provides a data-driven approach to this problem. Using the probability to rebuy, we define for each repurchase, if the customer was retained or resurrected.</p><h4><strong>Creating a less lagging indicator for retention</strong></h4><p>Retention takes time to measure. We are a 3 years old company and we can’t wait 12 months to measure retention. So with our model we can have an indicator with a drastically reduced observation period.</p><h4><strong>Segmenting customers for campaigns</strong></h4><p>Based on the probability to rebuy, we segment our customers to deliver a personalized campaign based on their engagement. Combined with the predicted lifetime value, we can send a more efficient message, which helps in optimizing ROI on retention campaigns.</p><h3>Summary</h3><p>In this post, we gave a very brief overview of the BG/NBD &amp; Gamma-Gamma models. We have mostly focused on the overall intuition, but if you are still curious you can find the complete details from the original authors <a href="http://brucehardie.com/papers/018/fader_et_al_mksc_05.pdf">here</a> and <a href="http://www.brucehardie.com/notes/025/gamma_gamma.pdf?source=post_page---------------------------">here</a>. The official documentation for the <a href="https://lifetimes.readthedocs.io/en/latest/">lifetimes’</a> package is also a great resource, and you can quickly run the models on your data. The package also offers useful methods for plotting &amp; evaluating model performance.</p><p>The BG/NBD &amp; Gamma-Gamma models, and also the lifetimes’ package, have been very useful here at Liv Up, and we think it might be as useful for any business that wants to understand the relationship with its customers.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1fc884e6a4ad" width="1" height="1" alt=""><hr><p><a href="https://medium.com/liv-up-inside-the-kitchen/looking-at-retention-lifetime-value-with-data-science-1fc884e6a4ad">Looking at retention &amp; lifetime value with data science</a> was originally published in <a href="https://medium.com/liv-up-inside-the-kitchen">Liv Up — Inside the Kitchen</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>