<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dimitris Apostolopoulos on Medium]]></title>
        <description><![CDATA[Stories by Dimitris Apostolopoulos on Medium]]></description>
        <link>https://medium.com/@dimitris_2542?source=rss-460972859609------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Mx9wYMKWA5FnHR52Gn3-yw.jpeg</url>
            <title>Stories by Dimitris Apostolopoulos on Medium</title>
            <link>https://medium.com/@dimitris_2542?source=rss-460972859609------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 01:01:58 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dimitris_2542/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Lead Scoring for Customer Segmentation: What it is and how we did it]]></title>
            <link>https://medium.com/moosend-engineering-data-science/lead-scoring-for-customer-segmentation-what-it-is-and-how-we-did-it-9a9090b2dcb0?source=rss-460972859609------2</link>
            <guid isPermaLink="false">https://medium.com/p/9a9090b2dcb0</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[customer-segmentation]]></category>
            <category><![CDATA[lead-scoring]]></category>
            <category><![CDATA[sales]]></category>
            <category><![CDATA[marketing]]></category>
            <dc:creator><![CDATA[Dimitris Apostolopoulos]]></dc:creator>
            <pubDate>Fri, 28 May 2021 15:57:32 GMT</pubDate>
            <atom:updated>2021-05-28T15:57:32.268Z</atom:updated>
            <content:encoded><![CDATA[<p>By nature, sales, and businesses in general, have an urgency to get as many leads as possible as quickly as possible. Therefore, they have to manage their time and contacts to achieve good numbers. However, either due to lack of time or because of the large number of leads, this is not always the case. Quite a few times time is wasted on leads that have little to no potential of becoming customers. That’s where Lead Scoring — and Business Value in general — come into play. Lead Scoring helps the sales rep to discern and focus on leads that are more probable to show an interest in the brand and eventually become clients.</p><h3>What is Lead Scoring</h3><p>Lead scoring is a methodology usually used in sales and marketing for ranking leads, potential, and existing customers in order to determine whether they’re ready to buy a product, interact with a brand, or engage with the business in general. The score is based on criteria that have to do with the interest the customers have shown in the business, their behavioral patterns, and their engagement with the business.</p><p>There are different ways of ranking customers, like using the Stars System (with 0 being the lowest and 5 the highest), or assigning them to a class/group, like “High”, “Mid”, or “Low”, always according to their score.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/0*Y-vOr4EK4GKefTNy" /><figcaption><a href="https://aeroleads.com/blog/10-best-practices-customer-segmentation/">source</a></figcaption></figure><h3>Where and why is it used?</h3><p>Lead scoring is more than a marketing methodology, and different teams can benefit from it if used correctly. First and foremost, it enables the Sales Team to pay attention and use resources to the leads that are truly in a ready-to-buy situation but need a little push. Secondly, it allows the Marketing Team to identify the campaigns and channels that work better and bring in potential customers that are more likely to buy or engage. Last but not least, in eCommerce, this means more purchases which results in more revenue with a more consistent customer base.</p><h3>Why Did We Do It</h3><p>The reason we took up this project was to build a tool to help our clients differentiate their audience/customer base according to their behavior and level of engagement. That way, clients will have a better understanding of their customers, will be able to segment their audience even more, and further personalize their experience. For instance, using Lead Scoring, a client can identify low engagers, motivate them, or even reward high engagers as a token of appreciation and loyalty.</p><h3>Current State of Affairs in the Lead Scoring World</h3><p>Lead Scoring can be broken down into categories — or types — according to what we choose to score and how we choose to do it. Also, there are many different approaches and methods or algorithms that can be used, some of them being widely known, while others not so much. Whichever method or algorithm one chooses to use, there is the possibility to use it as is or try to make any changes someone wants according to the business’ needs.</p><h3>Types of Lead Scoring</h3><h4>What we Score</h4><p>In terms of what we score, there is Implicit Lead Scoring. This generally involves scoring leads based on user behavior and metrics, e.g., their actions, the pages they visit, and the interests they show. The other type of lead scoring, depending on what we score, is Explicit Lead Scoring. This involves trying to match leads to clients who have already bought. To do that, we compare demographic data and other information we have available about them.</p><h4>How we Score</h4><p>When it comes to how we score, there are the methods of Traditional and Predictive Lead Scoring. On the one hand, in Traditional Lead Scoring, each lead is given a score using predetermined criteria composed of the aforementioned implicit and explicit data. On the other hand, in Predictive Lead Scoring, an algorithm is used to predict behavior based on historical data. Their main difference is that in Traditional Lead Scoring, we have to assign values to each criteria manually, while in Predictive Lead Scoring, the score is updated based on the history and behavior of the prospect.</p><h3>RFM / RFE in Theory</h3><p>RFM stands for “Recency, Frequency, Monetary” and is a marketing tool used for identifying a company’s best customers. By numerically scoring a customer in these three categories, “good” clients are expected to score high in all three of these categories, giving them a high overall score/rank.</p><h4>Recency</h4><p>Recency: This metric shows how recently a customer was active or made a purchase. The less the time elapsed since the customer’s last activity, the higher the score of said customer.</p><h4>Frequency</h4><p>Frequency: This category shows how often a customer purchases or engages in a particular period of time. Obviously, customers who score high in this category seem to be more loyal, as their actions of engagement are more frequent.</p><h4>Monetary</h4><p>Monetary: Also known as “Monetary Value”, this metric measures how much a customer has purchased or spent with the company over a specific period of time. Usually, the more someone spends, the better score they get in this category.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*I2eJFfPEsv_6U0bK" /><figcaption><a href="https://github.com/jlopezsi/RFM-Analysis-1">source</a></figcaption></figure><h3>Methodology</h3><p>The RFM factors are used to reasonably predict whether a customer will purchase again or not. Here’s how we achieve that: at first, we rank each customer by each of these factors individually — let’s say in the range of 1 to 5. So, we end up with each customer having a triad of values — with 5 being the best score and 1 being the worst for each factor — that show how said customer fares in each one. After that, we calculate the average of these values to find the final RFM score of the customer. One can also use a weighted average, depending on the nature of the business and the factor considered to be more important. Having the final score at our disposal, we can decide how to segment the customer base to discern the high-quality ones from those less likely to interact with the brand.</p><h3>RFE</h3><p>For Non-eCommerce customers, where there is no way to measure Monetary Value, there is RFE. RFE is an alternative to RFM that, instead of the “Monetary” component, uses the “Engagement” component of the customer, meaning the level of activity of this customer, e.g. number of clicks, page views, etc. Otherwise, the calculation and scoring process stays the same as in “traditional” RFM.</p><h3>Our Approach to RFM</h3><p>We decided to have a take on the RFM method, but we needed to work around some issues in the way we approach this. The main problem was that RFM uses some standard variable to classify a population, such as the Monetary Value. However, we didn’t have purchase data on this occasion, so we had to find a way to substitute the main variables RFM uses with the data we had in our disposal and classify the customers using these “custom” factors.</p><h4>Recency</h4><p>For the first variable, Recency, we tracked the date of the last action of each client. Then, we calculated the time between the last date the customer made an action and the last day of the training period. That way, each customer would be paired with a number that would indicate the number of days passed since the last action the customer took, e.g. (Customer1, 4). That means the smaller the number, the more recent the action, and thus, the better their Recency score should be.</p><h4>Frequency</h4><p>For Frequency, we counted the number of times each customer made an action during the time interval of the training period. Now, we have another pair of customer and number (Customer1, 36), which indicates the number of actions this customer made during the training period. This is pretty straightforward, the bigger the number, the more actions the customer took, so the bigger the Frequency score should be.</p><h4>Monetary (Score)</h4><p>As we mentioned before, we don’t have purchase data in order to score the customer for Monetary Value; however, we have an equivalent. What we consider conversion here is a certain action and not a purchase; therefore, we don’t need to calculate the revenue a customer brings to the business. We can use the score we assign the customers, as it is an indicator of the actions said customer made. So, the third pair of customer and number we get (Customer1, 28) indicates the score the customer achieved according to the action they took during the training period.</p><h4>Classification per Variable</h4><p>If we group these pairs by customer, we get a triad of numbers for each customer, which indicates the customer’s score in all three variables (Customer1, 4, 36, 28). Because these numbers have different ranges, we have to transform them in some way. For each variable, we take the distribution of the scores and classify them by the quadrant they belong to. In the case of Recency, the scores that belong to Q1 are the best ones because, as we said, the more recent the action, the better. For the other two variables, the better scores would be in Q4. We assume that the best quadrants are mapped with the number zero (0) for each variable, while the worst with the number three (3). This way, all variables have values that range between 0 and 3.</p><h4>Final Results</h4><p>After this mapping, the case of our “example customer” will be something like (Customer1, 0, 1, 1), which we turn into a pair of customer and number, like (Customer1, 011), with three zeros (000) indicating the best scoring customers, and a triad of threes (333) indicating the ones that engaged the least. Now, taking the distribution of these numbers, we can classify our customers into four classes. We map the quadrants as we did earlier, with Q1 being the quadrant with the best scoring customers while Q4 has the customers with the lowest activity during the training period. In the end, we map the classes in the range of 0 to 3, with 0 being the worst and 3 the best. This method helps us differentiate between the customers who would have similar scores and maybe would overlap or be mistakenly placed in the wrong quadrant.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/0*JFLTE22WV3dcR3A3" /></figure><h3>Our Custom Model</h3><p>We split our client base into two categories: eCommerce and Non-eCommerce. We made this decision based on the fact that there is a major difference in how each type of client measures customer engagement. The eCommerce ones acknowledge customer engagement mainly through their purchases. The rest measure engagement through traffic and certain actions, e.g., site visits, form completions, file downloads, etc.</p><p>This tells us that each category has a different endgame, a different state that is considered a conversion. Clients in the eCommerce categories want to find the customers that are more likely to make a purchase, while Non-eCommerce clients want to identify customers that are more likely to interact with their product. This interaction could be reading an article, opening and clicking a newsletter, or visiting a specific page in their site.</p><h4>eCommerce Clients</h4><p>Because eCommerce clients want their customers to purchase, we need to predict the probability of a specific customer doing so. This can easily be achieved through the use of our company’s Recommender Systems. A Recommender System calculates the probability of a customer purchasing certain products based on the customer’s purchase history and preferences. If the probability of purchasing a product is over a certain threshold, the system recommends the product to the customer for purchase. If we take the weighted average of a customer’s probability for each product, we get the probability of this customer purchasing in general. This metric can be used as a ranking tool in and of itself, thus helping us classify customers according to who is more likely to purchase a product.</p><p>Since there is a way to segment the customers of eCommerce clients, the approach I’m about to describe is about trying to predict high-quality customers for clients in the Non-eCommerce category.</p><h4>Non-eCommerce Clients</h4><p><strong>The Data</strong></p><p>At first, we decided to build a relatively simple and basic model that evaluates clients according to their campaign opens and clicks. We got all of the customer actions of each campaign and put them in a file, including information on who performed each action, when it was made, and finally, the type of action (open or click).</p><p><strong>Training Period and Scoring</strong></p><p>For our training period we used a time interval of 2 months, training on campaign data and statistics of randomly selected clients, assigning 1 point for each open and 5 points for each click the clients’ customers made. Of course, clicking is less probable than opening a campaign, which happens more frequently. Therefore, we assumed that the ratio of 1 click every 5 opens, or that a click is 5 times more rare or stronger than an open, is a good starting point.</p><p><strong>Aggregating and Classification</strong></p><p>After we formatted the data a little to add the respective score next to each action, we split the data into two sets: training and testing. In the training set, we grouped the data by customer and got the total score of each customer for all actions they made. Then, we normalized these total scores to get the final score, which was in the range between 0 and 1, and classified the customers in classes. There were three different Engagement Classes, depending on whether the customer’s final score was over .66, between .33 and .66, or under .33.</p><p><strong>Testing</strong></p><p>Having these first calculations at our disposal, we wanted to test them in the testing set, which is a set of the same customers we have in the training period. The difference is that the customers’ actions in this testing period were chosen in a time interval of one month, following the end of our training period. Using the same scoring and classification method as the one we used during the training period, we checked whether the customers who have been assigned in a specific class would end up again in that same class in the testing set. We had to pay special attention to the high scoring class, as it is the one that indicated the high-quality customers.</p><p><strong>Observations</strong></p><p>Looking at these first results, we observed that we had some problems. First of all, we didn’t have enough customers in the High Engagement classes, and as a result, we couldn’t conclude whether the model was performing well or not. Also, the Low Engagement classes included a considerable percentage of the customer base we trained the model on, so the suspicion of overfitting made its appearance. We immediately understood that we had to identify these issues and find solutions or, at least, countermeasures.</p><h3>Problems Encountered</h3><p>Of course, as in every initial approach, not everything goes as expected or planned. From the very beginning, we encountered some problems that we had to solve or, at least, find an alternative way to deal with.</p><h4>Data Skewness</h4><p>One of the problems we encountered was ending up with classes whose populations were drastically different because of the skewed distribution that characterizes the data. Skewness means that most data points gather on one end of the distribution, meaning they have very low or very high values. Ideally, we want our data to be as close to the Normal Distribution as possible. However, this is rarely the case, and ours was not an exception. There are ways to overcome the problem of skewed distributions and the issues they cause, but the main idea behind them is to transform our data — which wasn’t useful at all — with the kind of data we had to work with.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-KHDZy_-t0e-tzw5" /><figcaption><a href="https://en.wikipedia.org/wiki/Skewness">source</a></figcaption></figure><h4>Class Underpopulation</h4><p>Campaigns are characterized by skewness because not every customer opens a campaign that was sent to them, and even less click on said campaign. This results in them scoring low in our algorithm, which means most customers tend to gather at the lower end of the score distribution. In our case, this resulted in high engagement classes being underpopulated because of the majority of data points being on the lower end of the distribution and thus producing weird results in said classes or in general.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/541/0*kkZ2hbN2_xxBevxd" /><figcaption><a href="https://towardsdatascience.com/multi-class-text-classification-with-scikit-learn-12f1e60e0a9f">source</a></figcaption></figure><h4>Class Overlapping</h4><p>Also, that same skewness of the distribution causes some of the low engagement classes to overlap due to the high number of instances and their minimal scores. This means that because of the low-scoring customers being too close in terms of scoring, the low engagement class these customers will end up in is unclear. That way, we ended up with sets of classes that were short one or two classes, making it impossible to have useful results in order to find patterns or correlations or make correct decisions.</p><h4>Outliers and other Scoring Issues</h4><p>The previous issue also showed us that our scoring method had flaws and needed to change. In the case of low-engaging customers, their large numbers and very low score resulted in the problem, as mentioned above, where they weren’t always placed in the correct class. Moreover, we didn’t set limits in scoring, which means that some customers achieved too high a score because of their high activity, without said activity being of high value, necessarily. For example, a customer who opened many campaigns but never clicked was assigned a higher score than another customer who had clicked a few campaigns, even though they didn’t open as many campaigns as the first.</p><p>Also, the lack of limits allowed some customers to score too high because of the high number of actions they had made, thus making them outlier points in our data and altering the results or losing possibly valuable information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qmWN3-SjwLl4yiRL" /><figcaption><a href="https://cxl.com/blog/outliers/">source</a></figcaption></figure><h3>Tweaking</h3><p>After encountering the problems above and observing the first results, we decided to shake things up and try some well-known — and some not so popular — techniques that would improve our model and our results. As someone can imagine, not all things we tried showed the expected results and weren’t taken into account.</p><h4>Enhancements</h4><p><strong>Training Data Refinement</strong></p><p>First of all, we extended the time interval of the training period to 4 months, to have even more information available. Also, we removed customers who chose to unsubscribe from receiving campaigns because they would not interact with the clients’ campaigns anymore. As a result, we would not give any more information and add value to our model.</p><p><strong>Scoring Revision</strong></p><p>Following that, we changed the scoring method we used. We decided to score by considering the unique opens and clicks per campaign. So, if a customer received 10 campaigns during the training period, they would be scored for no more than 10 actions. This way, we set limits to our scoring method and avoided having a big deviation between customer scores, thus solving the problem with the outliers. One more issue this revision solves is the case when a customer that only opens campaigns but has high activity, scores higher than a customer that clicks campaigns but has low activity, although the actions of the latter are of higher value. This issue is solved entirely by using the enhancement that follows next.</p><p><strong>Time Decay</strong></p><p>We also decided to apply a time decay method to the customers’ scores to give more weight and value to the most recent customer activities. What time decay does is reduce the score of an action, after a certain time interval passes since said action.</p><p>For example, let’s assume we have two customers who opened a campaign during the training period but opened it with a 10-day difference. Opening a campaign means 1 point, so both customers start with 1 point for said action. We begin to count the days that have passed since the action was made, and every time a predetermined period passes (e.g., every seven days), we cut the score of the action in half. When we reach the present day, we will have the true score for each customer for this particular action. Of course, the customer that made the most recent action will have a higher score value.</p><p>Before, when we didn’t have the time decay method applied in our scoring method, these two customers would have the same score assigned for these actions. This is based on the notion that a customer who has recently engaged with a brand, is more likely to engage again. In our case, a customer that has interacted with the most recent campaigns sent to them is more likely to interact with the next one they receive.</p><p>Moreover, the same importance is given to the campaigns themselves: The older the campaign a customer interacts with, the less its impact on the final score.</p><p>Of course, the total score of a customer is an aggregate of these final scores per action, but these depend on how much time has passed since the action and at which interval we choose to cut the score. Therefore, we experimented with different periods and tried reducing the scores after seven (7), ten (10), and fifteen (15) days had elapsed since each activity or action.</p><p><strong>Quantile Classification</strong></p><p>Now, in terms of the distribution and classification issues we had to solve, we decided to classify the final scores according to the quartiles of their distribution. This means that we take all the scores, sort them, find the values (quartiles) that separate each of the four quadrants (quantiles) from the ones next to them and divide the scores to classes using these values as margins. This way, we have classes that don’t overlap with each other and have similar populations. Also, we ended up with five (5) classes, one each for the resulting quantiles, and one (1) for the customers with no activity at all.</p><p>We chose this method to classify the customers because some clients have customers who are more active and engage more than other clients’ customers do, which would cause an imbalance problem. For example, the customers of a client who sends 3 campaigns a month will be placed in the High Engagement classes with 3 opens or clicks. This is not the case for a client who sends 3 campaigns per week. Out of all of their customer base, the customers who interacted with these campaigns 3 times will probably be placed in the Low Engagement classes; on the contrary, the ones in the high Engagement classes will have over 10 interactions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/938/0*S8aXYzJnKVmx4Mql" /><figcaption><a href="https://www.sigmamagic.com/blogs/what-are-quartiles/">source</a></figcaption></figure><p><strong>Prediction Method</strong></p><p>Lastly, we decided to change what we predict in the testing period and how we do it. Instead of predicting if the customers will result in the same class as in the training period, we tried to predict whether they would make a certain action in the next campaign they would receive, regardless of when they received it. Regarding that, we also decided to have five (5) training windows. We took the training and testing periods and rolled them forward by seven (7) days, and we did that four (4) times. That way, we had similar results times five, but with different time intervals for the training and testing period. In the end, we calculated the weighted average of these five results of the five runs to get the final results of the model with the current settings and parameters.</p><h4>Didn’t Make the Cut</h4><p>Some things didn’t work out as we would have liked, or at least didn’t give us any more information, even though we had pretty high hopes for them, especially for the one we describe more elaborately.</p><p><strong>Industry-Score Correlation</strong></p><p>At first, we tried to identify patterns or any kind of correlation between the customers’ industry and their scores. For example, we ran the model for many customers, after an elaborate selection of them was made, according to some of the Industries we had available in our system. After that, we grouped these clients according to the Industry they belong to and tried to see if a particular Industry scores better in general or compared to the rest.</p><p><strong>Custom Fields Research</strong></p><p><em>What is a Custom Field</em></p><p>Custom Fields are, as the name implies, fields in a client’s database that are not part of a template but are custom made and sometimes unique per client, although there are some that are widely used among businesses. The client creates Custom Fields and uses them to serve their individual needs for information about their customers. Usually, Custom Fields are populated with values when the customers fill in forms or complete their user profile. Therefore, it is common for many of those Custom Fields to have a lot of NULL values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/0*GAdgwfbzZIBAOeB6" /></figure><p><em>Choosing Custom Fields</em></p><p>To find the best Custom Fields for each customer, we need to implement a feature selection method. Feature selection is a method to select those features of your data that contribute the most to your prediction output. Because each client has different Custom Fields, both in numbers and names, we need to identify the best ones to use in each case. That’s why we used a couple of techniques and algorithms to help us in this task.</p><ul><li>NULL Values</li></ul><p>Because of the unique characteristics that Custom Fields have, we had to single out the ones that would actually give us information about a customer’s profile and/or behavior. Therefore, after many tests and trials, we decided not to consider those Custom Fields whose population has less than 25% of their values equal to NULL.</p><ul><li>Cardinality</li></ul><p>Moreover, we checked Custom Fields in terms of cardinality. Cardinality is the diversity of values a Custom Fields has in comparison to the population. For example, the Custom Field “Telephone Number” has high cardinality because all customers have a different telephone number. This means that we cannot group customers according to this Custom Field and expect to see a pattern. For that reason, we decided to keep the Custom Fields, whose variation of values is under 20% of the population.</p><p><em>Clustering</em></p><p>After we keep the Custom Fields with the requirements mentioned above, which could give us valuable information, we use clustering algorithms to determine any clusters or patterns created among the customers. One way was to try and find a finite number of clusters, while the other was to try and explore if there was any natural density between the Custom Fields that would help us identify clusters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/0*KWlHRO0txGjkG2aQ" /><figcaption><a href="https://en.wikipedia.org/wiki/Cluster_analysis">source</a></figcaption></figure><ul><li>k-NN</li></ul><p>One algorithm we used for the case of a finite number of groups was k-NN (k-Nearest Neighbors), in which we give a number as input, and the algorithm tries to group the data in clusters of a number that is equal to the number we gave. Unfortunately, k-NN didn’t work because there is no absolute number of clusters, resulting in us not finding the correct number to give as input to the algorithm.</p><ul><li>DBSCAN</li></ul><p>Another algorithm used to examine the density between the Custom Fields was DBSCAN (Density-based spatial clustering of applications with noise). DBSCAN is a density-based clustering algorithm that takes no parameters into account and groups together data points closely packed together and have many neighbors, marking the points whose nearest neighbors are too far away as outlier points. However, DBSCAN didn’t produce any fruitful results as there was no correlation between the Custom Fields, and therefore could not classify the customers, or classified too few of them.</p><p><strong>Country/Region as a Factor</strong></p><p>In the case of the Region/Country of a customer, we tried to see if actions made in a certain Region or Country would result in better scores or give us a correlation. We checked if customers that performed an action from a certain Region or Country scored higher than others who did so in another region. Also, we tried to add Region and Country as a factor in the aforementioned Custom Fields to enhance the calculation we made there and try and extract more information. However, this attempt, too, was not fruitful.</p><p><strong>Combining Models</strong></p><p>After running tests for many clients in both the RFM and our Custom models and comparing their results, we observed that the RFM model was better in some classes while, in others, the Custom Model was giving better results. So, we decided to build a Mixed Model that calculates both models’ results but ends up forming the classes depending on the model that scored better for said class.</p><p><strong>Statistical Analysis</strong></p><p>Another method we tried was to enact a statistical analysis of the clients and their campaigns to find a pattern in one of these metrics. Some of the metrics were the Open, Click-Through, Click-to-Open, and Bounce rates, the number of Unsubscribed customers, the number of Complaints, Segmentation percentage, and the number of Campaigns the client sent. However, these statistics didn’t give us any insights regarding the behavior of the customers or the clients’ performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-uKyhYoj0XNLLRAG" /></figure><p><strong>Honorable Mentions</strong></p><p>A few honorable mentions are the following: We tried to use the exact amount of campaigns that were sent per customer (but didn’t have the data available for the time period we were interested in), or keep only the customers that had made more than one (1) actions in the 4-month training period (but didn’t produce better results as we reduced the data and lost information). Moreover, we examined if we could find any look-alikes between the customers, meaning building a customer profile based on other customers we already know converted. But to no avail, these attempts didn’t give the results we would have liked them to.</p><h3>Testing Framework</h3><p>The need to determine which of the aforementioned models performed better pushed us to test each model with many combinations of parameters and settings. This testing would give us some insights into which parameters to use for these models to be even better and the method that would better serve our needs according to our clients’ data and profiles.</p><h4>First Attempt</h4><p>Our first approach was to assign a score to the customers depending on their actions in a certain time interval and then classify them in classes according to their score. Next, we used the same scoring and classification methods as we did in training, trying to see if we would get similar results per class. In a perfect world, the results of the testing period would resemble the results of the training period, and all would be well. But as we mentioned earlier, that wasn’t the case and, along with some problems we encountered, we decided to change our approach entirely.</p><h4>A New Approach</h4><p>The new approach didn’t change the way we train our data. We score the customers according to certain actions and classify them depending on their score (or the distribution of their scores after the improvements we applied). However, we needed to change the way we thought about predicting the behavior of a customer. Therefore, we decided to predict not the class in which the customer will be after a certain time interval passes, but whether a customer will interact with the business or product in a certain way, considering said customer belongs in a particular class.</p><h4>Training Data</h4><p>At first, we use our Model to classify the customers of each client in Classes, depending on the Engagement Score they achieved during a certain time interval T. The Engagement Score is calculated according to how much the customers interacted with the campaigns they were sent during said time period T. These interactions could be based on whether the customers opened a campaign they were sent, clicked it, or did both. Having this opportunity, we tested our models with all possible types of interactions.</p><h4>Expected Performance</h4><p>After we generate the Classes, we calculate the Engagement Probability of each Class, meaning the probability of a customer that belongs in a specific class to make a certain action. That’s the Expected Performance of each Class and is calculated by dividing each customer’s score by the total number of campaigns the customers belonging to each Class received.</p><h4>Engagement Metric</h4><p>Before we calculate the Test Performance of our algorithm, which is checking if the customers in each Class engaged with the next campaign they received, we need to choose which interaction counts as engagement. The choices were the same as in the training period: opening a campaign, clicking it, or doing both. Of course, we decided to try all of them to have a clear image of the possible results.</p><h4>Test Performance</h4><p>The way we calculate the Test Performance for a Class is by dividing the unique clicks of the customers in the Class by the number of said customers. Because all customers are tested on whether they engaged with the next campaign they received or not, each customer’s score would be 0 or 1. It has no meaning to apply the time decay method here, as we only care if the action itself happened, not when it happened.</p><h4>Error Rate</h4><p>By comparing the Expected Performance and the Test Performance, we can see how well our algorithm predicted the customers’ actions. We take the absolute difference between Expected Performance and Test Performance for each Class and then calculate the weighted average of these results, which gives us the Error Rate of the Model. In the end, we take the average of the five Error Rates that were calculated after the five runs we programmed, getting the final performance rate of our Model with the selected settings and parameters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*URME9_o4zZ_SfyeP" /></figure><h4>Expectation Through Probability</h4><p>The above methodology is based on the assumption that the higher the class a customer belongs to, the more probable it is that this customer will make a specific action the next time they receive a campaign. For a customer to belong in a high class means that they achieved a pretty high score. But to achieve a high score, a customer has to have high activity, especially in the most recent campaigns.</p><p>In the marketing world, if a customer has engaged recently with a brand or product, it is quite probable that they will engage again with the brand or product in the near future. Therefore, the higher the class a customer belongs to, the higher the probability of making a certain action. In short, we reasonably predict that a customer will make a certain action based on an expected behavior we have observed previously, monitoring and measuring it with a chosen metric.</p><p>The metrics we had at our disposal were opens and clicks of the sent campaigns. Generally, a customer clicks a campaign fewer times than opening one. This means clicks are a more “valuable” action, especially considering that customer clicking is often viewed as conversion for many clients. In some instances, we also used both metrics in our scoring method to see how they would affect the training and the testing performance results.</p><p>Lastly, the customers’ interest varies depending on the client and their product. For example, there will be different interest and activity for the campaigns of a News Publisher and different for those of a Swimwear Retailer. This means that some customers will lose interest almost immediately in some customers’ products, while the appeal will be nearly consistent in other cases. So, this method of testing helps us cope with this as well.</p><h3>Results and Conclusion</h3><p>After all was said and done, we had to lay everything on the table and choose the model that gave us better results along with the best combination of parameters and settings. Of course, all the testing, trials, and errors we made along the way contributed to us reaching the final form of our implementation.</p><h4>Results</h4><p>After enhancing our Custom Model with the improvements mentioned earlier, building the Combination Model, and putting them together with our approach to the RFM model, we tested them all again and again, trying a wide variety of combinations of parameters and settings.</p><h4>Training Interval</h4><p>First of all, we had to choose the time interval of our training period. Between two (2) and four (4) months, we decided to go with the latter, as it would give us more information about the customers’ behavior and activity while at the same time it wasn’t slowing us down performance-wise.</p><h4>To Decay or not to Decay</h4><p>Also, regarding the time decay method we applied in our models, we had to check whether it was actually giving the expected boost to them. After running the models with and without including the time decay method in the implementation, it was clear as day that the time decay method was an integral part of our models’ performance. As we mentioned earlier, time decay is based on the factor that a customer who has recently engaged with a brand is more likely to engage with the brand again. In our case, a customer with this kind of activity and behavior should be assigned a better score than a customer who hasn’t interacted lately. That way, not only do we make our algorithm perform even better, but we also manage to have a more fair customer score.</p><h4>Days of Decay</h4><p>After we decided to include the time decay method in our implementation, a different question arose. We had to choose the time interval that would reduce the score — using the time decay method — after elapsing. This meant we had to test after how many days we would choose to cut the score in half. We decided to test three different time periods: seven (7), ten (10), and fifteen (15) days. After using all three values in the models and testing them with all the different settings and parameters, we observed that we had better results using the fifteen (15) days as the input than we had when using the other two.</p><h4>Opens vs. Clicks vs. Both</h4><p>One of the most important things we had to decide upon was the metric we would use when training and scoring the customers. We trained with customers who only opened a campaign, with those who just clicked it, and customers who made either of those actions. At this point, let’s keep in mind that when we talk about opens and clicks, we mean the unique opens and unique clicks a customer made to all the campaigns they received. As far as performance is concerned, when training with customers that either opened or clicked a campaign, we didn’t get the results we would’ve hoped for, given the added value the customers’ scores had.</p><h4>Opens vs. Clicks, The Final</h4><p>Regarding the other two metrics, we had to test them in tandem with the metric we would choose for the next action. This means that when we trained with opens, we would try to predict if the customer would open the next campaign, while when we trained with clicks, we would try to predict whether the customer would click the campaign. In the end, training and predicting with clicks proved to be the better performing parameters. However, because the option of training and predicting with opens was pretty close regarding the final results, we decided to implement a solution. This would allow our clients to choose whether they want to classify their customers using clicks or by training the model with unique opens and predicting the next open, thus using the second-best set of metrics in terms of performance.</p><h4>The Chosen One</h4><p>The question remains, though: Which model performed better even after optimizing the parameters and settings? That is our Custom Model. It went toe to toe with our RFM approach; however, it produced better results, especially in the High Engagement classes, which made it come on top and show the best results among all the models we tested. What is really important is the aforementioned outstanding performance in the High Engagement classes because this is where the high-quality customers are. After testing over 25 clients, with all combinations of the aforementioned parameters and settings, our Custom Model achieved an Error Rate of under 4%, which means it predicts accurately with over 96% accuracy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DrA4cbUliur40Men" /></figure><p>After all the trials and errors, studying and implementing, we managed to build a model that reasonably predicts whether a customer is bound to interact with a product or business soon with high accuracy, and classify that customer according to the probability of that prediction. This gives our clients the ability to segment their customers and act differently depending on this classification. This will be possible with our Custom Model, which scores the customers according to their activity the last 4 months, refactors the scores when a 15-day time interval has passed, and tries to predict whether the customer will click the next campaign they will receive with over 96% probability.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9a9090b2dcb0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/moosend-engineering-data-science/lead-scoring-for-customer-segmentation-what-it-is-and-how-we-did-it-9a9090b2dcb0">Lead Scoring for Customer Segmentation: What it is and how we did it</a> was originally published in <a href="https://medium.com/moosend-engineering-data-science">Moosend Engineering &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Product Recommender System: How The Journey Ends]]></title>
            <link>https://medium.com/moosend-engineering-data-science/product-recommender-system-how-the-journey-ends-13e3bc3264a9?source=rss-460972859609------2</link>
            <guid isPermaLink="false">https://medium.com/p/13e3bc3264a9</guid>
            <category><![CDATA[recommendations]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[recommender-systems]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Dimitris Apostolopoulos]]></dc:creator>
            <pubDate>Wed, 10 Jul 2019 11:47:00 GMT</pubDate>
            <atom:updated>2019-07-10T11:47:00.901Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Good day everyone!</strong> It’s me again, your favorite <strong>data scientist</strong> extraordinaire and I’m here with the <strong>very last article</strong> of <a href="https://medium.com/moosend-engineering-data-science/the-road-to-recommender-systems-d6bb79bd169d">our series</a>. Yes, I am sad too, but I won’t be away for too long, so stay tuned!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4vBsDON_I5R0Eq3zrhhMzQ.jpeg" /></figure><p>But back on today’s article. I am here to present how <strong>the whole recommender system</strong> works,<strong> combining all the methods</strong> that I came up with and described in all my <strong>previous articles</strong>.</p><p>Apart from that, I will also remind you why we went through so many adventures to build the<strong> recommendation engine</strong> we built: The <strong>end goal</strong> of our quest was to create a <strong>data-driven model</strong> for <strong>product recommendations</strong> that works for all <strong>eCommerce stores</strong> in <a href="https://moosend.com/"><strong>Moosend</strong></a><strong>’s</strong> database.</p><blockquote>This model should have the ability to produce personalized product recommendations based on the customer’s past interactions, enhance information from multiple shops and work for a plethora of different eCommerce stores.</blockquote><h3>Let’s recap first</h3><p>In order to come to a successful conclusion, I will need to describe the model in full.</p><p>First in order of business was <strong>preprocessing</strong>, <strong>data transformation</strong> and the <strong>preparation</strong> for the model itself.</p><p>We needed to get the products off the database and perform the <a href="https://medium.com/moosend-engineering-data-science/named-entity-recognition-matching-products-to-genders-a91fb466e3a3"><strong>Named Entity Recognition</strong></a>. The goal was to match a product into a customer gender, using the exchange loop we talked about previously, and then to create the <a href="https://medium.com/moosend-engineering-data-science/product-clustering-a-text-clustering-approach-c392c2ef4310"><strong>Product Clusters</strong></a>. These clusters contained the same or very similar products across all shops in the database.</p><p>Afterwards, we went ahead and tried to create <a href="https://medium.com/moosend-engineering-data-science/recommender-systems-optimized-ef68f1e19923"><strong>Shop Clusters</strong></a> that contained the most similar shops, in order to maximize the learning process and reduce calculation time and the size of the matrix itself.</p><blockquote>You see, these consumed more memory than we could afford.</blockquote><p>Lastly, we did our best to enhance the information of the customers’ interaction with <strong>individual products</strong> into <strong>product clusters</strong>. That way, we could share the findings that came up from studying a big eCommerce store with a smaller one. This was made possible, through <strong>sharing the purchase patterns</strong>.</p><p>Having all the information at the ready, me and my team sailed to another destination: <strong>Data transformation</strong>. We used the data in conjunction with the <strong>Interaction Score Formula</strong>, in order to generate the <strong>interest score </strong>for every customer-product interaction.</p><p>The last step of this part of the quest was the <strong>decay</strong> of the generated interest score, which we calculated based on the <strong>time interval</strong> of every interaction.</p><p>After we managed to process the interaction data, we fed them into our model. We fashioned an<strong> interaction matrix</strong> (R)<strong> </strong>for every <strong>shop cluster</strong>. The matrix had <strong>rows</strong>, each one representing every single <strong>customer</strong>, and <strong>columns</strong> used to represent the <strong>product clusters</strong>. We made sure to leave the cells empty, when there were no interactions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*mmHUze1s878NyWf6ULksNA.png" /></figure><p>The next step was to <strong>remove the bias</strong> from the R, then <strong>decompose the R matrix into P and Q.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*VEOO-8DD53qXaUCt7YHuMg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/465/1*OQmGi0jvpVjjz2_Un5RppA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*cWLwYmXggz186OEX9UgcjQ.png" /></figure><p>Our last step was to <strong>calculate the dot product of P and Q</strong> <strong>and add the bias </strong>we removed previously.</p><p>The name of this brand new little trinket-the generated matrix of the process, that is-was <strong>R-hat</strong>, meaning that it was the same as the <strong>matrix R</strong>, except it had all its cells filed with the predicted interest score for every <strong>customer-product cluster interaction</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*dd2b5-ZYt_BPKW23ANfPpw.png" /></figure><h3>Model Workflow</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5RoAD9Pilm2FyRvFI6j2cg.png" /></figure><h3>Behind The Recommender</h3><p>Our journey resulted in our creating the <strong>R-hat matrix</strong> with all the predicted interest scores for every customer-product interaction.</p><p>Generally speaking, what we had to do is to recommend the products with the highest predicted scores. In order to make this work though, and to make the model more “<strong>intelligent</strong>”, we had to add simple, man-made logic behind the algorithm.</p><blockquote>What was that, you ask?</blockquote><p>Well, it depended on what each scientist would like to achieve. If, for instance, you’d like to avoid recommending a product that someone has already purchased, this is where you’ll have to pinpoint your most important parameter.</p><p><strong>But why would you recommend a product that someone else has already purchased?</strong></p><p>This, my fellow adventurer, is something that depends solely on the nature of the product itself.</p><blockquote>For example, we would very much love to see a recommendation for the same coffee type or brand, or the same shampoo type or brand, but we can’t recommend the same t-shirt or the same construction tool.</blockquote><h3>Important Parameters To Keep In Mind</h3><p>Everything is fine and well so far, isn’t it? I bet you’ve got a key question though: What kind of parameters did we take into account?</p><p>Let me tell you a thing or two about parameters and <strong>parameter tuning</strong>. First off, here are the <strong>most important parameters</strong> that you should keep in mind along the way:</p><p>i) <strong>learning rate</strong> ( how our model changes beliefs ),</p><p>ii) number of <strong>latent factors</strong> (in how many terms we describe your customers and your product ),</p><p>iii) selection of <strong>gradient descent optimizer</strong> ( how aggressive or how smooth the model change the weights to find the optimum result [<strong>adagrad</strong>, <strong>adadelta</strong>] ),</p><p>iv) <strong>epochs</strong> ( how many times pass the data till the models learn ) and</p><p>v) the <strong>loss function</strong> to measure the error are the most important.</p><blockquote>Generally, there is no rules to tune the above parameters but I can mention some <strong>tips</strong> for you that I got through my experience.</blockquote><blockquote>Remember to <strong>keep the learning rate low. </strong>That is between 0.001 and 0.01.</blockquote><blockquote>As far as the <strong>latent factors </strong>are concerned: These factors are used to determine the terms in which we are going to describe every customer in the P, Q matrix. The size of the matrix and the complexity of the situation lead us to represent every customer and every product with <strong>100 latent factors</strong>.</blockquote><blockquote>Mind you, I discovered a pot of gold through that one: <strong>More latent factors</strong> can lead to <strong>better results</strong>, but adding more and more makes zero difference after a certain point.</blockquote><blockquote><strong>Gradient Descent Optimizer</strong> works best with the <strong>adadelta </strong>optimizer. Both <strong>adagard </strong>and <strong>adadelta </strong>proved to give the same results at the end of the process, however, adadelta needed <strong>less epochs </strong>and, by extension, <strong>less time</strong>.</blockquote><blockquote>Finally, when it comes to the loss of function, I personally chose to go with the <strong>Bayesian Personalised Ranking </strong>(BPR) option. The reason was that I got the best results from the <a href="https://medium.com/moosend-engineering-data-science/building-a-validation-framework-for-recommender-systems-a-quest-ec173a24b56f">simulation process</a> we managed to come up with.</blockquote><h3>Let’s talk Numbers!</h3><p>Here’s the model’s performance for cluster and individual product levels:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/725/1*al5XhLfmb6I2DgFyV0Nhfg.png" /></figure><p>Our new model is clearly improved, compared to the <a href="https://medium.com/moosend-engineering-data-science/recommender-system-the-information-enhancement-trick-79dd12ea88e3">previous one</a>, which is a huge win for <strong>various reasons</strong>. Like, for example the following one: Think of a <strong>recommender engine</strong> that lead a customer <strong>exactly </strong>where you wanted them-to the purchase of a product.</p><p>Think of that purchase having been made with around <strong>8% accuracy</strong> on the individual product-8 products out of 100 sets of 5 products our system recommended, that is.</p><blockquote>This can take the shop’s revenue on a whole new level!</blockquote><p>If my journey through <strong>recommender systems</strong> taught me anything, this was the fact that the<strong> strongest metric </strong>any data pirate like myself could ever have, is <strong>accuracy.</strong></p><p>I needed to analyze <strong>customer segments</strong>, in order to map out how the <strong>information of a customer can influence</strong> the recommendations in cluster level and individual product.</p><p>And if you need some <strong>more insigh</strong>t on <strong>customer segments</strong> and <strong>validation</strong>, <a href="https://medium.com/moosend-engineering-data-science/building-a-validation-framework-for-recommender-systems-a-quest-ec173a24b56f">here’s your trusted map</a>!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/726/1*-0iJGIpybTIU157DtQzt_g.png" /></figure><h3>Takeaway</h3><p>But what did we achieve through building this model, besides accuracy?</p><p>We managed to <strong>enhance the information</strong> from similar eCommerce stores, so that the small-medium eCommerce stores with not that much data, could benefit from the larger ones.</p><p>Not only that, but our system also works for a <strong>variety of different businesses</strong>, because it was “taught” how to adjust to different customer purchase logic.</p><blockquote>But is performance our only indicator of success? No, not by far.</blockquote><p>Our golden achievement is that our <strong>recommender system</strong> is very good in terms of <strong>memory and calculations</strong>, seeing as my crew and I managed to build a matrix that was <strong>smaller in size</strong>, thus making it more <strong>manageable</strong>.</p><p>This means that it can actually “use” the techniques I mentioned before, without being too large or too slow, as there was <strong>less complexity</strong> in calculations and<strong> less memory</strong> to be consumed.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=13e3bc3264a9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/moosend-engineering-data-science/product-recommender-system-how-the-journey-ends-13e3bc3264a9">Product Recommender System: How The Journey Ends</a> was originally published in <a href="https://medium.com/moosend-engineering-data-science">Moosend Engineering &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Recommender Systems: Optimized!]]></title>
            <link>https://medium.com/moosend-engineering-data-science/recommender-systems-optimized-ef68f1e19923?source=rss-460972859609------2</link>
            <guid isPermaLink="false">https://medium.com/p/ef68f1e19923</guid>
            <category><![CDATA[recommendation-system]]></category>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[optimization]]></category>
            <dc:creator><![CDATA[Dimitris Apostolopoulos]]></dc:creator>
            <pubDate>Mon, 24 Jun 2019 14:48:19 GMT</pubDate>
            <atom:updated>2019-06-26T06:24:15.082Z</atom:updated>
            <content:encoded><![CDATA[<p>Good day everyone and welcome to another one of my quests!</p><p>The <a href="https://medium.com/moosend-engineering-data-science/the-road-to-recommender-systems-d6bb79bd169d">Recommender Systems quest</a> <strong>continues </strong>with vigor. I’ve shown you the way past the Product Clustering Isles, through the endless waves of Named Entity Recognition, the Information Enhancement, the Validation Framework and how to build it and, at this point, I am going to show you <strong>how I improved the performance</strong> of my model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y-3RPJJ4-qdNTDT7WndsFQ.jpeg" /></figure><p>The brief explanation is the following: Taking a closer look into eCommerce <strong>purchase patterns</strong>, I discovered multiple “places” where me and my crew could sail and improve our results. In other words, <strong>we had multiple points we needed to take care of</strong>.</p><p>There was only one way to take care of that: By <strong>fracturing all of our data</strong> and <strong>compiling shop clusters</strong> and, afterwards, using a <strong>scoring formula</strong> that represents the interests of the user with a score and <strong>not a “yes” or “no” answer</strong>.</p><blockquote>What is more, I decided to calculate the decay of interest and remove the bias that exists in interactions and add these results to the mix.</blockquote><h3>The Shop Clustering Approach</h3><p><strong>How many shops</strong> can you count in one single journey? I’ll be honest with you, <strong>I’ve got no idea</strong>, seeing as there are so many <strong>different kinds</strong> of shops out there!</p><p>Although there was a significant decrease of the matrix after applying <a href="https://medium.com/moosend-engineering-data-science/product-clustering-a-text-clustering-approach-c392c2ef4310">Product Clustering</a>, that was not enough.</p><p>I had to <strong>reduce </strong>the size of the <strong>interaction matrix</strong> that we had somehow, in order to make it more manageable. So, I did the only thing that made sense: I ran a <strong>clustering algorithm in shops</strong>. This helped the crew and myself <strong>classify </strong>the most similar ones into <strong>categories</strong> (or groups, if you will).</p><p>This clustering method can have a great <strong>impact </strong>on the <strong>product recommendation</strong> this engine produces and it proved to be a great way to <strong>differentiate </strong>the <strong>clusters of shops</strong> buyers purchase products from, using a <strong>different modus operandi</strong> and <strong>optimize each cluster separately</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*83LImeh1an9oFZhxqmRacA.png" /></figure><p>Every <strong>shop</strong> and <strong>product </strong>each one of them sells was <strong>represented as a vector</strong>. Then, the crew and I decided to create a <strong>sparse matrix</strong>. In order to do that, we decided to <strong>combine all shop vectors</strong> and try to find the ones that were the most <strong>similar</strong>. Those were combined in <strong>couples</strong>.</p><p>After that, we started building the distribution of shop clusters.</p><p>And we used three “compasses”, three different ways to measure the similarities without getting lost in the process:</p><ol><li>The <strong>vector similarity</strong> was used to calculate the Euclidean distance.</li><li>The percentage of <strong>product similarity</strong> in each couple of shops was used to calculate how many shops could “learn” from each other.</li><li>The <strong>product interactions</strong> each shop could provide to others were used so that we could share the customer-product interaction information data.</li></ol><h3>More Formulas: The Interest Score Approach</h3><p>Much like a crew is represented by the color of the ship’s sails, each and every customer is represented by their product interactions in a 2D sparse matrix. We call that matrix, “R”.</p><p>Its <strong>rows </strong>represent the <strong>customers</strong>. Its <strong>columns</strong>, on the other hand, represent the <strong>products </strong>as vectors.</p><p>Views, add-to-cart, and purchases are <strong>key customer interests</strong>, but they can be measured <strong>inaccurately </strong>as equal. How is that possible, you ask?</p><p>Well, all of us have viewed a product by <strong>accidentally clicking</strong> on it. On the other hand, <strong>I don’t think that anyone can buy a product by mistake</strong>. And maybe, just maybe you can <strong>view a product that is of no actual interest</strong>.</p><p>Therefore, we came up with a life-saver. A <strong>formula </strong>to individuate the interactions and define the score.</p><p>The formula is pretty simple: Every <strong>view </strong>scores <strong>one point</strong> (3.5 points is the maximum amount), every <strong>“add-to-cart” </strong>scores <strong>4 points</strong> (4.5 maximum) and every <strong>purchase </strong>scores <strong>5 points</strong>. And here you have it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/553/1*3_KVQnQ6jpYg7QXTIET1aw.png" /></figure><p>Using these techniques, we managed to help our scoring formula get ready to receive the data and measure and balance every product interaction. <strong>This created the so-called “interest-score”.</strong></p><p>And then we were ready to fill the customer-product cell with the calculated value.</p><h3>Interest Decay: What Happens Now?</h3><p>There are many types of product views, but some depend solely on specific time-frames. This means that they’re only needed at a very specific point in time and therefore, couldn’t be characterized as a recurring need.</p><p>A <strong>fine example</strong> of that, would be the time one will be searching for a <strong>wedding gift</strong>. They’ll google “champagne glasses” today, but this search will be <strong>irrelevant after they attend that wedding</strong>.</p><blockquote>We needed our model to be able to understand the fact that personal interest changes and the dominant interest differs from time to time. In order for our machine to “learn” that new path, we had to transform the customer-product interest score.</blockquote><p>We almost crashed the ship in our attempt to teach the machine. It wasn’t easy. But after several weeks of working on this and only this, we came up with the <strong>“Exponential Decay” formula</strong>.</p><p>This formula pretty much managed to <strong>monitor how a customer interacts with various products</strong> and <strong>measure how interest changes through time</strong>.</p><p>The “Exponential Decay Formula” was applied to <strong>transform </strong>every <strong>interest score in the matrix</strong> we built.</p><p>Of course, the <strong>rate of decay</strong> was (and still is) <strong>different </strong>for <strong>every shop cluster</strong>. Customers have <strong>different buying behaviors</strong> for <strong>different shops</strong>. For example, a pirate will purchase rum way more often than… Well, all of you who are reading this article.</p><p>Or maybe you can think of <strong>how often you buy things from a supermarket</strong>, as opposed to the <strong>frequency with which you buy shoes</strong>. The frequency is completely <strong>different</strong>.</p><h3>Biased Customers: How To Tackle The Issue</h3><p>Up until now, I believe that everything I’ve described sounds a little easy-perhaps too easy to have taken us that long to master.</p><blockquote>There’s something you’ll need to keep in mind, though: Every person has different needs. Needs that we had to take into account while building our model, in order to keep it from crashing and burning.</blockquote><p>What we needed to do, essentially, was to <strong>keep all the valuable pieces of information</strong>, while <strong>eliminating the differences</strong>.</p><p>So, check out the thought process:</p><p>We’ve got <strong>Product “A”</strong>, which <strong>appeals to both James and Nick</strong>. James and Nick give the product a <strong>three out of five score</strong>.</p><p>However, <strong>James </strong>grades the products he’s interacted with, with an <strong>average of 2.8</strong>. <strong>Nick</strong>, on the other hand, grades with an <strong>average of 4</strong>.</p><p>So, even though both have given Product “A” the <strong>same grade</strong>, the <strong>grade itself </strong>means <strong>something different </strong>for either one of them.</p><p><strong>James </strong>is a <strong>sophisticated user </strong>and has <strong>higher standards</strong>, therefore he scores the products with <strong>lower grades</strong>. That being said, a <strong>3 out of 5 review </strong>is considered to be a <strong>good one for James</strong>.</p><blockquote>James’ standards are higher (and he’s a little tight-handed and strict when it comes to reviews).</blockquote><p><strong>Nick </strong>is the exact <strong>opposite</strong>: He’s generous and kind and <strong>doesn’t mind handing out good reviews</strong>. Therefore, giving it a <strong>three out of five</strong>, means that the product has a <strong>poor performance score</strong>.</p><blockquote>We had to remove the bias from our interaction terms. This is, actually, a matrix normalization technique that helps us remove personal interests, make unbiased calculations and add the customers’ extracted interest at the end of the matrix factorization process.</blockquote><p>So, here’s what we’ve got:</p><p>We created the <strong>“R” matrix</strong>, using <strong>customer-product interactions </strong>with the <strong>decay scores</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*mmHUze1s878NyWf6ULksNA.png" /></figure><p>Then, we <strong>removed the bias </strong>from the “R” matrix:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*VEOO-8DD53qXaUCt7YHuMg.png" /></figure><p>Decompose the R — Debiased matrix</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/465/1*OQmGi0jvpVjjz2_Un5RppA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*cWLwYmXggz186OEX9UgcjQ.png" /></figure><p>Then, we <strong>calculated the dot product of “P” and “Q” matrix </strong>and <strong>added the bias we removed before decomposition</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*dd2b5-ZYt_BPKW23ANfPpw.png" /></figure><p>Pretty simple, eh? Nah, it most definitely wasn’t.</p><h3>Takeaway</h3><p>In the next article, I am going to present my crew’s final finding: How the final model we implemented works, and how we can compare it to the previous models we made.</p><p>Read on and stay tuned for more adventures!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ef68f1e19923" width="1" height="1" alt=""><hr><p><a href="https://medium.com/moosend-engineering-data-science/recommender-systems-optimized-ef68f1e19923">Recommender Systems: Optimized!</a> was originally published in <a href="https://medium.com/moosend-engineering-data-science">Moosend Engineering &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Validation Framework For Recommender Systems: A Quest]]></title>
            <link>https://medium.com/moosend-engineering-data-science/building-a-validation-framework-for-recommender-systems-a-quest-ec173a24b56f?source=rss-460972859609------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec173a24b56f</guid>
            <category><![CDATA[evaluation]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[validation]]></category>
            <category><![CDATA[recommender-systems]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Dimitris Apostolopoulos]]></dc:creator>
            <pubDate>Mon, 10 Jun 2019 10:46:53 GMT</pubDate>
            <atom:updated>2019-06-10T10:46:53.905Z</atom:updated>
            <content:encoded><![CDATA[<p>Hello again and welcome to yet another one of my quests on <strong>building a Recommender System </strong>that will tackle all problems and bring optimal results!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8YNcNgZBuG4VDSmIt5N_LA.jpeg" /></figure><p>Today, I am going to show you how I guided my team through the process of building a <strong>Validation Framework</strong>.</p><p>Now, one of the trickiest (if not <strong>the </strong>trickiest) parts of Recommender Systems is <strong>measuring the quality of recommendations </strong>the model generates, seeing as the validation framework techniques used in Machine Learning don’t really apply to our case or, sadly,<strong> don’t describe the results as a whole</strong>, only partly.</p><blockquote>In this article I am going to present an alternative way on how you can evaluate a recommendation engine and built a validation framework for testing our models.</blockquote><p>This quest aimed to present another, <strong>alternative </strong>way of evaluating a recommendation engine and <strong>building a validation framework</strong>, all in favor of <strong>testing the models</strong>.</p><h3>Evaluation Techniques for Recommender Systems</h3><p>There are two ways to evaluate a recommendation system: The <strong>online </strong>way and the <strong>offline </strong>way.</p><blockquote>I won’t tell you which one steered the wheel to the right direction, but I will describe how they work. The decision, after that, will be yours to make:</blockquote><h4><strong>1. Online evaluation for Recommender Systems</strong></h4><p>In order to go through with this plan, you’ll need to <strong>deploy the algorithm to the production</strong>, <strong>track</strong> down any and all <strong>recommendations </strong>it generates and <strong>validate </strong>those <strong>through customer interaction</strong>.</p><p>This <strong>may represent the real performance</strong> of the algorithm and may seem like a good, solid choice, it is, however, <strong>time consuming</strong>. The <a href="https://moosend.com/blog/marketing-metrics/"><strong>metrics</strong></a><strong> </strong>Online Evaluation needs in order to work, are the following:</p><ul><li>Customer Lifetime Value (<strong>CLTV</strong>)</li><li>Click-Through Rate (<strong>CTR</strong>)</li><li>Return On Investment (<strong>ROI</strong>)</li><li>Purchases</li></ul><blockquote>Another thing you should keep in mind, would be that every time you make a <strong>modification </strong>to the algorithm, you have to <strong>deploy anew</strong> and <strong>wait for the evaluation</strong>. Also, you need to <strong>apply A/B testing principles</strong> and make sure you’ve fished out all the <strong>right data</strong>.</blockquote><p>It is only normal that <strong>this takes time to run</strong>, especially if you’re aiming for those <strong>long-term metrics</strong>.</p><h4><strong>2. Offline evaluation of Recommender Systems</strong></h4><p>Offline evaluation can be divided into two categories: <strong>Implicit</strong> and <strong>Explicit</strong> <strong>Feedback</strong>.</p><blockquote>Implicit feedback will help you <strong>estimate the results</strong> through each <strong>interaction with the product:</strong> Clicks, views, add-to-cart, purchases, etc.</blockquote><blockquote>Explicit feedback requires that you <strong>measure a different component:</strong> A <strong>score </strong>from the customer, or perhaps an <strong>upvote</strong>.</blockquote><p>In both cases, a Data Pirate like myself needed to <strong>split the dataset </strong>into <strong>two training and validation sets. </strong>This division is meant to <strong>measure the performance </strong>with a <strong>metric </strong>or a <strong>group of metrics.</strong></p><h4><strong>Offline Evaluation Metrics</strong></h4><p>There are many evaluation metrics to be used for offline evaluation. The ones that are most used are the <strong>RMSE, Normalized Discounted Cumulative Gain (NDCG@k), Precision@k, Recall@k </strong>and <strong>F1 Score</strong>.</p><ul><li><strong>RMSE</strong></li></ul><p>In order to calculate this metric, you need to <strong>have ratings</strong>. RMSE’s values range from 0 to 1 and the <strong>lower </strong>the RMSE, the <strong>better</strong>.</p><p>The <a href="https://en.wikipedia.org/wiki/Mean_squared_error"><strong>Mean Squared Error</strong></a><strong> </strong>is calculated between the <strong>predicted ratings (p) </strong>that our algorithm produced and the <strong>actual ratings (a).</strong></p><p>This is the route:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/553/1*etyb3dYfsrkegJewO8L_cA.png" /></figure><p>Onto our second metric!</p><ul><li><strong>NDCG@k</strong></li></ul><p>This metric is <strong>one of the best known</strong>, due to the <strong>information it can retrieve</strong>. It takes the <strong>order of recommendations </strong>into account and it’s very <strong>helpful </strong>in terms of measuring the <strong>quality </strong>of <strong>web search </strong>and <strong>recommendation engines</strong>.</p><p>The results are <strong>evaluated in terms of relevancy (rel)</strong> of the <strong>actual result</strong> compared to the <strong>predicted result list</strong> and its values range between 0 and 1. Of course, the <strong>best value</strong> you could aim for is <strong>1</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/553/1*CcksNtCq_FrsQR5Rn8whnA.png" /></figure><p>Next in order of business, and in order to <strong>normalize the results </strong>from the different lists, we <strong>divide </strong>the <strong>DCG </strong>by the <strong>ideal DCG </strong>(<strong>IDCG</strong>):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/553/1*4yNhmAOQpiaHYsAYd2ac4A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/553/1*7cxpoXXSceACJO43SQcQdg.png" /></figure><p>I wouldn’t want to tire you with references from all my previous adventures, and seeing as <strong>Accuracy, Precision@k and Recall@k</strong> have been analyzed in a previous article,<strong> I’ll </strong><a href="https://medium.com/moosend-engineering-data-science/the-road-to-recommender-systems-d6bb79bd169d"><strong>ship you there</strong></a>.</p><ul><li><strong>F1 score</strong></li></ul><p>This metric measures the tests’ <strong>accuracy</strong>.<strong> </strong>The F1 score is the <strong>harmonic average of (p) and (r) </strong>and the values, again, <strong>range between 0 and 1</strong>, with 1 indicating the <strong>perfect precision and recall</strong>.</p><p>The following equation is what guided us through this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/553/1*-Xg3CYfLvFZmpXbbwDfxgQ.png" /></figure><h3>All aboard the Simulation Ship</h3><p>We didn’t have a lot of time of course, <strong>not nearly enough</strong> to work on improvements and test as many scenarios as me and my crew would’ve liked, when it came to <strong>online evaluation of our Recommender System</strong>.</p><blockquote><strong>We decided to do it offline</strong>.</blockquote><p>However, the standard offline evaluation just <strong>couldn’t cut it.</strong></p><p><a href="https://moosend.com/"><strong>Moosend’s</strong></a> database is an <strong>unlimited source of data</strong>, and we needed a very solid plan to avoid getting shipwrecked.</p><p>The only way around that, was to <strong>create a simulation</strong>.</p><p>That simulation would <strong>measure the performance of the model</strong> and was meant to <strong>track down the improvements we perform</strong>, on top of that.</p><p>We needed to <strong>train </strong>our recommendation engine first. For this, we utilized 3 months’ worth of interaction data, geniously named <strong>“The Training Section”</strong>.</p><p>Right when this period <strong>ended</strong>, we entered the <strong>second phase</strong>: The <strong>“Recommendation Day”</strong>.</p><p><strong>After 29+ days</strong>, we created a different time period called <strong>“The Validation Set”</strong>.</p><p>This period is the time during which we <strong>wait</strong>, in order to check if the <strong>customer </strong>is going to <strong>interact </strong>with the <strong>recommended products</strong>.</p><p>Here’s the process:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*XWDf085fWL-9ng6iYugfbg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*Z3qiyCBnEUe2sV9n85S6cQ.png" /></figure><p>We repeated the process <strong>from the very beginning</strong>, in order to generate product recommendations for the <strong>following day</strong>.</p><blockquote>Let’s assume that you’ll need to <strong>recommend products on December 1st</strong>. Your <strong>training data</strong> should be from <strong>September 1st, up until November 30th</strong>.</blockquote><blockquote>And then, you <strong>wait</strong>. Up until December 30th, more specifically, to make sure that <strong>the customer is going to interact</strong> with the recommended product.</blockquote><p>This is the way to <strong>simulate the inner workings of your model under real circumstances</strong> and, at the same time, <strong>evaluate its performance</strong> during different time periods, seeing as you’ve already got the data.</p><p>But what metrics were used for the evaluation of the simulator? <a href="https://medium.com/moosend-engineering-data-science/the-road-to-recommender-systems-d6bb79bd169d"><strong>This</strong></a><strong> is where you’ll find out</strong>!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*E8POnl9AJJgRGKs8rTzAFQ.png" /></figure><h3>The Segmentation Trick for Recommender Systems</h3><p>Let’s say that you’ve got a chap that <strong>won’t interact with anything but one product</strong> and that alone.</p><p>The <strong>lack of information</strong> will send our recommendation engine to a new adventure, in search of <strong>similar customers</strong> that will be, down the line, <strong>irrelevant </strong>to our original customer.</p><blockquote>You see, there’s <strong>not enough data</strong> to determine whether or not there are any similarities.</blockquote><p>Our <strong>need to measure how our model performs</strong>, in regards to the information provided by the data of the “Training Section”, made me and my crew <strong>split the customers</strong> into three different categories, based on the product interactions.</p><p>So, we created the <strong>“New”</strong>, <strong>“Regular” </strong>and <strong>“VIP” </strong>categories.</p><p><strong>New</strong> customers were the ones with 1–4 product interactions. <strong>Regular </strong>customers were the ones with 5–10 product interactions. <strong>VIP </strong>customers were all the customers with more than 10 product interactions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*NiIJI7bpPhoZdYY9qlsHBA.png" /></figure><h3>Stay tuned!</h3><p>This journey enabled us to measure how the model performs through a period of time, by combining <strong>simulator</strong> and <strong>segments</strong>.</p><p>It <strong>describes </strong>the model in various ways and can provide a <strong>full report</strong> on our model, <strong>depending on the information</strong> that we have on each of the customers.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec173a24b56f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/moosend-engineering-data-science/building-a-validation-framework-for-recommender-systems-a-quest-ec173a24b56f">Building a Validation Framework For Recommender Systems: A Quest</a> was originally published in <a href="https://medium.com/moosend-engineering-data-science">Moosend Engineering &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Recommender System: The Information Enhancement Trick]]></title>
            <link>https://medium.com/moosend-engineering-data-science/recommender-system-the-information-enhancement-trick-79dd12ea88e3?source=rss-460972859609------2</link>
            <guid isPermaLink="false">https://medium.com/p/79dd12ea88e3</guid>
            <category><![CDATA[recommender-systems]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[recommendation-engine]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[recommendations]]></category>
            <dc:creator><![CDATA[Dimitris Apostolopoulos]]></dc:creator>
            <pubDate>Mon, 27 May 2019 14:10:57 GMT</pubDate>
            <atom:updated>2019-05-29T15:25:34.029Z</atom:updated>
            <content:encoded><![CDATA[<h3>Recommender Systems: The Information Enhancement Trick</h3><p>Hello and welcome to another article on my attempt to tame the seas of <strong>Recommender Systems</strong> and fight the Kraken (or product-you decide how to name it!)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zrz2q5XEosrO0auBbNLiaQ.jpeg" /></figure><p>In my two previous articles, you saw my <strong>methodology </strong>on how to master working with products, annihilating the monster that made things very difficult.</p><blockquote>You see, the more the data (the products and customers of the shops in our database, in our case), the larger and more complex the calculations. Large and complex calculations take up a <strong>huge amount of time</strong>. I had a proper Lernaean Hydra to tackle!</blockquote><p>I needed to take a lot of time to solve this problem, but I had another task at hand as well: Like any good hero would do, I needed to help out smaller shops with interaction data, as theirs were simply not enough.</p><p>In this article though, I will…</p><p>(Drumroll, please!)</p><p>…Investigate the <strong>impact </strong>my quest had on our <strong>model</strong>.</p><blockquote>Oh, and if you’re feeling lost, <a href="https://medium.com/moosend-engineering-data-science/the-road-to-recommender-systems-d6bb79bd169d"><strong>here’s your map</strong></a>.</blockquote><p>But let me remind you what <strong>Recommendation Systems</strong> are, first:</p><p>A recommendation system is a system that aims to <strong>predict the preferences</strong> of a user when it comes to an item.</p><p>It uses either the <strong>data from a user’s past interactions</strong> with an item (views, purchases, ratings, etc.), in order to make recommendations of similar products to the user, or some key components of the item the user has interacted with, which will result in <strong>recommending items with</strong> <strong>similar features</strong> or use.</p><p>But I know you’re not up for long narratives, this is action and we need to get straight to the point. So, without further ado:</p><h3>Product clustering and entity recognition steered my wheel in a different direction</h3><p>In order to avoid more traps, I used the same structure as my first model, but with a twist: I applied the principles of <a href="https://medium.com/moosend-engineering-data-science/product-clustering-a-text-clustering-approach-c392c2ef4310"><strong>Product clustering</strong></a> and <a href="https://medium.com/moosend-engineering-data-science/named-entity-recognition-matching-products-to-genders-a91fb466e3a3"><strong>Named Entity Recognition</strong></a> to the whole process.</p><p>Let me tell you, it did make things somewhat safer.</p><p>For those of you who don’t remember, <strong>Product Clustering </strong>is the process through which I managed to put all same or similar products of different brands under the same cluster.</p><p><strong>Named Entity Recognition </strong>is the process through which we seek and identify information units in unstructured text and classify them into predefined categories.</p><blockquote>Pretty handy, if I say so myself!</blockquote><p>Now, before <strong>merging the interactions</strong> of the customers with the products, I went through with product clustering.</p><p><strong>This created groups (clusters) of products.</strong></p><p>The next step was to assign the interactions of the customer to the cluster that the products they interacted with, belong.</p><p>This lead all the information that I already had for each product to be <strong>enhanced</strong>. I used the interaction data and shared it across all the products in the cluster.</p><blockquote><strong>Let me break it down a bit for you:</strong></blockquote><p>John bought a Nike t-shirt, Jane bought an Adidas t-shirt and Peter bought Adidas shoes.</p><p>The t-shirts ended up in the same cluster. However, the shoes didn’t.</p><p>The interactions were assigned like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*KnKBHHEmhzjedMGzuRhSew.png" /></figure><p>But there was also another thing to keep in mind: A cluster contained products from only <strong>one gender</strong>.</p><p>Therefore, it wasn’t possible for someone to select a product that belonged to another gender, with the exception of the neutral (unisex or otherwise not specified) products.</p><h3><strong>Model Workflow</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/1*AfUE3OFCjf2kU_RiObKIqA.png" /></figure><h3>The woes of selecting a product (and how I solved the problem)</h3><p>Selecting a product to recommend from a cluster is not all fun and games. These little buggers can get a bit tricky sometimes.</p><p>So, I had another problem in my hands that I just had to solve, in order for my endeavors to really matter.</p><p>I tried to find a way to score each individual product in the cluster <strong>according to popularity</strong>. But there was a parameter I didn’t keep in mind: This made me end up selecting the exact same product from the cluster, over and over again.</p><p>The problem with that was the fact that the highest score was assigned to <strong>only</strong> <strong>one product each and every time</strong> and there was no other kind of data to salvage the rest.</p><p>Now, as if that was not enough, this scoring method enabled another monster to appear:</p><blockquote><strong>A product for men ended up in a cluster with women’s products.</strong></blockquote><p>I had no idea how to get out of that one and the only logical solution at the time was to <strong>randomly select</strong> each individual product from the cluster.</p><h3>It ended up being a numbers game</h3><p>As evident, I was at a small loss at that point. However, there was no solution but to push through-even if this meant that I would crash and burn in the process.</p><p>I used the same metrics that I’d used for the <strong>first model</strong> I’d built, but there was a small problem stalling the whole process: the clusters.</p><p><strong>At that stage,</strong> I had product clusters instead of individual products. The clusters are comprised of the same or very similar products.</p><p>This left me with no choice but to measure the performance of my “ship” in two different levels: <strong>The cluster and the product level.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/1*h3VLlcrOMlbZRvBvHew-9g.png" /></figure><p>The results are pretty clear here. My first model was incomplete, but by adding product clustering and then the named entity recognition, I managed to <strong>improve</strong> my original model’s individual components and push through.</p><ul><li>Its <strong>accuracy </strong>was increased, from 2,3% to 7,58%</li><li>Its <strong>precision </strong>was increased, from 0,51% to 1,45%</li><li>Its <strong>recall</strong> <strong>rate</strong> was increased, from 1,8% to 2,06%</li></ul><blockquote>So far, so good and this trip was branded as a successful one!</blockquote><p>Or so I thought. But I was quick to judge…</p><h3>Numbers do lie (if you count wrong)</h3><p>Yes, the size of the matrix was decreased when I applied product clustering. The computation time was decreased as well (<strong>Hooray!</strong>). But this doesn’t mean I managed to bring it down to a working size, it was still too large to manage.</p><p>The other issue I faced, was the fact that the <strong>calculations</strong> were <strong>too complex</strong> for my plan to work properly.</p><blockquote>But it didn’t stop there.</blockquote><p>My calculations were way off on a thing or two (stay tuned to find out how I managed to fix all that!)</p><p>As I observed the products that model recommends and compared them to the product interaction information that I fed to my model, I realized that something was more than a bit off.</p><p>There was no distinction between the interactions with the products (views, add to cart, purchases, you name it!) and the fact that the customers were different individuals.</p><blockquote><strong>This was simply not a part of the equation at all.</strong></blockquote><p>To me, it sounded like a thunderstorm was approaching. I had no option but to find a solution.</p><p>And I did find it. But I won’t tell you how just yet. <strong>Tune in for my next adventure.</strong> More monsters, complex mechanics and heroic quests await!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=79dd12ea88e3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/moosend-engineering-data-science/recommender-system-the-information-enhancement-trick-79dd12ea88e3">Recommender System: The Information Enhancement Trick</a> was originally published in <a href="https://medium.com/moosend-engineering-data-science">Moosend Engineering &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Named Entity Recognition: Matching Products to Genders]]></title>
            <link>https://medium.com/moosend-engineering-data-science/named-entity-recognition-matching-products-to-genders-a91fb466e3a3?source=rss-460972859609------2</link>
            <guid isPermaLink="false">https://medium.com/p/a91fb466e3a3</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[text-clustering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[named-entity-recognition]]></category>
            <category><![CDATA[pattern-recognition]]></category>
            <dc:creator><![CDATA[Dimitris Apostolopoulos]]></dc:creator>
            <pubDate>Mon, 13 May 2019 14:14:48 GMT</pubDate>
            <atom:updated>2019-05-13T14:27:04.127Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bKzz5N-dtrNhBKPaVAO0eQ.jpeg" /></figure><h3><strong>What is named entity recognition?</strong></h3><p><strong>Named Entity Recognition</strong> is the process through which we seek and identify information units in unstructured text and classify them into predefined categories including persons, organizations, date-time expressions, locations, etc.</p><h4><strong>A little backstory</strong></h4><p>When I set out to put together the latest, fastest Product Recommendation system, despite my best intentions, <strong>I couldn’t have foreseen certain aspects</strong>.</p><p><strong>One of these is Named Entity Recognition</strong>, which is road bump number two along the way.</p><p>Hey, we’ve already done this before, in part one about <a href="https://medium.com/moosend-engineering-data-science/the-road-to-recommender-systems-d6bb79bd169d">Recommender Systems </a>and part two about<strong> </strong><a href="https://medium.com/moosend-engineering-data-science/product-clustering-a-text-clustering-approach-c392c2ef4310">Product Clustering</a><strong>, </strong>and now it’s time for part three.</p><p><strong>In this article, we will discuss the improvements we can make in regard to product clustering.</strong></p><p>Our main issue is that with the current clustering technique, we have the same cluster products attributed to different genders. Which means it’s not working. But it should. Oh, and it will.</p><blockquote>When it comes to social interaction, gender is one of the variables that can divide people into categories.</blockquote><p><strong>This is what the recommendation engine that we are currently implementing does</strong>, as we try to identify similar customers. Hence, these bits of information will be of major importance.</p><p>But before we continue, I’ll need to point out again that we have no information about the customer, other than their product interactions.</p><h3><strong>Named Entity Recognition: Working with Data from Products</strong></h3><p>The only data we can use as far as the product itself is concerned is the product title. This means that <strong>all the named entity extraction can be done only through text</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*LvXPJ48ZX9b4-_GY6h4D_w.png" /><figcaption>(<a href="https://moosend.com/blog/customer-segmentation-ecommerce/?utm_source=Medium&amp;utm_medium=Medium%20Article">source</a>)</figcaption></figure><p><strong>The only data we can use</strong> as far as the product itself is concerned <strong>is the product title</strong>. This means that all the named entity extraction can be done only through text.</p><p><strong>In Named Entity Recognition the most common recognizable elements are:</strong></p><ul><li>Organizations</li><li>Names</li><li>Brands</li><li>Geographical Locations</li></ul><p>So, we analyze and extract the selected entities of a text using a parser.</p><p><strong>But what are the entities we could recognize from product titles?</strong></p><blockquote>In our case, the <strong>named entity worth recognizing was the brand name</strong>, therefore we need to exclude it from the product clustering process.</blockquote><p>This means that the remaining terms stay intact.</p><p>However, product titles are too short and most brand names don’t have the appropriate format, so named entity recognition through a parser (for syntax analysis) will not work properly.</p><p>Therefore, we will need another process for the recognition aspect.</p><p>As <a href="https://moosend.com/?utm_source=Medium&amp;utm_medium=Medium%20Article">Moosend’s</a><strong> </strong>employee, I have access to its database which contains a lot of data, including one of the biggest brand lists.</p><p>And having a large brand list at your disposal means that you can simply remove the brands that you identify from the products, <strong>through a regular expression.</strong></p><p><strong>The second named entity recognized is the gender of a product</strong>, taking into account the consumers it is addressed to. So, we need to create 4 categories of products: <strong>male</strong>, <strong>female</strong>, <strong>kids</strong> and <strong>neutral</strong>.</p><p>Disclaimer: Now, for the purposes of clarity, I will be following a binary logic of distinguishing between male and female items.</p><p>But I kindly ask for the LGBTQ community’s understanding. Lipstick will fall into the female category, and razors under the male one.</p><p><strong>When designing a Product Recommender system</strong> for one of your clients, <strong>you will need to pre-define what falls into what category</strong>, be it in terms of gender, be it in terms of age, culture, and so on.</p><p>We proceed with the creation of some hard-coded rules to determine male, female and kids’ products.</p><p><strong>Some of these rules can be the words “Men”, “Women”, “Kids” in text</strong>.</p><p>The products that can’t be categorized from the title alone or are unisex products, fall under the “neutral” category.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*x9ZOpeibWv29BDB8g2Ur4Q.png" /></figure><h3><strong>Named Entity Recognition: Recognizing Customer’s Interest</strong></h3><p>We can categorize customers by their interests or, in our case, by the gender category they interact with the most.</p><p>Customer categorization uses the same categories as product categorization:</p><ul><li>Male,</li><li>Female,</li><li>Kids,</li><li>and Neutral</li></ul><p>When we complete the process of product categorization, we proceed to calculate the percentage of product genders every user interacts with.</p><p><strong>Τhe next step is to set a threshold to the percentages</strong>. Those that have a percentage that is higher than the threshold, we categorize to the gender category.</p><p>For these steps, <strong>I would recommend a threshold between 0.8–0.9</strong>, depending on the products you recognize in the previous process.</p><p>Below, we present 4 rows of data examples from the process:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*0AVOua0pcdjHlBY5X3Kcfw.png" /></figure><p><strong>Both James and Nick are males</strong> but they are interested in different product categories.</p><p>In fact, <strong>95% of James’ interactions take place with “male” products</strong> and there are 5% interactions with neutral products or products the gender of which we can’t determine, <strong>so we clearly categorize James as male.</strong></p><p>On the other hand, <strong>92% of Nick’s interactions are with female products</strong>, 7% of his interactions are with neutral ones, and 1% is with kids’ products.</p><p>Subsequently, <strong>we determine that Nick’s interested mostly in female products</strong>. Therefore, we categorize him as female.</p><h3><strong>Eternal loops between customers and products</strong></h3><p>We are about to feed everything into the algorithm and see how that goes.</p><p>One issue of the process is that, in many cases, the product title of a product is not a descriptive measurement for the gender category.</p><p>Not every item is conveniently named “MEN’S SHAVING CREAM”. Some are just named “SHAVING CREAM”, while others are named “DREAM CREAM” or “VENUS LEGS” (Marketing teams, what can you do.)</p><p>As a result, this leads to a great number of products ending up in the <strong>neutral</strong> category.</p><p>In order to reduce the distribution of the products in the neutral category, in the last step of the process, we create an “<strong>exchange loop</strong>” (don’t look it up, I made the name up) between products and customers.</p><p>More specifically, once we get the biggest part of our data from <strong>products </strong>and <strong>customers</strong>, we have to merge information together.</p><p>We represent all the recognized customers with their gender product interactions, formatting a vector for every single one, then we take the almost perfectly classified customers to the gender category ( more than 98% ).</p><h4><strong>Named Entity Recognition Example:</strong></h4><p>Let’s suppose that Jane purchases a lipstick, a dress, a skirt, and a bag.</p><p>We place every product in a vector (like the image below) and we label the recognized products with their gender category, after that, we calculate the percentage of items that fall within the same category.</p><p><strong>In detail:</strong></p><p><strong>Jane purchases show us that 98% of her products are labeled as female</strong> and 2% as neutral <strong>so we recognize her as female</strong> due to the fact that she interests mostly for “female” products.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*a52o08Ees4HP6LfJr8W8WQ.png" /></figure><p>Now, we only have<strong> the product gender categories</strong> that fall between the classified and neutral categories.</p><p>This leads to the assumption that neutral products will probably be categorized in the same gender category as the rest.</p><p><strong>If we detect the same pattern with the same product multiple times </strong>(90% of the data that contains that pattern), <strong>we classify the product to the gender category</strong>.</p><p><strong>Simply put:</strong></p><p>If someone has purchased four “female” category items, we will “deliberately” assume that the fifth one will also be “female”.</p><p>Correspondingly, when we see “replicas” of similar purchase patterns, we will also assume that these are one or the other, and therefore attribute them accordingly.</p><blockquote>After we have categorized all the products that we can identify, we repeat the customer categorization with the new data and we repeat the process till we can’t identify other products and customers.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*-TTbYKkl_g0P1n58wAIXWg.png" /></figure><h3>Conclusion</h3><p>In my next article, I will take you back to square 1 of <a href="https://medium.com/moosend-engineering-data-science/the-road-to-recommender-systems-d6bb79bd169d">Product Recommender systems</a>, only this time we will revisit everything that didn’t work last time.</p><p>Essentially, we will be monitoring the performance of our recommendation engine after we applied the <strong>BRAND NEW product clustering and the SHINY NEW named entity recognition</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a91fb466e3a3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/moosend-engineering-data-science/named-entity-recognition-matching-products-to-genders-a91fb466e3a3">Named Entity Recognition: Matching Products to Genders</a> was originally published in <a href="https://medium.com/moosend-engineering-data-science">Moosend Engineering &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Product Clustering: A Text Clustering Approach]]></title>
            <link>https://medium.com/moosend-engineering-data-science/product-clustering-a-text-clustering-approach-c392c2ef4310?source=rss-460972859609------2</link>
            <guid isPermaLink="false">https://medium.com/p/c392c2ef4310</guid>
            <category><![CDATA[text-clustering]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[clustering]]></category>
            <dc:creator><![CDATA[Dimitris Apostolopoulos]]></dc:creator>
            <pubDate>Mon, 15 Apr 2019 13:30:25 GMT</pubDate>
            <atom:updated>2019-05-15T06:36:51.501Z</atom:updated>
            <content:encoded><![CDATA[<p>In my previous article, I steered the wheel of my ship to “<a href="https://medium.com/moosend-engineering-data-science/the-road-to-recommender-systems-d6bb79bd169d">Recommender Systems</a>” islands, but crushed on the rocks <strong>of performance of my matrix systems.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*niuDGQA9hKcEFhczcotOHw.png" /></figure><p>So now, one by one, over the articles to follow, I will be picking up each of the variables which did not perform and try to fix them (Spoiler alert: Everything gets fixed in the end).</p><p>Let us take a quick scan at where we left off.</p><h3>“Painspotting”: Finding the issue</h3><p>As described in the first part of this series of articles, one of the issues we faced was the following: the interaction matrix from our previous article was too large to <strong>measure</strong>. This was due to the volume of data we gathered which contained duplicate information because the same products or similar ones can be found in different shops.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/935/1*X8PkYKu4Zhjqm6Vv0Rv2nw.png" /></figure><p>In other words, if you sell headphones and three of your competitors also sell the same brand of headphones, there will be duplicates on your matrix, slowing you down for no reason.</p><p>This is why the purpose of this article is to <strong>implement cross-shop identification of the same or very similar products</strong>.</p><h3>“Go over the motions”: Merging information</h3><p>We will use product information (namely Product Code, Product Title, Product URL and Product Price), as provided by our data set.</p><p>Now, every shop uses its own in-house system to track the products. Therefore, Product Codes are unique for every shop.</p><p>Worse, we can’t rely on Product Prices either, since prices vary from one shop to another.</p><p>This leaves us with two options: Product Title and the Product URL.</p><p>Product URL can be a good source of information, if one can build a Web Scraper to drag the data from the website page but, because of the “unstructured-ness” of the HTML, we can’t have a Web Scraper that can work for every website.</p><p>This leaves us with just one option, that is, for the whole clustering to be done using just Product Titles.</p><h3>“Doing the homework”: Text Pre-processing</h3><h4>What is Pre-processing?</h4><p>Text pre-processing refers to all adjustments that a text must undergo before it is fed to the algorithm.</p><h4>Follow These Text Clustering Pre-processing Steps</h4><p>The pre-processing steps that we go through, in regard to the data, are:</p><ul><li>First, we identify the brands and remove them from the title, so that we are left with the product name alone.</li><li>Then, we remove words which describe colors in order to reduce the noise of the data, since, at this point, we want to categorize products by color. For instance, we want to create a category of “Black Converse All Star shoes 10” and another one of “White Converse All Star shoes 10.5”.</li><li>Afterwards, we remove numbers and the units of measurements (if any) from the title, because we want to create groups with very similar products like “Cola 330ml” and “Cola 500ml”.</li><li>Finally, we stem the words, that is, remove the suffix of the word in order to find a common root, and remove stopwords altogether.</li><li>In order to feed the title data into an algorithm, we convert data to vectors. To achieve that, we use 2 different vectorizers: <strong>CountVectorizer</strong>, which creates a binary vector with {0,1}, and<strong> tf-idf Vectorizer</strong>, which assigns a weight to all words, based on the frequency of the word in all vectors. In this case, we use both vectorizers to find the one that works better for us.</li></ul><h3>Moving on: Text Clustering</h3><h4>What is Text Clustering?</h4><p>Text Clustering is the process through which one can generate groups within unlabeled data. In most techniques of clustering, the number of groups are predefined from the user, but in this case, the number of cluster groups has to change dynamically.</p><p>We can have clusters that contain one single product and clusters that contain 10 or more; the number depends on how many similar products we can find.</p><blockquote><em>Our needs lessen our options, in the world of clustering to </em><a href="http://mccormickml.com/2016/11/08/dbscan-clustering/"><em>DBSCAN</em></a><em>. The DBSCAN is a density based algorithm that relies on how close the vectors are to each other, in order to create the group.</em></blockquote><p>DBSCAN Result Groups:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*HgbsN_Prqgx7ZLc0opescw.png" /></figure><p><strong>I know what you’re thinking: Does the Coyote ever catch Road Runner?</strong></p><p>Not that? Ok, how about:</p><h3>Why does the DBSCAN fail to cluster the data correctly?</h3><p>The title of a product is a very short sentence (1–5 words). However, the vectors we create are very large because every unique word from our data composes our vocabulary. This will be the length of our vectors, so we lose all the information.</p><p><a href="https://www.geeksforgeeks.org/dimensionality-reduction/"><strong>Dimensionality reduction</strong></a> techniques such as PCA and SVD won’t help resolve this issue, seeing as every column of the transform matrix represents a word.</p><p>Therefore, when you remove some columns consequently, you will remove a lot of products.</p><p>As the ready solutions we have, don’t work properly, we decide to build a custom clustering process, aiming to find a solution to our problem.</p><h3>Break your sneakers: Training a vectorizer</h3><p>When you <strong>train a vectorizer</strong>, it learns the words that the given sentence contains.</p><p>For example, upon being given “Nike Capri Shoes”, the vectorizer learns only these 3 words. This means that when you transform all the other products, their vectors will be filled with 0 except for the ones that contain a word or all the words.</p><p>To find how similar 2 vectors are, we measure their similarity with the <strong>Euclidean distance</strong>. To set 2 words in the same group, the distance has to be higher than our threshold. The groups that we generate are called <strong>categories</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*_JdbZpL4OuQjFtY8o24akg.png" /></figure><p>Think of our data like a big bucket of products. Categories are useful, because they create smaller buckets which contain related data that we can process.</p><p>Now we create <strong>subcategories for </strong>every bucket by running the same process again with a higher threshold and create the subcategories. Subcategories are the final groups we will use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*Bu4vHOm3qSe2Tk0VnV0t5Q.png" /></figure><h3>Changing gears: Tips to Improve Your Speed</h3><p>The whole process is a bit time-costly. In order to save time, we will go through all the text pre-processing steps, except for vectorization.</p><p>After that, we sort our data based on number of words that the title contains, so the titles with 1 word will go to the top of the list and the titles with the most words at the bottom.</p><p>One-word titles will form the majority of our categories, reducing the volume of the data we process.</p><h3>Success! Now making my way out (with flair)</h3><p>On our <a href="https://medium.com/moosend-engineering-data-science/named-entity-recognition-matching-products-to-genders-a91fb466e3a3">next article</a>, we will continue working with any information we can extract from our product.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c392c2ef4310" width="1" height="1" alt=""><hr><p><a href="https://medium.com/moosend-engineering-data-science/product-clustering-a-text-clustering-approach-c392c2ef4310">Product Clustering: A Text Clustering Approach</a> was originally published in <a href="https://medium.com/moosend-engineering-data-science">Moosend Engineering &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Road to Recommender Systems]]></title>
            <link>https://medium.com/moosend-engineering-data-science/the-road-to-recommender-systems-d6bb79bd169d?source=rss-460972859609------2</link>
            <guid isPermaLink="false">https://medium.com/p/d6bb79bd169d</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[recommender-systems]]></category>
            <category><![CDATA[recommendation-engine]]></category>
            <category><![CDATA[recommendations]]></category>
            <dc:creator><![CDATA[Dimitris Apostolopoulos]]></dc:creator>
            <pubDate>Mon, 01 Apr 2019 13:38:42 GMT</pubDate>
            <atom:updated>2019-04-28T14:52:40.601Z</atom:updated>
            <content:encoded><![CDATA[<p>In this series of articles, I’m going to share the edifying experience of mastering <strong>Recommender Systems</strong> and the implementations deriving from basic to advanced models around recommendation systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jZoC9hp9kHOEueaF1CS5Dg.jpeg" /></figure><p>I will also briefly refer to the Recommender-system-related challenges you will be coming across when building such a system, and what I did to overcome them.</p><p>Lastly, I will walk you through the ideas, unsuccessful attempts, and the validation framework I used to <strong>track the performance of the model</strong> and the other machine-learning models I constructed, in order to reach the successful formula.</p><p>But first,…</p><h3>What is a Recommender system?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/935/1*yHEbhQgRQWc9zZLCGkTPBw.png" /><figcaption>Recommender systems for personalized customer experiences.</figcaption></figure><p>Recommender systems are arguably the most common application of <strong>big data</strong> in order to <strong>improve individual customer experiences</strong>, by suggesting content for your website.</p><p>Recommender systems are part of information filtering systems and Artificial Intelligence which aim to predict the preferences that someone will have over a plethora of choices.</p><h4>Where do we use Recommender systems?</h4><p>The most common area of use is <strong>product recommendation field filtering</strong> and learning through customers’ preferences, in order to apply this knowledge to others.</p><h3>The Backstory</h3><p>Enjoying life as I was, I decided I needed to shake things up a little.</p><p>Take on a challenge, you know.</p><p><em>Roam the worlds of data science and conquer AI battles.</em></p><p>Aaah, challenges.</p><p>You know, the type of thing you regret when it’s already too late.</p><p>Boy, did I take the plunge, and dove into Recommendation Systems, fought the sea monsters and here I am. Sinbad the sailor, if you will, survived to tell the story.</p><p>It must have been about a year or so ago when the Email Marketing and Automation company I work for (<a href="https://moosend.com/?utm_source=Medium&amp;utm_medium=article&amp;utm_campaign=RecSys">Moosend</a>) assigned me to a brand new project.</p><p>The concept was to <strong>create a data-driven recommendation engine</strong> that can fit and work for every e-commerce shop. This general-purpose system has to <strong>produce personalized product recommendations</strong> from their own customer product interactions (which we already have).</p><p>The challenge about this was that it had to be <em>entirely dynamic</em> and able to adapt to<strong> </strong>various patterns, namely to seasonal <strong>purchase patterns</strong> (e.g. during gift-giving periods such as Christmas, Hanukkah, Easter etc. ), but also to <strong>revenue maximization</strong>.</p><h3><strong>Understanding Recommendation Engines</strong></h3><p>Every time I get a new project the first thing I do is try to understand the basics; <strong>when </strong>and <strong>what </strong>it will be used for, the <strong>structure </strong>of a system, the <strong>variety</strong>, and the <strong>scalability </strong>that it could have.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*bbcb5xlPPWFe-B7QlwK4FQ.png" /></figure><p>There are different types of <strong>Recommender Systems (RecSys) </strong>and what you choose depends on the strategy you want to follow to approach your customers.</p><p>Now, never before has the world been driven by data. But now, most of the biggest e-commerce sites rely on data-driven decision systems to amplify their sales.</p><p>And personalized product recommendations are AI’s gift to the word of eCommerce, since they can help you <strong>increase your click-through rate (CTR) and sales rate</strong>.</p><p>Like I said, there are 5 different types of AI recommendation systems:</p><blockquote><strong>Content-Based</strong> recommender systems identify similar products based on their attributes, i.e. the characteristics of each product.</blockquote><blockquote>We represent every product with its attributes (for example, the attributes of a mobile phone are Screen size, Price, Camera, Software, etc.) and we try to find the most similar ones.</blockquote><blockquote>In this way, we <strong>suggest similar “mobile phones” to someone</strong> with an interest for a mobile with specific characteristics.</blockquote><blockquote><strong>Collaborative Filtering</strong> engines identify the preferences of similar customers and are based on the idea that people with similar behavior share similar interests.</blockquote><blockquote>In these systems, we represent every customer with their interactions and this way, we <strong>predict the probability of their interest for every product</strong>, that is, the likelihood of the customer actually appreciating the product recommended to them.</blockquote><blockquote>As a result, we can approach a new customer by recommending products for the most similar ones.</blockquote><blockquote><strong>Hybrid Systems</strong> are the combination of Content-Based and Collaborative Filtering recommender systems.</blockquote><blockquote>Score every given product from both models and weigh each result; the final recommendations will result from the linear combination of the 2 scores.</blockquote><blockquote><strong>Association Rules</strong> or <strong>Market Basket Analysis</strong> engines are slightly different from the previous ones.</blockquote><blockquote>Having a large data set of interaction data we can <strong>find patterns for items frequently purchased together as a sequence</strong>; for example if someone has added coffee to his cart but hasn’t added sugar, we recommend sugar.</blockquote><blockquote><strong>Repeat Purchase</strong> engines <strong>predict the specific time or approximate time by which a customer will have purchased a specific product</strong>.</blockquote><blockquote>This algorithm uses the product duration, purchase history and day statistics in order to predict the future date.</blockquote><blockquote>For example, if someone buys monthly contact lenses, we recommend the same product every 30 days in case they have forgotten to purchase them. This way, we encourage them to continue shopping with our store.</blockquote><h3><strong>Collaborative Filtering</strong></h3><p>To refresh your memory, this type of recommendation engine tries to <strong>identify users with similar interests</strong> by using their product interactions (purchases, product views and add to cart products).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/1*owm2XtnBo16V2ysP2oKgpA.png" /></figure><p>You can implement Collaborative Filtering for recommender systems with one of two methods; the <strong>memory-based</strong> one or the <strong>model-based</strong> one. In both methods, we represent customers with their interactions<strong><em> like vectors formatting a matrix.</em></strong></p><p>In the memory-based one, you measure the distance between all the vectors (customers) with each other and recommend products from the most similar.</p><p>In the model approach, widely known as <strong>matrix factorization model for recommender systems</strong>, we recognize <strong>latent factors</strong> in data.</p><p>In the world of statistics, <strong>latent factors</strong> are not variables that we observe or measure with a direct way, but a set of variables which explain (describe) other variables and their relationship in a lower dimensional space without losing the information.</p><p>In our case, latent factors find and decode patterns for every customer in order to identify the similarities between them.</p><h3>Recommender System Model #1 (My first shot)</h3><p>The first model I came up with was a <strong>standard matrix-factorization model.</strong></p><p>In this particular one, we represent a customer by their product interactions in a 2D sparse matrix named R ; <strong>sparse matrix</strong> is a well-known as computation-efficient and memory-efficient way to store a large amount of data all together and ready to process.</p><p>The <strong>rows of the matrix</strong> represent your customers and columns represent products like vectors, then we fill the customer-product interaction cell with the value of 1.</p><p>As you can imagine, the cell with the products that the customer did not interact with remains empty:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/1*K3gKJGy919Uj-ZKE9zpCTg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/465/1*OQmGi0jvpVjjz2_Un5RppA.png" /></figure><p>The next step of the process is to split the R matrix into two, one for the customers (P) and one for the products (Q) with their latent factors. Then, we tune both matrices with a <strong>lambda function</strong> and measure the error rate from the values of the original R matrix. When the error rate drops from the given threshold, we interrupt the process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/1*c2fNDOpgv82EakpylClUrA.png" /></figure><p>To format the R-hat matrix we calculate the <strong>dot product</strong> of P and Q. In linear algebra, dot product is the result of the matrix multiplication.</p><p>The last step is to recommend <strong>a group of products sorted by the highest purchase probability</strong> of the specific customer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/1*uNa-XTltjVZMiLXE-yVx2A.png" /></figure><h3><strong>Performance metrics</strong></h3><p>In order to monitor the performance of our model, we have to<strong> measure the quality of the recommendations</strong> that the model produces.</p><p>In recommender systems, we measure the performance of an engine with Precision@k and Recall@k, which are widely used in <strong>information retrieval scenarios</strong>.</p><p><strong>Precision </strong>is defined as the Number of recommended items that the customer has interacted with (:viewed, added to cart, etc.), divided by the number of items in the recommendation set k.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*tSuv7nflhqTw5516-c_0DQ.png" /></figure><p><strong>Recall </strong>is the number of recommended items in recommendation set k that the customer has interacted with, divided by the universal number of items that the customer has interacted with, even outside the recommendation set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*A-PVjstvfwjtKHMPJAlv_g.png" /></figure><p>Also, we used one more measurement for our system called <strong>Accuracy </strong>score in order<strong> to measure the overall performance</strong>. We define the accuracy score as the sum of recommendation sets with which a customer has interacted (minimum 1 interaction/set), divided by the number of total customer recommendations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/553/1*K2T-ZD4XvqwoU8NMB79o9A.png" /></figure><p>In all models, we <strong>measure the performance of the models</strong> in the top 5 (k=5) recommended products.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/1*2PA1na1-auZYO31Ws1Ldow.png" /></figure><h3><strong>The Good, the Bad and the Ugly</strong></h3><p><strong>Spoiler alert:</strong></p><blockquote>The cons of the model outnumber the pros.</blockquote><p><strong>The good thing</strong> about implementation is that the model and process are quite straightforward and simple for someone who can understand the basics and has some experience in the field. Also, the implementation of this model allows us to have all the information into a single “trained” matrix, which is ready for recommendations in production.</p><p><strong>Now, about the cons</strong>. When new shops join the recommendation engine, given the matrix sparsity, there is exponential growth in computations and time spent on them.</p><p>As a result, after dozens of sites, the system will have consumed a lot of memory and <strong>will take days to tune and work properly</strong>.</p><h3>Conclusion</h3><p>The <strong>most important thing</strong> when producing personalized product recommendations is <strong>data</strong>; as a case in point, a large number of medium-small shops do not have enough interaction data to produce personalized recommendations of their own.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/1*DxVS4r84-SievDelIFbs6Q.png" /></figure><p>Observing the disappointing results and working over the purpose of the system, I decided to change my course and <strong>focus on how I process and distribute the information.</strong></p><p>Thus, I can <strong>help smaller, data-weak shops</strong>, and decrease the size of the interaction matrix.</p><p>In the <a href="https://medium.com/moosend-engineering-data-science/product-clustering-a-text-clustering-approach-c392c2ef4310">next article</a>, I will walk you through the way that we can automatically merge the information of products and interactions between different shops.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d6bb79bd169d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/moosend-engineering-data-science/the-road-to-recommender-systems-d6bb79bd169d">The Road to Recommender Systems</a> was originally published in <a href="https://medium.com/moosend-engineering-data-science">Moosend Engineering &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Improving Subject Lines: An endless quest using Machine Learning (pt. 2/2)]]></title>
            <link>https://medium.com/moosend-engineering-data-science/improving-subject-lines-an-endless-quest-using-machine-learning-series-8b6cb3f563ba?source=rss-460972859609------2</link>
            <guid isPermaLink="false">https://medium.com/p/8b6cb3f563ba</guid>
            <category><![CDATA[decision-tree]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[email-marketing]]></category>
            <dc:creator><![CDATA[Dimitris Apostolopoulos]]></dc:creator>
            <pubDate>Fri, 13 Oct 2017 08:48:06 GMT</pubDate>
            <atom:updated>2017-10-16T06:32:37.396Z</atom:updated>
            <content:encoded><![CDATA[<p>The second part of this series for my loyal followers. Hi Mum. Hi Dad.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1jgzVu-yRgtOkiVrFQYiDw.png" /></figure><p><a href="https://medium.com/moosend-engineering-data-science/exploring-the-subject-line-series-de1560496d9"><strong>In the previous part</strong></a> I built a model which was able to <strong>predict the open rate</strong> of an email campaign <strong>based on the morphology of a subject line </strong>and external characteristics of a campaign.</p><p>Awesome, right? <em>I know, Ma’.</em></p><p>Today I’m taking the dose of action up a notch. Yep, things are about to get <em>real crazy</em>.</p><p>This time, I’ll try to <strong>predict the open rate</strong> by building a model that <strong>i) considers words, ii) measures the impact of each word, and iii) its performance in the subject line</strong>.</p><p>Get coffee.</p><h3>Part 2: A linguistic approach of a subject line</h3><p><em>Successor to “Non-linguistic approach to subject lines”</em></p><p><strong>Machine learning aims to understand learning principles</strong> through the computational process.</p><p>By “computational process”, I mean the representation of features on a word-less level, a purely mathematical level in order to <strong>unearth hidden patterns</strong> or <strong>optimize pattern combinations</strong>.</p><blockquote><em>But how do we make sense out of this computational process?</em></blockquote><p>The answer is through human theories. By the few, hand-picked fellas who see stories where there’s numbers.</p><p>Theories are the main source to render a model effective and productive. It’s like a sparkle that starts a chain reaction: good theories and mathematical representation in an <strong>n-dimensional feature space</strong> unlock the model’s knowledge as soon as data flows in.</p><p>*fist bump*</p><h3>Diving into the subject line</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PB2FPbIMbXSPxzogUoiHEw.jpeg" /></figure><p>Here at <a href="https://moosend.com/?utm_source=medium&amp;utm_medium=post&amp;utm_campaign=ml3"><strong>Moosend</strong></a><strong>,</strong> we spend a lot of time constructing theories, and exploring ideas that can make the <strong>impossible lose its prefix.</strong></p><p>Our mission is to serve a greater purpose for the email marketing community: break the subject line code!</p><h3>The goal of our latest model implementation was to break down elements and distinguish them into discrete categories.</h3><p>Think of the subject line as a sequence of placeholders:</p><p>We represent every <strong>element</strong> (word or symbol) as a placeholder and we measure the impact of each element.</p><p>In this way, we measure not only the impact of a word, but also the impact of a pattern or a pattern combination in it.</p><p><strong>This is how we constructed our model.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/953/1*orb-yIA2btRQTLBCPSnx0g.png" /></figure><p><em>“ The subject line is a sequence of placeholders. We can remove, replace or even find the best combination of elements in order to improve it.”</em></p><h3>Subject Line Pre-processing</h3><p>Before I started processing the text and removing elements around the subject line, I ran an <strong>entity recognition.</strong></p><p>In doing so, I extracted specific entities from each subject line which would help me identify pattern and pattern combinations, as well as their relationship influencing the open rate.</p><p>Some of these entities are <em>brands, currency, percentages, emojis, duration patterns,</em> and so on.</p><p>Our first step in text processing is to remove numbers and punctuation.</p><p>Then, we <strong>tokenize</strong> every subject line, <strong>remove stopwords</strong> and <strong>stem each word</strong>.</p><p>Stemming takes place in order to find the common root of words. As a case in point, the words: “claims”, “claiming”, and “claimed” have the same impact in a sentence, as they are derivatives of the same root.</p><p>Our goal here is to <strong>focus on the root of a word,</strong> not on the prefix or suffix. This way, we <strong>reduce the wordlist</strong> and have a better understanding of the impact of a word in a sentence.</p><h3>Feature Construction Area</h3><p>In part 1 of this series, we went over the importance of sender performance history. We’ll be using this feature in this <strong>model implementation, </strong>too.</p><p>We must also consider the number of recipients of a campaign because it’s the denominator of the open rate.</p><p>After subject line pre-processing, we create our wordlist and score each word.</p><p>To complete the scoring process, we establish <strong>2</strong> <strong>measurements</strong> for each word: <strong>word score</strong> and <strong>position score</strong>.</p><p>The word score:</p><p><em>For every subject line </em><strong><em>s_i,</em></strong><em> we score the </em><strong><em>w_ij</em></strong><em> word, where </em><strong><em>j</em></strong><em> is the position of the word in </em><strong><em>s_i</em></strong><em>, with the sum open rate from the campaigns that the word is found in, divided by the number </em><strong><em>N</em></strong><em> of campaigns.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/327/0*8k9aQEKUiARcyeA7." /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/697/0*sNNy8MiN5l9h6aAy." /><figcaption>Figure 1: Relationship of Word Score and Open Rate</figcaption></figure><p>The position score:</p><p>The second measurement is based on the theory that the score is influenced by the number of words in the subject line and also the position of each word in it.</p><p>Essentially, the words placed at the beginning of the subject line are considered <strong>stronger words, </strong>while those at the end are considered <strong>weaker.</strong></p><p>Thus, our measurement scores the word position with the sum of the open rate of the subject line divided by the word position in <strong>s_i</strong> and the product number of those, divided by <strong>N</strong> subject lines that the word is found in:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/386/0*0fVfbHEQCUOwwERJ." /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/703/0*TpKsgOU-ldUILEnl." /><figcaption>Figure 2: Relationship of Word Position and Open Rate</figcaption></figure><h3>Model Selection (An another forest)</h3><p>As in the maiden post of this series, I went for an ensemble method to approach this problem, only this time I chose <strong>Gradient Boosting Regressor</strong> (GBR) as it performs better without overfitting, compared with the other ensemble algorithms.</p><blockquote>GBR is more flexible in parameter tuning than Random Forest: some core parameters that enhance my model are the number of <strong>estimators</strong>, the <strong>learning rate</strong>, the <strong>depth</strong> and the <strong>loss function</strong>.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/581/0*aYqd1zLVttMdRHN9." /><figcaption>Figure 3: Model progress through training steps</figcaption></figure><p>A large number of estimators can boost our accuracy but it can also lead to <strong>overfitting </strong>— so, <em>how can we prevent this from happening?</em> The <strong>depth parameter</strong> limits the number of nodes in the tree and prevents overfitting of each tree in the model. The learning rate is how fast our model changes “beliefs”, that is, how fast it learns new things on the top of old one. There is an exchange in the learning rate, estimators, and depth, therefore, these parameters must be tuned with the best combination to produce results that prevent overfitting.</p><p>In Figure 3, we see how the loss of the model (Train Loss) is changing in association with the number of estimators (Iter):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/629/1*eO9Gb79unn4-TcRv0HpYdA.png" /></figure><h3>Evaluation</h3><p>To evaluate our results, we use the same measurements as in the previous model (R-Squared, MAE, MSE, Accuracy) and we will <strong>compare and contrast</strong> them so we can see how this method describes the whole problem:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/662/1*DCMXerBTUhvnDwo7aYQH_A.png" /></figure><p>Simply put:</p><p>This model implementation boosted model accuracy by 10%, dropping the Mean Absolute Error (MAE) by around 43% compared with the Random Forest implementation.</p><p>We can see clearly that the value of R-Squared increased, which means our <strong>independent</strong> variables (features) of this model describe the <strong>dependent</strong> variable (Open Rate) in a better way than the previous model.</p><p>This means, that we can use this model as a baseline tool for a whole system of marketing “assistants”. These marketing “assistants” could improve a subject line not only on pattern level, but also on word or syntax level.</p><p>The verdict?</p><p>Succe- -Ma, please, no, don’t call aunt Gertrude. No, it’s not Nobel-worthy. <em>But it’s getting there.</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2Frt7VK4v3cfHfa%2Ftwitter%2Fiframe&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2Fanimation-animated-cartoon-rt7VK4v3cfHfa&amp;image=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2Frt7VK4v3cfHfa%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="327" frameborder="0" scrolling="no"><a href="https://medium.com/media/7cbbb9ca3127d3db3c1ec2a789001a0c/href">https://medium.com/media/7cbbb9ca3127d3db3c1ec2a789001a0c/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8b6cb3f563ba" width="1" height="1" alt=""><hr><p><a href="https://medium.com/moosend-engineering-data-science/improving-subject-lines-an-endless-quest-using-machine-learning-series-8b6cb3f563ba">Improving Subject Lines: An endless quest using Machine Learning (pt. 2/2)</a> was originally published in <a href="https://medium.com/moosend-engineering-data-science">Moosend Engineering &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Improving Subject Lines: An endless quest using Machine Learning (pt. 1/2)]]></title>
            <link>https://medium.com/moosend-engineering-data-science/exploring-the-subject-line-series-de1560496d9?source=rss-460972859609------2</link>
            <guid isPermaLink="false">https://medium.com/p/de1560496d9</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[email-marketing]]></category>
            <dc:creator><![CDATA[Dimitris Apostolopoulos]]></dc:creator>
            <pubDate>Tue, 12 Sep 2017 14:08:02 GMT</pubDate>
            <atom:updated>2017-10-16T06:33:25.584Z</atom:updated>
            <content:encoded><![CDATA[<p>I open up about open rate prediction in email campaigns. Glam is a tough job, but someone’s got to do it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sHgJZ0QKKOrGLhsF38oVnQ.jpeg" /></figure><h3><strong>Part 1: Non-linguistic approach to subject lines</strong></h3><p><em>Precursor to “Linguistic approach to subject lines”</em></p><p><strong>Today’s article picks up on email open rate prediction.</strong></p><p>Which emails do you open? What influences your decision? Is it the subject line? Is it the timing? Is it a lot more?</p><p>I’m setting out on a quest to <strong>discover what influences open rate prediction</strong>.</p><p>I will be sharing what this road map brings back with you. I’d love to hear your comments, love to reply to questions, love to help out. <em>I’m the outgoing type of data scientist, yes.</em></p><p><strong>*Guitar intro solo*</strong></p><h3>I am fascinated by <em>modern</em> marketing tools:</h3><p>developed from data-driven Machine Learning algorithms, modern marketing tools encompass all things Future; the potential is crazy and mainly untapped. What a great time to be alive, right?</p><p>I tend to see modern martech tools as “doors” to some kind of radically new perspectives that lead to solutions, and opportunities for businesses and marketers. Ultimately, we are talking about safer, calculated decisions even when the user has low or even no experience in the field.</p><blockquote>Amazing how data can reveal hidden patterns, right? Measuring things that humans generally ignore or the human mind can’t understand.</blockquote><p>In the past few weeks <em>I’ve been trying</em> to predict the open rate of email campaigns based on the subject line of the email.</p><p>By subject line, I’m referring to it as a whole, without considering the words used or their meaning.</p><p><strong>I work for </strong><a href="https://moosend.com/?utm_source=medium&amp;utm_medium=medium_organic&amp;utm_campaign=mediumblog"><strong>Moosend</strong></a><strong>, so I have access to some millions of emails and insights to come up with various implications for further research.</strong></p><p>In today’s case, what I did was to<strong> look for a pattern in the subject lines of the most opened emails from our database</strong>. I completely disregarded the meaning of words.</p><p>Disregarded. Ignored.</p><p>Instead, I focused on morphology alone. I went looking for the grammar-and-syntax pattern there is across thousands of subject lines.</p><h3>So this is why I went for patterns over meaning. At first.</h3><p>Consider some subject lines with the word “<strong>free</strong>”. The word “free” has shades of meaning, which influence the semantic value of the word depending on the context. Also, depending on the country or a national advertising trend, subject lines with that word could boast a higher open rate. Or fall into “spam”, either by email clients or perception-wise. Worse, the word could even be bound by several cultural implications per English-speaking country, <em>which I would not get into, in the first place.</em></p><p>Focusing purely on morphology of the subject line, namely the number of characters, the number of words, the factor/influence of punctuation and other parameters like the influence of segmentation, total recipients, the industry the sender is in, and the day the campaign was sent.</p><h3>Open rate as a variable</h3><p>Open rate is a delicate continuous variable and as such, it can be influenced by a variety of factors. Can’t think of more than the subject line itself? Think again.</p><blockquote>Every time you open your inbox you read certain emails and ignore certain others. Why did you open email A and ignored B?</blockquote><p>I am sure there can be <strong>a bunch of reasons why</strong> email A open rates outnumber those of email B. Maybe you didn’t like the subject line, or thought that the sender is a spammer, maybe the profile picture looked fishy.</p><h3>These all make sense BUT.</h3><p>All the aforementioned hypotheses make that regression problem much more complex and much more unstable because we can’t measure them. This practically means that, <em>in all probability, we cannot achieve a high-accuracy model.</em></p><h3>So we move on to Feature Construction.</h3><p>The features of our model are the single most important group of variables that influence the results of the model.</p><p><strong>Constructing a good feature space is a prerequisite for higher accuracy, </strong>and better features means higher flexibility and better results even when the approach of a problem is not excellent.</p><p>External characteristics of an email campaign (see sender industry, day email was sent, total recipients, segmented list vs non-segmented list) are mostly predefined and don’t require any special transformation but the subject line have to be transformed to multiple features so we can use them as inputs for our model.</p><p>In order <strong>to analyze the morphology of a subject line</strong> we must create a function that takes the subject line and looks for specific features, such as the ones mentioned earlier. This becomes much easier to implement, using these two core libraries for data manipulation: NumPy and Pandas.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/694/0*j0hfQLCmJo9xNTRi." /><figcaption>Figure 1: The effect of emoji on open rate of email campaigns</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/0*-nxL2-83lFranjTY." /><figcaption>Figure 2: The redistribution of open rate through days</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/0*1VKWW9WMcL8NYwus." /><figcaption>Figure 3: How the number of words of a subject line effects the open rate</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/694/0*R6mp41vmgDsXEtKF." /><figcaption>Figure 4: The effect of exclamation mark on open rate of email campaigns</figcaption></figure><h3>Sender integrity adds value.</h3><p>We cannot overstate the importance and value of the <strong>email sender</strong>. When we receive an email in our inbox the sender and the subject line are the only clues we have.</p><p>To measure the sender, I could either assign a unique ID like category value, or measure their performance history. Measuring the latter using a continuous value can work as normalization term for our model. This is because every user has established their “public image” across their subscribers and the open rate doesn’t change unexpectedly, except in some cases with good segmentation, but can be changed over time.</p><p>So my model takes the sender’s performance history as a baseline for the final prediction and measures the distance above or below the vertical axis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/699/0*FUmW2Bw-xI7N658R." /></figure><h3>Normalization</h3><p>Data preprocessing is crucial as the success of the model itself lies in the quality of preprocessing. In many cases it depends on the method you use and what you want to accomplish. It’s a technique that makes things work and maybe a tad more complicated.</p><p>In our feature space scaling varies: we have binary, multiclass and continuous variables so we choose a normalization technique for the preprocessing step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*YJ_4kjJUJJwejLhm." /></figure><p><em>Normalization is a simple mapping (rescaling) of data between 0 and 1 using the following formula</em></p><h3>Model Selection</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nPjvCTrTNBu8BuvuvEtICA.png" /></figure><p>To boost accuracy, the model approach is based on the <strong>Ensemble method</strong>. Ensemble learning is generally used to average the predictions of different models to get a better prediction to a more complex problem. Ensemble methods are like using the predictions of small “expert” models in different parts of the input space, so we choose the <strong>Random Forest Regressor</strong> as the perfect tool for the job. Forest outperformed all Ensemble algorithms. <em>It’s in its nature.</em> <em>(I didn’t say I’m a funny guy.)</em></p><p>The next step is to employ Hyperparameter Tuning of machine learning algorithms which can optimize the performance of the model based on the nature of the research at hand. Optimization is made easy by Python with SciKit Learn package, and GridSearchCV to find the best parameter combination.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/628/1*QW9rlP0QjBXwhnez7YFCzA.png" /></figure><h3>Evaluation</h3><p>To evaluate our model I used <strong>4 measurements</strong>, namely R-Squared, Mean Absolute Error (MAE), Mean Squared Error (MSE), and accuracy. In regression, accuracy can be measured directly through R-Squared, MSE and MAE which give us enough information for a regression model. In our case, it’s individual campaigns we are talking about, so, to measure the accuracy of our regressor, I created a threshold within a +/-1% range from the predicted value in order to measure the accuracy of our model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/611/1*kKFcEHbWfiDViuRHFEcNpw.png" /></figure><h3>Conclusion</h3><p>Even a simple model like this demonstrates the power of machine learning, since we can predict 7/10 campaigns correctly in our threshold. From the array above, <strong>we see clearly the importance of sender performance history</strong> which boosted our accuracy by 17% and decreased the MAE by 0,90.</p><h3>What this means</h3><p>This means that its accuracy is higher, but there is more to be done.</p><p>The main shortcoming of the model right now is that the morphology model I came up with considers form and placement in the sentence, not meaning. Essentially, it can make suggestions about adding emojis, or personalization, and the effect they will have on the impact of the subject line, but it cannot guide you through the best alternative for a word.</p><p>Let’s consider this example: “Get 5% off”.</p><ul><li>Personalization</li><li>Emojis</li><li>Amount in dollars over percentages</li><li>various equivalents to “get”: “claim”, “discover”, “here’s”, and so on,</li></ul><p>…are a few of the changes that can be incorporated to test the impact of our subject line.</p><p><strong>I will be coming back with more of these posts on this series.</strong></p><p><strong><em>What are your thoughts?</em></strong></p><p>What would you like me to work on in a future post? Leave your comments below and I will make sure to tag you on the next post as an Inspiration Contributor!</p><p>In the meantime, feature engineering, preprocessing, and parameter tuning, essential to machine learning, won’t cease to amaze us any time soon. Data and the aforementioned process make us all intrigued and powerful to <strong>unveil solutions to virtually any data-related challenge.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=de1560496d9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/moosend-engineering-data-science/exploring-the-subject-line-series-de1560496d9">Improving Subject Lines: An endless quest using Machine Learning (pt. 1/2)</a> was originally published in <a href="https://medium.com/moosend-engineering-data-science">Moosend Engineering &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>