<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[delvify - Medium]]></title>
        <description><![CDATA[Transform your business with our AI powered solutions. We help global brands turn consideration into conversion with intelligent insight-driven Programmatic AI with visual AI solutions for eCommerce. - Medium]]></description>
        <link>https://medium.com/delvify?source=rss----f66732270a67---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>delvify - Medium</title>
            <link>https://medium.com/delvify?source=rss----f66732270a67---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 02:30:12 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/delvify" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Reinforcement Learning with Q-Learning (Part I)]]></title>
            <link>https://medium.com/delvify/reinforcement-learning-with-q-learning-part-i-36c2a7de5ba3?source=rss----f66732270a67---4</link>
            <guid isPermaLink="false">https://medium.com/p/36c2a7de5ba3</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[game-design]]></category>
            <dc:creator><![CDATA[Annette Catherine Paul]]></dc:creator>
            <pubDate>Tue, 26 Oct 2021 08:02:25 GMT</pubDate>
            <atom:updated>2021-10-26T08:02:22.422Z</atom:updated>
            <content:encoded><![CDATA[<h3>Reinforcement Learning with Q-Learning — Part 1</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1ZWe8yuPB3yX0Ru3gGpS9Q.png" /></figure><p>How many of you have grown up playing arcade games such as Ms.Pac-Man, Prince of Persia, Dave? My personal favorite is Claw, the pirate cat who sets on a quest to find an ancient amulet containing nine gems while fighting off enemies. My sister used to be the gamer, and I used to be the strategy manager 😉</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F5laSotIO2J0%3Fstart%3D51%26feature%3Doembed%26start%3D51&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D5laSotIO2J0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F5laSotIO2J0%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/68873a10352a555586e3bbacd1546b12/href">https://medium.com/media/68873a10352a555586e3bbacd1546b12/href</a></iframe><p>My sister and I used to spend hours, day and night trying to complete all the levels. The more we played, the better we understood the strategy, we were able to roll out a sequence of decisions. We heavily relied on experience and many many virtual deaths! Let’s face it, we have all done trial and error playing video games before.</p><p>Imagine if these sequences of decisions rolled out could be learned and attributed by a machine in such a manner where you learn from your mistakes and ensure maximum rewards as possible? It is very much possible with reinforcement learning, which is one of the three types of the machine learning paradigm.</p><blockquote>Reinforcement learning (RL) is an area of machine learning concerned with how intelligent agents ought to take actions in an environment in order to maximize the notion of cumulative reward. — [1]</blockquote><h4>Intelligent agents! Wonder what that means?</h4><p>Before we dive in, there are some key terminologies used across RL world. To get us started, let’s dive into an example, a Halloween special!</p><p>Let’s travel back to your childhood, the peak of your love for candy (well for some of us the peak has never saturated 😉). Halloween is around the corner. Since your mother does not trust you around chocolates, you know she has hidden them somewhere around the house, but you face the following challenges:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FNr7RuWKQA8lb2%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FNr7RuWKQA8lb2%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FNr7RuWKQA8lb2%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="337" frameborder="0" scrolling="no"><a href="https://medium.com/media/83bae297a3ac56245c5fc546287eceed/href">https://medium.com/media/83bae297a3ac56245c5fc546287eceed/href</a></iframe><ol><li>You have no idea where it is hidden.</li><li>You have to cross through the hallway across your tattletaling sister, into the living room where your mom is watching TV into the dining area where your father is munching and reading the newspaper and finally into the kitchen, the last place where the candies could be hidden.</li></ol><p>Let the hunt begin!</p><p>In short, your objective as an <em>agent </em>is to maximize <em>positive</em> <em>rewards </em>(candies) as much as possible whilst minimizing <em>negative rewards </em>(encountering your family) by being fully aware and alert of your <em>environment, </em>thoroughly updating your <em>policy</em> from where you will decide on an <em>action</em> to perform, whilst <em>feedbacking </em>and learning from each action.</p><p>Now that’s a mouthful! To summarize:</p><ul><li><strong>Agent</strong>: The system that interacts with the environment</li><li><strong>Policy</strong>: Set of definitions or rules on the actions of the agent given a specific state</li><li><strong>Reward</strong>: On performing an action in a given state(s), the reward you get for enacting the same</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2F1wmMklzKmFOG22utS6%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2F1wmMklzKmFOG22utS6%2Fgiphy-downsized-large.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2F1wmMklzKmFOG22utS6%2Fgiphy-downsized-large.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="294" frameborder="0" scrolling="no"><a href="https://medium.com/media/0d2a3d66de8bae8875ef3d5e5f242677/href">https://medium.com/media/0d2a3d66de8bae8875ef3d5e5f242677/href</a></iframe><p>What if you remember a strategy from the previous year that you used failed miserably and your mother grounded you for two weeks! You play back the whole <em>episode </em>in mind, reminding yourself how the <em>system dynamics </em>changed over one action, and want to ensure that this time it does not happen again. At the same time, you run through various scenarios running the <em>expected amount </em>of positive rewards awaiting you within each strategy.</p><ul><li><strong>Episode</strong>: When you visualize end-to-end sequence of actions given a state(s) until reaching the termination stage</li><li><strong>Value</strong> <strong>function</strong>: At a given state, the total amount of expected rewards that the agent can collect at the end of the episode <em>or </em>given an action, the total amount of expected rewards the agent can collect at the end of an episode. The former is known as <em>state value function</em> and the latter, <em>action value function</em>.</li><li><strong>System dynamics — Model:</strong> Given an action, how the environment can change.</li><li><strong>State space</strong>: All possible states come from a set called <em>state space</em> in a sequence of states called <em>chain</em> which must be distinguishable between each other and unique, thereby following Markov process assumptions.</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2Fcm6gkgSpc18Ay2TKFo%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2Fcm6gkgSpc18Ay2TKFo%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2Fcm6gkgSpc18Ay2TKFo%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="244" frameborder="0" scrolling="no"><a href="https://medium.com/media/4cef8bbb4319f78c755b1cecfaeb40a7/href">https://medium.com/media/4cef8bbb4319f78c755b1cecfaeb40a7/href</a></iframe><p>You suddenly recollect that last time she hid most candies around her and some scattered across from her. If your strategy is to maximize the expected value of candies you can collect, do you go ahead with aiming for the ones around her given that, majority of the candies are placed in that location or do you avoid that and thus avoid the risk of getting caught? The former right? This is why the expected value is not just an additive process without any intuition. The candies near your mother is <em>discounted</em> as we are not certain whether we will be caught or not.</p><p><strong>Discount: </strong>The value of the future rewards at present value. For a higher discounted value, the agent gives more attention to the long term reward rather than the short term one.</p><p>What about beyond this particular use case? There are some cases where the task is non terminal wherein the learning process of best action occurs alongside the interaction with the environment, until an external intervention is initiated.</p><p>Interesting so far right? However, there are some challenging aspects to implementing RL in real world scenario.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FDLG7XAxSMlJudB7g4r%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FDLG7XAxSMlJudB7g4r%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FDLG7XAxSMlJudB7g4r%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="367" frameborder="0" scrolling="no"><a href="https://medium.com/media/f7fb4b0a6923258dc21b1a9ad4e7f649/href">https://medium.com/media/f7fb4b0a6923258dc21b1a9ad4e7f649/href</a></iframe><ol><li><strong>Explore vs exploit?</strong> In the above scenario we have an advantage as we know our own house. But what if we are unaware of the environment? Do we go for an explore and experiment approach or do we adopt a more restrictive approach. Maybe “the road not taken?” This tradeoff is an active dilemma in the filed of RL.</li><li><strong>Short term investment or long term investment?</strong> Rewards are not always immediate. Sometimes they can accumulate into a larger sum which will be given at a later stage. Do you adopt a longer look ahead objective or a shorter immediate return objective?</li><li><strong>Does being intelligent really pay off?</strong> Are the odds really favorable for the intelligently thought out and executed strategy or for the impulsive one? The answer to this is really unknown if you are not aware of your environment.</li></ol><h4>How does it learn?</h4><p>We have mentioned words such as <em>feedback</em> and <em>learn </em>multiple times now. But what does this mean? What strategy do you use to learn? We can either accumulate the rewards at the end of an episode to be able to predict the expected reward in the upcoming episodes or we can estimate this reward output at each step along the way. The former approach is the <em>Monte Carlo approach </em>and the latter is the <em>Temporal Difference Learning. </em>The choice depends on your environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/855/1*eGpYpBO41MGt7q9yb6U4tg.png" /></figure><p>Now that we have an understanding of the various key elements and learning methodologies, let’s take a look at the various strategies of approaching RL. We already covered the <em>value</em> based methodology in the beginning where the objective is to maximize the expected rewards in the future for each given state. So based on a particular state, you would ideally aggregate the candies you have and choose a state that is the most favorable.</p><h4>Are there other means of choosing a strategy?</h4><p>Remember we spoke of policy earlier? Policy is the set of rules or definitions an agent follows for a particular time period. A strategy where it revolves around the policy, is known as <em>Policy Based Strategy</em><strong>. </strong>Policies can be of two types: deterministic and stochastic, where in the former case you have a definitive response action which will never change for that policy (once the response action is defined in a policy, you cannot change it) and the latter case where you would have an idea or a soft margin, i.e., probability for the same actions for output (For example, this response action has a chance of 70%).</p><p>Unlike policy oriented approach in a <em>Model Based Strategy</em> , the RL behavior tends to be learning for an optimal characteristic by observing and performing actions and their outcomes to ensure the next state and its immediate reward. In this case, RL tries to lower the gap of lack of prior knowledge.</p><h4>Q-Learning</h4><p>Let’s dive more into a value based algorithm known as Q-Learning. This algorithm is focused heavily on the player and exploration strategy. Every time you conduct a trial and error exploration, the knowledge about the environment is absorbed and thus, you know the best path ahead. Q-Learning uses a methodology called Q-matrix. At every point, you estimate the maximum expected reward in the future for every action and every state. Hence, in this manner you would know the next best action to perform. This matrix is constantly improved and absorbs deeper understandings.</p><p>The following depiction of Q-learning formula is a great place to start with!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*q7_vEh_Onikc7F4Xhg30KQ.png" /><figcaption>Image Source: Q-Learning Formula, Wikipedia [9]</figcaption></figure><p>Let‘s take a deeper look into this formula.</p><ul><li><strong>Learning Rate</strong>: The size of the step that decides how much you want to override the old knowledge. If the value is 0, you are basically not learning anything whereas the value of 1 will give more importance to the most recent information. Like short term memory loss!</li><li><strong>Discount Factor</strong>: Remember the chocolates near your mother? How important that is, is determined by this factor. Again, if it is 0, then you are only thinking about what’s near you, whereas if it is 1 you are waiting for a high rewarding chocolate!</li></ul><p>We update the new Q matrix with states and actions for a time period <em>t</em> by adding the old value and the temporal difference between the new value and the old value based off of the learning rate provided. In this manner the entire table acts as a “cheat sheet” as rightfully demonstrated in this great article [7]!</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2Fd9QiBcfzg64Io%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2Fd9QiBcfzg64Io%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2Fd9QiBcfzg64Io%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="435" frameborder="0" scrolling="no"><a href="https://medium.com/media/b8c4151f1d990fdc40c644fc08384363/href">https://medium.com/media/b8c4151f1d990fdc40c644fc08384363/href</a></iframe><p>Interesting right? In our next part, we will dive into Open AI Gym which is a toolkit for tasks related reinforcement learning algorithms wherein there are many great environments you can use to run your algorithm on! Stay tuned for the Part II article!</p><p>Thank you for reading this article! How do you use reinforcement learning in your use cases? Feel free to share in comments!</p><h3>Ready To Know More?</h3><p><a href="https://delvify.ai/get-started/">Get in touch with our team</a> of experts to find out more on how implementing reinforcement learning can be beneficial for your brand.</p><p><a href="http://www.delvify.ai/">www.delvify.ai</a></p><h3>Resources</h3><p><a href="https://en.wikipedia.org/wiki/Reinforcement_learning">[1] Reinforcement Learning, Wikipedia </a><br><a href="https://www.kdnuggets.com/2018/03/5-things-reinforcement-learning.html">[2] 5 things you need to know about reinforcement learning, KDNuggets, Shweta Bhatt </a><br><a href="https://medium.com/free-code-camp/an-introduction-to-reinforcement-learning-4339519de419">[3] An introduction to Reinforcement Learning, FreeCodeCamp, Thomas Simonini </a><br><a href="https://www.oreilly.com/library/view/statistics-for-machine/9781788295758/e8f0bbd1-b60a-47d2-9cd4-0d0661fc9212.xhtml">[4] Statistics for Machine Learning, O’Reilly, Pratap Dangeti </a><br><a href="https://simoninithomas.github.io/deep-rl-course/">[5] Deep RL course, Thomas Simonini</a><br><a href="https://link.springer.com/referenceworkentry/10.1007%2F978-0-387-30164-8_556">[6] Model Based Reinforcement Learning, Springer, Soumya Ray, Prasad Tadepalli</a><br><a href="https://medium.com/the-official-integrate-ai-blog/understanding-reinforcement-learning-93d4e34e5698">[7] What is Model-Based Reinforcement Learning, Integrate AI</a><br>[8] Diving deeper into Reinforcement Learning with Q-Learning, FreeCodeCamp, Thomas Simonini<br><a href="https://en.wikipedia.org/wiki/Q-learning">[9] Q-Learning, Wikipedia</a><br><a href="https://gym.openai.com/">[10] Open AI Gym</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=36c2a7de5ba3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/delvify/reinforcement-learning-with-q-learning-part-i-36c2a7de5ba3">Reinforcement Learning with Q-Learning (Part I)</a> was originally published in <a href="https://medium.com/delvify">delvify</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to Clustering Algorithms and its Use Cases]]></title>
            <link>https://medium.com/delvify/introduction-to-clustering-algorithms-and-its-use-cases-35c1655c91e7?source=rss----f66732270a67---4</link>
            <guid isPermaLink="false">https://medium.com/p/35c1655c91e7</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[customer-segmentation]]></category>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[ecommerce]]></category>
            <dc:creator><![CDATA[Annette Catherine Paul]]></dc:creator>
            <pubDate>Fri, 17 Sep 2021 08:33:00 GMT</pubDate>
            <atom:updated>2021-09-20T01:09:39.839Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xoamlA8x7Rbj0zx8llkhDg.png" /></figure><p>Have you ever been in the middle of a conversation, and you suddenly forget what an object is called? If you speak multiple languages and constantly shift between them, then you probably have dealt with this situation. At this point, you would try to recollect and well, let’s admit it, most of the time we would fail and would try and describe that object by associating it with something similar.</p><p>We tried to identify similar objects or associations between this object and other perceivably similar objects to group them together. This concept is typically what a clustering algorithm does.</p><blockquote>Cluster analysis or clustering is the task of grouping a set of objects in such a way that objects in the same group called a cluster are more similar to each other than those in other clusters. — [1]</blockquote><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2Fl4dqF0Aw5S8xXKHP5j%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2Fl4dqF0Aw5S8xXKHP5j%2Fgiphy.gif%3Fcid%3Decf05e47kpmiprd8m5coe9n2x981d4mu6kgn4ragqn3dbj70%26rid%3Dgiphy.gif%26ct%3Dg&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2Fl4dqF0Aw5S8xXKHP5j%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="362" frameborder="0" scrolling="no"><a href="https://medium.com/media/d45dc2f5f0ff8e0e654c77088151850f/href">https://medium.com/media/d45dc2f5f0ff8e0e654c77088151850f/href</a></iframe><h4>But…what if I don’t know what that object is to begin with?</h4><p>When you don’t have a label to work with, you are dealing with an unsupervised problem. If you have some labels, then it would be a semi-supervised problem. Clusters cannot be accurately defined as opposed to a definitive label which is why there are many algorithms that significantly differ on how these clusters are defined.</p><p>Some ways how a cluster is defined are as follows:</p><ol><li><strong>Distance</strong>: How far is an instance away from each other?</li><li><strong>Centroids</strong>: How far is an instance away from the center of each cluster?</li><li><strong>Density</strong>: How dense is each cluster?</li><li><strong>Distribution</strong>: How does the shape impact the cluster?</li></ol><p>When the margin is definitive in the sense that either the instance belongs to the cluster or it does not, then this type of clustering is termed as <em>Hard Clustering</em>. Whereas, when an instance can belong to a particular cluster to a certain degree it is termed as <em>Soft Clustering</em>. The latter provides a score for each instance per cluster. This score could be based on a distance measure or a similarity measure such as RBF.</p><p>Without further ado, let’s analyze a dataset to dive into a better understanding on clustering analysis. Mall customer segmentation data! — [2]</p><p>The complete analysis can be seen from my <a href="https://www.kaggle.com/annettecatherinepaul/introduction-to-clustering-kmeans">Kaggle notebook here [3]</a>. This dataset consists the basic parameters to try and tackle customer segmentation approaches. A snapshot of the dataset is seen below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/1*0PXHHPEHjmPU1M--Xxtw7Q.png" /><figcaption>Mall customer segmentation data snapshot, Image source: By author</figcaption></figure><p>The spending score is based on parameters such as customer engagement and purchasing behavior. It ranges from 1–100. This dataset has 200 such customers and there exists no missing values for any fields.</p><p>Before we begin with any clustering analysis its always a good idea to understand the shape and distribution of the features you are dealing with. Let’s take a look at the pairplot of this dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/709/1*D1UbZVAeEpVrHVUE2GVWAA.png" /><figcaption>Pairplot for data, Image source: By author</figcaption></figure><p>What can we infer from these plots? There seems to be a definitive interesting pattern or groupings between spending score and annual income, age and annual income, as expected. Spending score seems to follow roughly a normal distribution whereas annual income tends to have a skewed nature. Does the scale of these differ by a large amount? Not really, although feature scaling can definitely benefit any model. We need to ensure there isn’t any significant outlier affect on the same when we consider feature scaling methods.</p><p>But what about gender? Gender is a categorical variable, so what if we could try and visualize the same plot but specific to the gender distributions?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/788/1*6OplhrNHeZFZvmxJG2LOWg.png" /><figcaption>Pairplot of the data by gender, Image source: By author</figcaption></figure><p>Now the patterns are visibly changing more and more. However, does gender really seem to have an impact of significance with this dataset? Let’s take a look at the histogram of age by gender to identify if there is indeed any significant relationship for this set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/382/1*1nSjaHGbffq9RPkvZWTjyw.png" /><figcaption>Histogram analysis for age and gender, Image source: By author</figcaption></figure><p>As you can see on the left, age seems to have a significant relationship with the dataset. But how do we visualize this? Age is a numerical field in the dataset and to visualize the same we would need to take a page from stratified sampling to form multiple stratas whilst not affecting the distribution too much.</p><p>Based on this distribution on a cumulative manner, we can create stratas to have a definitive understanding on how many of these customers fall into these various stratas. As seen from the image below, I used, 15–30, 30–45, 45–60 and Above 60 as the stratas based on the distribution shown. The two plots follow roughly the same proportion within each bins.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/773/1*Ch1QstUOTt4us5hzjzUF9w.png" /><figcaption>Strata comparison for age and age range, Image source: By author</figcaption></figure><p>Now that we have created the stratas, let’s visualize the impact of age range on this dataset using the pairplot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/781/1*sikZVMjOUaNc_XJ7IAX0Uw.png" /><figcaption>Pairplot of the data by age range, Image source: By author</figcaption></figure><p>This plot reveals the possibility of cluster analysis and the impact already! If we deep dive into the spending score vs annual income plot the cluster formations, although vague at this point, is visibly significant. Now its time for us to model our clusters!</p><h3>k-Means</h3><p>Before heading to the analysis, let’s try to capture the depth of this beautiful and simple algorithm. The idea of this algorithm is as follows:</p><ol><li>You start with a centroid randomly and provide the number of clusters to be initialized. This value is called ‘k’, hence the name k-Means.</li><li>Using the distance measure to this centroid, you group an incoming instance.</li><li>Based on step 2, you update the centroid again of each cluster based on their distance.</li><li>Iteratively perform 2 and 3 until the centroids stop changing.</li></ol><p>Using the Pycaret package on python, lets run the k-Means model and see what the results look like. When we set up the data we can provide an optional parameter which asks whether we would like a profiling to be performed or not. The profiling results are a part of the Kaggle notebook, if you would like to know more interesting information from the data. The profiling too confirms the insignificance of gender within this dataset based on correlation matrices. The model by default is assigned k = 4. Let’s visualize how this model looks like on our dataset:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*8o_sUEoVZ6CxZ1sjzDHVZw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*eo0FO8xat3MRUFdFyG6I1w.png" /><figcaption>3D and 2D k-Means model cluster plots for the data with k = 4, Image source: By author</figcaption></figure><p>From this visualization, cluster 2 seems to be well segregated from the other clusters. However there seems to be a spread of cluster 1. The marginal borders between cluster 1, cluster 3 and cluster 4, respectively seems to be quite thin. Also do you notice an instance to be what seems like an anomaly from the 3D plot? As you can see, there is a heavy dependency on the value of k for this algorithm which requires you to run the same couple of times in order to arrive at a solution that you are comfortable with.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FCPskAi4C6WLHa%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FCPskAi4C6WLHa%2Fgiphy.gif%3Fcid%3Decf05e47nx6jzlpe4ujevjr3exse3tdsxkrmgzg1axpr5gt7%26rid%3Dgiphy.gif%26ct%3Dg&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FCPskAi4C6WLHa%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="198" frameborder="0" scrolling="no"><a href="https://medium.com/media/e87e8b00b96e40f8925395c19e4e4cc9/href">https://medium.com/media/e87e8b00b96e40f8925395c19e4e4cc9/href</a></iframe><h4>How do I know what model to be comfortable with if I don’t have labels to begin with?</h4><p>As we all know, with any analysis, we need to define the performance measure, or ways to evaluate it in order for us to know how this model would perform for our use cases. We definitely want to limit any surprises after we deploy the model! This algorithm provides you with a guaranteed solution. But is it the most optimal solution? Let’s find out by understanding how to evaluate a cluster model.</p><h4><strong>Inertia</strong></h4><p>It is the mean squared distance between each instance and its closest centroid. The lower the value, the the more denser the cluster is. But is this always a good measure? As the number of clusters increases this value will always decrease. Hence inertia is heavily dependent on the value of k as well. How do we decide on a value for k?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/515/1*hqmXbndemHYk-8vLcxPsZw.png" /><figcaption>Elbow plot to determine the optimal k for the model, Image source: By author</figcaption></figure><h4><strong>Elbow plot</strong></h4><p>Plotting inertia vs number of clusters will reveal a much more informative plot. We can see that there is an elbow at k = 5. After k = 5, the distortion score approaches saturation and this suggests that an optimal k could be k = 5. But this approach is sort of brute force and of course computationally expensive as you have to run the model multiple times with different k.</p><h4><strong>Silhouette score</strong></h4><p>A better measure is the mean of silhouette coefficient across all the instances. This coefficient measures the mean distance between intra-cluster distance, i.e. the mean distance of the instance with other instances within the same cluster and the mean distance of the nearest cluster over the maximum value between these two measures. This measure varies from -1 to 1 and can be depicted as follows:</p><ol><li>Value ~ 1 → The instance is well packed within its defined cluster</li><li>Value ~ 0 → The instance is closer to the boundary of its defined cluster</li><li>Value ~ -1 → The instance most probably does not belong in its defined cluster</li></ol><p>When we use this information and plot every instance’s silhouette coefficient we can obtain a much more informative evaluation of the cluster, and we get a silhouette diagram. Let’s take a look at the significance of various elements within this plot:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/494/1*uzr2aeesKdLpTYgH25zk2Q.png" /><figcaption>Silhouette diagram for Kmeans model with K = 4, Image source: By author</figcaption></figure><ul><li><strong>Height: </strong>Each cluster label has a knife looking formation. The height of this depicts the density or the number of instances within that cluster.</li><li><strong>Width: </strong>The width of these formations are the respective silhouette coefficients. The wider it is, the better.</li><li><strong>Dashed line: </strong>Furthermore, the dashed line is the average silhouette coefficient. If a majority of the instances within a cluster belong to a lower coefficient than this line, we can confidently say that the cluster formations have room for improvement as its mean would be closer to other clusters. Hence majority of the instances should be existent beyond this line and closer to 1.</li></ul><p>In this manner, Silhouette Score and its diagram act as a good source of choice for optimal k.</p><p>Based on the above information, our model has an average silhouette score of 0.4 which shows room for improvement which is also suggested by the elbow plot with k = 5. Most of the instances do not lie beyond the dashed line. Cluster 1 seems to be performing the worst as some instances tends towards -1 and indicates that they probably belong to their neighboring clusters. Cluster 1 has heavy room for improvement.</p><p>As suggested by these plots let’s re-run the algorithm with k = 5. Let’s take a look at their 3D and 2D plots first.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*O4hBGVJrUt3PEy7p6G8mdg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*qQIWtt2-LDN8YtKft9h2og.png" /></figure><p>Visually the cluster grouping seems to be performing better with k = 5. However there seems to be some marginal instances between cluster 1 and cluster 3. What about the silhouette diagram?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/494/1*bE1QEJITPdF-e2eKGlDflg.png" /></figure><p>Even though the elbow plot suggested to approach with k = 5, model seems to be performing worse than before as majority instances do not lie beyond the average silhouette score. There is a tendency in cluster 3 towards -1 which suggests that there are some instances which may belong to the neighboring groups.</p><p>There is definitely room for improvement. As you probably guessed, the influence of the k value, the number of instances within each cluster and the shape of the cluster all has significance on the choice of k-Means to fit with the data.</p><h4>How can I use clustering analysis in my business use cases?</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FA01J1PFAXrvLa%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FA01J1PFAXrvLa%2Fgiphy.gif%3Fcid%3Decf05e471p3afithbo1hcowcxqlqmmhml69rs8g91ek2mo35%26rid%3Dgiphy.gif%26ct%3Dg&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FA01J1PFAXrvLa%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="244" frameborder="0" scrolling="no"><a href="https://medium.com/media/9e5fa1558b7ebb6f4248fdd0b3aaab82/href">https://medium.com/media/9e5fa1558b7ebb6f4248fdd0b3aaab82/href</a></iframe><p><strong>Exploratory data analysis</strong> — What to do with this data? We have asked this question so many times whenever dealing with a new dataset. Clustering forms an integral part to analyzing data especially in the exploratory data analysis stage. It can help you understand the nuances between your features to further give you a better understanding on how to take it from there. After all, any good analysis should be able to increase the number of questions to ask!</p><p><strong>Customer segmentation — </strong>As shown in the example above, understanding a buyer persona and grouping it can help attain various growth objectives, be it, marketing, retargeting audience, demand planning and even customer retention. When a customer lands on your page for instance, based on their interaction we would be able to map them back to a cluster and take strategic decision on how to guide the customer towards your objective KPI. Whether that is improving recommendations in this manner for an uplift in your conversion or whether it is to engage them better and achieve a higher retention rate, clustering analysis to the rescue!</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FBDBYgw22ciY%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DBDBYgw22ciY&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FBDBYgw22ciY%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/f8f7de5c8600307ef939f34ba7f4bb15/href">https://medium.com/media/f8f7de5c8600307ef939f34ba7f4bb15/href</a></iframe><p><strong>Visual search — </strong>Have you ever taken a stroll in the park and noticed some amazing shoes? You suspect that its from your favorite brand, so you snap a photo and try to search online where you would get all similar looking shoes if not for the same one, for you to choose from. Clustering algorithms can group images into various clusters based on their similarity and when shown with a new image, it would simply try to identify which group does this image resemble the most to?</p><p><strong>Fraud detection </strong>— An anomaly is anything that deviates from a what we define as a standard or anything that is not expected. When an instance is shown to a cluster model wherein it performs strange in the sense that it belongs to none of the clusters, then that instance could be something to watch out for or simply might need your attention on! Flagging these instances could potentially save you from a danger. But make sure the instance is an anomaly and not a novelty instance (curious about this? More on that later!</p><h3>Beyond k-Means</h3><p>What are the limitations of k-Means?</p><ul><li><strong>Dependency on k and on initial value assignment: </strong>There is a heavy dependency of the algorithm on k value which means we need to run the algorithm multiple times in order to evaluate and choose a model that’s best fitting. Additionally, there is a dependency of the model on the initial assignment of these centroids, which is randomized.</li><li><strong>Numerical influence: </strong>k-Means can only handle numerical data. Although the purpose of this analysis was to introduce clustering analysis and means of evaluation of the same, as with any algorithm with appropriate data preprocessing, categorical variable handling and feature engineering can improve the performance of the model.</li><li><strong>Shape of the cluster: </strong>k-Means is heavily shape dependent. The algorithm has an assumption that we are mainly dealing with spherical formations. When the shape of the data and its distribution do not align with this assumption we would need to look for other algorithms more suiting in terms of the shape of the clusters. But k-Means act as a great first step in understanding a possible underlying shape of each clusters.</li><li><strong>Number of members:</strong> The number of members within each cluster is an important parameter for this algorithm. The algorithm assumes roughly that each cluster would have the same number of instances which may be a very idealistic thought process for various use cases in the real world.</li><li><strong>Dimensions: </strong>We worked with very limited features. There may be numerous other factors impacting the spending score. When dealing with higher dimensions especially with a distance based similarity measure, it always saturates and converges to a constant value which will heavily bias your model.</li><li><strong>Outliers: </strong>As we saw in our analysis above, outliers can have a significant impact on the centroid as it drags the shape of the cluster. Sometimes, outliers may even get their own cluster!</li><li><strong>Convergence to local minima</strong>: k-Means will give a guaranteed convergence. But this may be a local minima and not global minima. Hence not always will the solution be an optimal solution.</li></ul><p>What’s beyond k-Means? I will deep dive into more clustering algorithms in the coming weeks. Stay tuned!</p><p>Thank you for reading this article! How do you use clustering algorithms in your use cases? Feel free to share in comments!</p><h3>Resources</h3><p><a href="https://en.wikipedia.org/wiki/Cluster_analysis">[1] Cluster analysis, Wikipedia </a><br><a href="https://www.kaggle.com/vjchoudhary7/customer-segmentation-tutorial-in-python">[2] Mall customer segmentation data, Vijay Choudhary, Kaggle </a><br><a href="https://www.kaggle.com/annettecatherinepaul/introduction-to-clustering-kmeans">[3] Clustering analysis, Annette Catherine Paul, Kaggle </a><br>[4] Unsupervised learning techniques, Aurelien Geron, Hands-On Machine Learning with Scikit Learn, Keras &amp; Tensorflow, O’Reilly <br><a href="https://developers.google.com/machine-learning/clustering/algorithm/advantages-disadvantages">[5] k-Means advantages and disadvantages — Clustering in Machine Learning, Google developer courses</a></p><h3>Ready To Know More?</h3><p><a href="https://delvify.ai/get-started/">Get in touch with our team</a> of experts to find out more on how we use horizontal partitioning in our solutions and how we can support your business today.</p><p><a href="http://www.delvify.ai/">www.delvify.ai</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=35c1655c91e7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/delvify/introduction-to-clustering-algorithms-and-its-use-cases-35c1655c91e7">Introduction to Clustering Algorithms and its Use Cases</a> was originally published in <a href="https://medium.com/delvify">delvify</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Conceptual Introduction into Association Rule Mining — Part 2]]></title>
            <link>https://medium.com/delvify/a-conceptual-introduction-into-association-rule-mining-part-2-96c73c4ce87b?source=rss----f66732270a67---4</link>
            <guid isPermaLink="false">https://medium.com/p/96c73c4ce87b</guid>
            <category><![CDATA[data-mining]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[visualization]]></category>
            <category><![CDATA[association-rule-mining]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[Annette Catherine Paul]]></dc:creator>
            <pubDate>Fri, 13 Aug 2021 07:21:37 GMT</pubDate>
            <atom:updated>2021-08-13T07:21:37.041Z</atom:updated>
            <content:encoded><![CDATA[<h3>A Conceptual Introduction into Association Rule Mining — Part 2</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c_T2xwnhbkisBzQNP_7gOg.png" /></figure><p>In the <a href="https://medium.com/delvify/a-conceptual-introduction-into-association-rule-mining-part-i-ed5f61a7600a">Part-I</a> of this series we covered what associations are and aren’t. We also looked into how to validate and evaluate the association rules we have generated and link it to the business use case we have in mind. Now lets deep dive into implementing these concepts using the Apriori algorithm related to Association Rule Mining.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*tCmK7_uWBDeSQmSF7vAmKQ.png" /><figcaption>Image source: By author</figcaption></figure><p>To recap, the main idea of association rules are to recognize associations within different items in a general form of <em>“if antecedent, then consequent”</em>. Obviously in a large dataset when we use such rules, dimensionality reduction and optimizations become a challenge. The main challenge is to address these challenges and try and tackle it efficiently.</p><h3>Apriori Algorithm</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FUOzYwtqCc49lTa0MeZ%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2Ffoilarmsandhog-foil-arms-and-hog-fah-sean-finegan-UOzYwtqCc49lTa0MeZ&amp;image=https%3A%2F%2Fmedia1.giphy.com%2Fmedia%2FUOzYwtqCc49lTa0MeZ%2Fgiphy.gif%3Fcid%3D790b761192a24e853cd970caf728a913f98ffd7c4d13d123%26rid%3Dgiphy.gif%26ct%3Dg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="244" frameborder="0" scrolling="no"><a href="https://medium.com/media/89288e796171dd4ca6cd5bf059e05ba7/href">https://medium.com/media/89288e796171dd4ca6cd5bf059e05ba7/href</a></iframe><p>Have you ever seen the ‘you may also like..’ suggestions across platforms? This algorithm is the main hero behind suggesting items which are relevant to <em>you. </em>In Part 1 of this series it was mentioned that association rule mining cannot map back to the customer level. So what does this really mean? The algorithm looks for a frequent set of items within the data and then uses this information to correlate and arrive on some associations.</p><blockquote>Apriori is an algorithm for frequent item set mining and association rule learning over relational databases. It proceeds by identifying the frequent individual items in the database and extending them to larger and larger item sets as long as those item sets appear sufficiently often in the database. — [2]</blockquote><p>Unlike other association rule based algorithms, Apriori relies on databases consisting of transactions which is further seen as a set of items. Using these transactions and a pre-defined threshold parameter, say k, the algorithm tries to find itemset(s) which are a part of at least k subsets. This is a great illustration of the Apriori principle:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OQBl8F21t6dFVRN-.png" /><figcaption>Image source: Akhil Ram — Affinity Analysis, Kaggle [7]</figcaption></figure><p>Apriori is commonly associated with Market Basket Analysis also known as affinity analysis. The idea here is to recognize useful correlations between different items based on their presence in the dataset. So what are these rules? How do they help businesses grow?</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2F245oYTbugpaxvSBjTS%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2F245oYTbugpaxvSBjTS&amp;image=https%3A%2F%2Fmedia0.giphy.com%2Fmedia%2F245oYTbugpaxvSBjTS%2Fgiphy.gif%3Fcid%3D790b7611836b6dd083a1c8244168a6cc3a82b29db74ef7a3%26rid%3Dgiphy.gif%26ct%3Dg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="244" frameborder="0" scrolling="no"><a href="https://medium.com/media/28f3ca39c6b8b63226ff7ba2eef444ea/href">https://medium.com/media/28f3ca39c6b8b63226ff7ba2eef444ea/href</a></iframe><p>Let’s use an example to understand this algorithm. You can find the full Kaggle notebook analysis <a href="https://www.kaggle.com/annettecatherinepaul/apriori-algorithm-association-rule-mining">here</a> [4]. The dataset used is The Bread Basket [3] dataset. This dataset is ideal for market basket analysis as seen by the snapshot of the dataset below. The dataset has 20,507 transactions based on the items ordered from a bakery.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/1*U24C_ACw28jJ2SxTTy1-Bw.png" /><figcaption>Image source: By author — Snapshot of the Bread Basket dataset</figcaption></figure><p>Before heading into the market basket analysis, it is always a good idea to get a sense of the top contenders to understand how diverse the products are. The following network diagram plots the top 30 products by total count in transactions with the thickest edge belonging to the topmost contender. In this case Coffee is the most frequently occurring item.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*Gjj19XR_G17tVy39-fQulQ.png" /><figcaption>Image source: By author — Top 30 Items on count</figcaption></figure><p>From this diagram one would assume to think straight into looking for associations only with ‘Coffee’. Although this logic definitely makes sense as with high probability there is a chance that most of the associations would be based of a coffee, you may be missing out on some other niche / long tail cart analysis by only focusing on the top most contenders, especially if your main revenue is driven by E-commerce sources.</p><h4>So what is long tail presence?</h4><blockquote>The long tail is a business strategy that allows companies to realize significant profits by selling low volumes of hard to find items to many customers, instead of only selling large volumes of reduced number of popular items. — [5]</blockquote><p>The same reason why Netflix is able to keep up with a stable increase of subscribers every quarter! Their business model and strategy is based on this theory. The online experience is a great way to promote the presence of this long tail theory to exploit and engage customers with longer retention! The following image depicts the importance of considering long tail presence.</p><blockquote>“Even as consumers flock to mainstream books, music and films, there is a real demand for niche fare found only online.” — [6]</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/0*jyvXPSntIOKhMUji.gif" /><figcaption>Image source: Chris Anderson, Wired — Pic depicting the importance of long tail presence</figcaption></figure><p>Hence, association rule mining can help uncover such niche relationships and learn at the same time to help recommend similar niche / long tail products.</p><p>Before we apply the Apriori algorithm we need to pre-process the dataset to a transaction * items matrix with each row representing whether the item was in transaction or not (0/1). A snapshot of the same is shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z7sfAIcTyAKlzNLXiUHGFg.png" /><figcaption>Image source: By author — Snapshot of the transaction * items matrix</figcaption></figure><p>Now the dataset is ready for the Apriori algorithm. First, the rules were generated with a support of 5%, i.e. the item in consideration should be present in at least 5% of the transactions and then with 1%. We need to first generate the frequent item sets using, which the association rules would be generated based off a metric and minimum threshold.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/1*P_Wa030eXRM6Q27-o6KMPw.png" /><figcaption>Image source: By author — Frequent item sets with support 5% (left) and 1% (right) respectively</figcaption></figure><p>As expected, ‘Coffee’ dominates this list. Now let’s take a look at the rules generated by each of the frequent item sets respectively with the metric used as <em>lift</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/863/1*yMeGIP25fe89d6Ia-38meA.png" /><figcaption>Image source: By author — Snapshot depicting the rules generated with support 5% threshold and metric as lift</figcaption></figure><p>The first frequent item set with 5% generated only 4 rules. In practical transactions, 5% is a very constraining threshold and would need a good amount of confidence to sustain. From the above snapshot, we can say that any cart consisting ‘Cake’ would likely have ‘Coffee’ too with 52% confidence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*4QI3C830apqpk_1Txai3zQ.png" /><figcaption>Image source: By author — Snapshot depicting the rules generated with support 1% threshold and metric as lift</figcaption></figure><p>Compared to the previous set of rules generated, with 1% support threshold, naturally more rules are could be generated. The interesting observation here is the top contender has replaced Cake → {Coffee} to Toast → {Coffee} and this time with 70% confidence, with Cake → {Coffee} moving to 6th place.</p><h4>Great! What is the next step?</h4><p>From these rules, if we want to visualize the top associated rules, provided that there are two or more items as antecedents, which in turn answers the question, which item is more likely to be added to the basket next?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/703/1*sqwzS6dCxaCj_Ga_5VkAsw.png" /><figcaption>Image source By author — Visualization depicting the next best item to be in cart for rules generated with support threshold 1% and metric used lift</figcaption></figure><p>Visualizations are very important with association rule mining as visualizations make the conclusions more intuitive and business driven so that people across industries can interpret and make intuitive business decisions. From the above image we can see some possibilities of given antecedents what would the consequents be. For example, {Tea, Coffee} → {Cake}. This can help in many ways, one of which is demand planning / product placements in retail stores.</p><h4>How can I summarize my rules in some concise visualizations?</h4><p>We were able to interpret the above plot as the association rules generated were concise and small given the thresholding we did. What if we had more rules identified and/or lower thresholding which resulted in more identified association rules?</p><p>Let’s try and visualize this scenario for the same dataset but with a lower thresholding of 0.5% as compared to the 1% before.</p><h4>1. Parallel Category Plots</h4><blockquote>Parallel categories diagram is a visualization of multi-dimensional categorical datasets wherein each variable in the dataset is represented by a column of rectangles, where each rectangle corresponds to a discrete value taken on by that variable. — [8]</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LCVwMBOqAOm6DGT4MwSDuQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IUeolbkwc0705ev1TdJKXQ.png" /><figcaption>Image source: By author — Parallel category plots using min support of 0.5% and PyARMviz library</figcaption></figure><p>The above plots depicts the relationship between items for which rules were identified. For example: {Bread, Cake} → {Coffee} relationship is clearly noted. PyARMViz library is an advanced python association rule visualization library which uses the Efficient-Apriori algorithm as its backend. This interactive visualization displays the count of the rules as well when hovered upon.</p><h4>2. Association Rule Strength</h4><blockquote>Association rule strength is a scatterplot with support and confidence on its axes, in addition to the third measure which is lift by default indicated by the color scale. — [10]</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yYAa7a4VGg6FgfSbdIYvCw.png" /><figcaption>Image source: By author — Association rule strength scatter plot with min support threshold set at 0.5% using PyARMViz library</figcaption></figure><p>From the plot above, it is clear that rules with high lift have a relatively low support. In fact, in the original paper by Bayardo Jr and Agarwal [11], they depict how the most interesting rules exist on the border of support/confidence. The interactive features of the library helps these rules stand out from a large base of rules.</p><h4>3. Network Graph</h4><blockquote>Graph-based techniques visualize association rules using vertices and edges where vertices annotated with item labels represent items, and Itemsets or rules are represented as a second set of vertices. — [10]</blockquote><p>These powerful visualizations show a beautiful representation of the rules mined. However, they tend to get cluttered and messy. Hence, it is only useful to visualize small set of association rules mined.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EK6UKKoUHnXYoIu5OECbTw.png" /><figcaption>Image source: By author — Network graph with min support threshold as 0.5% using PyARMViz library</figcaption></figure><p>The above network plot is a snapshot with the coffee node hovered. The complexity of the network is very high that the interactive tool helps to understand intuitively which rules offer the most business value.</p><p>In this manner, Apriori algorithm can be useful to many use-cases across industries. I hope you enjoyed this article. Let us know how you plan to use the concept of association rule mining in your environment!</p><h3>References</h3><p>[1] <a href="https://medium.com/delvify/a-conceptual-introduction-into-association-rule-mining-part-i-ed5f61a7600a">A Conceptual Introduction into Association Rule Mining — Part I, Annette Catherine Paul for Delvify</a><br><a href="https://en.wikipedia.org/wiki/Apriori_algorithm#:~:text=Apriori%20is%20an%20algorithm%20for,sufficiently%20often%20in%20the%20database.">[2] Apriori Algorithm, Wikipedia</a><br><a href="https://www.kaggle.com/mittalvasu95/the-bread-basket">[3] The Bread Basket Dataset, Aditya Mittal — Kaggle </a><br><a href="https://www.kaggle.com/annettecatherinepaul/apriori-algorithm-association-rule-mining">[4] Apriori Algorithm — Association Rule Mining, Annette Catherine Paul — Kaggle</a><br><a href="https://www.investopedia.com/terms/l/long-tail.asp#:~:text=What%20Is%20the%20Long%20Tail,reduced%20number%20of%20popular%20items.">[5] Long Tail, Adam Hayes — Investopedia</a><br><a href="https://www.wired.com/2004/10/tail/">[6] The Long Tail, Chris Anderson — Wired</a><br><a href="https://www.kaggle.com/akhilram7/affinity-analysis-of-market-basket">[7] Affinity Analysis of Market Basket — Akhil Ram, Kaggle</a><br><a href="https://plotly.com/python/parallel-categories-diagram/#:~:text=The%20parallel%20categories%20diagram%20(also,taken%20on%20by%20that%20variable.">[8] Parallelogram — Plotly</a><br><a href="https://github.com/mazeofthemind/PyARMViz">[9] Advanced Python Association Rule Visualization Library — Mazeofthemind </a><br><a href="https://cran.r-project.org/web/packages/arulesViz/vignettes/arulesViz.pdf">[10] arulesViz, CRAN R project</a><br>[11] Bayardo, Jr RJ, Agrawal R (1999). “Mining the most interesting rules.” In KDD ’99: Proceedings of the fifth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 145–154. ACM</p><h3>Ready To Know More?</h3><p><a href="https://delvify.ai/get-started/">Get in touch with our team</a> of experts to find out more on how we use horizontal partitioning in our solutions and how we can support your business today.</p><p><a href="http://www.delvify.ai/">www.delvify.ai</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=96c73c4ce87b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/delvify/a-conceptual-introduction-into-association-rule-mining-part-2-96c73c4ce87b">A Conceptual Introduction into Association Rule Mining — Part 2</a> was originally published in <a href="https://medium.com/delvify">delvify</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Does Product Taxonomy Boost Product Discovery?]]></title>
            <link>https://medium.com/delvify/does-product-taxonomy-boost-product-discovery-9d8bedbe0f0b?source=rss----f66732270a67---4</link>
            <guid isPermaLink="false">https://medium.com/p/9d8bedbe0f0b</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[visual-search]]></category>
            <category><![CDATA[ecommerce]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Jasmine]]></dc:creator>
            <pubDate>Fri, 23 Jul 2021 08:11:32 GMT</pubDate>
            <atom:updated>2021-07-23T08:11:32.710Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3TBsZvKQH36BniMZ.png" /></figure><h4>Since the days of Aristotle in the West, humans have used classification to help organize their experience in the world. In the retail world, a good taxonomy has helped us organize and group our products into similar categories so that the products are easy to find. Cereals in a grocery store are never next to detergent. Everything we need is all found in close proximity to each other. That is why taxonomy is what we will be exploring in our blog this week, and how it important it is for the consumer’s convenience.</h4><p>What is a taxonomy, really? A taxonomy usually will have three elements: a category tree, a facet and a value list. What do these mean? A category tree is a simple hierarchy you might use to arrange your products in groups and sub-groups such as men’s clothes&gt;trousers&gt;jeans. If the jeans are stone-washed, that is a facet or an attribute and if the length is 33 inches, that is a value. If you want to peruse all 1,000+ categories, Ebay, the grandaddy of them all has produced a site <a href="https://www.isoldwhat.com/getcats/full-category-tree">here</a> where you can explore such categories.</p><h3>So Much To Do, So Little Time…</h3><p>Taxonimies have evolved over time to fit the online requirements of searchability. We have heard them described as <strong>search retrieval mechanisms</strong>, and this requires an awful lot of thinking and manual optimization. If you have a keyword-based search function on your site, you are in a straitjacket of pain (for the escape method see below). You have to ensure that your taxonomies are search engine optimized. You don’t want customers who abandon your site because they can’t find a product or come up with zero results! So, in order to optimise your product taxonomy, you include additional content like Meta Titles or Descriptions or stuff your URLs with so many words you lose count. By and large, this is done manually and extremely inefficient, we have named a few faults that could occur:</p><ol><li>Words are forgotten or copy/pasted to irrelevant categories.</li><li>Suppliers have different information in different, changing formats that requires reformatting and translating to your own feed.</li><li>Meanings of some categories are unclear or overlap in unusual categories.</li><li>Errors in product names or misspelling or odd characters lead to products that do not update or system rejection.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*7OQJPT-mjUVHe2M4" /></figure><p>And the list goes on. The end result is a miserable experience for the staff who want to highlight how beautiful and unique a product is, frustration for the customers who can’t find what they are looking for and lower profitability directly related to inferior CX. Fortunately, there is a solution! Drop your worries and bucket and use the power of AI to power your new search functionality. There are multiple examples of AI and machine learning used to rank and group objects efficiently and logically. Computer vision can quickly group similar shaped objects into a group, saving dozens of hours of manual labor.</p><h3>Introducing…NLP</h3><p>The real power for search in your eCommerce shop is Natural Language Processing (NLP). You must probably be wondering, how can NLP help?</p><p>NLP is one of the most powerful tools in AI today, and it can help in many ways. Through the use of NLP, you can entirely remove the tedious and error-prone manual tagging of products. The first problem that can be solved are the vague, too broad or missing categories in a product feed. Through the use of SVMs (a machine learning method to group similar items), we can classify products into a set of known categories through supervised learning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*c7VZiNVsJ5Ge8sc1" /></figure><p>Given a product with informational details such as name and descriptions, we can group the product into a particular category with similar products, e.g. ‘Women’s Clothing’ or ‘Automotive’. To do this, we analyse product feed information from different data sets and build a model that will automatically classify your products. This is not perfect, but accuracy rates now reach above 85% and are certainly better than human operators. Delvify improves on these numbers by working with our customer’s unique datasets.</p><h3>What’s My Story?</h3><p>Your products are unique and each item has a story or description attached to it. The description of each product holds the emotional weight of brand designers who spend hours thinking carefully about the products they develop and the merchandisers who work with them to develop concepts and themes for each and every product.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*r1KhGoDFOzjK1IYQ" /></figure><p>If I looked at Monet’s Water Lilies and described it as ‘Oil Painting &gt; French &gt; Impressionist &gt; Flowers, you would be baffled. So, why do we insist on searching using keywords? With the use of NLP, we can take apart sentences and find the connections and meaning in them. It is a way to match what your customers “feel” with the descriptions of your products. With NLP, you can search your product database with <strong>no tags needed!</strong> Yes, none at all!</p><p>How can you find your products without tags? With NLP, your customers need to only describe what they want and the algorithms get to work matching the search query meaning with the description that most likely matches. For example, in a normal in-site search engine, searching for “natural food for my Dalmatian” will require a lot of faceting! But an NLP engine would understand natural as a synonym for organic and food for a Dalmatian as dog food, and return results for organic dog food from a pet food selection. With these accurate results, what your brand gets is a happy customer (and most likely, returning customer)! The NLP search functionality can match the qualitative meaning of your products so brands can have a super efficient way to avoid manually labelling their products and concentrate on what is important — delivering accurate results to consumers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/369/0*vyNjrv31R74yxTlp" /></figure><h3>Ready To Know More?</h3><p><a href="https://delvify.ai/get-started/">Get in touch with our team</a> of experts to find out more on how we use horizontal partitioning in our solutions and how we can support your business today.</p><p><a href="http://www.delvify.ai/">www.delvify.ai</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d8bedbe0f0b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/delvify/does-product-taxonomy-boost-product-discovery-9d8bedbe0f0b">Does Product Taxonomy Boost Product Discovery?</a> was originally published in <a href="https://medium.com/delvify">delvify</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Conceptual Introduction into Association Rule Mining — Part I]]></title>
            <link>https://medium.com/delvify/a-conceptual-introduction-into-association-rule-mining-part-i-ed5f61a7600a?source=rss----f66732270a67---4</link>
            <guid isPermaLink="false">https://medium.com/p/ed5f61a7600a</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[business-analytics]]></category>
            <category><![CDATA[business-intelligence]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Annette Catherine Paul]]></dc:creator>
            <pubDate>Fri, 09 Jul 2021 08:01:44 GMT</pubDate>
            <atom:updated>2021-07-09T08:07:02.992Z</atom:updated>
            <content:encoded><![CDATA[<h3>A Conceptual Introduction into Association Rule Mining — Part 1</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aOF6moy098NJfSKsAZLnlg.png" /></figure><p>As humans, we inherently try to connect between various observations and concepts. These connections are built over some associations we relate to. The strength of these associations are directly related to how analytical we are with and without biases. Sometimes, it is a direct representation of anything we can relate to. Other times, if we put efforts, we can try and comprehend things beyond our scope, like the time we argued about whether the dress was blue and black or white and gold.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FhsbBp0SVB6HzOUkI8B%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2Fmemecandy-hsbBp0SVB6HzOUkI8B&amp;image=https%3A%2F%2Fmedia4.giphy.com%2Fmedia%2FhsbBp0SVB6HzOUkI8B%2Fgiphy.gif%3Fcid%3D790b761151a6fd5a7f1b90c3b26cb77215843039f613896a%26rid%3Dgiphy.gif%26ct%3Dg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="436" frameborder="0" scrolling="no"><a href="https://medium.com/media/609b219ef0ab63ddc5346bda8f97d109/href">https://medium.com/media/609b219ef0ab63ddc5346bda8f97d109/href</a></iframe><p>In my family, my mother is not fond of owning pets. However, when we got our tabby cat, my cat<em> associate</em>d with my mother the most on the grounds that she used to pet him the least, but when she did, he enjoyed it the most. <br>Whenever guests who were not comfortable with cats came over, we would have to keep him inside my room with the door closed. When we returned, we were always shocked to find all the papers in the room destroyed, probably cursing on us in secret cat language.</p><p>At no point did we communicate with our cat that when guests come, we will lock him up for a brief while. But after observing and registering repetitive actions, he made associations and chose the appropriate rule to apply (action) based on these associations.</p><h4>What is association rule learning?</h4><p>Now that we understand associations from our perspective, as humans we tend to jump to conclusions quickly. In other terms, we tend to conclude associations as <em>causality</em> or <em>correlations</em>. With association rule learning that is not the case. Let us first understand what association rule learning is and why it does not necessarily imply causality.</p><p>When we walk into a supermarket, there is a reason why dairy items are all placed together and, not far from it, breakfast cereals are placed too. These product placements are a result of associations based on previous customer transactions with the one motivation that a customer should not spend time searching for relevant items. Also, if the product placements are intuitive, we would end up buying more than what we need. We may come with the intention to buy milk, but seeing the breakfast cereals, we would see an association and would want to purchase those too.</p><p>For example, this happens frequently when shopping in <a href="https://en.wikipedia.org/wiki/LuLu_Group_International">Lulu Hyperpermarket</a>. Their product placements are so intuitive that there is no way you are going to step out without buying more than what you came for and not to mention every other hypermarket you walk into, you compare with Lulu unknowingly as in your mind that is how you default the arrangement of any hypermarket.</p><blockquote>Association rule learning is a rule based machine learning method for discovering relations between variables in large databases. It is intended to identify strong rules discovered in databases using some measures of interestingness. — [1]</blockquote><h4>Does that mean my customer will always buy breakfast cereals because they bought milk?</h4><p>Association rule learning tries to quantify the strength of co-occurrence. An example of co-occurrence is, you may be motivated to join the gym if your friend joins too (who doesn’t like to workout together, right?) but that does not mean, you <em>will </em>join without thinking twice. Additionally, it does not contain any order preference as it is a <em>set of elements </em>and it emphasizes on capturing all the items within a transaction as a group itself. Hence, you cannot map it back to a customer level and arrive at preferential inferences.</p><p>When I think back to my cat’s actions. Did he always react in that manner whenever we closed him in my room or were the <strong><em>frequency</em></strong> of his actions different at different times? Certainly it would be unfair to assume that all of his actions will be same. Some associations would be stronger than others.</p><h4>How do we validate this?</h4><p>Let us assume (if there existed multiple itemset to his reactions to being locked in a room) these are the possible item sets:</p><p>Itemset1: {locked in room, anger}<br>Itemset2: {locked in room, calm}<br>Itemset3: {locked in room, anger, hiss}<br>Itsemset4: {locked in room, anger, sleep}</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FL5wOS2OajUQaWshcTH%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2Fthefastsaga-L5wOS2OajUQaWshcTH&amp;image=https%3A%2F%2Fmedia3.giphy.com%2Fmedia%2FL5wOS2OajUQaWshcTH%2Fgiphy.gif%3Fcid%3D790b7611c1933e8d039ccc205c14c4efb24d1a94d5c56332%26rid%3Dgiphy.gif%26ct%3Dg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="186" frameborder="0" scrolling="no"><a href="https://medium.com/media/d52f287c47f69228b737ac607b802f0f/href">https://medium.com/media/d52f287c47f69228b737ac607b802f0f/href</a></iframe><p><strong>Support: </strong>From my experience with him, the co-occurrence of him being locked in a room and angry is higher than the co-occurrence of him being locked in a room and calm. The <em>number of times</em> he was angry most definitely outweighs the number of times he was calm. Additionally, anger would most certainly result in {hiss} as compared to pushing it off by taking a nap ({sleep}). In other words, itemset3 has more <em>support</em> than itemset4. <br><strong>Impact of support</strong>: If I know that beyond a certain proportion my cat tends to behave in a certain way then, I can consider it as a significant support. However, if the support is low, then the strength of the association would tank too as we cannot really arrive at any conclusion based on that scenario.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FAmCJYLARBYBsBQ1Bgt%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2Ftheoffice-AmCJYLARBYBsBQ1Bgt&amp;image=https%3A%2F%2Fmedia3.giphy.com%2Fmedia%2FAmCJYLARBYBsBQ1Bgt%2Fgiphy.gif%3Fcid%3D790b7611d09e25b0a686d02e10f0209700088fa5e5f1823b%26rid%3Dgiphy.gif%26ct%3Dg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="362" frameborder="0" scrolling="no"><a href="https://medium.com/media/571b4d298c285bd127157c4f3ac3c0ee/href">https://medium.com/media/571b4d298c285bd127157c4f3ac3c0ee/href</a></iframe><p><strong>Confidence:</strong> We all want some level confidence to relate to anything in our life right? Given that his itemset has {hiss}, how many times was he actually locked in the room ({locked in room})? <br><strong>Impact of confidence: </strong>This conditional probability speaks of the confidence with which you can speak of an association rule with a frequent item set implying on to another one. Sometimes, this value can be incoherent with intuitive knowledge. If the number of occurrences itself is limited, but has a relatively high overlap with various items. then this number can be high, even close to 1. Although intuitively we know that the association is misleading, just as in the case of given {sleep}, what is the confidence of {locked in the room}. Hence we need another measure in lieu with this concept to make sure we are not comparing apples and oranges, and that measure is lift.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FHQ9dS2mcEiqYw%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2Fexercise-cat-funny-HQ9dS2mcEiqYw&amp;image=https%3A%2F%2Fmedia2.giphy.com%2Fmedia%2FHQ9dS2mcEiqYw%2Fgiphy.gif%3Fcid%3D790b76116d5ee5c782bbe5ffc2799a2890330119b8111458%26rid%3Dgiphy.gif%26ct%3Dg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="359" frameborder="0" scrolling="no"><a href="https://medium.com/media/3d4f452f75c00e22ec480070ac27c6e4/href">https://medium.com/media/3d4f452f75c00e22ec480070ac27c6e4/href</a></iframe><p><strong>Lift: </strong>The support of an itemset that we are trying to associate to is controlled by lift. Or, what is the lift to the <em>confidence </em>of {sleep} in having {locked in the room} within the itemset. We are trying to analyze the change in likelihood of having {locked in room} in the set knowing that {sleep} is also present in the same set, <em>above</em> the likelihood of having {locked in room} in a set without any prior knowledge of existence of {sleep}.<br><strong>Impact of lift: </strong>Whenever the presence of one item actually pushes to having another item in the set, lift would be greater than 1, which implies high association, greater the cooccurrence and the reliability of the confidence with which we can speak of {X} → {Y}. This is what will truly help in the product placement scenario we spoke of before.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/797/1*gIzbeRcOpoYPeldbMuMvtw.png" /><figcaption>Image Source: Patterns of User Involvement in Experiment-Driven Software Development [2]</figcaption></figure><h4>What is the scalability of these rules with respect to time?</h4><p>Optimizing these two aspects is always a challenge within any technological firm. While finding answers is indeed important, the relevancy and optimization of those answers is also very important, as we do not have all the time and resources in the world to scale solutions that are unrealistic. Hence, generating these rules from a massive dataset could take a lot of time and effort. Are there any ways wherein we can leverage our data and optimize what we feed into the model? By leveraging itemset(s) that are frequent enough when compared to a minimum support threshold value, we can significantly optimize. A related concept when we deep dive into this is <strong><em>Maximal Frequent Itemset </em></strong>and <strong><em>Closed Itemsets</em></strong>.</p><blockquote>A maximal frequent itemset is a frequent itemset for which none of its immediate supersets are frequent. — [3] <br>An itemset is closed in a dataset if there exists no superset that has the same support count as its original itemset — [4]</blockquote><p><strong>Impact: </strong>Maximum Frequent Itemset as the name suggests is the most compressed form of an itemset holding the maximum information possible. This is ideally the most optimized way to use. But one main disadvantage is while every subsequent Itemsets can be derived, their support is lost. If your business goals and insights revolve around the requirement of support, we need to adopt another optimized approach which is Closed Itemsets. This approach ensures lowering inessential items without losing the support values.</p><h4>What are some use cases of association rules mining?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/0*3w3TRzH9I4kB7ydD.png" /><figcaption>Image Source: Association Rules Apriori Algorithm [5]</figcaption></figure><ul><li><strong>Market Basket Analysis (Affinity Analysis): </strong>By looking for combinations of items that occur together frequently in transactions, we try to uncover associations between these items, for improving product placements for offline shopping. The graph shown outlines these associations visualized as rules.</li><li><strong>Improving recommendations by showcasing relevant products: </strong>Recommender systems are intuitive and complex architectures leveraging many aspects of an ecosystem, inclusive of customer behavior, product behavior, content behavior etc. With the help of collaborative filtering, we can attribute the relationship back to a customer and intuitively recommend better products at a user level.</li><li><strong>Formal Concept Analysis: </strong>When we have a principled way of deriving a concept hierarchy from a collection of objects and their properties, we conduct a formal concept analysis. A snapshot of this is when we tried to relate to concepts using my cat — [7]</li><li><strong>Text Analysis:</strong> A novel method used to visualize associations between words to try and capture using a directed graph method. This can help uncover some thoughts on which words can go hand in hand often, especially in a text search where the query heavily depends on a user’s behavioral and cultural background. — [8]</li><li><strong>Fraud Analysis in Insurance:</strong> In this setting, association rules are used to identify and flag fraud rings which maps the claim identifier to the parties involved as items with a goal to identify frequently occurring associations between these items when handling the claims. For example a fraud would be: If insured by A and police offer X then auto repair shop <em>could </em>imply fraud. At least it gives us a list to validate — [9]</li></ul><p>This article is meant to be a gentle introduction into association rule learning. For Part 2 of this article we will be implementing this concept using various models such as Apriori, Eclat and FP growth algorithms.</p><p>How do you implement association rules within your firm? Feel free to leave your comments.</p><h3>References</h3><p><a href="https://en.wikipedia.org/wiki/Association_rule_learning">[1] Association Rule Learning, Wikipedia </a><br><a href="https://www.researchgate.net/publication/337999958_Patterns_of_User_Involvement_in_Experiment-Driven_Software_Development">[2] Patterns of User Involvement in Experiment-Driven Software Development</a><br><a href="https://www.geeksforgeeks.org/maximal-frequent-itemsets/">[3] Maximal Frequent Itemsets, Geeksforgeeks.org</a> <br><a href="http://www.hypertextbookshop.com/dataminingbook/public_version/contents/chapters/chapter002/section004/blue/page002.html#:~:text=An%20itemset%20is%20closed%20in,count%20as%20this%20original%20itemset.">[4] Closed Frequent Itemset, Chap 2: Association Analysis, The Data Mining Hypertextbook<br></a><a href="https://www.kdnuggets.com/2016/04/association-rules-apriori-algorithm-tutorial.html">[5] Association Rules Apriori Algorithm, KDNuggets</a><br><a href="https://towardsdatascience.com/a-gentle-introduction-on-market-basket-analysis-association-rules-fa4b986a40ce">[6] A Gentle Introduction on Market Basket Analysis — Association Rules, Towards Data Science, Medium </a><br><a href="https://towardsdatascience.com/a-demystifying-introduction-to-formal-context-analysis-fca-ab8ce029782e">[7] A Demystifying Introduction to Formal Concept Analysis (FCA), Towards Data Science, Medium<br></a><a href="https://towardsdatascience.com/visualizing-association-rules-for-text-mining-6cb80dc71171">[8] Visualizing Association Rules for Text Mining, Towards Data Science, Medium </a><br><a href="https://www.dataminingapps.com/2016/05/how-can-association-rules-be-used-for-fraud-detection-can-you-give-an-example/">[9] How can association rules be used for fraud detection?, Datamining apps</a></p><h3>Ready To Know More?</h3><p><a href="https://delvify.ai/get-started/">Get in touch with our team</a> of experts to find out more on how we use horizontal partitioning in our solutions and how we can support your business today.</p><p><a href="http://www.delvify.ai/">www.delvify.ai</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ed5f61a7600a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/delvify/a-conceptual-introduction-into-association-rule-mining-part-i-ed5f61a7600a">A Conceptual Introduction into Association Rule Mining — Part I</a> was originally published in <a href="https://medium.com/delvify">delvify</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Do You Do Horizontal Partitioning in SQL?]]></title>
            <link>https://medium.com/delvify/sql-horizontal-partitioning-725d1efe560c?source=rss----f66732270a67---4</link>
            <guid isPermaLink="false">https://medium.com/p/725d1efe560c</guid>
            <category><![CDATA[postgres]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[jagesh maharjan]]></dc:creator>
            <pubDate>Mon, 28 Jun 2021 09:40:26 GMT</pubDate>
            <atom:updated>2021-06-28T09:40:26.117Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BXq8ntvosACJ3UhtaYPC4g.png" /></figure><p>Software developers constantly optimize their infrastructure in order to gain high(er) throughput and low(er) latency. Application architecture and databases are crucial areas to continually optimize. In a previous post, we discussed the master-slave strategy to scale a database using horizontal partitioning.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FILyP03-hj7U&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DILyP03-hj7U&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/9c710f4f1d887bad914458c2efe6721b/href">https://medium.com/media/9c710f4f1d887bad914458c2efe6721b/href</a></iframe><p>As the name suggests, it partitions the table horizontally, just like slicing the cake with a knife. Why do we want to do this? Imagine you have a table with hundreds of millions of records. Querying a record from such a huge table takes an almost unsupportable amount of time. Slow reading this data creates long wait times and an unsatisfactory experience.</p><p>However, if we partition the table, querying will be much faster as the database engine will only scan a specific partition and not the entire table. The partitioning information is stored in the database as metadata. But instead of a long description, let us jump straight into how we can create horizontal partitioning in a Postgres database.</p><p>To spin up the Postgres database we will want to use a Docker, which pulls the Postgres image from a Docker-hub. Additionally, we will pull the pgAdmin from the Docker for a dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/873/1*KVdN7TpIM8M_JU6iRP2gog.png" /><figcaption>Figure 1: Postgres image from a Docker-hub</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/1*BUbJ-F8rtMI44ZdtaIA6ww.png" /><figcaption>Figure 2: pgAdmin image from a Docker-hub</figcaption></figure><p>Once you spin up the containers, you can go to the browser to login into pgAdmin, which is found at the HTTP with port 5050.</p><p>We then connect to the Postgres database through a TCP connection. Let us go inside the running container so we can create a sample database and tables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/603/1*AXs0dkfzWmR866g9kZk0bw.png" /><figcaption>Figure 3: Connecting PSQL via container</figcaption></figure><p>We will create a simple table called “Customer”. To make it easier to use, let us rename it customer_opt (opt as an optimized version of table) and we can alter the existing table quickly. The table has two integer fields. One is the auto-increment (serial) field and the other the customer field named “cid”. We will assume our table already has several records. If this is not the case, we can use the random number generator, to create values in cid from 0 to 99. Once we run the command, the table becomes a partitionable table, and then we can define a rule such that certain records stay in the designated partitioned tables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*4AHw024q2FlDMxVENUjHYQ.png" /><figcaption>Figure 4: Create a table that enables partition</figcaption></figure><p>For example, we can create four partition tables and evenly split the cid into those four partitions with values ranging from 0 to 24 in the first partition, 25 to 49 in the second partition, 50 to 74 in the third partition, and 75 to 99 in the fourth partition.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/1*jHc0GR_GeClwN-ANUBvxLQ.png" /><figcaption>Figure 5: Partition tables</figcaption></figure><p>We can attach these partitions to the main table with a rule for a value in cid within a designated range.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/631/1*LlXzwYjxCGoCgUuPZB22Wg.png" /><figcaption>Figure 6: Attaching the partitions in a table</figcaption></figure><p>Int he case where we did not have any records in our table, then here is how we can insert records:</p><p># inserting a 10 millions records in a table<br>insert into customer(cid) select floor(random()*100) from generate_series(0, 10000000);</p><p>Now, we can query and check the performance on the partitioned table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/653/1*d-PkInVZOSj9RZWF72FVSQ.png" /><figcaption>Figure 7: Query time</figcaption></figure><p>This was a practical look into how to partition your tables horizontally and achieve much faster data lookup and shorter wait times. At Delvify, we use this type of partitioning to improve the data experience for users. We hope it will be useful for your customers as well.</p><h3>Ready To Know More?</h3><p><a href="https://delvify.ai/get-started/">Get in touch with our team</a> of experts to find out more on how we use horizontal partitioning in our solutions and how we can support your business today.</p><p><a href="http://www.delvify.ai/">www.delvify.ai</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=725d1efe560c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/delvify/sql-horizontal-partitioning-725d1efe560c">How Do You Do Horizontal Partitioning in SQL?</a> was originally published in <a href="https://medium.com/delvify">delvify</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to Generative Adversarial Networks (GANs)]]></title>
            <link>https://medium.com/delvify/introduction-to-generative-adversarial-networks-gans-ecb1ac0b0d83?source=rss----f66732270a67---4</link>
            <guid isPermaLink="false">https://medium.com/p/ecb1ac0b0d83</guid>
            <category><![CDATA[generative-adversarial]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ecommerce]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Annette Catherine Paul]]></dc:creator>
            <pubDate>Fri, 11 Jun 2021 06:52:55 GMT</pubDate>
            <atom:updated>2021-06-11T06:52:55.002Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LiyfE1aMXqDJ2KdXF7CIxA.png" /></figure><p>Humans have successfully relied on machines for tasks such as predictions, recognitions etc. AI is becoming an integral part of various industries across the world so much so that it has not only been able to match human performance, but also surpass it. Remember Age of Ultron? Something like that, but less destructive.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2F1fM9ePvlVcqZ2%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2F1fM9ePvlVcqZ2%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2F1fM9ePvlVcqZ2%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="217" frameborder="0" scrolling="no"><a href="https://medium.com/media/07cfec01e4ac6ce6546837b1a8add8c8/href">https://medium.com/media/07cfec01e4ac6ce6546837b1a8add8c8/href</a></iframe><p>‌If you talk to any ML / DL professional or enthusiast, they would all concur as to how they are trying hard to optimize with the limited resources and data that they have (as occupying more of either would be expensive and challenging). Not to mention the time and resources it takes to label the data (so much pain!).</p><h4>What if machines could generate more data? Wouldn’t that help?</h4><p>Yes! Thanks to <a href="https://en.wikipedia.org/wiki/Ian_Goodfellow">Ian Goodfellow</a>, who put together the idea of Generative Adversarial Networks (GANs), which enables machines to create pragmatic data with remarkable results by using two discrete neural networks.</p><h4>What is Generative Adversarial Networks?</h4><p>Let’s think back to the times you tried to trick your mother by <strong><em>generating </em></strong>lies over lies. More often than not, your mother would clearly be able to <strong><em>discriminate</em></strong> the fake from the truth by just observing your facial expressions. Mothers will be mothers, aka the ultimate champions!</p><p>Your goal is to keep generating flawless lies while your mother’s goal is to figure out those lies and set you on the right path.</p><p>Similarly these two networks, i.e., the <strong>generator</strong>, which is trained to generate data and the <strong>discriminator </strong>which is trained to identify the fake from the real one, work together in a competitive manner (hence the term adversarial) to form the architecture of Generative Adversarial Networks, aka GANs.</p><blockquote>A GAN is a class of machine learning frameworks wherein two neural networks contest with each other in a game in the form of a zero-sum game, where one agent’s gain is another agent’s loss. — Wikipedia [2]</blockquote><p>Let’s use the lying scenario to better understand a simple implementation of the GAN architecture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wuao9_U1rT6ucolQukpr9w.png" /><figcaption>Vanilla GAN Architecture to depict the Lying Kid — Identifying Mother Scenario. Source for training set images: [3]</figcaption></figure><ol><li>The training set in this case would be all the facial expressions you would exhibit when you lie which acts as the first input to your mother (the discriminator)</li><li>You (the generator), would generate the facial expression and acts as the second input to the discriminator. Your goal is to produce expressions that apprehends the data distribution of the training set.</li><li>The discriminator then thinks about all the facial expressions you have had when she has caught you lying and compare it with your current expression (from the generator). Your mother then assigns a probability which she uses to judge you.</li><li>This loss / classification error is used to learn from yours and her mistake. If your goal was successful, then she would learn from her mistake and try to identify that particular face next time. If her goal is successful then you would try your best to avoid this situation in the future.</li></ol><h4>But what does iterative learning really mean?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/499/0*AjgEYbJRU8yiLlUp" /><figcaption>Source: <a href="https://mmmsimpsons.tumblr.com/">https://mmmsimpsons.tumblr.com/</a>, GIPHY</figcaption></figure><p>As most of us would agree, coming to a consensus with our mother is a difficult task, it could be a life long “iterative learning”!</p><p>In technical terms, this iterative learning is associated with the weights and biases of the network. For the generator this means to update both these parameters with a goal to maximize the probability with which the discriminator classifies the incoming input as real and for the discriminator this means to maximize the probability of an accurate prediction.</p><p>Your gain is your mother’s loss and vice versa. So when do we stop learning if one network gets better as the other gets worse? When the generator produces fake data which is hard to distinguish, the discriminator has a fighting chance as much as a flip of a coin. It’s 50/50! This state is called Nash equilibrium and is achieved when neither you nor your mother improve the situation by further actions. When this happens, GAN has converged and to achieve convergence is to date an open challenge in its practical applications.</p><blockquote>Nash equilibrium is a decision making theorem within game theory that states a player can achieve the desired outcome by not deviating from their initial strategy. — Investopedia [4]</blockquote><h4>But, how to evaluate the discriminator?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/606/1*NTXPcozsmvMCNCSR8xxOcg.png" /><figcaption>TP: True Positive, FP: False Positive, FN: False Negative, TN: True Negative</figcaption></figure><ul><li>Discriminators’ goal: (max(TP) &amp; max(TN)) | (min(FP) &amp; min(FN))</li><li>Generator’s goal: max(discriminator’s -&gt; FP)</li></ul><h4>Simple implementation of GAN</h4><p>AI is transforming the e-commerce industry especially during this challenging pandemic times. How? Check out the articles in <a href="https://delvify.ai/blogs/"><strong><em>Delvify’s Blogs</em></strong></a>! Let us use the Fashion MNIST dataset and see how it performs on a simple GAN implementation. This dataset of Zalando’s article images has 60,000 training examples with an associated label from 10 classes. Each of the image is 28*28 in size and is grayscale.</p><p>For the full implementation of a simple vanilla architecture refer to this <a href="https://www.kaggle.com/annettecatherinepaul/simple-gan-fashion-mnist">Kaggle notebook</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u0uX8O9rikGfcGCUfa0hBA.png" /><figcaption>Output images of GAN data generated using Fashion MNIST at iteration 1000, 5000, 10000, 15000 and 2000 respectively</figcaption></figure><p>We can see that with such a simple implementation of GAN, the results obtained in the last iterations are quite impressive. Here we are trying to create artificial data from the fashion MNIST dataset. As you can see the last iteration i.e. iteration: 20,000 is how our output looks like. In other words, these are the new images that our model has learnt to synthesize.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/673/1*qPU2cEm7sJ26w-Y2MqXCLA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/675/1*mVwIG60DAznQ9Tw7aBi9bA.jpeg" /><figcaption>Plots depicting the discriminator’s loss, generator’s loss and the discriminator’s accuracy</figcaption></figure><p>The generator loss is typically higher. Losses for both cases tend to stabilize after a point. The discriminator loss tends to be around 0.5 which is a good sign considering this implementation using a simple architecture. The generator loss is highly erratic in the beginning but seems to be stabilizing at around 2. The discriminator accuracy is around 85% which is a considerable amount given the simple architecture [7].</p><p>Although the images are indeed subject to improvement, we exhibited how powerful GAN is with even a simple architecture. By using more complex and powerful network architecture for both Generator and Discriminator such as Deep Convolutional GANs (DCG), we can do wonders!</p><h4>What are some other business use cases of GANs?</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2F3o85xsGXVuYh8lM3EQ%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2F3o85xsGXVuYh8lM3EQ%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2F3o85xsGXVuYh8lM3EQ%2F200.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="317" frameborder="0" scrolling="no"><a href="https://medium.com/media/a5d2d90b7a73a89c239ae49b106701ea/href">https://medium.com/media/a5d2d90b7a73a89c239ae49b106701ea/href</a></iframe><ul><li><strong>Generating missing parts of images:</strong> Have you played the game, guess the animal with some parts of the animal hidden? GANs can help you complete the missing parts of image. Within the eCommerce landscape, this could be beneficial in visual searches wherein we need to identify the missing parts of an image uploaded by a customer, to better recommend products.</li><li><strong>Auto Text sequencing:</strong> What about the game, finish my story ? GANs can help automatically generate different sequences of a story line in making. This means that if you ever encounter yourself in the film industry and is deprived a convincing climax, in the future GANs could provide one! [8]</li></ul><p>With more and more use cases GAN is conquering the world of possibilities!</p><p>As you can see, GANs can be used in multiple facets within the eCommerce landscape and can help interpret your dataset through the generator and the discriminator. This is an exciting field that is changing rapidly as time goes on and we are seeing this in all sorts of dimensions, including the use of it to discriminate deepfakes! Hope you found this article insightful to understand an introductory foundation of GANs.</p><p>If you would like to find out more on Delvify’s research on GANs, <a href="https://delvify.ai/get-started/">contact our team today</a>!</p><h3>References</h3><p>[1] GANs in Action, Jakub Langr &amp; Vladmir Bok, Manning<br>[2] Generative adversarial networks, Wikipedia<br>[3] How to Tell Someone’s Lying to you by watching their face — here are 12 dead giveaways, Business Insider<br>[4] Nash Equilibrium Definition, Investopedia <br>[5] Delvify Blogs, <a href="http://www.delvify.ai/">www.delvify.ai</a><br>[6] Fashion MNIST dataset, <a href="https://www.tensorflow.org/datasets/catalog/fashion_mnist">Tensorflow datasets</a><br>[7] How to identify and diagnose GAN failure modes, <a href="https://machinelearningmastery.com/practical-guide-to-gan-failure-modes/">machine learning mastery</a><br>[8] Practical applications of GANs, Packt<br>[9] Simple GAN Fashion MNIST, <a href="https://www.kaggle.com/annettecatherinepaul/simple-gan-fashion-mnist">Kaggle</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ecb1ac0b0d83" width="1" height="1" alt=""><hr><p><a href="https://medium.com/delvify/introduction-to-generative-adversarial-networks-gans-ecb1ac0b0d83">Introduction to Generative Adversarial Networks (GANs)</a> was originally published in <a href="https://medium.com/delvify">delvify</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Modeling Customer Behavior & Analysis: Analytical and Use Case Approach]]></title>
            <link>https://medium.com/delvify/modeling-ecommerce-customer-behavior-analysis-analytical-and-use-case-approach-5ed2fe3c361c?source=rss----f66732270a67---4</link>
            <guid isPermaLink="false">https://medium.com/p/5ed2fe3c361c</guid>
            <category><![CDATA[customer-analytics]]></category>
            <category><![CDATA[ecommerce]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Annette Catherine Paul]]></dc:creator>
            <pubDate>Fri, 28 May 2021 07:17:57 GMT</pubDate>
            <atom:updated>2021-05-28T07:25:32.820Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HOpeCMgWAZZLvrUgQT8G5Q.png" /></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FxT9C25UNTwfZuk85WP%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FxT9C25UNTwfZuk85WP%2Fgiphy.gif&amp;image=https%3A%2F%2Fmedia2.giphy.com%2Fmedia%2Fv1.Y2lkPTc5MGI3NjExZm9vdHI4bGIxOHdjYWY0OXZjcjM5Y2lwZWlteGJ6eWx0YWt3N2NxeSZlcD12MV9naWZzX2dpZklkJmN0PWc%2FxT9C25UNTwfZuk85WP%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="311" frameborder="0" scrolling="no"><a href="https://medium.com/media/d9defb87d03fb6ac94afc553be6da7a9/href">https://medium.com/media/d9defb87d03fb6ac94afc553be6da7a9/href</a></iframe><p>Data is like a language with grammar. The correct formation can lead to insights and useful conclusions which can improve your product / business strategy!</p><p>A picture is worth a thousand words. Which is why data visualization is a key solver for many problems. Sometimes it helps to identify a problem which you would have otherwise never known.</p><blockquote>Data Visualization is useful for data cleaning, exploring a data structure, detecting outliers and unusual groups, identifying trends and clusters, spotting local patterns, evaluating model output, and presenting results. It is essential for exploratory data analysis and data mining to check data quality and to help analysts become familiar with the structure and features of the data before them. — MIT Press [1]</blockquote><p>Once I have identified my trends and captured the data, what do I do next? <br>I like to think of it like an iterative approach wherein you visualize and then change your approach, test, validate and repeat the process. But then what exactly is approach? It can be introducing a model for deep dive or even simply changing the parameters of reach of your email campaign.</p><h3>Great. But, how does that help my business grow?</h3><p>I personally love data science more when we relate it back to a use case which can be improved. Let’s dive into an example and see how data and visualizations help understand your consumers better and what it means both qualitatively and quantitatively. In the end we will see how the qualitative will meet with the quantitative.</p><p>The data set I’ll be using is Online Shoppers Intention [2] from UCI Machine Learning Repository. You can find the complete codes to this analysis on my Kaggle notebook, <a href="https://www.kaggle.com/annettecatherinepaul/online-shoppers-behavior-prediction">here</a> [3]. This analysis is done using R. For the models, 80% of the data was used as a training set and the remaining 20% as validation set.</p><p>Now let’s take a look at the power of visualizations using Business Intelligence and Recommendations!</p><h4>1. Impact of Bounce Rate and Exit Rate</h4><blockquote>Bounce rate is an Internet marketing term used in web traffic analysis. It represents the percentage of visitors who enter the site and then leave rather than continuing to view other pages within the same site — Wikipedia [4]</blockquote><blockquote>Exit rate as a term used in web site traffic analysis is the percentage of visitors to a page on the website from which they exit the website to a different website. The visitors just exited from that specific page — Wikipedia [5]</blockquote><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2Fl396SbpG7USk2iAN2%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2Fl396SbpG7USk2iAN2%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2Fl396SbpG7USk2iAN2%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="333" frameborder="0" scrolling="no"><a href="https://medium.com/media/953b7e31ac9841254e88d1a81babe793/href">https://medium.com/media/953b7e31ac9841254e88d1a81babe793/href</a></iframe><p>Bounce Rate is calculated by aggregating one page visits by overall entrance visits whereas Exit Rate is calculated by aggregating total exits from a page by the total visits to that page.</p><p>So closely intertwined right? Think of it like exit rate is related to the overall % of visitors that were within the previous session whereas bounce rate is the % of visitors that were part of that one and only session.</p><p>Hence one can say that <em>all bounces can be exits but the vice versa is not true.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*KC8hcD1W__CmuGFtkHn3nA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*K7L6PZ0DdNtPC8wUQUBCOg.png" /><figcaption>(Image Source: By Author) Plots depicting the relationship between exit rates and bounce rates</figcaption></figure><ul><li>High bounce rates could be related to user satisfaction due to many reasons. A few being the UI of the website, technical glitches or even slow throughput.</li><li>High exit rate could depict performance issues across funnels, which further opens gate for improvement with optimization.</li></ul><p>One interesting point to note is that all numerical data seems to exhibit relatively high positive skewness.</p><p>Now that we have identified the problem. What do we recommend?</p><p><strong>Recommendation 1: </strong>More optimization of the landing pages by various methods such as making it more UI friendly, icons wherever needed, color impact, dynamic price approach and shipping cost friendly.</p><p><strong>Recommendation 2: </strong>Based on the funnel optimization, email retargeting with a personalized touch. Personalization brings in loyalty and in turn better conversion and retention!</p><p><strong>Recommendation 3: </strong>Introducing pop-ups offering personalized discounts when a customer bounces more than a threshold times or tries to leave the page.</p><h4>2. Weekend syndrome and loyal customers</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FlqFY9hBTLX7os%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FlqFY9hBTLX7os%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FlqFY9hBTLX7os%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="234" frameborder="0" scrolling="no"><a href="https://medium.com/media/2303edd3bdf9e811f3879ae04f16ecb4/href">https://medium.com/media/2303edd3bdf9e811f3879ae04f16ecb4/href</a></iframe><p>The following plot signifies that most customers, although could be revenue drivers, are not really returning customers. The need to focus on a balance between conversion and retention is apparent from this plot. Why? While retention points out to the brand growth and value, conversion impacts sales and revenue growth. Most of the customers purchase during a weekday and the brand could exploit this further to engage customers during weekends.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*67E1heggNhuYtpQW.png" /><figcaption>(Image Source: By Author) Plots depicting the revenue distribution by visitor type and weekend status</figcaption></figure><p><strong>Recommendation 4: </strong>Use your loyal customers to engage new customers by offering more reward and referral programs where rewards are mutual.</p><p><strong>Recommendation 5: </strong>Time dependent campaigns / discounts to engage customers during the time where they are least inactive.</p><h4>3. Lower conversions during holidays</h4><p>As we can see from the plots below, the engagement during certain months were high starting from February, and it becomes a decreasing trend after May. A high engagement level appears again as Black Friday approaches. The engagement when the demand is high seems to be high but conversion is significantly lower as you can see that most of the revenue drivers are returning customers. Hence, this could mean that there is a good loyalty program if any, but as you can see, conversions are significantly on the lower end. In short, there are a lot of people viewing the product but not taking the next step to purchase.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*_7o7JZaUTOvreUoHYdwsdg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*7elqbSsfXNkIvGYkmcUdnA.png" /><figcaption>(Image Source: By Author) Plots depicting the trend lines for revenue status based on month and visitor type based on months</figcaption></figure><p><strong>Recommendation 6: </strong>Seasonal promotions alongside promotional events/offers could bring in more conversion worthy customers whilst ensuring retention.</p><h4>4. Other revenue drivers!</h4><p>The main concern on these plots could be the impact of Google SEO optimization as only 45% of these were revenue driven showing more room for improvement with SEO / Ads.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*cSSiSZEXfUHSI4NrWo7cZw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*jwW3iUzUuHGf5jIGtOc-oQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*1eYYxKUvOpOsBtYAVVcowg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*x_fgDpv35Uk6PZ7RDf-rnQ.png" /><figcaption>(Image Source: By Author) Plots depicting the relationship of revenue with OS, browser, region and traffic type</figcaption></figure><p><strong>Recommendation 7:</strong> Validating smooth experiences across all browsers and OS.</p><p><strong>Recommendation 8:</strong> Personalized ads and reach using A/B testing to ensure the reach for further conversions/retentions within smaller regions.</p><p><strong>Recommendation 9:</strong> SEO from different sources like Google, Bing, Baidu etc.</p><h3>So how do we prioritize these recommendations? Can modelling help?</h3><p>Feature importance can help identify which data attributes could be prioritized more for your efficient business model deployment.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FxT5LMSKD0k8io8oTRe%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FxT5LMSKD0k8io8oTRe%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FxT5LMSKD0k8io8oTRe%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="244" frameborder="0" scrolling="no"><a href="https://medium.com/media/b7baa6c34b1c60851626d873d4a0ac11/href">https://medium.com/media/b7baa6c34b1c60851626d873d4a0ac11/href</a></iframe><p>Using the scaled data, we can feed it into a decision tree to help us understand which features of our model should be prioritized as the model speaks. Decision trees are good with classification as they are quite easy to optimize and scale.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*q_ckUS9AukVlCQWC.png" /><figcaption>(Image Source: By Author): Feature Importance depicted using a Decision Tree Approach</figcaption></figure><p>As we can see, the importance is accurately in line with our qualitative recommendation. However, the high importance of PageValue can suggest for one more additional recommendation:<br><strong>Recommendation 10:</strong> Customers may be viewing significantly different products and their recommendations which suggests that, there is a room for improvement on this area using recommendation engines and bundle packages. Additionally, including a wide range products to exploit the long tail effect can also be crucial to drive revenue.</p><p>The modelling helped in identifying an additional recommendation which we would have not been able to capture on a first level and now we can prioritize our recommendations based on the importance of the features:</p><ul><li>Recommendation 10 — Recommendation engines and inclusion of “long tail” driven products</li><li>Recommendation 9 — SEO optimization and Ads via Social Media</li><li>Recommendation 1 — Optimization of landing pages</li><li>Recommendation 7 — Smooth technical operations and user-friendly UI</li><li>Recommendation 6 — Seasonal promotions and discounts</li><li>Recommendation 5 — Weekend/Time based promotions and discounts</li><li>Recommendation 4 — Discounts for newcomers and connections of loyal customers</li><li>Recommendation 8 — Region based A/B testing and market reach</li><li>Recommendation 2 — Personalized email retargeting</li><li>Recommendation 3 — Exit rate strategy with personalized pop-ups</li></ul><p>In this manner we can see how EDA and modelling have been intuitively linked to Business Intelligence and the power it has together!</p><p>How would you use visualizations to interpret your audience better? Would love to know your thoughts in the comments 😊</p><h3>References</h3><p>[1] MIT Press, <a href="https://hdsr.mitpress.mit.edu/pub/zok97i7p/release/3">Why is Data Visualization Important? What is Important in Data Visualization?</a><br>[2] UCI Machine Learning Repository — Kaggle, <a href="https://www.kaggle.com/henrysue/online-shoppers-intention">Online Shoppers Intention</a><br>[3] Kaggle, <a href="https://www.kaggle.com/annettecatherinepaul/online-shoppers-behavior-prediction">Online Shoppers Behavior Prediction</a><br>[4] Wikipedia, <a href="https://en.wikipedia.org/wiki/Bounce_rate">Bounce Rate </a><br>[5] Wikipedia, <a href="https://en.wikipedia.org/wiki/Exit_rate">Exit Rate</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ed2fe3c361c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/delvify/modeling-ecommerce-customer-behavior-analysis-analytical-and-use-case-approach-5ed2fe3c361c">Modeling Customer Behavior &amp; Analysis: Analytical and Use Case Approach</a> was originally published in <a href="https://medium.com/delvify">delvify</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Let’s Deep Dive Into Transformers]]></title>
            <link>https://medium.com/delvify/lets-deep-dive-into-transformers-e81992e63ca0?source=rss----f66732270a67---4</link>
            <guid isPermaLink="false">https://medium.com/p/e81992e63ca0</guid>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[jagesh maharjan]]></dc:creator>
            <pubDate>Fri, 14 May 2021 07:29:39 GMT</pubDate>
            <atom:updated>2021-05-14T07:29:38.971Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YgbTUEU1hPsoFuON2aDOYw.png" /></figure><p>In mid-2019, I published a blog on T<a href="https://www.linkedin.com/pulse/transformer-jagesh-jugs-maharjan/?trk=public_profile-settings_article_view">ransformers</a> which was more of summary of the high-level architecture. As I discussed this with my colleagues at <a href="https://www.delvify.ai/">Delvify</a>, we decided I should write a blog to delve into more detail about this exciting technology. In this blog I will delve into the details of Transformers and describe more of the low-level design of the transformer architecture.</p><p>There are already a lot of blogs explaining the concepts and architecture of Transformers so you might be familiar already with Transformers, but I hope this will add some detail to your understanding.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/542/1*GN67yes31xNfhcfI6GMOdw.png" /><figcaption>Figure 1: Transformer Architecture</figcaption></figure><p>Transformers are simple stated, an Encoder-Decoder model. The Encoder takes high-dimensional inputs transforms them and extracts features to make a latent representation. This is is then passed to a Decoder to do the reverse. Encoding is done by passing data through fully-connected (dense) layers or convolutional layers. We will talk about multi-head self attention layers later.</p><p>Let us discuss the left-hand side of the Transformer in the Figure 1 first. This is the Encoder. The Encoder takes in inputs which are represented by the text below the red button on the left. Transformers are often but not exclusively used for Natural Language Processing (NLP) tasks. In computer vision these Transformers are called Vision Transformers (ViT). For our example let us just assume we are using the Transformer for NLP tasks. Thus the inputs would be sentences or a text corpus. Consequently, the output will also be text. One example might be language modeling where you want to predict the next word in a sentence.</p><p>The input to our encoder is a word embedding. To generate this embedding we might start with a Word2Vec model. Because the semantic similarity of words and concepts can be quantified by how close they are to each other as points in vector spaces, our Word2Vec model represents each distinct word with a particular list of numbers called a vector or and when we put them all together in a finite dimensional space we get vector embedding. Thus each word has its own vector embedded in or model. In Figure 2 below we show a 16x16 patch embedding. During model training, the parameters in each <strong>of </strong>these embedding subspaces are trained separately as part of a particular self-attention mechanism.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I7lOIKFMh1jH0PcfhPcqiA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TPuKUfjNROo4j3OIMlIo7A.png" /><figcaption>Figure 2: Input Embedding</figcaption></figure><p>Next the word embedding undergoes a similar process with positional embedding to include information on its position in the text. Positional embedding is calculated using the equations below:</p><p>pos_embed(pos, 2i) = sin(pos/10000^(2i/dmodel))<br>pos_embed(pos, 2i+1) = cos(pos/10000^(2i/dmodel))</p><p>As you can see, even positions takes use the sine (sin) function and and the odd position cosine (cos) function to create a positional encoding vector. By adding these calculations to the original word vectors we gain positional embedding and this is represented in Figure 3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WrSMwFpqW6oBI8Ti_OVYHQ.png" /><figcaption>Figure 3: Positional Embedding</figcaption></figure><p>It is useful to remember that the sin and cos functions have a linear property. This is what positional encoding looks like in practice:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/672/1*9kC1OH8gqiGHS6jC0cS7zg.png" /></figure><p>Next up in the Encoder layer is a multi-headed self attention layer. Self-attention is a mechanism used to build representations based on the the pair-wise correlations between the elements in a sequence. In particular, we compute an attention score between the query vector Q of <em>each</em> element to the key vector K of <em>every other</em> element in the sequence. Eventually this is passed to the feed-forward network. During this process we also undergo layer normalization operations to normalize the values of the representations produced within the Transformer layer.</p><p>Each attention mechanism performs a lookup just as you would do a dictionary lookup, or database query and produce a set of possible answers. The most probable answers have the highest attention scores. The computed attention scores are then used as weights. Thus the output of the attention mechanism is the product of the attention weights and the value of the V vector of the sequence. Really this is just a weighted sum of the entire sequence. How does it work in practice?</p><p>The Query(q) and Key(k) vectors are transformed using a dot product to form a scoring matrix as shown in the figure below. If we were thinking of images we might say it is the relative relation of a patch with other patches in an input image. The scores in a matrix are scaled down by the square root of the dimensions of the K vector. This process ensures that we avoid an exploding gradient. The resulting matrix is called a scaled matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LTklzINh63tn1lRQw0xSCw.png" /><figcaption>Figure 4: Attention Score</figcaption></figure><p>The scaled matrix is passed through a Softmax function to generate the probability weights which is now called the attention score matrix (self-attention represented by the red matrix below). Because it was run through a Softmax, values in the matrix will be between of 0 to 1. Next, the attention-score matrix is multiplied by the V vectors that were passed through the linear layer at the beginning resulting in an output vector which is then processed through the linear layer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UCP_uqDoG8Ezo-qJookp4Q.png" /><figcaption>Figure 5: Output Vector</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/659/1*z2h_4pSO8FmL9PtXik2O8A.png" /></figure><p>The Q, K and V vectors are generated for <em>every</em> element in the input sequence by passing the representation of that element (a vector) through a trainable dense layer. Crucially, we do not pass the entire vector, but a subsection of the vector (along the embedding dimension). We do this to enable multiple attention heads!</p><p>The above illustration is a single-headed attention network. For multi-headed attention network, multiple single-headed networks can be concatenated to give a single vector, which then goes through the linear layer. Multi-headed attention gives an impactful representation of the input.</p><p>Still don’t get it? Check out this fantastic representation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/819/0*Y4G0mSaguuNEWFI2.png" /><figcaption><em>Source: </em><a href="https://peltarion.com/blog/data-science/self-attention-video"><em>Getting meaning from text: self-attention step-by-step video, Peltarion blogpost</em></a></figcaption></figure><p>Now, we use the residual connections, feed-forward layer and normalization layer. The positional embedding vector we created in our first step is allowed to propagate deeper in the model. We normalize the value of the representations produced, then the rectifier linear unit (ReLU) activation layer, and then to the linear layer, and finally normalized again. This gives us the final output from the Encoder layer. An Encoder can be stacked as many times as needed to give meaning and a rich representation of any input. A basic BERT model, a new and state of the NLP model uses 12 encoders stacked on top of each other. The BERT large uses 24 encoders stacked together!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*5Spc_MH1uf6ow4bJyYUYfQ.png" /><figcaption>Figure 6: Encoder</figcaption></figure><p>Now let us jump over to the right-hand side of the Transformer, or in other words, the Decoder. The Decoder takes the output embedding we just calculated as its input. The process is similar to that of the Encoder, where it undergoes a position encoding and is passed through the multi-headed attention layer. However, there are few differences inside the multi-headed attention in a Decoder.</p><p>By default, the self-attention mechanism can freely look to any position in the input sequence. In other words, the entire value vector V is fully visible to the attention mechanism. We don’t want this if, for example we are guessing the next word in a sequence. So we “mask” the output of some positions. We do this by making it a highly negative number. In effect these positions will receive an insignificantly small weight after the Softmax operation. Because these future token’ attention score will be 0, this tells the network not to focus on the future words that have never attend at this instance of time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WY2dJzILydAN1QcKypyk1A.png" /><figcaption>Figure 7: Masking Output Attention Matrix</figcaption></figure><p>In the end, the masked score matrix is multiplied with the V vector to generate the output vector just like that of the Encoder layer. Next, on the second multi-headed layer of the Decoder, the output of the Encoder is passed along with the output vector from the previous multi-headed vector, to be passed through the normalization layer. Further this is passed through the linear network with residual vector and the layer normalization just as we did with the Encoder. Lastly, the output is passed through the last linear layer as a classifier, which will be equal to the length of the number of classes and after it is passed through the softmax so we generate the probability of the output.</p><p>One hack is that the Decoder can also be used <em>N </em>number of times as that of the Encoder. This improves the Decoder’s output.</p><p>So we have given a bit more information on the newest of AI tools, the transformer. But what we described was the simplest version. There are many others and as we see below the family is growing quickly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Fw01MEzHUXlj_SmV.png" /><figcaption>ref: Timothy Liu, SUTD</figcaption></figure><p>Not all of these are practical for everyday use if you don’t have the resources of Microsoft or Google, but many are within reach. At Delvify we use the BERT model commercially in our operations and hope you can find the perfect Transformer for your use.</p><p>References:<br>1. <a href="https://arxiv.org/pdf/1706.03762.pdf">Attention is all you need</a><br>2. <a href="http://jalammar.github.io/illustrated-transformer/">Jal Ammar Blog</a> <br>3. <a href="https://www.linkedin.com/pulse/transformer-jagesh-jugs-maharjan/?trk=public_profile-settings_article_view">Jagesh (Jugs) Blog</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e81992e63ca0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/delvify/lets-deep-dive-into-transformers-e81992e63ca0">Let’s Deep Dive Into Transformers</a> was originally published in <a href="https://medium.com/delvify">delvify</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Programmatic Predictions for 2021]]></title>
            <link>https://medium.com/delvify/programmatic-predictions-for-2021-dd1d50fbd4a8?source=rss----f66732270a67---4</link>
            <guid isPermaLink="false">https://medium.com/p/dd1d50fbd4a8</guid>
            <category><![CDATA[marketing]]></category>
            <category><![CDATA[digital-marketing]]></category>
            <category><![CDATA[advertising]]></category>
            <category><![CDATA[programmatic]]></category>
            <category><![CDATA[strategy]]></category>
            <dc:creator><![CDATA[Jasmine]]></dc:creator>
            <pubDate>Fri, 19 Feb 2021 08:31:04 GMT</pubDate>
            <atom:updated>2021-02-19T08:31:04.141Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_NUItfo6ikYReoLD.png" /></figure><h4>With 2020 drawing to a close, it’s been a tough year for all and especially for advertisers who have had to be agile and<a href="https://delvify.ai/how-has-the-future-of-programmatic-been-affected-by-covid-19/"> shift priorities throughout the year to deal with the changing landscape</a>. However, although it may seem like Groundhog Day, the digital industry has not been put to a standstill with many innovations developing, we look forward to the key trends we expect to dominate in 2021 for Programmatic advertising.</h4><h3>The Cookie Crumbles</h3><p>When Google announced they would be <a href="https://blog.chromium.org/2020/01/building-more-private-web-path-towards.html">removing third party cookies</a> on Chrome, this sent many marketers and brands into a spin. What does this mean? How will it affect my campaigns? Will I be able to target audiences? How will measurement work?</p><p>After the initial furore, like many who have come to terms with endless Zoom calls in 2020, the industry started to discuss and bring about alternatives to the cookie and improving privacy across the web.</p><p>There are now various initiatives in motion from industry experts to build a new standard for identity and measurement in digital. From Google’s own privacy sandbox to Unified ID 2.0, and as we move into 2021 we will have a clearer picture of how the removal of cookies will affect digital advertising.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*7kVjQGIf2AX-uOzY" /></figure><h3>The Return Of Contextual Targeting</h3><p>Without cookies and a clear alternative thus far, we expect a return to contextual targeting for many advertisers. Shunned in the past as an old way of reaching your audience, contextual targeting can provide relevant messaging at the right time and place to your audience.</p><p>This will also see a shift towards more private deals and packages set up by publishers. We have seen some publishers move away from the Open exchange in a bid to make their inventory more premium and sought after, and applying other signals to their inventory such as sentiment analysis and brand mentions to the deals. The downside is that CPMs will become more expensive for brands if they wish to capture their audience on these channels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*b31hiO4ynbDLRrXWEFF4aQ.png" /></figure><h3>The Year Of The Mobile (Again)</h3><p>It is almost a running joke in the digital industry that *this* year will be the year of mobile, although we can attribute 2020 with an increase in use of mobile and connected devices more than ever and we expect this trend to continue into 2021.</p><p>With the “story” format being a standard across many social platforms and more recently LinkedIn, brands have even more ways to connect and reach their audiences across mobile devices.</p><p>However, similar to the removal of third party cookies on Chrome, with Apple’s announcement of planned privacy updates in early 2021, mobile advertising has to start preparing and looking for ways to run effective campaigns with limited ad tracking (LAT) enabled. This is a major industry shake up, with no concrete alternative on the table yet and something to look out for in 2021.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/0*2hFUdxDCb6spvqDa" /></figure><h3>Rise In Programmatic TV And Audio</h3><p>Traditionally, TV ad buying has been linear in both format and process, with the rise in connected TV driving the growth in Programmatic TV advertising, it opens up a wide range of opportunities and a way to democratize the TV ad buying world.</p><p>An <a href="https://www.emarketer.com/content/connected-tv-advertising">eMarketer forecast </a>indicated that the number of connected TV users in the U.S. will climb to 204.1 million, 60% of the country’s population!</p><p>With increased streaming and time at home to consume more media, connected TV presents advertisers with a new way to reach their audiences at scale. We expect innovations in being able to link TV consumption data, whether it is sentiment or viewing content and activate cross-channel campaigns whereby Programmatic platforms become truly omni-channel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*b31hiO4ynbDLRrXWEFF4aQ.png" /></figure><h3>Bring On 2021!</h3><p>2021 will be a time for Programmatic to become firmly the de-factor way of buying digital advertising, whether that is on desktop, mobile, connected TV or Out-of-Home billboards. Let’s put all the negativity from 2020 behind us and look forward to creating a better 2021!</p><p>This is the last Programmatic AI blog of 2020, so our team would like to wish you Happy Holidays and a Happy New Year. Stay Safe!</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*b31hiO4ynbDLRrXWEFF4aQ.png" /></figure><h3>Ready To Know More?</h3><p><a href="https://delvify.ai/get-started/">Get in touch with our team</a> to see how you can scale your brand with Programmatic advertising in 2021!</p><p><a href="http://www.delvify.ai/">www.delvify.ai</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FZkNdEw1ugaASXFB595YjA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dd1d50fbd4a8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/delvify/programmatic-predictions-for-2021-dd1d50fbd4a8">Programmatic Predictions for 2021</a> was originally published in <a href="https://medium.com/delvify">delvify</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>