<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Rhydham Gupta on Medium]]></title>
        <description><![CDATA[Stories by Rhydham Gupta on Medium]]></description>
        <link>https://medium.com/@rhydhamgupta?source=rss-62e949c6123b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Ek5YQIljSiFSSJ5QyK1H4w.jpeg</url>
            <title>Stories by Rhydham Gupta on Medium</title>
            <link>https://medium.com/@rhydhamgupta?source=rss-62e949c6123b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 10:02:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rhydhamgupta/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Mastering the Art of Pricing Optimization — A Data Science Solution]]></title>
            <link>https://medium.com/data-science/mastering-the-art-of-pricing-optimization-a-data-science-solution-eb8befb79425?source=rss-62e949c6123b------2</link>
            <guid isPermaLink="false">https://medium.com/p/eb8befb79425</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[pricing-analytics]]></category>
            <category><![CDATA[analytixtime]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[retail]]></category>
            <dc:creator><![CDATA[Rhydham Gupta]]></dc:creator>
            <pubDate>Mon, 28 Aug 2023 14:16:33 GMT</pubDate>
            <atom:updated>2023-08-29T13:12:55.026Z</atom:updated>
            <content:encoded><![CDATA[<h3>Mastering the Art of Pricing Optimization — A Data Science Solution</h3><h4>Unlocking Secrets of Real-World Data Science Solutions for Pricing Optimization in Retail</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GoDYJfNMY26rm1brWkPm5A.png" /><figcaption>Image By Author</figcaption></figure><p>Table Of Contents:</p><p><a href="#df73"><strong>1. Overview</strong></a><strong><br></strong><a href="#ff4f"><strong>2. Elasticity Modeling</strong></a><strong><br></strong><a href="#03a7"><strong>3. Optimization</strong></a></p><h3>1. Overview</h3><p>Pricing plays a very crucial role in the world of business. Making a balance between sales and margins is very important for the success of any business. How can we do it in the data science way? In this section, we will build the intuition of an effective data science solution for pricing optimization and then we will go into the details and code of each component.</p><p>Note — There are different types of pricing strategies but in this article, we will focus on building the pricing strategy for conventional businesses/established brands with enough data on price change history. Let’s look at the basic approach that we are trying to follow before going into details —</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/961/1*W4tuuWqMrxQ1FLTYDDNPDw.png" /><figcaption>Image by Author</figcaption></figure><p>We have plotted sales and prices for item 1. In the past 9 months, there have been 2 price changes and clearly, we can see the impact on the sales. When prices were lower sales were higher. Now the question is how to <strong>quantify the impact on sales due to price changes in the past and predict the optimal price for the item in the future.</strong></p><p>An interesting observation from Jan-Apr, the price has been fixed at $5 but we still observe sales fluctuations. It’s very normal as in a practical world there are a lot of external factors that impact sales like seasonality, holidays, promotional events, marketing expenditure, etc. <strong>Hence, we don’t model the actual sales but the baseline sales which we derive using the different models.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/967/1*8KLnALEfl1pZccfJ3sjFwg.png" /><figcaption>Image by Author</figcaption></figure><p>You can observe that we are looking at a smoother trend of sales in the baseline sales series. Is it 100% accurate? Definitely Not! Data science is all about how close we can get to reality. Let’s move to the process now —</p><p>Suppose, we are hired and asked to provide the prices for thousands of items across stores for the Retailmart group. The prices of the same item can be different across different stores. The company has provided us with data for the past 5 years. What should be our approach to solving it?</p><p>Let’s understand this with a price meter example. Assume that we have a price meter and we have fixed the minimum and maximum values and the dial can move in between these two extremes. Currently, the dial is pointing toward the current price. Our objective is to stop the dial at a point where we can maximize profits.</p><p>Now as we move the dial toward the right (i.e. we are increasing the price) the sales will start to go down and margins will increase but</p><ol><li>Can we quantify this decrease? yes, we can and it is known as the price elasticity of an item. In simple terms, <strong>Price elasticity</strong> for an item would mean the percentage change in sales for a 1% change in the price.</li><li>In the real world, sales are often driven by promotional events, holidays, extra discounts, etc. but for optimizing the price we would need to exclude the effect of all these external factors and compute <strong>baseline sales.</strong></li><li>Once we quantify the change in sales w.r.t the change in price, <strong>we need the answer, Where do I stop the dial?</strong> For that, we need an objective which in most cases is maximizing the profits. Profits = Sales * Margin so we need to stop at the position where our profits value gets maximum. Mathematically, this is a concept of non-linear optimization where value can move within bounds.</li><li>Business Rules are important, we have to make sure that the final recommended prices adhere to these rules.</li></ol><p>So these are the major steps that we will follow to derive the right price for each item in a given store. Let’s look at these steps in a little more detail —</p><h4>1. Baselines Sales/Base Units</h4><p>This step is a pre-step for the subsequent steps. As stated, we want to model the impact on sales due to changes in the prices. The ideal scenario is to have sales that are only impacted due to prices but in the practical world, it is never the case —</p><p>So we want to simulate the sales for our ideal scenario and we do it using a time series model on the below equation —</p><blockquote>Sales ~ function[Baseline_sales + (promotional effect) + (holidays effect) + (any other effect) ]</blockquote><p>Please note that sometimes we don’t have actual data on the external factors that are impacting the sales. In such a scenario, we can use dummy variables to account for all such factors. A simple example can be if, in a certain month, we see a sudden increase in sales but the price remains constant, we can introduce a simple dummy variable having 1s for that month and 0s for the remaining months.</p><h4>2. Price Elasticity</h4><p>Price Elasticity refers to the % change in sales w.r.t the % change in the price of an item for a given store.</p><p>As an example, consider two products milk and ABC green tea. Which do you think will have high price elasticity?</p><p>Milk, being an essential everyday item with high competition, exhibits high price elasticity. Even a slight change in price can significantly influence sales due to its widespread demand. On the other hand, ABC green tea, which might be available in a limited number of stores, experiences low price elasticity. A small price change for ABC green tea is unlikely to have a substantial impact on sales due to its niche market presence.</p><p>How will we model this?</p><blockquote>Baseline_Sales ~ function[(price) + trend]</blockquote><p>The coefficient of the price variable will be used as the price elasticity. Trend variable is used to account for sales increase due to a long-term trend and not necessarily due to price changes. We will discuss more details on computing elasticities in the Price Elasticity section below.</p><h4>3. Non-linear optimization within bounds</h4><p>In this step, we will get the answer to where the dial should be stopped.</p><p>We first define our objective function — <strong>maximizing the profits</strong></p><p>Then we define the start and end point of the price meter that defines the LB and UB of the price</p><p>We have already computed baseline sales and price elasticities which quantify the sales sensitivity to price. We will put all these inputs into our non-linear optimization function and we will get the optimized price.</p><p>In very simple words, the algorithm will try different price points within the bounds and check the value of the objective function which in our case is profits. It will return us to the price point where it could get maximum value for our objective function. (In linear optimization, visualize how the gradient descent works). We will discuss more details on computing the optimized prices in the Optimization section below.</p><h4>4. Business Rules</h4><p>So can we directly implement the optimized prices in our stores?</p><p>No, but what is left now? Adhering to business rules is one of the most important requirements for any business.</p><p>But what kind of rules are we talking about in pricing —</p><ol><li>Ending Number Rules — It is a common practice to price the product at $999 or $995 instead of $1000. There are several psychological reasons for doing so hence we will need to make sure that our final recommended prices adhere to any such rules if these are applicable.</li><li>Product Gap Rules — Can you sell the one-pack Maggi priced more expensive per unit than the 4-pack Maggi? No, Right. Often if the pack’s size increases the per-unit cost should go down or at least stay the same.</li></ol><p>So, these are examples of some of the business rules that the business wants to apply. We will do some post-processing steps on the optimized prices to get to the final recommended prices.</p><p>Now that you are aware of the overall process, it is time to dive into more details and coding.</p><h3>2. Elasticity Modeling</h3><p>In this section, we will understand how we can use this concept to derive optimized prices for 1000s of items across multiple stores. Let’s say we have to determine price elasticity for a snack item Yochips selling in a California store for the past 3 years. Let’s first see the definition of <strong>price elasticity</strong>:-</p><blockquote>Price elasticity is defined as the percentage change in sales with a 1% change in the price.</blockquote><p>Now you may be wondering, which algorithm can I use for computing the price elasticity of an item like Yochips?</p><p>Let us look into some details from the economic book on <strong>the constant price elasticity model</strong> and see if we can relate it to some data science algorithm.</p><p>The multiplicative form of the demand function will be:-</p><blockquote>Yi = α*Xi^β (where y will be sales/demand, x will be price and β price elasticity)</blockquote><p>Taking the log on both sides</p><blockquote>log(Yᵢ) = log(α*Xᵢ^β)</blockquote><blockquote>log(Yᵢ) = log(α) + β*log(Xᵢ) ……….Eq(1)</blockquote><p>log(α) can be considered as an intercept as β₀</p><blockquote>log(Yᵢ) = β₀ + β₁*log(Xᵢ) ………….Eq(2)</blockquote><p>Now taking the differentiation on both sides, we will get</p><blockquote>δY/Y = β₁*δX/X</blockquote><p>The term on the left-hand side represents the % change in Y which is the % change in sales while the term on the right side represents the %change in price. Now when</p><p>%change in price = 1%; then δX/X = 1</p><p><strong>δY/Y = β₁</strong></p><p>This would mean the percentage change in the sales will be <strong>β₁ and that is our elasticity.</strong></p><p>Now if you would have noticed, Equation 2 is a <strong>Regression Equation where log(sales) are regressed against log(price) and the coefficient of log(price) will be our price elasticity.</strong></p><p>Hurray! Now we know that computing elasticity is as simple as training a regression model.</p><p>But there is one more catch. The demand function equation has some assumptions and one of the assumptions is that sales are only impacted by price but that is generally never the case in the real world because there are generally multiple factors impacting the sales like promotions, holidays, events, etc. So what is the solution, we need to compute the sales component wherein we can remove the impact of all these additional events.</p><p>One thing to clarify is that base sales refer to unit sales and not dollar sales. So in equation 2, we need to regress the price against base units instead of actual unit sales. Now the question is how can we derive the base units from the actual sales units?</p><p>Let’s understand it using an example. Below you can see the weekly plot of the time series for sales units and sell prices:-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o-vVIWcPaRgsVCVSev-IjA.jpeg" /><figcaption>Image by Author</figcaption></figure><p>Can you see any pattern in the above plot, it is difficult to tell because there are too many fluctuations in the sales units series. Now these fluctuations can be due to multiple factors like holidays, promotions, events, FIFA World Cup, etc. To isolate the effect of price changes, we need to compute sales that exclude the influence of these additional factors.</p><p>Using the prophet model, we can decompose the time series and extract the trend component which represents the base sales. By applying this technique, we separate the long-term impact of price changes from other short-term influences. Let’s see what we are talking about:-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yEzzJ4urkrAt3nFO5Adq4g.jpeg" /><figcaption>Image by Author</figcaption></figure><p>In the above plot, have decomposed the original log sales units (gray) into log base units (yellow line plot)</p><p>Here is a code using which you can decompose the time series and fetch the trend component which will become base sales:-</p><pre># Defining the inputs<br>timestamp_var = &quot;week_ending_sunday&quot;<br>baseline_dep_var = &quot;ln_sales&quot;<br>changepoint_prior_scale_value = 0.3<br>list_ind_vars_baseline = [&#39;event_type_1_Cultural&#39;,&#39;event_type_1_National&#39;,&#39;event_type_1_Religious&#39;,&#39;event_type_1_Sporting&#39;]</pre><pre># Preparing the datasecloset<br>df_item_store = df_item_store.rename(columns={timestamp_var: &#39;ds&#39;, baseline_dep_var: &#39;y&#39;})<br>df_item_store[&#39;ds&#39;] = pd.to_datetime(df_item_store[&#39;ds&#39;])<br><br># Initializing and fitting the model<br>model = Prophet(changepoint_prior_scale= changepoint_prior_scale_value) #Default changepoint_prior_scale = 0.05<br><br># Add the regressor variables to the model<br><br>for regressor in list_ind_vars_baseline:<br>    model.add_regressor(regressor)<br>model.fit(df_item_store)<br><br># Since we are only decomposing current time series, we will use same data is forecasting that was used for modelling<br># Making predictions and extracting the level component<br>forecast = model.predict(df_item_store)<br>level_component = forecast[&#39;trend&#39;]</pre><p>Below are the inputs that we need to define:</p><ul><li>changepoint_prior_scale_value — This controls the smoothness of the trend. You can read more about it in the prophet model documentation.</li><li>list_ind_vars_baseline — These include all the additional events that have impacted the sales like some festivals, sports events, cultural events, etc.</li></ul><p>Here is how changepoint_prior_scale_value impacts the trend. when the value is small then it leads to an almost straight line and when the value is high then the trend is less smooth.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o_q7Xn0vYHMFk9GnN8NhHw.jpeg" /></figure><p>The code is simple, Initially, we rename the “ln_sales” variable as “y” and the “week” variable as “ds” to meet the prerequisites for utilizing the Prophet model. Next, we initialize the Prophet model, specifying the “changepoint_prior_scale” parameter. Subsequently, we incorporate additional event and holiday variables into the model. Finally, we generate forecasts using the same dataset on which we trained the model and extracted the trend component.</p><p>Great. We now have the base units series and we can fit a linear regression model between base units (which are already at log scale because we decomposed the log_base_units series) and log(price). Below is the equation:-</p><blockquote>log(base units) = intercept + elasticity*log(price)</blockquote><p>With the above equation, we can compute the elasticity value. In practice, not all the series are good to model so you may expect some unexpected values of elasticity for various items. Then what is the solution? If we can somehow perform a regression with constraints on the elasticity value. But how can we implement it? Using Optimization function.</p><p>For any optimization, below are the basic requirements:-</p><ol><li>An objective function — This is the equation that we try to minimize/maximize. In our case, it will be the loss function that we use in linear regression MSE (pred-actual =&gt; [<strong>intercept </strong>+ <strong>elasticity</strong>*ln_price — actual]²)</li><li>The initial values of the parameter that we are trying to optimize, in our case it is intercept and elasticity. These can be any random values initially.</li><li>Bounds for the parameters, These are min and max bounds for both intercept &amp; elasticity</li><li>Optimization algorithm — This is dependent on the library but you can use the defaults and that should give you the right results</li></ol><p>Let’s now see the code:-</p><pre># Preparing the matrix to feed into optimization algorithm<br>x = df_item_store_model<br>x[&quot;intercept&quot;] = 1<br>x = x[[&quot;intercept&quot;,&quot;ln_sell_price&quot;,&quot;ln_base_sales&quot;]].values.T<br># x_t = x.T<br><br>actuals = x[2]</pre><pre>from scipy.optimize import minimize<br><br># Define the objective function to be minimized<br>def objective(x0):<br>    return sum(((x[0]*x0[0] + x[1]*x0[1]) - actuals)**2) # (intercept*1 + elasticity*(ln_sell_price) -ln_base_sales)^2<br> <br># Define the initial guess<br>x0 = [1, -1]<br><br># Define the bounds for the variables<br>bounds = ((None, None), (-3,-0.5))<br><br># Use the SLSQP optimization algorithm to minimize the objective function<br>result = minimize(objective, x0, bounds=bounds, method=&#39;L-BFGS-B&#39;)<br><br># Print the optimization result<br>print(result)<br><br># Saving the price elastitcity of an item in the dataframe<br>price_elasticity = result.x[1]<br>df_item_store_model[&quot;price_elasticity&quot;] = result.x[1]</pre><p>Notice that we have defined the initial parameter value for intercept as 1 and elasticity as -1. There are no bounds defined for intercept whereas bounds of (-3,-0.5) are defined for elasticity. This is the main reason why we are performing the regression through the optimization function. After running the optimization, we save the optimized parameter value of the price elasticity. Hurray! we have computed the price elasticity!</p><p><strong>So our Yochips at the California store have a price elasticity of -1.28.</strong></p><p>Let’s look at the price elasticity for some other series as well:-</p><p><strong>Low Price Elasticity</strong>: No significant change in sales with an increase in price. Below is a plot for an item having a price elasticity of -0.5</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1021/1*l4auh4yQtZ9geIpQkBBWYA.jpeg" /><figcaption>Image by Author</figcaption></figure><p><strong>Medium Price Elasticity</strong>: There is a moderate drop in sales with an increase in price. Below is a plot for an item having a price elasticity of -1.28</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BnBqa_dEXBcUsMtKjtq9PQ.jpeg" /><figcaption>Image by Author</figcaption></figure><p><strong>High Price Elasticity</strong>: There is a high drop in sales with an increase in price. Below is a plot for an item having a price elasticity of -2.5</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1021/1*k8lMKTtXepzfJk1KaPw3oA.jpeg" /><figcaption>Image by Author</figcaption></figure><p>Using the same approach, we can compute the price elasticity for all the items. In the next article, we will explore how we can utilize these elasticity values to determine the optimized prices for each item.</p><h3>3. Optimization</h3><p>In the previous section, we had already determined the price elasticity of Yochips for our California store. But that doesn’t help the store manager, he wants to know how he should change the price of Yochips to maximize revenue. In this article, we will understand the methodology to optimize prices.</p><p>But before that, there are a few questions, we must ask the store manager.</p><p>Q: Is there some limit on the minimum and maximum price change that should be considered while optimizing prices?</p><p>Based on the discussions with the store manager, we established that the price decrease should not exceed 20% and price increases should also be limited to 20%.</p><p>The current price for the Yochips is $3.23 and now we know that the optimized price has to be between $2.58 — $3.876. But how can we derive an optimized price?</p><p>But how can we derive an optimized price that maximizes revenue? Let’s do some maths:-</p><blockquote>Optimized Revenue = Total units sold * (<strong>Optimized Price</strong>)</blockquote><p>We need to optimize the price so that we can maximize revenue. But the total units sold will also change with a change in price. Let’s re-write the above equation and we can call the total units sold at an optimized price as optimized units:-</p><blockquote>Optimized Revenue = Optimized units* (<strong>Optimized Price</strong>)……………(eq1)</blockquote><p>We already know that —</p><blockquote>Elasticity = %change in the units sold/ %change in the price</blockquote><p>Therefore:-</p><blockquote>Optimized units = Base units + change in units at an optimized price</blockquote><p>Here, Base units refer to total unit sales at the current price which is $2.58</p><blockquote>Optimized units = (Base units + (Base units * price elasticity * (% change in the optimized price vs regular price) ………. (eq2)</blockquote><p>Let’s impute the eq2 in eq1</p><blockquote>Optimized Revenue = (Base units + (Base units * price elasticity * (<strong>% change in the optimized price vs regular price</strong>) * (<strong>Optimized Price</strong>) …….. (eq3)</blockquote><blockquote>Optimized Revenue = (Base units + (Base units * price elasticity * [(<strong>Optimized Price </strong>— Current Price)/ Current Price]* (<strong>Optimized Price)</strong>…………..eq(4)</blockquote><p>Below are the key parameters in the optimization equation (eq4):</p><p><strong>Base units </strong>= Average unit sales at the current price.</p><p><strong>Price elasticity </strong>= computed value for the item’s price elasticity</p><p><strong>Current price </strong>=<strong> </strong>Latest selling price</p><p>Great! In our equation, apart from the optimized price, we have data for all other variables. So which algorithm can we use to compute an optimized price that maximizes revenue? <strong>We can simply use the optimization algorithm.</strong></p><p>What are the critical components needed for optimization:-</p><ol><li><strong>The objective function that needs to be minimized/maximized</strong>:- We have already defined the objective function which is maximizing the optimized revenue as defined in eq(4).</li><li><strong>Bounds</strong>: As defined by the store manager, we need the optimized price to not change by more than 20%. So the Lower Bound = Current Price (1–0.2) &amp; Upper Bound = Current Price (1+0.2)</li><li><strong>Optimization Algorithm</strong>: We will use Scipy.optimize library from Python to implement the optimization.</li></ol><p>Let’s look at the code:-</p><pre># Taking latest 6 weeks average of the base sales<br>#--------------------------------------------------<br><br># Ranking the date colume<br>df_item_store_optimization[&quot;rank&quot;] = df_item_store_optimization[&quot;ds&quot;].rank(ascending=False)<br><br># Subset latest 6 weeks of data<br>base_sales_df = df_item_store_optimization.loc[df_item_store_optimization[&quot;rank&quot;] &lt;= 6].groupby(&quot;id&quot;)[&quot;base_sales&quot;].mean().reset_index()<br><br>df_item_store_optimization_input.rename(columns = {&quot;base_sales&quot;:&quot;base_units&quot;}, inplace=True)<br><br># Deriving the min and max bound for the sell_price<br>#--------------------------------------------------<br><br># Creating UB and LB as with the range of 20%<br>df_item_store_optimization_input[&quot;LB_price&quot;] = df_item_store_optimization_input[&quot;sell_price&quot;] - (0.2*df_item_store_optimization_input[&quot;sell_price&quot;])<br>df_item_store_optimization_input[&quot;UB_price&quot;] = df_item_store_optimization_input[&quot;sell_price&quot;] + (0.2*df_item_store_optimization_input[&quot;sell_price&quot;])</pre><p>The above code helps us in data prep for optimization. Firstly, we are computing the base units as the average of base_sales (trend component of the decomposed series) for the latest 6 weeks. We have already discussed the methodology to compute the base_sales in <a href="https://medium.com/p/98f4f0514bfc/edit">t</a>he above section.</p><p>Next, we are defining the LB_price &amp; UB_price by decreasing and increasing 20% from the current sell price respectively.</p><p>Let’s define the code to carry out optimization.</p><pre>from scipy.optimize import minimize<br><br># Define the objective function to be minimized<br>def objective(opti_price):<br><br>    df_item_store_optimization_input[&quot;opti_price&quot;] = opti_price<br>    df_item_store_optimization_input[&quot;optimized_units&quot;] = df_item_store_optimization_input[&quot;base_units&quot;] + (df_item_store_optimization_input[&quot;base_units&quot;]*\<br>                                                                                                        ((df_item_store_optimization_input[&quot;opti_price&quot;]/df_item_store_optimization_input[&quot;sell_price&quot;]) - 1)*\<br>                                                                                                       (df_item_store_optimization_input[&quot;price_elasticity&quot;]))<br>    <br>    df_item_store_optimization_input[&quot;optimized_revenue&quot;] = df_item_store_optimization_input[&quot;optimized_units&quot;]*df_item_store_optimization_input[&quot;opti_price&quot;]<br>    <br>    return -sum(df_item_store_optimization_input[&quot;optimized_revenue&quot;])<br><br># Define the initial guess<br>opti_price = df_item_store_optimization_input[&quot;sell_price&quot;][0]<br><br># Define the bounds for the variables<br>bounds = ((df_item_store_optimization_input[&quot;LB_price&quot;][0], df_item_store_optimization_input[&quot;UB_price&quot;][0]),)<br><br># # Use the optimization algorithm to minimize the objective function<br>result = minimize(objective, opti_price, bounds=bounds)<br><br># Print the optimization result<br>print(result)</pre><p>The above code will give us the optimized price. Can you guess that in the objective function why are we defining negative optimized revenue? What is -(-1), it is 1. We are minimizing the objective function and using the negative sign for the optimized revenue will lead to maximizing the optimized revenue.</p><p>Moreover, we can initialize the opti_price variable with any random value, just to facilitate fast convergence, we are initializing it with the current sell_price. In the bounds, we are defining the LB &amp; UB that we have created in the above code.</p><p>Hurray! We have found out the optimized price for Yochips and we are ready to propose it to the California store manager.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0qLtCGyPOcy__d0NbuLnfQ.jpeg" /><figcaption>Image by Author</figcaption></figure><p>Our recommendation will be to drop the Yochips prices by 10.2% to $2.9. This will result in maximum revenue.</p><p>This is the last step in the price optimization approach and the overall approach is so powerful that it can help us to return optimized prices for each item at every store.</p><p>One of the limitations of the above approach is for the items where we don’t have sufficient price change history. In that scenario, we use other techniques but if that fraction of items is less, then the average price elasticity at category level can be used for such items.</p><p>Hope you enjoyed this article! Now you know how to do Pricing Optimization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*8wfU0owwVDjtNOx5e2tkiA.jpeg" /><figcaption>Image by Author (Fonts made from <a href="http://www.onlinewebfonts.com/">onlinewebfonts</a>)</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eb8befb79425" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/mastering-the-art-of-pricing-optimization-a-data-science-solution-eb8befb79425">Mastering the Art of Pricing Optimization — A Data Science Solution</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Twitter Recommendation Algorithm — Simplified]]></title>
            <link>https://rhydhamgupta.medium.com/twitter-recommendation-algorithm-simplified-e8111e60f8f3?source=rss-62e949c6123b------2</link>
            <guid isPermaLink="false">https://medium.com/p/e8111e60f8f3</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[twitter-recommendation]]></category>
            <category><![CDATA[recommender-systems]]></category>
            <category><![CDATA[analytixtime]]></category>
            <dc:creator><![CDATA[Rhydham Gupta]]></dc:creator>
            <pubDate>Tue, 13 Jun 2023 06:03:54 GMT</pubDate>
            <atom:updated>2023-06-13T06:03:54.004Z</atom:updated>
            <content:encoded><![CDATA[<h3>Twitter Recommendation Algorithm — Simplified</h3><h4>Decoding the data science in the Twitter recommendation algorithm</h4><p>Do you know how the Twitter recommendation algorithm works? You might be wondering why should I know about it. Here’s why, knowing it will help you to come up with better ideas in your future projects and will make you look more knowledgeable and smarter in the team, On a lighter note why should you not grab the $44 billion algorithm when it is available for free :)</p><p>One of the challenges in understanding the Twitter recommendation algorithm is that either the content is too business focused like Twitter blogs which makes it very difficult to relate to the actual implementation or if you start digging into the research papers for the algorithms that are mentioned, then good luck finishing it ever. In this article, the intent would be to strike a perfect balance between the two so that you walk away feeling confident with a solid understanding of the algorithm.</p><p>There are majorly 3 components of the Twitter recommendation algorithm:-</p><ol><li>Candidate sourcing — selecting the top 1500 from millions of tweets that should be recommended to a user</li><li>Ranking Candidates — Then we do the ranking for the selected candidates based on the probability for a user to interact with all the selected 1500 tweets</li><li>Applying post rules — If you have blocked a user, and still you see his tweets, what are you gonna do? You cannot kill Twitter and Twitter would not want you to uninstall it. So better to ensure that it doesn’t happen.</li></ol><p>Suppose there is a user named Rohan, In this article, we will deep dive into how the algorithm will select which tweets to recommend to Rohan in what order. Let’s Begin!</p><h3>Candidate sourcing</h3><p>This is a very crucial step because selecting only 1500 tweets from millions is of course a very hard job. The Twitter algorithm handles it in two major ways:-</p><ol><li>In-network tweets (50%)</li><li>Out-of-network tweets (50%)</li></ol><p><strong>In-network tweets:</strong></p><p>These are easy because it is sourced from the people you follow. In case there is a higher number of tweets from followers which exceeds the threshold of 1500 tweets, the priority of tweets is defined based on the recency and popularity of the tweet.</p><p><strong>Out-of-network tweets:</strong></p><p>This is the hard part. Remember that a user can both create a tweet(producers) and read/interact with tweets (consumers). In this article, we will refer to users with at least 1 tweet as producers, and all users by default will also be consumers.</p><p>To understand it, we will start with the final output and then we will go into more details regarding its implementation.</p><p>We will discuss the Simcluster algorithm which is a clustering algorithm. Yes don’t be shocked, we will use the clustering to choose the right set of tweets for the user. Here’s how it works:-</p><ol><li>Cluster the producers into communities — These communities are built based on the producers that share a common topic of interest, e.g. politics, sports, AI, movies, etc.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/436/1*aus6jjZSB68ARr2DIoKxIg.jpeg" /></figure><p>2. Create the community vector for Rohan— Rohan will have some interaction with tweets from producers in the past. For ease of explanation, we have only considered the number of likes as an interaction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*3oWYApse1lz1ABAzmvb-kg.jpeg" /></figure><p>Now, a little bit of mathematics. Take the dot product of the Rohan-producer matrix with the producer-community matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/406/1*ly68IL4a48qwPAZ4Es-9uw.jpeg" /></figure><p>What does the community vector for Rohan represents? He likes 33% of the tweets from Community 1 and 66% of the tweets from Community 2.</p><p>3. Selecting the tweets from each community based on the Rohan community vector — Now to put it simply, if the algorithm has to source 1500 tweets for Rohan, it can pick up 500 tweets from Community 1 and 1000 tweets from Community 2. Now this tweet selection will be based on the popularity and recency of a tweet in the community. For e.g. a recent tweet that is being liked by a lot of users. Hurray, we have selected the 1500 tweet candidates for Rohan.</p><p>Did you notice something in the above process, once you assign the producers to communities, the rest of the steps simply flow based on the historical data. Twitter uses the SimCluster algorithm for clustering the producers into communities.</p><p>The basic idea is that u<strong>sers who belong to the same community are more likely to follow each other and tweet about similar topics. </strong>So there are two crucial elements of a community, Users within a community:-</p><ul><li><strong>tweet about similar topics </strong>and</li><li><strong>tend to follow more people within the community </strong>as against other communities.</li></ul><p>We will first try to capture both of the above elements in the data and then use an algorithm to assign the users to communities.</p><ol><li><strong>Create a user-topic similarity matrix </strong>— Determine how similar the user’s tweets are to each topic.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/789/1*4EGRtae3eaSUyn-F7gKdmw.jpeg" /></figure><p>Now, you might be wondering how we know that P1 has 4 tweets about politics or P2 has 2 tweets on politics and 2 tweets about sports.</p><p>It will be done using a text classification model or simply based on the similarity with the corpus of words. Based on the exploratory data analysis, we can define the corpus of words for each topic, e.g. sports can include words like cricket, basketball, ball, match, ball, etc. Now any given tweet will be compared with every topic corpus and assigned topic wherever the match is highest.</p><p><strong>2. Network structure between users</strong> — If two users follow each other and they belong to the same community, then it adds up the score but if they follow each other and are tagged to a different community then it assumes it to be an undesirable state. Let’s look at the network structure between users below:-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/586/1*RPEeSSdTdmhWR-rUQOZSYw.jpeg" /></figure><p>Great! We have defined both elements. Now how do we cluster the producers into communities so that producers in each community tweet about similar topics and tend to follow more users within the community?</p><p>It uses a probabilistic method called the Metropolis-Hastings algorithm to sample from the posterior distribution of the communities given the tweet data and the network structure. Here is how it works:</p><ul><li>It starts with randomly assigning the users to the different communities and calls it the initial state</li><li>It then generates a new state from a distribution, to put it simply we reassign the users from one community to another which is basically different than the initial state</li><li>Then we calculate the score (probability) for both the initial state and the new state, if the new state score is higher than the initial state, we replace the initial state with the new state otherwise retain the initial state. This process happens for the defined number of iterations. Now this score will be higher if the users within a community tweet about a similar topic and follows most of the people within the same community.</li></ul><p>The most important part of the above algorithm is how are we defining the score for each state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9BHjax545au5oMAfwfItGA.jpeg" /></figure><p>The above snapshot explains how the score is computed. Note that if you are not able to understand it fully, I will also write a separate article explaining this process in detail along with the code. For now, just know that using the above algorithm we are clustering the producers to communities and each community will have users tweeting about similar topics and following each other more. Moreover, in practice, we generally compute log_probability instead of probability.</p><p>Hurray! We have completed the 1st part and we have already filtered out 1500 tweets for Rohan. Now let’s score them in the next step.</p><h3>Ranking Candidates</h3><p>The idea is that we compute the probability of a user interacting with all the selected tweets and rank it based on descending order of the probability of interaction. Difference interactions could be:</p><ol><li>Like Probability</li><li>Reply Probability</li><li>Retweet Probability</li><li>View Probability</li></ol><p><strong>Intuition:</strong></p><p>In the past <strong>which type of users</strong> have <strong>which type of interaction</strong> with <strong>which type of tweets? </strong>which type is determined by different features on which we can train our model? The features are mainly:-</p><ol><li>User Features — user location, preferred language, age, total followers, etc.</li><li>Tweet Features — tweet text features, media features, likes, replies, retweets, etc.</li></ol><p><strong>Now the question is how can we convert it to the classification model? This will involve preparing the modeling data that includes defining dependent variables, independent variables, and the granularity of the model.</strong></p><p><strong>Granularity — </strong>User * tweet</p><p><strong>Dependent Variable — </strong>In reality, we create a multiclass classification model. But for ease of understanding, we can consider only one interaction scenario, like. If for a given tweet, the user has liked it, then mark it as 1 else 0.</p><p><strong>Independent Variables — </strong>All the user features and tweet features become independent variables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tC2fnI97Hkd4L-_K77Uo9A.jpeg" /></figure><p>For model training, we will create a dataset as above based on the past data. We can train our model on millions of user*tweet combinations. Moreover, in the above example, there are certain text variables, so we can create the embeddings for these variables or one-hot encoding before training the model. We can use any classification model.</p><p>Now let’s predict the probability of Rohan liking all the 1500 tweets that were selected in step 1. The higher the probability better it will be ranked.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cv2E4Ojj3mN0HYJJME2GLQ.jpeg" /></figure><p>User features will be based on Rohan, Twitter features will be based on the selected 1500 tweets and we will use the trained model to predict the probability of Rohan liking each tweet.</p><p>Please note that in our example we have used only 1 interaction variable for prediction but in practice, we predict the probability of all the interactions and create a combined score for Ranking. For e.g.</p><ul><li><strong>Like probability</strong>: 0.8</li><li><strong>Reply probability</strong>: 0.2</li><li><strong>Retweet probability</strong>: 0.1</li><li><strong>View probability</strong>: 0.05</li></ul><p><strong>Score</strong>: 0.8 * 0.4 + 0.2 * 0.3 + 0.1 * 0.2 + 0.05 * 0.1 = 0.385</p><p><strong>Interesting Fact:</strong></p><p>Elon Musk mentioned in one of his tweets that the logistic regression model used for ranking has been trained many years ago. Isn’t it amazing, a model trained many years ago is still giving so relevant predictions?</p><h3>Applying post rules</h3><p>This step is mostly rule-based step where</p><ol><li>Certain tweets blocked by users are filtered out from the ranked list</li><li>Advertisements are mixed in with the ranked tweets</li></ol><p>This is the overall process that Twitter goes through to provide you with the recommendation on the “For You” page. Please note that in the tweet sourcing step, they also use some other models but I have discussed one of the most popular techniques of Simclusters that they use in the recommendation. Moreover, for ranking, although they train a logistic regression model, it is trained using the Neural Networks where the last layer is sigmoid and gives the probability of each interaction. I will share a separate article on the implementation of the Simcluster algorithm &amp; Neural Network model for Ranking.</p><p>Hope you have got a conceptual understanding of the Twitter recommendation algorithm &amp; hopefully you can also leverage it in your future projects.</p><p><em>Please note that the above explanation might not be 100% the same as used by Twitter but the intent of this article was to understand the conceptual details of different models that might get used in other projects. I hope that you liked this article.</em></p><p>Happy Learning!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e8111e60f8f3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to build a Recommender System from scratch in 2 easy steps— E-commerce (3)]]></title>
            <link>https://rhydhamgupta.medium.com/how-to-build-a-recommender-system-from-scratch-in-2-easy-steps-e-commerce-3-e0ebc844e32a?source=rss-62e949c6123b------2</link>
            <guid isPermaLink="false">https://medium.com/p/e0ebc844e32a</guid>
            <category><![CDATA[recommendation-system]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[recommendation-engine]]></category>
            <category><![CDATA[analytixtime]]></category>
            <category><![CDATA[content-based-filtering]]></category>
            <dc:creator><![CDATA[Rhydham Gupta]]></dc:creator>
            <pubDate>Sat, 03 Jun 2023 13:48:52 GMT</pubDate>
            <atom:updated>2023-06-03T13:48:52.397Z</atom:updated>
            <content:encoded><![CDATA[<h4>Understanding the approach to build a Powerful Recommendation Engine in 2 Easy Steps for recommending similar items</h4><p>Imagine you are a data scientist in an e-commerce company. Currently, they use a popularity-based recommendation engine on their website that recommends the top-selling items to users. Recently the marketing team has been receiving negative feedback on customer experience and one of the prominent reasons is lack of personalization. Your manager comes to you and said, Rhydham the business wants a better recommendation system at the earliest and you are tasked with the responsibility.</p><p>You start having more conversations with the business to understand the requirements and realized that they want the recommended items to be based on different variants and other similar items to what the customer is browsing. What will be your approach that you will recommend to your manager? Don&#39;t worry, if you don’t know, after this article you will be able to propose an efficient solution to your manager confidently.</p><p>Let’s start with some real recommendations results, you would have observed that, on an e-commerce website when you select a product, there is a list of recommended products below, which are often very useful something we really want.</p><p>Below is a glimpse of what we are trying to create. These are the two recommendations snapshot for a popular e-commerce platform Flipkart (owned by Walmart) in India:-</p><p>When user have selected the Samsung Smartphone:-</p><blockquote><strong>Selected Item : </strong>SAMSUNG Galaxy F13 (Waterfall Blue, 128 GB) (4 GB RAM)</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*91Ak1hrvgcLxUY1hrSiCGA.jpeg" /></figure><blockquote><strong>Selected Item: </strong>WellManStore LED Temperature Display with Double Wall Insulated Water Bottle Stainless Steel 500 ml Bottle (Pack of 1, Black, Steel)</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0vcKZxcZ2EE97zdvJcLgpg.png" /></figure><p>Did you notice something in the above examples:</p><ol><li>Similar products belong to the same category of items, e.g. If the selected product is a smartphone, then the similar products are smartphones as well. Moreover, all recommended smartphones are in the comparable price range.</li><li>In 2nd example, the searched and similar products are both bottles in a similar price range.</li></ol><p>Now let’s dive into the methodology for creating the above recommendations. We will try to understand it with a smartphone example:-</p><p>Our recommendation framework will be a two-step approach:-</p><ol><li>Subsetting the relevant candidates from all items pool</li><li>Ranking the candidates</li></ol><h3>Subsetting the relevant candidates from all items pool</h3><p>The objective of this step is to shortlist a few most relevant items from all other items. Basis the recommendations, what do you think will be the most obvious candidates for the selected items? A very simple condition can be filtering the same item category and comparable price range. For e.g.</p><p>Category = Smartphones</p><p>Price Range = +-20% current price of the selected product</p><p>These conditions will give us the list of n number of items. Please note that the above point was just an illustration, these conditions are defined based on the business requirements.</p><h3>Ranking the candidates</h3><p>Now, we have already reduced the scope of items that needs to be recommended, but still, there can be 100s of smartphones and an average customer will only look at the top 10–20 recommendations. We will rank the qualified items to fulfill this objective.</p><p>First, let’s define the most important features based on which we can say that the two items are similar. Some of the features can be:</p><ol><li>Brand</li><li>Operating system</li><li>Ram</li><li>Storage capacity</li><li>Camera Quality</li><li>Display size</li><li>Battery capacity</li><li>etc.</li></ol><p>Now the basic idea is that we will try to find the similarity between the selected item and all other items in the candidate&#39;s list based on the above features.</p><p>If you have noticed carefully in the recommendation example on Flipkart, the first two similar items are just different variants of <em>SAMSUNG Galaxy F13 (Waterfall Blue, 128 GB) (4 GB RAM), </em>this is due to the fact that these will have the most common features and the highest similarity score.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u-g5GApqf2kCZgCkSbe5ig.jpeg" /></figure><p>Consider the above example, you can observe that we have the selected item features and the relevant candidates&#39; features (in practice it would be 100s of items). Now the objective is to compute the similarity between Smart Phone 1 — and all other smartphones and define which should be recommended to the customer</p><p>If you are thinking in the right direction, then you must be asking yourself these questions:-</p><ol><li>Which similarity metric should I use?</li><li>How do I deal with the Brand and Operating system column, these are strings and how will I define their similarity?</li><li>Do I need to standardize variables?</li></ol><p>Here’s what we will do:-</p><ol><li>We will use cosine similarity (Reason: It is simple to explain and standardize the similarity score between 0–1)</li><li>We will one-hot encoding for the categorical variables</li><li>Yes, that will be required otherwise variables with higher magnitude will get higher weights, which is not desirable.</li></ol><p>Below is the formula for computing the cosine similarity</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/525/0*n7GCV5vxjTExvPFP.png" /></figure><p>Let’s understand it in detail with the example:</p><ol><li>Let’s create sample data into a data frame that we have seen in the above example —</li></ol><pre>import pandas as pd<br>import numpy as np<br>from sklearn.metrics.pairwise import cosine_similarity<br>from sklearn.preprocessing import StandardScaler<br><br># Dummy data for 5 products<br>data = {<br>    &#39;Product&#39;: [&#39;Smartphone1&#39;, &#39;Smartphone2&#39;, &#39;Smartphone3&#39;, &#39;Smartphone4&#39;, &#39;Smartphone5&#39;],<br>    &#39;Brand&#39;: [&#39;Samsung&#39;, &#39;Apple&#39;, &#39;OnePlus&#39;, &#39;Samsung&#39;, &#39;Xiaomi&#39;],<br>    &#39;OS&#39;: [&#39;Android&#39;, &#39;iOS&#39;, &#39;Android&#39;, &#39;Android&#39;, &#39;Android&#39;],<br>    &#39;RAM&#39;: [8, 4, 6, 8, 6],<br>    &#39;Storage&#39;: [128, 64, 128, 256, 128],<br>    &#39;Camera&#39;: [12, 16, 12, 20, 16],<br>    &#39;Display&#39;: [6.2, 5.8, 6.5, 6.1, 6.4],<br>    &#39;Battery&#39;: [4000, 3200, 4300, 4000, 4500],<br>    &#39;Price&#39;: [1000, 2200, 900, 1100, 800]<br>}<br><br>df = pd.DataFrame(data)</pre><p>2. Standardize the numerical features and one-hot encode the categorical feature</p><pre># Standardize numerical features<br>numerical_features = [&#39;RAM&#39;, &#39;Storage&#39;, &#39;Camera&#39;, &#39;Display&#39;, &#39;Battery&#39;, &#39;Price&#39;]<br>scaler = StandardScaler()<br>df[numerical_features] = scaler.fit_transform(df[numerical_features])<br><br># Encode categorical features<br>df_encoded = pd.get_dummies(df[[&#39;Brand&#39;, &#39;OS&#39;]])<br><br># Combine encoded features with standardized numerical features<br>df_final = pd.concat([df_encoded, df[numerical_features]], axis=1)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gqEDwNcsGkbjzyUE07ae2A.png" /></figure><p>3. Let’s find out the cosine similarity scores of the smartphone1 with all other smartphones</p><pre># Calculate cosine similarity matrix<br>cos_sim = cosine_similarity(df_final)<br><br># Define the selected product index (Product 1 in this case)<br>selected_product_index = 0<br><br># Get the similarity scores for the selected product<br>similarity_scores = cos_sim[selected_product_index]</pre><p>Here’s the output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/1*JKNpsIIAHcKpdL0Rx73XiQ.jpeg" /></figure><p>SmartPhone 4 is most similar to SmartPhone1, so that should be recommended to the customers.</p><p>And that’s it. We have completed designing the recommendation engine for similar items. As a bonus, just now you have also understood the concept of content-based filtering recommendation algorithm.</p><p>Hope you liked the simple and effective approach for populating the similar items section. This is the 3rd article in this series on creating the recommender system from scratch. If you are interested, you can read the 1st article in this series where I have listed down all the different building blocks of an effective recommender system, you can find it <a href="https://rhydhamgupta.medium.com/building-a-recommender-system-from-scratch-1-dd140492afa6">here</a>. Moreover, you can try at your end to think of the approach that you will apply in each section.</p><p>HAPPY LEARNING!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0ebc844e32a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Recommender system from scratch — Customers who bought this item also bought (2)]]></title>
            <link>https://rhydhamgupta.medium.com/building-a-recommender-system-from-scratch-customers-who-bought-this-item-also-bought-2-626e83abc73b?source=rss-62e949c6123b------2</link>
            <guid isPermaLink="false">https://medium.com/p/626e83abc73b</guid>
            <category><![CDATA[ecommerce]]></category>
            <category><![CDATA[recommender-systems]]></category>
            <category><![CDATA[content-based-filtering]]></category>
            <category><![CDATA[collaborative-filtering]]></category>
            <category><![CDATA[analytixtime]]></category>
            <dc:creator><![CDATA[Rhydham Gupta]]></dc:creator>
            <pubDate>Fri, 26 May 2023 16:52:15 GMT</pubDate>
            <atom:updated>2023-05-28T11:56:45.132Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building a Recommender system from scratch — Customers who bought this item also bought (2)</h3><h4>Approach to creating a recommender solution based on “customers who bought this item also bought” for any e-commerce businesses</h4><p>As a data scientist in a top e-commerce company, you’re tasked with improving the recommendation engine. To help you design an efficient solution, your manager arranged a 1-hour call with the warehouse manager because he wants you to get insights from real experience. The warehouse manager shares an interesting fact: when there’s a high demand for competitive exam books, there’s also a surge in the demand for earphones. You quickly realized that this insight aligns with collaborative filtering.</p><p>Here’s a challenge, without knowing collaborative filtering, can you design a recommendation engine based on the warehouse manager’s insights?</p><p>Let’s design the solution and we will use item-item collaborative filtering. We have two different options to implement it:-</p><ol><li>Using cosine similarity</li><li>Using Matrix factorization</li></ol><p>I personally find approach 1 easier to explain especially to the business people but the 2nd approach is also very powerful in the sense it captures the impact of latent variables. (Don’t worry we will understand the concept of latent variables)</p><h3>Approach 1: Using Cosine Similarity</h3><p>Let’s look at a very simple example. Below is the use-item matrix. If the customer has purchased the item then it is 1 else 0.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/1*JcVqjuDvkLeXhKRunsNIKg.jpeg" /></figure><p>Now, if I ask you which item is more related to Fitness Tracker? or in other words if the customer has purchased Fitness Tracker then which out of 4 products customer is more likely to buy given the above matrix? What would be your answer and why?</p><p>At first, Yoga Mat seems to be more related to Fitness Tracker, can you guess why? Because User1, User3, and User4 have purchased Fitness and the same customers have bought Yoga Mat as well. Let’s check if our hypothesis is correct or not</p><p>We will implement it in python to find the similarity between the different items</p><p><strong>Step 1</strong>: We will first create the above table as a data frame.</p><pre>import pandas as pd<br><br># Create the purchase history matrix<br>data = {<br>    &#39;User 1&#39;: [1, 1, 0, 1, 0],<br>    &#39;User 2&#39;: [0, 1, 1, 1, 1],<br>    &#39;User 3&#39;: [1, 0, 0, 1, 1],<br>    &#39;User 4&#39;: [1, 0, 1, 1, 0]]<br>}<br>df = pd.DataFrame(data, index=[&#39;Fitness Tracker&#39;, &#39;Earphones&#39;, &#39;Smartwatch&#39;, &#39;Yoga Mat&#39;, &#39;Running Shoes&#39;])</pre><p><strong>Step 2</strong>: Next, let’s compute the cosine similarity:-</p><pre>from sklearn.metrics.pairwise import cosine_similarity<br><br># Compute the similarity matrix using cosine similarity<br>similarity_matrix = cosine_similarity(df)</pre><p>Below is the output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/886/1*7oDX8_S-n3WQ6AeXAVVzMQ.jpeg" /></figure><p>As we have guessed rightly, the Yoga Mat is the most similar item to the Fitness Tracker out of all the given products. Interestingly all other items have the same similarity score and of course, the similarity of Fitness Tracker with itself should be 1.</p><p>In practical scenarios, this matrix will be much large and sparse and these similarity scores will clearly show high similarity towards only towards few items.</p><p><strong>Food for thought: </strong>Instead of finding the cosine similarity score, we would have also computed that for each pair of items, how many customers have bought both the items and the one with the highest number could have been assigned maximum similarity. Could you figure out what is the benefit of using cosine similarity over this simple approach?</p><p>Let’s take the example of a Fitness tracker and Yoga Mat, also assume a third example water bottle that was purchased by customers 1,3,4. If you look at it all the customers who have bought Fitness trackers have also bought Yoga mats and water bottles. Then which one do you think is better to recommend to a user buying a Fitness tracker? You might say the Yoga mat seems a more promising option as more users have bought it. Whereas the cosine similarity score of the Yoga Mat is 0.87 and the water bottle would have been 1. Why is it so? Because the cosine similarity method penalizes the mismatches as well. Since Customer 2 bought a Yoga mat but didn’t buy a fitness tracker hence its score goes down whereas this would not have been the case with a water bottle.</p><h3>Approach 2: Using Matrix Factorization</h3><p>While trying to understand Matrix Factorization, I usually tend to get lost in mathematics then after getting frustrated I would leave it. But then I found that starting with the implementation and thinking about the practical applications makes it more intuitive. That’s exactly what we will do in this article. We will continue the same example that we have used in the cosine similarity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/1*JcVqjuDvkLeXhKRunsNIKg.jpeg" /></figure><p><strong>Step 1</strong>: Let’s load this table into the NumPy array:-</p><pre>import numpy as np<br>import pandas as pd<br><br># Example data<br>products = [&#39;Fitness Tracker&#39;, &#39;Earphones&#39;, &#39;Smartwatch&#39;, &#39;Yoga Mat&#39;, &#39;Running Shoes&#39;]<br>users = [&#39;User1&#39;, &#39;User2&#39;, &#39;User3&#39;, &#39;User4&#39;]<br><br># User-item purchase matrix<br>purchases = np.array([[1, 1, 0, 1, 0],<br>                      [0, 1, 1, 1, 1],<br>                      [1, 0, 0, 1, 1],<br>                      [1, 0, 1, 1, 0]])</pre><p><strong>Step 2</strong>: Next, we will use the singular value decomposition method to break the user-item purchase matrix into the user-latent variables matrix and item-latent variables matrix.</p><p>What are latent variables?</p><p>These are the factors/features that we do not explicitly mention but are automatically deciphered based on the customer&#39;s purchasing pattern. One thing to note here is that these latent features don’t necessarily have a very clear meaning of the driver instead it can be a mix of multiple factors. But for understanding, we can try to give it some meaning. Let’s take our current example:-</p><p>products = [Fitness Tracker, Earphones, SmartWatch, Yogamat, Running shoes]</p><p>When we say that based on customer purchase patterns, Fitness Tracker and Yogamat are highly related. It could be due to multiple factors, both products fall into the premium segment, both these products could be compact &amp; durable, or maybe both these products have very high ratings. It is basically trying to <strong>decipher the underlying drivers for similarities that are not explicitly defined.</strong> Let’s see it in action</p><pre># Compute matrix factorization<br>k = 2  # Number of latent factors<br>U, S, V = np.linalg.svd(purchases)<br><br>U = U[:, :k]<br>S = np.diag(S)[:k, :k]<br>V = V[:k, :]<br><br># Compute predicted purchases<br>predicted_purchases = np.dot(np.dot(U, S), V)<br><br># Convert to DataFrame<br>df_predicted_purchases = pd.DataFrame(predicted_purchases, index=users, columns=products)</pre><p>Below are our decomposed matrix</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/1*X0xueLGQ3q_H-XXq-ua0jw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/508/1*K1d2nFlyFafaLbKQn852tw.jpeg" /></figure><p><strong>Step 3</strong>: Now, we can simply take the dot product of Fitness Tracker latent variables with other items latent variables and the one with the highest score will be recommended to the user who has purchased Fitness Tracker. Let’s try to compute the most similar product for Fitness Tracker,</p><pre># Select a target item<br>target_item = &#39;Fitness Tracker&#39;<br><br># Get similar items based on item latent factors<br>item_index = products.index(target_item)<br>item_latent_factors = V[:, item_index]<br>similar_items = pd.DataFrame({&#39;Product&#39;: products, &#39;Similarity&#39;: np.dot(V.T, item_latent_factors)})<br><br># Exclude the target item itself<br>similar_items = similar_items[similar_items[&#39;Product&#39;] != target_item]<br><br># Sort by similarity in descending order<br>similar_items = similar_items.sort_values(&#39;Similarity&#39;, ascending=False)<br><br>print(f&quot;Recommended items for {target_item}:&quot;)<br>print(similar_items)</pre><pre>Recommended items for Fitness Tracker:<br>         Product  Similarity<br>3       Yoga Mat       0.375<br>1      Earphones      -0.125<br>2     Smartwatch      -0.125<br>4  Running Shoes      -0.125</pre><p>Above, you can see the order of similar items for the Fitness Tracker. Now if you remember the results from the cosine similarity method-</p><p>Yoga Mat was most similar to the Fitness Tracker and the rest all the items have a similar score. We are observing the same results with this method as well. Great :)</p><p><strong>You might be wondering when to use cosine similarity vs matrix factorization.</strong></p><p>The cosine similarity approach is simple and offers ease of explainability to business people but it will not be able to capture the complex relationships that exist within data whereas the matrix factorization performs well for identifying these relationships but it becomes really difficult to explain it to business people. Moreover, it needs more data for giving more accurate results.</p><p>Now you know how to design our recommendation engine based on “Customer who bought this item also bought …..”</p><p>This is the second article in our series about building different components of a recommendation engine for an e-commerce company. You can read out the introductory article <a href="https://medium.com/@rhydhamgupta/building-a-recommender-system-from-scratch-1-dd140492afa6">here </a>which covers all the components that we will cover in this series. You can also try to give a shot at what would be your approach for each component.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=626e83abc73b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Recommender System from Scratch — (1)]]></title>
            <link>https://rhydhamgupta.medium.com/building-a-recommender-system-from-scratch-1-dd140492afa6?source=rss-62e949c6123b------2</link>
            <guid isPermaLink="false">https://medium.com/p/dd140492afa6</guid>
            <category><![CDATA[recommendation-engine]]></category>
            <category><![CDATA[data-scientist]]></category>
            <category><![CDATA[analytixtime]]></category>
            <category><![CDATA[recommendation-system]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Rhydham Gupta]]></dc:creator>
            <pubDate>Mon, 22 May 2023 16:02:49 GMT</pubDate>
            <atom:updated>2023-05-22T16:02:49.563Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building a Recommender System from Scratch — (1)</h3><h4>A Practical Guide to Creating an Amazon &amp; Flipkart-like Recommendation Engine</h4><p>Personalization is the new formula of success for many companies in today’s business landscape. A significant proportion of ML/AI-based use cases require building an efficient and explainable recommender system and hence there is a very good demand for people who are proficient in building recommendation systems for personalization.</p><p>However, most available resources on recommender systems often focus on movie ratings that do not reflect real-world scenarios. How often do you provide ratings of your purchase — very often or never, right? So we need some other metrics to gauge the customer behavior and intent. In this article, we will understand the methodology to build the recommender system from scratch with some real-used metrics.</p><p>A good recommender system is built upon multiple building blocks. In any e-commerce application like Amazon, we have two sets of broad recommendations —</p><ol><li>Homepage Recommendations (Which is a combination of trending products, past purchases, recommendations based on past purchases, etc.)</li><li>Product-Specific Recommendations (When you select a product, there is a list of product recommendations listed as you scroll down the page). If you observe carefully, these recommendations serve as a building block for the final recommendations on the Homepage</li></ol><p>So in this series of articles, we will start with understanding the product-specific recommendations. we will come back to the homepage recommendations towards to end.</p><p>Some of the most popular and common recommender systems techniques are mentioned below</p><ul><li>Collaborative Filtering</li><li>Content-Based Filtering</li><li>Market Basket Analysis</li></ul><p>The real art is defining the analytical problem from different aspects before applying these techniques, let&#39;s understand what I mean by analytical problem —</p><h3>Below are some of the common sections from Amazon and Flipkart that fall under product-specific recommendations:</h3><ol><li>Customers who bought this item also bought these items (Collaborative Filtering)</li><li>Customers who viewed items in your browsing history also viewed/Inspired by your browsing history (Collaborative Filtering)</li><li>What other items do customers buy after viewing this item (Collaborative Filtering)</li><li>Products related to this item/Similar Items (Content-based filtering)</li><li>Featured items you may like/Popular products based on this item (Content-based filtering)</li><li>You might be interested in similar categories (Content-based filtering)</li><li>Frequently bought together (Market Basket Analysis)</li></ol><h3>1. Customers who bought this item also bought these items</h3><p>By analyzing purchase histories, we can identify patterns where customers who bought a particular item also purchased other related items. The basic hypothesis of this section is that many customers have similar purchasing behavior. If there are two customers A and B who have purchased 70% of common items, then there is a high chance that A will like the remaining 30% of items that B has purchased.</p><blockquote><strong>Selected Item:</strong> Samsung Galaxy M33 5G (Mystique Green, 6GB, 128GB Storage) | 6000mAh Battery | Upto 12GB RAM with RAM Plus | Without Charger</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-sdx-uOdBR385C9Q6-8Jzw.jpeg" /></figure><p>The recommended smartphones are very similar to the selected item with the OnePlus smartphone coming as a surprise. This is one of the biggest advantages of this section that this is driven by similarity in customer purchasing behavior, and because of that, even some unexpected recommendations are likely to do very well.</p><h3>2. Customers who viewed items in your browsing history also viewed/Inspired by your browsing history</h3><p>Understanding customer browsing behavior allows us to recommend items based on their interests and preferences. By analyzing patterns in users’ browsing histories, we can suggest relevant products, increasing engagement and conversion rates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MZENi7no723DaO5Gu4ereg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*46bfejWPWYaO3Eev7eOauQ.jpeg" /></figure><p>Notice, that based on my recent browsing history, recommended items are also from different categories.</p><h3>3. What other items do customers buy after viewing this item</h3><p>Analyzing the purchase sequence of customers who viewed a particular item provides valuable insights into purchase patterns. This is a very interesting section because Amazon believes that what customers viewed previously is a strong indicator of what customers will purchase next.</p><blockquote><strong>Selected Item: </strong>Samsung Galaxy A54 5G (Awesome Graphite, 8GB, 256GB Storage) | 50 MP No Shake Cam (OIS) | IP67 | Gorilla Glass 5 | Voice Focus | Without Charger</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OI64TtzSa2Dq9ZKk7h6pnA.jpeg" /></figure><p>Generally, before buying a smartphone, I often search for other similar smartphones to compare their features and prices. That might be the reason that there are so many smartphones in the recommended items. Isn’t it interesting that travel adapters are the most purchased items after viewing this smartphone?</p><h3>4. Products related to this item/Similar Items</h3><p>Utilizing the content and attributes of products, we can create recommendations based on similarity. By analyzing item characteristics such as category, brand, or features, we can suggest related items to customers, enhancing their overall shopping experience.</p><blockquote><strong>Selected Item: </strong>SAMSUNG Galaxy F13 (Waterfall Blue, 128 GB) (4 GB RAM)</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*91Ak1hrvgcLxUY1hrSiCGA.jpeg" /></figure><blockquote><strong>Selected Item: </strong>WellManStore LED Temperature Display with Double Wall Insulated Water Bottle Stainless Steel 500 ml Bottle (Pack of 1, Black, Steel)</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0vcKZxcZ2EE97zdvJcLgpg.png" /></figure><p>This section compares the product attributes to get similar items. Interestingly, for the smartphone, the top two similar products are different variants of the same smartphone.</p><h3>5. Featured items you may like/Popular products based on this item</h3><p>Incorporating popularity and trends, we can showcase featured items that align with a customer’s preferences. By highlighting popular products related to their past purchases, we can capture their attention and drive sales.</p><blockquote><strong>Selected Item: </strong>Samsung Galaxy A54 5G (Awesome Graphite, 8GB, 256GB Storage) | 50 MP No Shake Cam (OIS) | IP67 | Gorilla Glass 5 | Voice Focus | Without Charger</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pcYa3TZFG3vcbxomgS49mw.jpeg" /></figure><p>Combining similarity in product attributes with popularity.</p><h3>6. You might be interested in similar category products</h3><p>By understanding user preferences, we can recommend items from similar categories. Expanding beyond specific items, this approach broadens the scope of recommendations and helps customers discover new and relevant products.</p><blockquote><strong>Selected Item: </strong>SAMSUNG Galaxy F13 (Waterfall Blue, 128 GB) (4 GB RAM)</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ocic8ScH11xJrUQllSp1gA.jpeg" /></figure><p>This section of recommendation plays smart, Instead of recommending similar items it recommends the associated categories of products based on the selected item. In the above example, when a smartphone is selected, it recommends purchasing mobile skin stickers, wired earphones, and smart glasses.</p><h3>7. Frequently bought together</h3><p>Leveraging user purchase history, we can identify products frequently bought together. This association provides valuable insights for cross-selling and enhancing customer satisfaction.</p><blockquote><strong>Selected Item: </strong>Samsung Galaxy A54 5G (Awesome Graphite, 8GB, 256GB Storage) | 50 MP No Shake Cam (OIS) | IP67 | Gorilla Glass 5 | Voice Focus | Without Charger</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jTyoOkd9g69VvcR_H2U2Fg.jpeg" /></figure><p>This section is about recommending items that are bought together in the same transaction.</p><p>In addition to the above strategies, there are several other strategies that can be used to pop up products in your recommendation system:-</p><ul><li>Trending Items</li><li>Frequently repurchased items (like grocery items)</li></ul><p>This is an introductory article related to a real recommendation engine. In the series of articles, we will go in-depth into each section and explore the methodology to build it from scratch.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dd140492afa6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Right time to React in your project]]></title>
            <link>https://rhydhamgupta.medium.com/power-of-expression-in-data-science-d131cdf49f9a?source=rss-62e949c6123b------2</link>
            <guid isPermaLink="false">https://medium.com/p/d131cdf49f9a</guid>
            <category><![CDATA[people-skills]]></category>
            <category><![CDATA[corporate-strategy]]></category>
            <category><![CDATA[growth]]></category>
            <category><![CDATA[analytixtime]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Rhydham Gupta]]></dc:creator>
            <pubDate>Wed, 16 Nov 2022 04:56:04 GMT</pubDate>
            <atom:updated>2022-11-30T02:11:14.418Z</atom:updated>
            <content:encoded><![CDATA[<p>There is a thing about introverted data scientists that they put so much emphasis on technical work that they often overlook aspects of people skills which cost them dearly in their careers. I want to share one such experience which has taught me very valuable lessons in my life.</p><p>Before this experience, whenever I read about such stories, I somehow managed to convince myself that it can never happen to me but when it did, it hits me very hard. This happened to me in one of the projects. My technical skills advantage was bulldozed and I ended up being ranked as a below-par performer.</p><p>More surprising was the fact that during that period, everything seemed to work against me. It was not that I didn’t try to change my ways but nothing worked out and that was mainly because I was finding all the faults in myself.</p><p>When the project gets completed and I got rolled off, it wasn’t until 3 months into my next project that I realized what was wrong with my previous project and the team. By the time project was completed –</p><ul><li>2 out of 6 people in the team have resigned from the company</li><li>The client who had continued for the last 4 years did not renew the contract</li><li>I had a fight with my seniors</li><li>Another colleague had frequent health problems in the past 3 months</li></ul><p>Something was definitely wrong but the question was, What? and who should be blamed?</p><p>Fortunately, I had a good team and manager in my next project. I have observed the same situations as the previous project, but now things are different. Everyone on the team was enjoying their work, the client was happy with the progress, and pressure situations went smoothly with collaborations.</p><p>I have summarised some key differences below based on my experience –</p><h3><strong>1. Saying NO </strong>—</h3><p>I remember that during one of the client calls, an additional task came in and I was asked to take up that task. While I was going to say yes, just then I got a ping from my manager who was also present in the meeting to say NO. I tried, fumbled, and said NO indirectly.</p><p>Immediately, after this call, I got a call from my manager and he told me, Rhydham I observed that you were finding it difficult to say NO, <strong>you should learn when to say NO and say it confidently</strong>, say something as simple as “I already have other tasks on my plate so won’t be able to pick up the additional task in this sprint.” He also mentioned that w<strong>e should recognize the client&#39;s priorities and try to accomplish the important tasks first</strong>. It seems like a small thing but this creates a huge difference.</p><p>I recalled that in my previous project, Neither my manager said NO to any justified/unjustified asks of the client and nor do I say NO when my plate was already overflowing with tasks. So in the end, we would fall short of targets and things would spiral to worse, all this despite working late into the night every day.</p><h3><strong>2. Setting the right timelines does not mean the most optimal timelines —</strong></h3><p>In my second project when I was going through the tasks list, I observed that some of the tasks include building an understanding and identifying the changes before diving into the implementation. I was shocked because that was never the case in my previous project.</p><p>I recalled that in my previous project, <strong>one of the major issues was that we could not clearly distinguish the tasks</strong>. A lot of changes were happening in the methodology but no time was kept for experimentation and ideation. <strong>The timelines were built according to the most optimal scenario case and that generally would never happen</strong>. Since there were no buffers, any delay would affect the entire plan which was not accounted for so the only option would be working extra time to deliver within unrealistic timelines.</p><h3><strong>3. Mistakes are not 100% your fault so don’t always let others put the entire blame on you instead discuss why it is happening —</strong></h3><p>It is easy when you deliver perfect work but when you make mistakes, it becomes challenging. I used to think that I will never make mistakes but this myth was also shattered.</p><p>I have always held this notion that if you make a mistake, you should take 100% responsibility for it, Is that a good idea? Yes, but <strong>you should understand the difference between taking responsibility vs taking the blame.</strong></p><p>Because when you take 100% of the blame on yourself, you are giving a free pass to other issues that must be resolved otherwise the errors will not stop. And this exactly happened to me. When delivery requirements aren’t made 100% clear and you are working on a tight schedule, mistakes are bound to happen. At that point, take a pause and analyze what needs to change, and confidently present it to your team.</p><p>Is there a common theme in all the above points? Yes, it is the “<strong>Power of Expression</strong>.”</p><p>You should have the ability to express your opinions with reasons in a timely manner. First, make it clear in your mind that your opinion matters. Secondly, be open-minded to these situations and learn to deal with different scenarios. Of course, this is easier said than done but observation is key. So start there.</p><p>This will give you better growth in your career and it is as important as developing your technical skills. Always remember these 3 points as they will help you immensely in the corporate world.</p><p>Hope you liked this article!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d131cdf49f9a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Churn Prediction (in Retail)– Approach in a real project]]></title>
            <link>https://rhydhamgupta.medium.com/churn-prediction-in-retail-approach-in-a-real-project-3d1718594bc6?source=rss-62e949c6123b------2</link>
            <guid isPermaLink="false">https://medium.com/p/3d1718594bc6</guid>
            <category><![CDATA[analytixtime]]></category>
            <category><![CDATA[approach]]></category>
            <category><![CDATA[retail]]></category>
            <category><![CDATA[churn-prediction]]></category>
            <category><![CDATA[propensity-model]]></category>
            <dc:creator><![CDATA[Rhydham Gupta]]></dc:creator>
            <pubDate>Sun, 30 Oct 2022 05:48:14 GMT</pubDate>
            <atom:updated>2022-10-30T05:48:14.075Z</atom:updated>
            <content:encoded><![CDATA[<p>In simple terms, Churn Prediction means predicting the customers who will stop purchasing in near future. But why do we need it?</p><p>Say we own a grocery store named ATmart, and we’ve posted solid growth of 10% for the past 2 years. But current year estimates project a negative ~-1% growth at the current pace. What should you do?</p><p>There can be 2 strategies, <strong>1) Increase new customers and 2)Increase the retention rate of old customers</strong>. Which strategy should be our priority? Customer acquisition is often the most expensive step in the customer journey so let’s focus on the second strategy, The first step to increasing the retention rate is to identify the customers who will stop buying in the coming months. <strong>This is what we refer to as Churn Prediction. </strong>Let’s see it in action.</p><p>In ATmart, from churn analysis, we have got a list of customers who are at risk of leaving. We further segment them into 3 groups and send them the following offers — <br>Group1- Additional 5% discount next month, <br>Group2-Gift Coupon on min purchase of 2K, <br>Group3-Get a 10% discount on combo packs. <br>Hopefully! This should put us back on the path of growth.</p><p>Since we have already seen the importance, let’s see how to do Churn Prediction Analysis the right way.</p><h4>Steps-</h4><ol><li>Defining the modeling construct</li><li>Input dataset with dependent and independent variables</li><li>Choosing the right classification model</li><li>Evaluation</li></ol><h3>Defining the modeling construct</h3><p>This is a very important step in the churn prediction model. This part is highly subjective and I have seen the average data scientist rush through it but a good data scientist spends a good amount of time on this step. This step basically involves making decisions. Let’s understand it in more detail.</p><p>For our ATmart, Few questions that we would like to answer —</p><ol><li><strong>How do we define an eligible customer for our case</strong>? <br>For e.g. these could be customers who have transacted in the <br>current month <br>or the past 2 months <br>or past 1 year or <br>maybe a more complex condition like someone who has done a transaction in at least 3 months in past 1 year. You get the point.</li><li><strong>What is the definition of churn that we want to predict? <br></strong>For e.g. a churned customer can be one <br>who will not do any transaction in the next 1 month <br>or who will not do any transaction in the next 2 months etc</li></ol><p>Now the common approach to getting answers to these questions is to ask business and dig deep into the data. So for ATmart, we contacted the store manager and they said that usually if a customer does not make any transaction in 2 months then chances of him/her coming back is very low. <em>So our churn customer becomes the one who does not do any transaction in 2 months.</em></p><p>Secondly, for the definition of eligible customers, we will need to go into the data but what to look for? So understand this point very carefully. Our objective is to learn the <strong>characteristics</strong> of<strong> customers who have left us in the past</strong>. So if a churned customer starts showing change in buying pattern from 3 months back, <em>so our definition of customer can be someone who has had atleast 1 transaction in the last 3 months</em>. <strong>Note that we finalize it using detailed analysis that I will cover in another article.</strong> Hurray! we got the answers, let’s proceed to the next step.</p><h3>Input dataset with dependent and independent variables</h3><p>Well if you are confused as to why we are talking about so many definitions then let’s connect the dot in this step. Any model needs dependent and independent variables to go into modeling but we only have a transactional dataset. How do we get there —</p><p>Note — <strong>We have Data till Oct’22 and we are trying to predict the churn for Nov-Dec’22</strong></p><p>Let’s begin with the prediction dataset. So for each customer, we need an output that gives us churn probability. So one thing is clear data has to be aggregated at a customer level. Here’s how the data will look like</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/1*nAVn-fVWjTEhFTqqSzQW9w.jpeg" /></figure><p>Since we only have data till Oct’22, we will aggregate the independent variables for Aug-Oct’22, feed it into the model, and will the churn predictions for the next two months.</p><p>Now you might be wondering how to create a training dataset. Let’s see it</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0NEtszEtYwfMQeg7wK0tOQ.jpeg" /></figure><p><strong>Dependent Variable </strong>— We create a flag using the 2 months churn condition. Since we have data till Oct’22 so we will use the last 2 months to create dependent flag. If any eligible customer<em>(who has done at least 1 transaction in the preceding 3-month period ‘June-Aug’22&#39;) </em>has not done any transaction in Sep-Oct’22 will be a churned customer <strong>hence we will put 1 corresponding to those customers and 0 for other customers.</strong></p><p><strong>Independent Variables </strong>— Now we can have multiple independent variables but note that any aggregation should only be for a 3-month period which is June-Aug’22. Some examples could be the Total transaction amount, #unique products purchased, Frequency of purchase, Recency, Payment Preference, etc. This exercise should be exhaustive and you should try to include all the variables that you think can be the potential indicators of customers which churned.</p><h3>Choosing the right classification model</h3><p>We have already done the heavy lifting. Time to choose a model for our training dataset. In my experience, tree-based algorithms such as Random Forest, and XGBoost work well for most cases. As they provide good accuracy and interpretability.</p><p>Sometimes, the event rate (Proportion of churn customers in training dataset) becomes very small so you often have to use techniques to handle the imbalance dataset.</p><h3>Evaluation</h3><p>As a general practice, we use cross-validation to ensure that model is not overfitting. For evaluation, we can use different metrics like Precision, Recall, F1-score, ROC Curve, etc.</p><p>Another very popular technique for validating churn models is using <strong>decile analysis, Gain Charts, and lift Charts</strong>. The idea is simply that if I pick up 10% of the customers at random then the probability of picking up the churned customers will be the same as 10% of the total churned customers. Conversely, if we select the 10% of the customers&#39; based on the descending order of the predicted probability, then what % of total churn customers I am able to capture? Like with 10% of customers I am able to capture 30% of the total churn customers hence I get a positive lift.</p><p>Hope you enjoyed this article! Check out my <a href="https://www.analytixtime.com/">blog </a>for more such articles.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3d1718594bc6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Clustering — Profiling & Cluster Movement Diagram]]></title>
            <link>https://rhydhamgupta.medium.com/how-do-you-select-optimal-clusters-in-kmeans-clustering-in-a-real-project-34f268c1da5f?source=rss-62e949c6123b------2</link>
            <guid isPermaLink="false">https://medium.com/p/34f268c1da5f</guid>
            <category><![CDATA[k-means-clustering]]></category>
            <category><![CDATA[real-project]]></category>
            <category><![CDATA[k-means]]></category>
            <category><![CDATA[profiling]]></category>
            <category><![CDATA[analytixtime]]></category>
            <dc:creator><![CDATA[Rhydham Gupta]]></dc:creator>
            <pubDate>Thu, 13 Oct 2022 07:43:40 GMT</pubDate>
            <atom:updated>2022-10-21T02:45:11.562Z</atom:updated>
            <content:encoded><![CDATA[<p>Kmeans clustering is one of the most widely used techniques by data scientists. Its simplicity, easy interpretation, and business value are simply amazing. But still, I find that when I ask people how do you select the optimal number of clusters, they’ll answer like the elbow curve. My next question to them is what else? And most often they look at me in amazement and say nothing more.</p><p>There’s nothing wrong with this but it is very incomplete because remember any technique is only valuable when it provides value to the business. In this article, I am going to share a practical approach to selecting the optimal number of clusters in any real project.</p><p>Broadly speaking there are four major steps -</p><ol><li>Defining the clustering objective</li><li>Using statistical measures to select the optimal range of clusters</li><li>Profiling of the clusters</li><li>Cluster movement diagram</li></ol><h3>Defining the clustering objective</h3><p>Clustering is no magic but<strong> grouping similar data points together. </strong>Now it is the business need that will determine the parameters for similarity. Let’s take a simple example —</p><p>Let’s say an e-commerce company (Flipkart, Amazon) wants to send discount coupons to selected customers. They have a fixed budget of $100K. Now as a data scientist you are asked to give them the list of customers who should be targeted for discounts and what level of discount should be offered.</p><p>One of the very practical solutions is to <strong>segment the customers</strong>. One objective could be that we want to offer these discounts to <strong>high-value customers</strong> and <strong>target the younger customer base</strong>. We have defined the parameters, now let&#39;s put them as variables-</p><p><strong>High-Value customers— </strong>Average Monthly sales, Average ticket value, Payment method, #unique categories purchased, frequency of purchase</p><p><strong>Younger customer base — </strong>Age, Location(City Tier)</p><blockquote>Avoid putting the irrelevant variables into clustering because it will only drift your analysis.</blockquote><h3>Using statistical measures to select the optimal range of clusters</h3><p>Now that we have the variables selected, the next question is how many clusters. The statistical measures are a good way to obtain <strong>the optimal range of clusters</strong> that can be selected. <strong>ELBOW CURVE</strong> and <strong>silhouette score</strong> are some popular methods that can be used. Use it to only get the range like 5–7, 4–6, etc. <em>Choosing the K (number of clusters) only from these measures is not always a good idea as it sometimes leads us to ignore some important information.</em></p><p>Let’s just extend the above example, Based on the elbow curve, we select 5 as optimal clusters. Now in profiling (which we will understand in the next section), we realized that one of the groups has high average sales but it includes customers from all age groups. Remember, while defining the objective, we were planning to target young customers as they can deliver high lifetime value. But because we choose a 5-cluster solution, we didn’t get the chance to look at a 6-cluster solution where that cluster was breaking down further based on the age criteria. So that’s the reason other steps are very important.</p><h3>Profiling of the clusters</h3><p>Profiling of clusters is a very important step in any real project clustering solution. Below is a snapshot of what we mean by profiling.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/973/1*-c8xjVq8huseU7IWaWmTPg.png" /></figure><p>It is <strong>equivalent to summarizing different variables across different clusters.</strong> Every cluster has some unique characteristics that are different from the other group.</p><p>Let’s say we are talking about Cluster-5, although it accounts for only 5% of the customers, the average monthly sales and ticket value are high. So it could be one of the potential groups to give discounts.</p><p>If we look at Cluster-4, we will see that it belongs to customers who are relatively young and come from Tier-2 and Tier-3 cities. In addition, their average sales are good but they are buying fewer categories of products on average. So we can strategize to give discounts to these customers for the categories they are not currently buying.</p><p>I think you got the idea.</p><h3>Cluster movement diagram</h3><p>Last but not the least, this step is very crucial to check the robustness of the existing clusters. It basically checks how the existing cluster size splits if we increase the number of clusters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/1*Be3KF72-iD2r3tebHd0LFQ.jpeg" /></figure><p>The number represents the count of customers in each cluster before(<em>5-cluster</em>) and after(<em>6-cluster</em>).</p><p>Clearly, we can see that if we increase the number of clusters from 5 to 6, it is mainly Cluster-5 that breaks down further.</p><p><strong>Implementation </strong>— A very valid question one might ask is how K-means know that c<strong>luster_no-1</strong> in the 5-cluster solution and 6-cluster solution is the same. It doesn’t but we know that if an existing <strong>cluster_no-1</strong> is breaking into 1,2,3,4,5,6. Suppose we see the max count of customers for the combination 1–6, so we will simply swap cluster 1 by 6 in the 6-cluster solution. Do the same exercise for the rest of the clusters.</p><p>Hope you liked this article :)</p><p>You can visit my <a href="https://www.analytixtime.com/">blog </a>for more such articles.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=34f268c1da5f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Science Interview Experience- Top 5 statistics questions asked]]></title>
            <link>https://rhydhamgupta.medium.com/data-science-interview-experience-top-5-statistics-questions-asked-2a7ef54716a4?source=rss-62e949c6123b------2</link>
            <guid isPermaLink="false">https://medium.com/p/2a7ef54716a4</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[interview]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[interview-questions]]></category>
            <category><![CDATA[analytixtime]]></category>
            <dc:creator><![CDATA[Rhydham Gupta]]></dc:creator>
            <pubDate>Tue, 20 Sep 2022 09:46:02 GMT</pubDate>
            <atom:updated>2022-09-20T09:51:59.138Z</atom:updated>
            <content:encoded><![CDATA[<p>Statistics is one of the topics that become very daunting in the data science interview. If you have some regression-related projects on your resume, you are bound to lead into the arena of statistics. In countless interviews, statistics questions became the reason for my non-selection.</p><p>The most tricky thing about statistics is that its implementation in projects is intrinsic in nature so it becomes difficult to develop a clear intuition, but still, if you can cover 5 main questions/concepts related to statistics, you will be able to sail through this section in the interviews.</p><p>Before going into details of questions, there is a fact about statistics that is important to understand.<strong> </strong>Statistics is a great deal about <strong>using probability to make the best guesses</strong>. Why the need to guess, you may ask? Because it is practically not possible to collect statistics for the population. <strong>The most intuitive example is opinion polls during elections.</strong> Aren’t you amazed that a survey among a few thousand people gives results so close to those of millions of voters? The power of statistics!</p><p>Let’s get into it –</p><p><strong>Q1: What is the central limit theorem? And what is a normal distribution? Difference between normal distribution vs standard normal distribution?</strong></p><p>Ans: Central limit theorem is the foundation of inferential statistics (making guesses on population using sample). It states that if the random sample is drawn from a population n number of times, then its mean will follow a normal distribution. <em>(Some people make the mistake of assuming that if the samples are coming from same distribution, their means should be equal. Mind that we are randomly picking samples so it will not happen)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ifGVw3CUzkGCpwf6P_wl-A.jpeg" /></figure><p>A normal distribution is bell-shaped distribution meaning data is distributed symmetrically around the central tendencies (mean, median, mode).</p><p>Standard normal distribution is a special case where mean = 0, standard deviation = 1.</p><p><strong>Bonus Tip </strong>— I used to mention that the normal distribution concept is also used when we standardize variables. An intuitive way to think is that the variable is now transformed in such a way that each row represents how many standard deviations the data point is from the mean.</p><p><strong>Q: What are the t-test and z-test? When do we use z-test vs t-test?</strong></p><p>Ans: Both the t-test and z-test are part of hypothesis testing. There is a null hypothesis (by default accepted fact) and an alternative hypothesis that we are trying to prove using the sample data. Specifically, using a t-test we can compare the sample means of two samples.</p><p>Explain it using a practical example in the interview –</p><p>For e.g. A company has recently implemented a recommendation system(RS) on a website and they want to know if it has had any real positive impact on sales. We define the null hypothesis as RS has no impact on the sales and the alternative hypothesis as RS has a positive impact on the sales. We get the data of sales for 20 random customers who use the old popularity-based system and the new recommendation system.</p><p>Popularity-based sample =&gt; x1(mean) = 800, SD1 = 100, n1=20</p><p>Recommendation Engine sample =&gt; x2(mean) = 900, SD2=200, n2=20</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/230/1*-BVDUm86O8zPIk7MT4JZrA.png" /></figure><p>t = (900–800)/sqrt(200²/20 + 100²/20) = 100/50 = 2</p><p>degree of freedom = (n1 + n2–2) = 38</p><p>Define α = 0.05, significance level</p><p>Using the (α, degree of freedom) we get the critical t-value from the tables (in this case, 1.68) and if the computed t-value is more than the critical t-value, we reject the null hypothesis and say RS has a positive impact. But if the t-value is less than the critical value then we say that the current sample does not give conclusive evidence to reject the null hypothesis.</p><p>Z-test is used over t-test when two conditions are followed-</p><ol><li>We have the population standard deviation</li><li>The sample size is greater than 30</li></ol><p><strong>Bonus Tip —</strong></p><p>What will happen to the computed t-value if we lower the number of data points in the sample? In the formula, reduce the value of n1 and n2 and recompute the t-value.</p><p><strong>Q: Explain the p-value and how is it relevant in the context of regression.</strong></p><p>The P-value for a certain event is the sum of the probability of a <strong>given event</strong>, <strong>equally rare event, </strong>and <strong>more rare event</strong>.</p><p>Some other definition states that it is the probability of getting a sample as or more extreme than the given sample. I know this might sound confusing. Check out this <a href="https://www.analytixtime.com/post/p-value-explained-clearly-regression-pdf-discrete/">article</a> for more details.</p><p>In regression, the p-value of the coefficient determines whether the coefficient is statistically significant. We test the hypothesis on each coefficient where the null hypothesis is that the coefficient is 0. Now if the p-value of the coefficient is below the threshold, we reject the null hypothesis and say that the coefficient for the variable is significant.</p><p><strong>Q: What is ANOVA? Where do you use ANOVA?</strong></p><p>ANOVA is used to compare means when we have more than 2 samples. It is part of hypothesis testing.</p><p>Now, the mathematical details of the ANOVA process are somewhat complex and long so you cannot expect a question on it but it is good to have knowledge of the process. If you get time, read about it.</p><p>Fundamentally, like the t-value, In ANOVA, we try to reject the null hypothesis that all the sample means are equal by computing the f-value which in simple terms as follows—</p><p>F-value= Variance between the samples/Variance within the sample</p><p>Check out the interesting version of an ANOVA-related question that was asked in one of my interviews. It can be found <a href="https://www.analytixtime.com/post/using-anova-test-in-the-real-project/">here</a>.</p><p><strong>Q: Explain the chi-square test and what it is used for.</strong></p><p>The Chi-square test is another test of hypothesis testing and it can be used to compare the categorical variables.</p><p>There are two main use-cases of chi-square-</p><ol><li>To determine goodness of fit (Let’s say someone gives you data and asks you to check if it comes from the normal distribution)</li><li>To test the independence of two variables</li></ol><p>Try to explain it with a practical example like—</p><p>Suppose research claims that sending cards to customers on their birthday increases the retention rate. Can you test this claim?</p><p>You design an experiment in which for a certain week you send cards to 1 group of customers while not sending them to another set of customers. Next week you check how many customers are retained from both groups. Let’s say below is the data</p><p>Null Hypothesis — Both the variables are independent meaning sending cards has no impact on the retention rate of the customer</p><p>Alternate Hypothesis — Both the variables are not independent</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*hDKmOmQQtJJTh-WHrRFHfA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*dQ26dJMHPzp_-tIe.png" /></figure><p>Expected data is based on the total frequency of rows and columns. For e.g. Expected(Received Cards,Retained) = (Total Retained X Total Received Cards)/Total Customers</p><p>= (110 X 100)/200 = 55</p><p>Calculate the Chi-square value using the above formula—</p><p>X² = (60–55)²/55 + (50–55)²/55 + (40–45)²/45 + (50–45)²/45 = 2.02</p><p>degree of freedom = (r-1)(c-1)</p><p>Using the (α, degree of freedom), we check the critical X² value from the table and compare it to the calculated X² value. If the calculated X² value is higher then we can reject the null hypothesis.</p><p>Hope you get to benefit from this article in your interview!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2a7ef54716a4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Science Interview Experience — Top 5 tricky ML questions asked]]></title>
            <link>https://rhydhamgupta.medium.com/data-science-interview-experience-top-5-tricky-ml-questions-asked-120c3ab400c5?source=rss-62e949c6123b------2</link>
            <guid isPermaLink="false">https://medium.com/p/120c3ab400c5</guid>
            <category><![CDATA[interview-preparation]]></category>
            <category><![CDATA[data-science-interview]]></category>
            <category><![CDATA[interview-questions]]></category>
            <category><![CDATA[experience]]></category>
            <category><![CDATA[analytixtime]]></category>
            <dc:creator><![CDATA[Rhydham Gupta]]></dc:creator>
            <pubDate>Mon, 12 Sep 2022 02:22:29 GMT</pubDate>
            <atom:updated>2022-09-15T06:33:03.272Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data Science Interview Experience — Top 5 tricky ML questions asked</h3><p>Recently, I have been giving many interviews for the role of Data Scientist and related roles. When I started, I was disappointed by the low success rate of passing the interviews. In initial interviews, everything will be going smoothly but then the interviewer will ask a tricky ML question and my fumbled answer and subsequent line of questioning will knock me out. Soon, I realized that though these tricky questions were asked in different shapes and contexts, the fundamentals of the underlying topics remained the same and are limited.</p><p>I worked on it and as a result, after initial hiccups, I started tasting success in the interviews. Of course, this was not the only factor for clearing the interviews but now the ML section no longer leads to my elimination instead it bought me extra points.</p><p>Based on my experience, I am going to share the top 5 questions/concepts which are most focused on by the interviewers in the ML section. I am confident that if you have a solid grasp of these concepts and the related concepts, the ML section of the interview will become your strength.</p><p><strong>Let’s dive into these questions —</strong></p><p><strong>Q: What is the difference between XGBoost and Gradient Boosting</strong></p><p>In one of the initial interviews, I remember myself answering that XGBoost is the faster and optimized implementation of the Gradient Descent. Although, this is technically correct but the interviewer is actually expecting an answer more from a data science perspective.</p><p>Ans:</p><ul><li>XGBoost has a provision to apply Regularisation</li><li>XGBoost has an in-built mechanism to deal with the Null values using a technique known as the sparsity awareness</li><li>It uses gradients to create the tress hence the criteria of node split are different and based on the similarity scores</li><li>Parallelisation to find the (variable, threshold) combinations using the weighted quantile sketch approach on large datasets</li><li>Optimisation to speed up the computations</li></ul><p><strong>Q: How do you control overfitting? Can we use cross-validation to control overfitting?</strong></p><p>Ans: You should know that cross-validation can be used to check the over-fitting but cannot control it. Then how can you control it —</p><ul><li>Feature Engineering &amp; Feature Selection</li><li>Parameter tuning</li><li>Treating outliers if it is a linear algorithm</li><li>Regularisation</li><li>Early stopping, Pruning</li><li>If possible, get more data</li></ul><p><strong>Bonus tip </strong>— Try to explain any scenario that you have faced while working on a project. I used to tell that in one of the projects I was facing this issue of overfitting because test accuracy was low while train accuracy was fine. On in-depth examination, I identified that there were highly correlated variables that were causing this to happen. So I used PCA to combine them into 1 and hence got rid of overfitting.</p><p><strong>Q: Different evaluation metrics used for regression and which one is preferred when?</strong></p><p>Ans: Different evaluation metrics used for regression are —</p><ul><li>R2 (Most commonly reported metric, it explains what percentage of variance in dependent is explained by the independent variables)</li><li>Adj. R2 (R2 that gets penalized for the high number of variables)</li><li>MSE (Loss function)</li><li>RMSE (Root mean squared error)</li><li>MAPE (Mean absolute percentage error, this metric is easy to explain to the business as it indicates what is the percentage error in prediction on an average. For e.g. 10% MAPE indicates that for each record on average prediction values deviate by 10% from the actual)</li></ul><p><strong>Bonus </strong>— <strong>When to use MSE vs RMSE?</strong></p><p>RMSE is the metric that is at the same scale as that of the actual dependent whereas MSE is on the squared scale.</p><p><strong>Q: What is the difference between Bagging &amp; Boosting</strong></p><p>Ans:</p><p>Bagging —</p><ol><li>It creates multiple trees parallelly and takes the majority vote for the final prediction</li><li>It creates trees on the actual dependent value</li><li>It provides more robustness and might not give very good results on the imbalanced dataset</li></ol><p>Boosting —</p><ol><li>It is the sequence of weak learners and the next tree is dependent on the residuals of the prediction till the last tree</li><li>It creates trees on the residuals</li><li>Since, it creates the subsequent trees on the residuals, hence more importance is given to the misclassified samples. So it works well on the imbalanced dataset</li></ol><p><strong>Bonus </strong>— In which case, will you prefer Bagging over Boosting and vice-versa?</p><p>The answer lies in the above points.</p><p><strong>Q: What are precision and recall? An interesting version of this question, </strong><br><strong>It is reported that 12 of the predicted 18 frauds were classified correctly and 80% of the total frauds were caught. Tell the precision and recall.</strong></p><p>Ans:</p><p>You might get confused about this question if you have only a surface-level understanding of the precision &amp; recall. Let’s try to create the confusion matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*55BWTeyZLgEzRm6CcmJSqg.png" /></figure><p>Precision = TP/(TP + FP) = 12/18 = 0.66</p><p>Recall = TP/(TP + FN) = 12/15 = 0.8</p><p>Hurray! Importantly if you have a sound understanding of precision and recall, you would realize that the answer is directly present in the question.</p><p>Recall — What % of the actual 1s are predicted correctly = 80% = 0.8</p><p>Precision — What is the accuracy rate of the predictions, out of 18 predictions 12 were right, hence 12/18 = 0.66</p><p>Interestingly, the fun part of this question is that TN is not a question and it is not even required for precision and recall.</p><p>Hope you liked this article. You can find out a detailed explanation in my <a href="https://www.analytixtime.com/">blog</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=120c3ab400c5" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>