<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Taingocbui on Medium]]></title>
        <description><![CDATA[Stories by Taingocbui on Medium]]></description>
        <link>https://medium.com/@taingocbui_39735?source=rss-e57460c7940c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*RTqz_Dr2t9BVfAJ0</url>
            <title>Stories by Taingocbui on Medium</title>
            <link>https://medium.com/@taingocbui_39735?source=rss-e57460c7940c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 12:47:30 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@taingocbui_39735/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building a Movie Recommendation System with the Surprise Library]]></title>
            <link>https://medium.com/@taingocbui_39735/building-a-movie-recommendation-system-with-the-surprise-library-64d6437c3531?source=rss-e57460c7940c------2</link>
            <guid isPermaLink="false">https://medium.com/p/64d6437c3531</guid>
            <dc:creator><![CDATA[Taingocbui]]></dc:creator>
            <pubDate>Sun, 12 Jan 2025 23:30:52 GMT</pubDate>
            <atom:updated>2025-01-12T23:30:52.605Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*m1eQqc0TyMkipMF_kwsM-w.jpeg" /></figure><p>Recommendation systems have become a cornerstone of modern digital experiences, driven by advancements in data availability, machine learning, and computational power. From their origins in simple rule-based filtering, they have evolved into sophisticated models using collaborative filtering, content-based methods, and hybrid approaches. Innovations such as matrix factorization, deep learning, and graph-based techniques have further enhanced their accuracy and scalability. In this post, we’ll walk through the process of building a simple movie recommendation system using the <strong>Surprise</strong> library and the <strong>MovieLens </strong>dataset<strong>.</strong></p><h4><strong>Recommendation System Overview</strong></h4><p>A recommendation system is a subclass of machine learning systems designed to predict user preferences for items or services. By analyzing patterns in historical data, these systems provide personalized suggestions, improving user engagement and decision-making. Today, recommendation systems are used across diverse industries to deliver personalized experiences. In <strong>e-commerce</strong>, they drive product suggestions and increase sales. <strong>Streaming platforms</strong> like Netflix and Spotify rely on them for personalized movie, TV show, and music recommendations. In <strong>social media</strong>, they power friend suggestions and content discovery. Other applications include healthcare (personalized treatment plans), education (course recommendations), and travel (destination suggestions).</p><p>The rapid growth of recommendation systems has been fueled by the need to handle massive user-item datasets and provide context-aware, real-time recommendations, making them indispensable for businesses aiming to enhance user engagement and satisfaction. According to market.us, the global recommendation system market was valued at 2.8 billion dollars in 2023 and is projected to grow at a compound annual growth rate (CAGR) of 28.5% in the next decade.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/1*F1Nd4MCaN-fQ2O5HnqeTIA.jpeg" /></figure><h4><strong>Why MovieLens Dataset?</strong></h4><p>The <a href="https://grouplens.org/datasets/movielens/latest/"><strong>MovieLens</strong></a><strong> </strong>dataset is a widely-used benchmark dataset for research and development in recommendation systems. Created by the GroupLens Research team, it contains user ratings, timestamps, and metadata for movies. Its popularity stems from its clean structure, diverse user interactions, and well-documented format, making it ideal for testing algorithms, performance bench-marking, and exploring user-movie relationships in recommendation systems. The dataset used in this tutorial is the smaller sample with over 100,000 ratings across over 9,000 movies.</p><h4>What is Surprise?</h4><p>The <strong>Surprise</strong> library is a Python framework for building and analyzing recommendation systems, particularly collaborative filtering methods. It supports various algorithms, including memory-based approaches (like k-nearest neighbors) and model-based methods (like SVD, SVD++, and NMF). Surprise simplifies tasks such as dataset preparation, model training, evaluation, and hyper-parameter tuning, with built-in metrics like Root Mean Squared Error (RMSE) and precision. Its applications range from academic research to prototyping systems for personalized recommendations, such as in e-commerce, streaming services, or any domain requiring tailored user-item predictions. Its flexibility and ease of use make it a popular choice for exploring recommendation algorithms. Before we dive in, make sure appropriate libraries are installed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/1*YkfQ7gP9HlRIXDLryJBjRw.png" /></figure><p>Besides Surprise library, all libraries used in this tutorial will be loaded into the notebook as follow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/819/1*Vc4WkRFlzk4DilW2SZlGog.png" /></figure><h4><strong>Content-Based Filtering vs. Collaborative Filtering</strong></h4><p>Content-based filtering and collaborative filtering are two foundational approaches in recommendation systems. <strong>Content-based filtering</strong> relies on item attributes and a user’s preferences, creating recommendations by matching the features of items (e.g., genres, keywords) with the user’s profile. It excels in personalization and works well for users with known preferences but struggles with recommending diverse items or handling new items without descriptive features. <strong>Collaborative filtering</strong>, on the other hand, relies on user-item interactions, such as ratings or clicks, to find patterns among users or items. It can be user-based (recommending items liked by similar users) or item-based (suggesting items similar to those the user has liked). Collaborative filtering provides diverse recommendations without needing item attributes but faces challenges like sparsity and cold starts for new users or items. Both approaches have strengths and limitations, and combining them often leads to more effective recommendation systems. This tutorial focuses only on use-based Collaborative filtering approach.</p><h4>Preprocessing</h4><p>The <strong>Surprise </strong>library requires data to be loaded from a specific format, typically a pandas DataFrame, because it has a strict structure for how it processes and models data. The library expects data in a structured format where each row represents an interaction, while columns typically include user_id, item_id and rating. A pandas DataFrame is a natural way to represent and manipulate such tabular data before passing it into the Surprise library. All csv files will be loaded into DataFrame using Pandas’ read_csv() method.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/753/1*BhULJGFLD6YtTh7Az5nTXA.png" /></figure><p>The load_from_df() method allows users to load user-item-rating interaction data from a pandas DataFrame into its internal Dataset format. This method ensures flexibility and ease of integration, as the DataFrame format is commonly used in machine learning workflows. It supports custom rating scales through the Reader class and simplifies preprocessing tasks like filtering or normalizing data. Using load_from_df() is ideal for dynamic datasets, offering more control compared to loading data directly from static files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/743/1*4Sigw2G9kYEh2Zhk_Itcrg.png" /></figure><h4>K-Nearest Neighbors Model</h4><p>The <strong>K-Nearest Neighbors (KNN)</strong> model is a type of algorithm used in both classification and regression tasks. In the context of recommendation systems, <strong>KNN models</strong> are used for memory-based collaborative filtering, where they identify “neighbors” (similar users or items) to make predictions. For <strong>user-user collaborative filtering</strong>, KNN identifies the most similar users to the target user and predicts ratings based on these users’ preferences. On the other hand, KNN finds similar items to the target item for <strong>item-item collaborative filtering</strong>, and predicts a user’s rating based on their ratings of these similar items. Due to limitations of memory-based model, KNN requires the user-item matrix to be available in memory; thus, having limited scalability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/402/1*3fuiTyKc6fimtD0N7vSVew.png" /></figure><ol><li><strong>Base Model — KNNBasic</strong></li></ol><p><strong>KNNBasic</strong> is a simple KNN model. It predicts ratings based on the weighted average of neighbors’ ratings. It requires a similarity measure like <strong>cosine</strong> or <strong>Pearson </strong>correlation. Cosine similarity and Pearson similarity are both metrics for measuring similarity but differ in focus and application. Cosine similarity measures the cosine of the angle between two vectors, focusing on their direction while ignoring magnitude, making it ideal for comparing high-dimensional or sparse data where the scale is irrelevant. In contrast, Pearson similarity evaluates the linear correlation between two vectors, accounting for their mean and relative deviations, making it suitable for analyzing relationships or trends in the data. While cosine similarity emphasizes vector orientation, Pearson similarity captures how well one variable’s variations predict another’s, regardless of scaling or translation.. Here I use cosine similarity for KNNBasic with user-based approach and 5 splits cross-validation. This set up is the same as using a train_test_split() with 80% train set and 20% test set. The average RMSE for KNNBasic model is 0.973, meaning that it was off by roughly 1 point for each guess it made for ratings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*-lBKSbqN1XTVMLoryDt1bA.png" /></figure><p><strong>2. KNNBaseline</strong></p><p>To further improve the neighborhood-based model, I will apply the <strong>KNNBaseline</strong> method, adding in bias term to reduce over-fitting. KNNBasic directly computes similarity between users or items based on raw ratings, making it simple and intuitive but less effective in datasets with systematic biases, such as users who consistently rate items higher or lower than average. On the other hand, KNNBaseline accounts for these biases by incorporating baseline estimates for ratings, which adjust for global, user-specific, and item-specific biases. I will also use KNNBaseline with cosine similarity. By adding in bias term, KNNBaseline method achieved the best RMSE of 0.8710 with cosine similarity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*xS99O7aWBxsBKvY_vycDpA.png" /></figure><h4>Matrix Factorization (Singular Vector Decomposition)</h4><ol><li><strong>Understanding SVD in Recommender Systems</strong></li></ol><p>Singular Value Decomposition (SVD) is a matrix factorization technique that decomposes the user-item interaction matrix into three matrices:</p><ul><li><strong>U</strong>: represents the user feature matrix, where each row corresponds to a user and each column to a latent feature.</li><li><strong>Σ</strong>: is a diagonal matrix containing singular values, which signify the importance of each latent feature. The latent features capture underlying patterns in user preferences and item characteristics. By reducing the dimensions, SVD identifies these patterns, which help in predicting missing ratings.</li><li><strong>V^T</strong>: is the transposed item feature matrix, where each column corresponds to an item and each row to a latent feature.</li></ul><p>The SVD algorithm predicts the missing values in the user-item interaction matrix by approximating the original matrix through the dot product of these three matrices. During training, the algorithm optimizes the matrices, and to minimize the difference between the actual and predicted ratings, typically using a loss function such as Mean Squared Error (MSE).</p><p>The following code sample will shuffle and split data into train and test set. To improve the performance of the SVD model, I also tune its hyperparameters. Surprise provides a GridSearchCV class for this purpose</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/815/1*Q-VBrJZvGsdSCezOFPeZhA.png" /></figure><p>The algo variable below holds the best parameters after training with GridSearchCV. The RMSE score on test set with the best parameters achieve 85.19%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/833/1*rQMTjo-ia8rIxtMun61VZg.png" /></figure><p>In the final part of this blog, I built a simple function which takes in user id, the predictions, and the number of recommendations that users want to return. The predictions parameter is the estimated ratings that the model calculates for user-item pairs in the test set or for unseen user-item pairs. In practice, after training the model, test method in Surprise can be used to generate these predictions for a set of user-item pairs. The result includes the true rating (if available), the predicted rating, and additional information such as the prediction details.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/861/1*-AdZZl0gTvf1ZcKAPnG7Tg.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=64d6437c3531" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Logistic Regression: A Comprehensive Guide]]></title>
            <link>https://medium.com/@taingocbui_39735/understanding-logistic-regression-a-comprehensive-guide-f01a3df9eeef?source=rss-e57460c7940c------2</link>
            <guid isPermaLink="false">https://medium.com/p/f01a3df9eeef</guid>
            <dc:creator><![CDATA[Taingocbui]]></dc:creator>
            <pubDate>Tue, 04 Jun 2024 02:14:51 GMT</pubDate>
            <atom:updated>2024-06-21T00:13:56.277Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*Qh5-YY0O7hP0_Wtw06zkmA.png" /></figure><p>With the rapid growth of machine learning, logistic regression stands out as one of the most fundamental and widely used algorithms today. Despite its simplicity, it offers a powerful solution for binary classification problems, making it a cornerstone technique in the data scientist’s toolkit. Whether you’re predicting customer churn rate, diagnosing diseases, or detecting spam, logistic regression provides a robust and interpretable method to tackle these challenges.</p><p>This blog aims to demystify logistic regression, breaking down its core concepts, mathematical underpinnings, and practical applications. By the end of this post, you’ll have a solid understanding of how logistic regression works, why it’s effective, and how you can implement it in your own projects. Let’s embark on this journey to explore the fascinating world of logistic regression and its pivotal role in machine learning.</p><h4>What is Logistic Regression?</h4><p>Logistic regression is classified under Supervised Learning, a class of machine learning algorithms that can learn a task through labeled training data. In Supervised Learning, any data used must have a label. By comparing its predictions against the actual labels, the algorithm can learn to make less incorrect predictions and improve the overall performance of the task its learning to do.</p><p>The field of Supervised Learning is further broken down into two categories including Classification and Regression. Logistic Regression is one of the most fundamental supervised classifications algorithms in which there are one or many independent variables that determine a binary outcome. Unlike linear regression, which predicts a continuous output, logistic regression predicts the probability of a binary event occurring by fitting data to a logistic curve.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*4aSTE-m3qml2oyV4ldC5WA.png" /><figcaption>A Logistic Regression Curve classifying monthly income based on age</figcaption></figure><h4>How Does Logistic Regression Work?</h4><p>At its core, logistic regression models the relationship between a dependent binary variable and one or more independent variables by using an activation function. This activation function, often known as Sigmoid function, is an S-shaped curve that transforms its real-valued input into a value between 0 and 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*HsEh0zOASpIni1ssBT2AUA.png" /><figcaption>Several different activation functions used in Machine Learning</figcaption></figure><p>Both the blue and green curves are graphs of sigmoid functions. This is a continuous function and bounded within a range (0,1). Among the family of sigmoid functions,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/119/1*ZJ5DsmF_Mm9zCpX4zviZ7g.png" /></figure><p>is mostly preferred due to its simple derivative (showing below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/189/1*cG8uRk2sZplJt7hXCT7v-Q.png" /></figure><p>In the context of logistic regression, x is a linear combination of the input features (independent variables). If we have two variables x₁ and x₂, the model can be represented as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/164/1*YqzhVrbV3o2-_XeiIR8ATg.png" /></figure><p>where <em>β</em>₀ is the intercept, and <em>β</em>₁<em> </em>and <em>β</em>₂ are the coefficients. The probability of the binary outcome is given by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/338/1*YA6DD16Rsu5wmIPEvvL-EQ.png" /></figure><h4>Loss Function and Maximum Likelihood Estimation</h4><p>Training a logistic regression model involves finding the optimal coefficients (β) that maximize the likelihood of observing the given data. This process is known as maximum likelihood estimation (MLE). To achieve this goal, we need to minimize a loss function that quantifies the difference between the predicted probabilities and the actual binary outcomes. The common loss function often used in logistic regression is the logistic loss (aka binary cross-entropy loss). For a single training example (x<em>⁽ⁱ⁾, </em>y<em>⁽ⁱ⁾) </em>where y<em>⁽ⁱ⁾</em> is the true label , the logistic loss function is defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/412/1*Ed4D4erAJeSE6oppXtOs6Q.png" /></figure><p>For the entire dataset, the total loss function is the average of the logistic loss over all m training examples.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/415/1*96TpQBSZIkABNCCurPnRnw.png" /></figure><p>or:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/495/1*kY08PRT2xt4cifzSvHNVtg.png" /></figure><p>where:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/213/1*oeHia1O1ajSWyS03V04yhQ.png" /></figure><p>This loss function can be minimized by using gradient descent. Gradient descent updates the weights iteratively in the opposite direction of the gradient of the cost function with respect to the weights. The gradient of <em>J(</em>w<em>) </em>with respect to w is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/302/1*FEe4DhVGtQb43LPv76nR-A.png" /></figure><p>The weights are updated according to gradient descent using the following rule:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/152/1*rWANyI8xc5U5yFUtVx7AaA.png" /></figure><h4>Interpretation</h4><p>One of the key advantages of logistic regression is the interpretability of its coefficients. Each coefficient represents the change in the log odds of the outcome for a one-unit change in the predictor variable, holding all other predictors constant. The log odds can be converted to odds by exponentiating the coefficient, making it easier to understand the impact of each predictor. For example, if coefficient <em>βⱼ</em> = 0.5, then the odd ratio is e⁰·⁵ ≈1.65, indicating a one-unit increase in X<em>ⱼ</em> implies the odds of output Y = 1 increase by 65%.</p><h4>Evaluation Metrics for Classification</h4><p>Evaluating the performance of a logistic regression model involves using various metrics to assess its accuracy, reliability, and predictive power. One fundamental metric is <strong>accuracy</strong>, which measures the proportion of correctly predicted outcomes (both true positives and true negatives) over the total number of predictions. Accuracy is useful because it allows us to measure the total number of prediction a model gets right, including both true positives and true negatives.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/401/1*IgbCIvLWsPGZ-d2X97cYrg.png" /></figure><p>Besides accuracy, precision and recall are also commonly used as the model’s performance metrics. While <strong>precision</strong> measures the proportion of true positive predictions among all positive prediction made by the model, <strong>recall</strong> quantifies the proportion of true positives that the model correctly identifies among all actual positives. Due to their definitions, precision and recall have an inverse relationship. In fact, these two metrics can be tricky and misleading in certain scenarios. For example, a conservative model classifies only 5 people out of 10,000 patients are sick. Assume this population contains 6,000 sick patients. Even though the model achieve a precision of 100% with 5 out of 5 correct classifications, it missed 5,995 cases where the patients actually had the disease. On the other hand, a model would easily score 100% for recall by just classifying every single patient in the population as having the disease. Sure, it would have many false positives, but it would also correctly identify every single sick person as having the disease.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/297/1*rKp1Mxz2CbOmRUiLb_VGrw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/323/1*Iq_QafaPta1b7GawcJDeRw.png" /></figure><p>To harmonize precision and recall into a single metric, the F1 score is used to measure model’s performance. This score is a harmonic mean of precision and recall. Therefore, the F1 score cannot be high without both precision and recall also being high. When a model’s F1 score is high, you know that your model is doing well all around.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/310/1*7tyIGZWTaw7zAqO6nmI18g.png" /></figure><p>Another metric used in measuring model performance is the ROC curve and its area under the curve (AUC). The <strong>ROC (Receiver Operating Characteristic) curve</strong> plots the true positive rate against the false positive rate at various threshold settings, illustrating the model’s discriminatory ability across different thresholds. The <strong>AUC (Area Under the Curve)</strong> of the ROC curve summarizes the model’s overall discriminative power; an AUC close to 1 indicates excellent performance, while 0.5 suggests no discrimination ability beyond random chance. These metrics collectively offer insights into a logistic regression model’s effectiveness in predicting binary outcomes and help gauge its practical utility across different domains and applications.</p><h4>Advantages and Limitations</h4><p><strong>Advantages</strong>:</p><ul><li><strong>Simplicity</strong>: Easy to implement and understand.</li><li><strong>Efficiency</strong>: Computationally efficient and requires less training time compared to more complex models.</li><li><strong>Interpretability</strong>: Coefficients are straightforward to interpret.</li></ul><p><strong>Limitations</strong>:</p><ul><li><strong>Linearity</strong>: Assumes a linear relationship between the independent variables and the log odds of the dependent variable.</li><li><strong>Binary Outcome</strong>: Primarily suited for binary classification; extensions like multinomial logistic regression are needed for multi-class problems.</li><li><strong>Sensitivity to Outliers</strong>: Can be sensitive to outliers, which can affect the model’s performance.</li></ul><h4>Conclusion</h4><p>Logistic regression is a robust and versatile algorithm that remains a cornerstone in the toolkit of data scientists and analysts. Its simplicity, efficiency, and interpretability make it an excellent starting point for binary classification tasks. While it may not always be the most powerful method available, its ease of use and effectiveness ensure its continued relevance in the ever-evolving landscape of machine learning.</p><p>Whether you are new to data science or an experienced practitioner, understanding logistic regression is essential. By grasping its principles and applications, you can better appreciate the strengths and limitations of this classic algorithm, and effectively apply it to solve real-world problems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f01a3df9eeef" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Demystifying Data Dilemma: An Introduction to A/B Testing]]></title>
            <link>https://medium.com/@taingocbui_39735/demystifying-data-dilemma-an-introduction-to-a-b-testing-51e4733da2a6?source=rss-e57460c7940c------2</link>
            <guid isPermaLink="false">https://medium.com/p/51e4733da2a6</guid>
            <category><![CDATA[a-b-testing]]></category>
            <dc:creator><![CDATA[Taingocbui]]></dc:creator>
            <pubDate>Sun, 24 Mar 2024 07:16:20 GMT</pubDate>
            <atom:updated>2024-03-24T07:16:20.566Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VhPfZIYkcnV-vWZYaAvgjg.jpeg" /><figcaption>Source: <a href="https://diggintravel.com/airline-ab-testing/">https://diggintravel.com/airline-ab-testing/</a></figcaption></figure><p>In the ever-evolving era of digital marketing, business decisions are increasingly being driven by data. Business leaders don’t want to make decisions unless they have evidence. Amidst this data-driven revolution, A/B testing emerges as one of the most common statistical methodologies in supporting businesses to optimize their strategies and user experience. In this article, I will paint a brief introduction of A/B testing, exploring its significance, best practices, and real-world applications.</p><h4>What Is A/B Testing?</h4><p>At its core, A/B testing involves comparing two versions of a webpage, app interface, email campaign or any other digital assets, to figure out which performs better in terms of predefined metrics. A/B testing is a structured approach to experimentation, combined with statistical principles with clear objectives. The test can help businesses gain invaluable insights into user behavior, preferences and the effectiveness of different strategies.</p><p>One of the first applications of A/B testing is its use in ad campaign marketing using promotional coupons started with Claude Hopkins, an American author and advertiser. However, the key principles we use in A/B testing today had not been developed until 1920s by Ronald Fisher, a statistician and biologist. One of the first applications of A/B testing is its use in ad campaign marketing using promotional coupons started with Claude Hopkins, an American author and advertiser. However, the key principles we use in A/B testing today had not been developed until 1920s by Ronald Fisher, a statistician and biologist.</p><blockquote>“He wasn’t the first to run an experiment like this, but he was the first to figure out the basic principles and mathematics and make them a science.”</blockquote><blockquote>— <strong>Kaiser Fung</strong>, founder of the applied analytics program at Columbia University</blockquote><p>Fisher ran agricultural experiments to determine if his land needs more fertilizer. These newly found principles were later adapted by scientists in the 1950s to run clinical trials in medicine. The concept became popular in the 1960s and 1970s as a tool to evaluate direct response campaigns by marketers. In fact, the math behind the tests hasn’t changed throughout the past century. The only difference is that A/B testing is used on a different scale today in terms of number of participants and experiments.</p><h4><strong>How does A/B Testing Work?</strong></h4><p>A/B testing involves several key steps to ensure accurate results and meaningful insights. The test starts by defining clear metrics and hypotheses. You need to be clear which specific metrics can be used to evaluate the test subject’s performance. Once the specific metric has been selected, it’s time to define a null hypothesis. The null hypothesis is the claim that a researcher is hoping to refute. A proper null hypothesis should ensure that it is written with quantitative measurement. After selecting proper metrics and forming null hypothesis, researchers can proceed by creating two different versions with only one differing element between them. These versions will be presented to two similarly sized groups of users assigned at random: the control group and the experimental group. The results collected from the two groups will be analyzed to see the effectiveness of the element over a specific period.</p><p>Some common statistical methods used to determine the significance of differences between elements are t-test, Chi-square test, or z-test. T-tests are mostly used to compare the means of two groups to determine if there is a statistically significant difference between them. In the scenarios where the sample size is large or the population standard deviation is known, z-tests are particularly more useful than t-test. Lastly, Chi-square tests are used when comparing categorical data between two or more groups. This test assesses the significant differences in the distribution of categorical variables across groups.</p><p>Besides selecting the appropriate statistical test, researchers also need to investigate various surrounding parameters such as alpha, power, sample size and effect size. Alpha represents the probability of committing a Type I error, accidentally rejecting a null hypothesis when it should not have been. When conducting an A/B testing with a t-test or chi-square test, the observed p-value is compared to the predetermined alpha level to determine if there is a statistically significant difference between the variations. Unlike alpha, the power of a statistical test is defined as the probability of correctly rejecting the null hypothesis when it is indeed false. A higher power indicates a greater ability to detect true effects, while a lower power increases the risk of Type II errors, failing to reject a false null hypothesis. Another crucial parameter of the test is effect size, which provides insights into the practical significance of differences observed between the two groups. Effect size also helps conducting the power analysis to determine the sample size that a study requires to achieve a certain probability of achieving a true effect.</p><h4>How to Interpret the Results of an A/B Test?</h4><p>Interpreting A/B testing results with a p-value involves understanding the statistical significance of the observed difference between two groups. P-value represents the probability of obtaining the test statistic at least as large as the one observed, by random chance, assuming that the null hypothesis is true. A low p-value (typically less than 0.05 threshold) suggests that the observed difference is unlikely to be due to random chance alone. In this case, researchers can reject the null hypothesis, concluding that there is a significant difference between the groups. Conversely, a high p-value suggests that the observed difference is likely due to random chance, and there is insufficient evidence to reject the null hypothesis.</p><p>Some A/B testing software programs may also report two conversion rates: one for the control group and the other for the experimental group. For instance, a 16% (+/- 2.3%) experimental group’s conversion rate meant that 16% of the experimental group took the desired action with a margin of error of 2.3%. This can be interpreted as if the A/B tests are run multiple times, 95% of the ranges will capture the true conversion rate. If the control group has a significantly lower conversion rate, it is likely a good decision for researchers to adapt the new version used with experimental group. However, such decisions also depend on other factors such as the costs of implementing the new version.</p><h4>What’s the Role of A/B Testing Today?</h4><p>With the growth of data in today’s business world, A/B testing plays a pivotal role in modern marketing and data-driven decision making for businesses. By conducting A/B tests, companies can experiment with different versions of their marketing strategies, advertisements, website designs, and product offerings to determine which resonates best with their target audience. This approach provides valuable insights into customer preferences and the effectiveness of various marketing strategies. Through rigorous analysis of A/B testing results, businesses can optimize their campaigns by allocating resources more efficiently, ultimately driving higher conversion rates, increased customers satisfaction, and improve ROI.</p><h4>Data Ethics</h4><p>When using A/B testing as a method to optimize websites, ad campaigns and more, researchers may face potentially dangerous situations when it comes to ethics. Often the purpose for A/B testing is to maximize revenue or optimize some other metric, but it’s also important to consider user impact when building A/B tests. Traditionally when researchers experimentally manipulate conditions to understand their impacts on people, they must follow ethical standards set by their institutional review boards to ensure that ethical practices are followed.</p><p>The ethical standard originally came from the Belmont Report of 1976, a notoriously unethical study that recruited African American men with syphilis and withheld modern treatment to study the effects of the untreated disease. This report lists three practices that are necessary for ethical human subject research including informed consent, assessing of risks and benefits, and ethical selection of subjects. To meet informed consent standard, research subjects must be provided with information about the research procedure and must comprehend the information that they are given. Moreover, their participation in the research must be voluntary. The second ethical standard states that there must be an assessment of risks and benefits. The research should not only anticipate benefits to society in the form of knowledge to be gained from the research but also risks including physical harm, psychological harm, legal harm, social harm, and economic harm. Finally, the last standard implies that subjects must be ethically selected on a social and individual basis. Researchers must consider distributive justice in selecting subjects due to social, racial, sexual and cultural biases. Vulnerable subjects should receive extra protection due to their illness or economic conditions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=51e4733da2a6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unlocking Efficiency: Automating Excel Tasks with Python]]></title>
            <link>https://medium.com/@taingocbui_39735/unlocking-efficiency-automating-excel-tasks-with-python-37d2f899dc61?source=rss-e57460c7940c------2</link>
            <guid isPermaLink="false">https://medium.com/p/37d2f899dc61</guid>
            <dc:creator><![CDATA[Taingocbui]]></dc:creator>
            <pubDate>Thu, 01 Feb 2024 00:18:08 GMT</pubDate>
            <atom:updated>2024-02-01T00:54:03.212Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2KQlw1d2xKc64bGMdSn5LA.jpeg" /></figure><p>For several decades, Microsoft Excel has played a significant role in data analysis and business processes across organizations. The software provides a structured platform for organizing and storing data around the world. Moreover, it also allows users to create visualization of data through charts and graphs, aiding the interpretation and communication of insights. However, with the emergence of big data, many functionalities and capabilities that Microsoft Excel offers have become obsolete compared to new data processing technologies. One of the biggest limitations that Excel has is it lacks automated features in handling complex data cleaning and data transformation. Analysts often spend considerable time on manual cleaning tasks. Those days when hours wasted just to clean data have gone! I hope this small article will help you grasp the power of Python and its amazing libraries for Excel automation, which will take your productivity to new heights.</p><h3><strong>The Pandas Library</strong></h3><p>Python has an extensive ecosystem of libraries which allow users to load, clean, and transform data with little effort. The first library which all Excel users should fall in love with is Pandas, a versatile and user-friendly library in Python. Pandas has transformed the way data is handled and manipulated. At the core of Pandas lies the DataFrame, a two-dimensional data structure that simplifies handling tabular data. One of Pandas strength is its ability to handle inconsistent data effectively. For instance, it is very difficult to validate text data and filter based on a set of conditions using Microsoft Excel. However, Pandas let users set up conditions for pattern searching, filter and group them effortlessly.</p><p>Another use case in which Python helps data analysts simplify a very complicated task is to split row data into multiple rows based on certain pre-set conditions. One of the challenges I have when working in the hospitality industry is to transform million rows of reservation data with only check-in and check-out date into aggregated data by stay-in-date. For such task, it is almost impossible to handle using Microsoft Excel . However, Pandas can handle such problem with ease. The following code sample allows users to split each row of reservations into a list of dictionaries.</p><pre>def split_row(row):<br>  l = list()<br>  for i in range(row[&#39;Occupied_Nights&#39;]):<br>    d = dict()<br>    #Each field of row  will be treated as keys of dictionary<br>    d[&#39;Group Name&#39;] = row[&#39;Group Name&#39;]<br>    d[&#39;Check-in date&#39;] = row[&#39;Check-in date&#39;]<br>    d[&#39;Check-out date&#39;] = row[&#39;Check-out date&#39;]<br>    d[&#39;Date&#39;] = row[&#39;Check-in date&#39;] + timedelta(days = i)<br>    l.append(d)<br>  return l<br><br>l_result = list()<br>for index, row in df.iterrows():<br>  l.extend(split_row(row))<br>df_result = pd.DataFrame.from_dict(l_result)<br>df_result.to_excel(&quot;Example1.xlsx&quot;, index = False)</pre><p>Each dictionary will store data for a single night of reservation with the average rate for that night. After the iteration through all reservations, the list of dictionaries will be combined into a different complete DataFrame. At the end of the process, newly split data will be exported to excel using to_excel function. In fact, Pandas has many other great functionalities to either replace Microsoft Excel’s capabilities or enhance its efficiency.</p><h3><strong>The COM Module</strong></h3><p>The second Python’s library I want to mention in this article is win32com.client. This is a powerful module in Python to streamline Excel repetitive tasks. The win32com package is a set of Python source files that use the PythonCOM module. The win32com.client is a subpackage, which helps Python programs to establish an interface with Microsoft Excel and manipulate the VB Macros. Note that users will need to install the “pywin32” library via pip or Anaconda. Below is a brief instruction on how to create a new workbook, add data and save the file.</p><pre>import win32com.client<br>#create a com object<br>excel = win32com.client.gencache.EnsureDispatch(&#39;Excel.Application&#39;)<br>excel.visible = True<br>#create a new workbook<br>workbook = excel.Workbooks.Add()<br>#get active sheet<br>sheet = workbook.ActiveSheet<br><br>#Add example data to the Excel sheet<br><br>data = [[&#39;Customer&#39;, &#39;RoomCategories&#39;, &#39;RoomRevenue&#39;],<br>        [&#39;A&#39;,&#39;Ocean Front&#39;, &#39;2750&#39;],<br>        [&#39;B&#39;, &#39;Garden View&#39;, &#39;1550&#39;],<br>       ]<br>for row_num, row_data in enumerate(data, start = 1):<br>  for col_num, cell_val in enumerate(row_data, start = 1):<br>    sheet.Cells(row_num, col_num).Value = cell_val<br><br>workbook.SaveAs(&#39;Example2.xlsx&#39;)<br>workbook.close()</pre><p>Besides simple tasks as mentioned above, the module win32.client also allows users to create and manipulate Excel pivot table. Some may wonder why we don’t use pandas’ pivot table instead of the Excel one. In fact, while users can conveniently create a pivot table with very few codes using pandas, the hard-coded pivot table created may not be a good tool for stakeholders as they cannot modify the Pivot Table Fields themselves. The com object, created with the module win32, allows users to create whole new pivot table with provided data source, insert certain VBA codes into the pivot table to refresh the data, adjust the filter or even make changes to the calculated fields. This pivot table automation not only streamlines workflows, but also reduces the likelihood of errors, ensures data accuracy and consistency. Below is a brief instruction on how to create a pivot table, and insert input data to its calculated fields.</p><pre>#create a pivot table<br>pvt_range = sheet.UsedRange<br>pvt_table_range = sheet.Cells(1,5)<br>pvt_table = workbook.PivotTableWizard(<br>  source_range = pvt_range,<br>  table_destination = pvt_table_range,<br>  table_name = &quot;Pivot Table Example&quot;<br>)<br><br>#Add fields to pivot table<br>pvt_table.PivotFields(&#39;Customer&#39;).Orientation = win32com.client.constants.xlRowField<br>pvt_table.PivotFields(&#39;RoomCategories&#39;).Orientation = win32com.client.constants.xlColumnField<br>pvt_table.PivotFields(&#39;RoomRevenue&#39;).Orientation = win32com.client.constants.xlDataField<br>pvt_table.PivotFields(&#39;RoomRevenue&#39;).Function = win32.client.constants.xlSum<br>pvt_table.PivotFields(&#39;RoomRevenue&#39;).NumberFormat = &quot;#,##0&quot;<br><br>#Add calculated field<br>calculated_field = pvt_table.CalculatedFields().Add(&quot;ADR&quot;,&quot;=RoomRevenue/OccupiedNights&quot;)<br>calculated_field.Orientation = win32com.client.constants.xlDataField<br>calculated_field.NumberFormat = &quot;#,##0.##&quot;<br><br># Autoformat the pivot table<br>pvt_table.ShowTableStyleRowStripes = True<br>pvt_table.TableStyle2 = &quot;PivotStyleLight16&quot;<br><br>workbook.SaveAs(&quot;Example_pivot_table.xlsx&quot;)<br><br># Close Excel<br>excel_app.Quit()</pre><h3>Conclusion</h3><p>In conclusion, the ability to programmatically interact with Excel via Python‘s libraries offers Excel users a transformative and efficient approach to handling data and automating tasks. Pandas and win32 empower users to efficiently handle large datasets, perform effective data analyses, and produce visually compelling presentations. This integration not only enhances the functionality of Excel but also opens a world of possibilities, enabling users to unlock the full potential of Excel for data management, analysis and reporting.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=37d2f899dc61" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>