<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Emmanuel Olaoluwa O. on Medium]]></title>
        <description><![CDATA[Stories by Emmanuel Olaoluwa O. on Medium]]></description>
        <link>https://medium.com/@emmanueloladele994?source=rss-5c8982be20d7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*4jacS_umnhZPFFzsZVkTJg.jpeg</url>
            <title>Stories by Emmanuel Olaoluwa O. on Medium</title>
            <link>https://medium.com/@emmanueloladele994?source=rss-5c8982be20d7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 10:28:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@emmanueloladele994/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How I Built My First Machine Learning Model: A Supervised Learning Capstone]]></title>
            <link>https://medium.com/@emmanueloladele994/how-i-built-my-first-machine-learning-model-a-supervised-learning-capstone-2e6be3aa4ba3?source=rss-5c8982be20d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/2e6be3aa4ba3</guid>
            <dc:creator><![CDATA[Emmanuel Olaoluwa O.]]></dc:creator>
            <pubDate>Sun, 01 Sep 2024 18:52:22 GMT</pubDate>
            <atom:updated>2024-09-01T20:30:39.534Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Project by Emmanuel Olaoluwa O" src="https://cdn-images-1.medium.com/max/1024/0*BD899w_GDzK4GAB5" /><figcaption>Photo by <a href="https://unsplash.com/@growtika?utm_source=medium&amp;utm_medium=referral">Growtika</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>Machine Learning (ML)</strong> is a branch of artificial intelligence (AI) that focuses on building systems that can learn from and make decisions based on data. Instead of being explicitly programmed to perform a task, a machine learning model learns patterns and relationships from data to make predictions or decisions.</p><h3>Types of Machine Learning</h3><ol><li><strong>Supervised Learning</strong>: In supervised learning, the model is trained on a labeled dataset, meaning that each training example has a corresponding output or label. The goal is for the model to learn a mapping from inputs to outputs so that it can accurately predict the output for new, unseen data. <strong><em>Examples</em></strong><em>: Classification (e.g., spam detection, image recognition), Regression (e.g., predicting house prices, stock prices).</em></li></ol><p><strong>2. Unsupervised Learning</strong>: In unsupervised learning, the model is trained on an unlabeled dataset, meaning there are no predefined outputs. The model’s goal is to find hidden patterns, structures, or relationships within the data. <strong><em>Examples</em></strong><em>: Clustering (e.g., customer segmentation, grouping similar items), Association (e.g., market basket analysis)</em></p><p>3. <strong>Reinforcement Learning</strong>: Reinforcement learning involves training a model to make a sequence of decisions by interacting with an environment to achieve a specific goal. The model learns from feedback in the form of rewards or penalties to improve its strategy over time. <strong><em>Examples</em></strong><em>: Robotics, game playing (e.g., AlphaGo), self-driving cars.</em></p><p>Today, I will focus solely on <strong>Supervised Learning</strong>. This approach is widely used for tasks where labeled data is available, and the goal is to predict outcomes based on input features. I’ll demonstrate how to build and evaluate a supervised learning model step-by-step.</p><h3>GOAL: Create a model to predict whether or not a customer will Churn.</h3><p><strong>Churn</strong> refers to the rate at which customers stop using a company’s product or service over a given period. In a business context, predicting customer churn is crucial because retaining existing customers is often less expensive and more profitable than acquiring new ones. By understanding and predicting which customers are likely to churn, businesses can take proactive steps to improve customer retention.</p><h3>Objectives of the Churn Analysis</h3><ol><li><strong>Segmenting Customers by Tenure</strong>:<br>The first step in our analysis is to segment customers based on their <strong>tenure</strong> — the number of months they have been with the company. We will create “cohorts,” or groups, of customers based on their tenure, allowing us to examine differences in behavior and churn rates across these cohorts.</li><li><strong>Exploring Contract Types</strong>:<br>We want to identify the different <strong>contract types</strong> available to customers (e.g., Month-to-Month, One-Year, Two-Year) to understand how these contracts may influence customer retention. Customers on longer contracts may be less likely to churn compared to those on shorter, more flexible contracts.</li><li><strong>Visualizing Tenure Distribution</strong>:<br>We’ll create a histogram to visualize the distribution of customer tenure. This helps us understand how long customers typically stay with the company and if there are any patterns, such as a high concentration of customers within certain tenure ranges.</li><li><strong>Analyzing Churn by Contract Type and Tenure</strong>:<br>By separating the histogram by <strong>Churn</strong> (whether a customer has left the service) and <strong>Contract Type</strong>, we can explore how these factors interact with tenure. This helps us identify trends, such as whether customers with shorter contracts are more likely to churn early.</li><li><strong>Scatter Plot of Charges</strong>:<br>We will create a scatter plot to visualize the relationship between <strong>Total Charges</strong> and <strong>Monthly Charges</strong> and differentiate by <strong>Churn</strong> status. This will help us see if there are any trends, such as whether customers with higher monthly charges are more likely to churn.</li><li><strong>Creating Cohorts and Calculating Churn Rates</strong>:<br>We will treat each unique tenure length as its own cohort (e.g., all customers who have been with the company for 1 month, 2 months, etc.). For each cohort, we’ll calculate the <strong>churn rate</strong> — the percentage of customers who have left the service. This will help us understand how the churn rate varies with tenure. We expect to see a general trend where the churn rate decreases as tenure increases, as customers who have been with the service longer are less likely to leave.</li><li><strong>Visualizing Churn Rate by Tenure</strong>:<br>Finally, we’ll create a plot to show the churn rate for each tenure group (from 1 to 72 months). This visualization will help illustrate the overall trend and provide insights into which customer segments are most at risk of churning.</li></ol><h3>Exploring Supervised Learning</h3><p>Throughout this analysis, we are focusing on <strong>supervised learning</strong> techniques, where we have labeled data indicating whether each customer has churned. The insights gained from this exploratory analysis can later be used to train a supervised machine learning model (such as logistic regression, decision trees, or random forests) to predict the likelihood of churn for new customers based on their features.</p><h3>Part 0: Imports and Read in the Data</h3><h3><strong>TASK: Import libraries and read in your data</strong></h3><p>import numpy as np<br>import pandas as pd<br>import matplotlib.pyplot as plt<br>import seaborn as sns</p><h3>Load the data</h3><p><em>df = pd.read_csv(‘../DATA/Telco-Customer-Churn.csv’)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LXh_o5rXHAVYXQ8pjLRMDw.jpeg" /><figcaption>df.head()</figcaption></figure><h3>Part 1: Quick Data Check</h3><p><strong>TASK: Confirm quickly with .info() methods the datatypes and non-null values in your dataframe.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/673/1*bUIlWapPfwLEYyDApRv-FQ.jpeg" /><figcaption>df.info()</figcaption></figure><p><strong>TASK: Get a quick statistical summary of the numeric columns with .describe() , you should notice that many columns are categorical, meaning you will eventually need to convert them to dummy variables.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*MDAymrHTpfRVzYtcBDl9Mw.jpeg" /><figcaption>df.describe()</figcaption></figure><h3>Part 2: Exploratory Data Analysis</h3><h3>General Feature Exploration</h3><p><strong>TASK: Confirm that there are no NaN cells by displaying NaN values per feature column.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*H_SLzLGN5xdgI2QHiZA-mA.jpeg" /><figcaption>df.isna().sum()</figcaption></figure><p><strong>TASK: Display the balance of the class labels (Churn) with a Count Plot.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/710/1*lFTrrr6eXj1s2X9BLThlAw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*5Wo7wcENu1U-2fD7GzJ-HQ.jpeg" /><figcaption>sns.countplot(data=df,x=’Churn’)</figcaption></figure><p><strong>TASK: Explore the distribution of Total Charges between Churn categories with a Box Plot or Violin Plot.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/1*YSgOzbwBhag9baQNB2MFiw.jpeg" /><figcaption>sns.violinplot(data=df,x=’Churn’,y=’TotalCharges’)</figcaption></figure><p><strong>TASK: Create a boxplot showing the distribution of TotalCharges per Contract type, also add in a hue coloring based on the Churn class.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nKaz5Q_MeD16BpfeZXcxLw.jpeg" /><figcaption>plt.figure(figsize=(10,4),dpi=200)<br>sns.boxplot(data=df,y=’TotalCharges’,x=’Contract’,hue=’Churn’)<br>plt.legend(loc=(1.1,0.5))</figcaption></figure><p><strong>TASK: Create a bar plot showing the correlation of the following features to the class label. Keep in mind, for the categorical features, you will need to convert them into dummy variables first, as you can only calculate correlation for numeric features.</strong></p><pre>[&#39;gender&#39;, &#39;SeniorCitizen&#39;, &#39;Partner&#39;, &#39;Dependents&#39;,&#39;PhoneService&#39;, &#39;MultipleLines&#39;, <br> &#39;OnlineSecurity&#39;, &#39;OnlineBackup&#39;, &#39;DeviceProtection&#39;, &#39;TechSupport&#39;, &#39;InternetService&#39;,<br>   &#39;StreamingTV&#39;, &#39;StreamingMovies&#39;, &#39;Contract&#39;, &#39;PaperlessBilling&#39;, &#39;PaymentMethod&#39;]</pre><p><strong><em>Note, I specifically listed only the features above, you should not check the correlation for every feature, as some features have too many unique instances for such an analysis, such as customerID</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DHvSQ4w80U_uc9No3-oE-w.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/662/1*mrZzauPQajvWjW958BdQ_A.jpeg" /><figcaption>corr_df[‘Churn_Yes’].sort_values().iloc[1:-1]</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WoqBW7BWtYrnq6AmqIdCSQ.png" /><figcaption>plt.figure(figsize=(10,4),dpi=200)<br>sns.barplot(x=corr_df[‘Churn_Yes’].sort_values().iloc[1:-1].index,y=corr_df[‘Churn_Yes’].sort_values().iloc[1:-1].values)<br>plt.title(“Feature Correlation to Yes Churn”)<br>plt.xticks(rotation=90);</figcaption></figure><h3>Part 3: Churn Analysis</h3><p><strong>This section focuses on segmenting customers based on their tenure, creating “cohorts”, allowing us to examine differences between customer cohort segments.</strong></p><p><strong>TASK: What are the 3 contract types available?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/881/1*OLEhSVF7OUYA2pStrS_lmg.jpeg" /></figure><p><strong>TASK: Create a histogram displaying the distribution of ‘tenure’ column, which is the amount of months a customer was or has been on a customer.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tc2k5CXtyyeH92O14FjfUw.jpeg" /><figcaption>plt.figure(figsize=(10,4),dpi=200)<br>sns.histplot(data=df,x=’tenure’,bins=60)</figcaption></figure><p><strong>TASK: Now use the seaborn documentation as a guide to create histograms separated by two additional features, Churn and Contract.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KVIiRawXivI_cUDcTF-YsA.png" /><figcaption>plt.figure(figsize=(10,3),dpi=200)<br>sns.displot(data=df,x=’tenure’,bins=70,col=’Contract’,row=’Churn’);</figcaption></figure><p><strong>TASK: Display a scatter plot of Total Charges versus Monthly Charges, and color hue by Churn.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*47TgUl9UCcQCusEVy2uh7Q.jpeg" /><figcaption>plt.figure(figsize=(10,4),dpi=200)<br>sns.scatterplot(data=df,x=’MonthlyCharges’,y=’TotalCharges’,hue=’Churn’, linewidth=0.5,alpha=0.5,palette=’Dark2&#39;)</figcaption></figure><h3>Creating Cohorts based on Tenure</h3><p><strong>Let’s begin by treating each unique tenure length, 1 month, 2 months, 3 months…N months as its own cohort.</strong></p><p><strong>TASK: Treating each unique tenure group as a cohort, calculate the Churn rate (percentage that had Yes Churn) per cohort. For example, the cohort that has had a tenure of 1 month should have a Churn rate of 61.99%. You should have cohorts 1–72 months with a general trend of the longer the tenure of the cohort, the less of a churn rate. This makes sense as you are less likely to stop service the longer you’ve had it.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/986/1*Uhtj6Jru2BS9r0KBo5yplg.jpeg" /></figure><p><strong>TASK: Now that you have Churn Rate per tenure group 1–72 months, create a plot showing churn rate per months of tenure.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h1JIj-5LJuGxhOYO6SdHoA.jpeg" /><figcaption>plt.figure(figsize=(10,4),dpi=200)<br>churn_rate.iloc[0].plot()<br>plt.ylabel(‘Churn Percentage’);</figcaption></figure><h3>Broader Cohort Groups</h3><p><strong>TASK: Based on the tenure column values, create a new column called Tenure Cohort that creates 4 separate categories:</strong></p><blockquote>‘0–12 Months’</blockquote><blockquote>‘24–48 Months’</blockquote><blockquote>‘12–24 Months’</blockquote><blockquote>‘Over 48 Months’</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*kqjQ3XRl_ZgI8cUzcjG0wg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/355/1*RYbtyVdP6w13fUQ74hMTsQ.jpeg" /><figcaption>df.head(10)[[‘tenure’,’Tenure Cohort’]]</figcaption></figure><p><strong>TASK: Create a scatterplot of Total Charges versus Monthly Charts, colored by Tenure Cohort defined in the previous task.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QHj7ho0qxoaIXs_yLmvn9Q.jpeg" /><figcaption>plt.figure(figsize=(10,4),dpi=200)<br>sns.scatterplot(data=df,x=’MonthlyCharges’,y=’TotalCharges’,hue=’Tenure Cohort’, linewidth=0.5,alpha=0.5,palette=’Dark2&#39;)</figcaption></figure><p><strong>TASK: Create a count plot showing the churn count per cohort.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hqiYQYoE3Z0-lyTjfsB9CA.jpeg" /><figcaption>plt.figure(figsize=(10,4),dpi=200)<br>sns.countplot(data=df,x=’Tenure Cohort’,hue=’Churn’)</figcaption></figure><p><strong>TASK: Create a grid of Count Plots showing counts per Tenure Cohort, separated out by contract type and colored by the Churn hue.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TgEwtk0fgMaarbm_jmY-qw.jpeg" /><figcaption>plt.figure(figsize=(10,4),dpi=200)<br>sns.catplot(data=df,x=’Tenure Cohort’,hue=’Churn’,col=’Contract’,kind=’count’)</figcaption></figure><h3>Part 4: Predictive Modeling</h3><p><strong>Let’s explore 4 different tree based methods: A Single Decision Tree, Random Forest, AdaBoost, Gradient Boosting. Feel free to add any other supervised learning models to your comparisons!</strong></p><h3>Single Decision Tree</h3><p><strong>TASK: Separate out the data into X features and Y label. Create dummy variables where necessary and note which features are not useful and should be dropped.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*mg8GWDdZmot1AyJrbRP41Q.jpeg" /></figure><p><strong>TASK: Perform a train test split, holding out 10% of the data for testing. We’ll use a random_state 101</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JPMW4hMTZ_JWjuBL1Wzdbw.jpeg" /></figure><p><strong>TASK: Decision Tree Performance. Complete the following tasks:</strong></p><ol><li>Train a single decision tree model (feel free to grid search for optimal hyperparameters).</li><li>Evaluate performance metrics from decision tree, including classification report and plotting a confusion matrix.</li><li>Calculate feature importances from the decision tree.</li><li>OPTIONAL: Plot your tree, note, the tree could be huge depending on your pruning, so it may crash your notebook if you display it with plot tree.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3MHMtLc-rp6zrhLvcuoG_Q.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*vRU32TYL9Kh_xqLHTY46TA.jpeg" /><figcaption>print(classification_report(y_test,preds))</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/621/1*cYtW5oQUGuUSX4rI9Q2n3w.jpeg" /><figcaption>plot_confusion_matrix(dt,X_test,y_test)</figcaption></figure><p><em>imp_feats = pd.DataFrame(data=dt.feature_importances_,index=X.columns,columns=[‘Feature Importance’]).sort_values(“Feature Importance”)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D_fFZpX93ZzS1uJex_qjfg.png" /><figcaption>plt.figure(figsize=(14,6),dpi=200)<br>sns.barplot(data=imp_feats.sort_values(‘Feature Importance’),x=imp_feats.sort_values(‘Feature Importance’).index,y=’Feature Importance’)<br>plt.xticks(rotation=90)<br>plt.title(“Feature Importance for Decision Tree”);</figcaption></figure><p><em>from sklearn.tree import plot_tree</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A8YN7IREEm0FvzibnsVg_w.png" /><figcaption>plt.figure(figsize=(12,8),dpi=150)<br>plot_tree(dt,filled=True,feature_names=X.columns);</figcaption></figure><h3>Random Forest</h3><p><strong>TASK: Create a Random Forest model and create a classification report and confusion matrix from its predicted results on the test set.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*tjUvzu6uBGROk3kaJZ4_pA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*9Ub1N_r4iijRWv3E5MyxGw.jpeg" /><figcaption>print(classification_report(y_test,preds))</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/530/1*0_ozOhXYhz7XSPKzp-_uow.jpeg" /><figcaption>plot_confusion_matrix(dt,X_test,y_test)</figcaption></figure><h3>Boosted Trees</h3><p><strong>TASK: Use AdaBoost or Gradient Boosting to create a model and report back the classification report and plot a confusion matrix for its predicted results</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/958/1*n_ICfwtpqcLNFp6Dog_FIg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/696/1*M55s6lc58GJTHoWU-UkdvQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/653/1*RT7WI-y2YCKWcn7qsPhPkg.jpeg" /><figcaption>plot_confusion_matrix(dt,X_test,y_test)</figcaption></figure><p><strong>TASK: Analyze your results, which model performed best for you?</strong></p><p>With base models, we got best performance from an AdaBoostClassifier, but note, we didn’t do any grid searching AND most models performed about the same on the data set.</p><p><em>Thank you.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2e6be3aa4ba3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My First Python Data Analysis.]]></title>
            <link>https://medium.com/@emmanueloladele994/my-first-python-data-analysis-d617c0ecca02?source=rss-5c8982be20d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/d617c0ecca02</guid>
            <dc:creator><![CDATA[Emmanuel Olaoluwa O.]]></dc:creator>
            <pubDate>Sun, 01 Sep 2024 13:43:41 GMT</pubDate>
            <atom:updated>2024-09-01T20:55:06.147Z</atom:updated>
            <content:encoded><![CDATA[<h3>My First Python Data Analysis on Movie Trends-Reel Insights</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7NXeAAN6gYmjqfaj" /><figcaption>Photo by Jakob Owens on Unsplash</figcaption></figure><h3>Overview</h3><p>If you are planning on going out to see a movie, how well can you trust online reviews and ratings? <em>Especially</em> if the same company showing the rating <em>also</em> makes money by selling movie tickets. Do they have a bias towards rating movies higher than they should be rated?</p><h3>Goal:</h3><p><strong>Your goal is to complete the tasks below based off the 538 article and see if you reach a similar conclusion. You will need to use your pandas and visualization skills to determine if Fandango’s ratings in 2015 had a bias towards rating movies better to sell more tickets.</strong></p><h3>Part One: Understanding the Background and Data <strong>TASK: Read this article: </strong><a href="http://fivethirtyeight.com/features/fandango-movies-ratings/"><strong>Be Suspicious Of Online Movie Ratings, Especially Fandango’s</strong></a></h3><p><strong>TASK: After reading the article, read these two tables giving an overview of the two .csv files we will be working with:</strong></p><h3>The Data</h3><p>This is the data behind the story <a href="http://fivethirtyeight.com/features/fandango-movies-ratings/">Be Suspicious Of Online Movie Ratings, Especially Fandango’s</a> openly available on 538’s github: <a href="https://github.com/fivethirtyeight/data">https://github.com/fivethirtyeight/data</a>. There are two csv files, one with Fandango Stars and Displayed Ratings, and the other with aggregate data for movie ratings from other sites, like Metacritic, IMDB, and Rotten Tomatoes.</p><h4>all_sites_scores.csv</h4><p>all_sites_scores.csv contains every film that has a Rotten Tomatoes rating, a RT User rating, a Metacritic score, a Metacritic User score, and IMDb score, and at least 30 fan reviews on Fandango. The data from Fandango was pulled on Aug. 24, 2015.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qmwfJxyCkqYZ47OW_MzHLw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yO9iJD5LLcxp42SbHjbo4Q.jpeg" /></figure><p><strong>TASK: Import any libraries you think you will use:</strong></p><p>import numpy as np<br>import pandas as pd<br>import matplotlib.pyplot as plt<br>import seaborn as sns</p><h3>Part Two: Exploring Fandango Displayed Scores versus True User Ratings</h3><p>Let’s first explore the Fandango ratings to see if our analysis agrees with the article’s conclusion.</p><p><strong>TASK: Run the cell below to read in the fandango_scrape.csv file</strong></p><p>fandango = pd.read_csv(“fandango_scrape.csv”)</p><p><strong>TASK: Explore the DataFrame Properties and Head.</strong></p><figure><img alt="fandango.head()" src="https://cdn-images-1.medium.com/max/607/1*NTNwNYGxpDpa9AseXQX2Mw.jpeg" /><figcaption>fandango.head()</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/1*2E79uEnFxAHtfLzv4Ns0EQ.jpeg" /><figcaption><strong>fandango.info()</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/1*PixDxd-qnSm9Hm2Shex06g.jpeg" /><figcaption>fandango.describe().T</figcaption></figure><p><strong>TASK: Let’s explore the relationship between popularity of a film and its rating. Create a scatterplot showing the relationship between rating and votes.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hqvnrcFA7axmYeIKSBjy-g.jpeg" /><figcaption><strong>plt.figure(figsize=(10,4),dpi=150)<br>sns.scatterplot(data=fandango,x=’RATING’,y=’VOTES’);</strong></figcaption></figure><p><strong>TASK: Calculate the correlation between the columns:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/370/1*X00e6PxksCuWvpbT6xWFPw.jpeg" /><figcaption><strong>numerical_fandango=fandango.select_dtypes(exclude=[‘object’])<br>correlation_matrix=numerical_fandango<br>correlation_matrix.head()</strong></figcaption></figure><p><strong>TASK: Assuming that every row in the FILM title column has the same format: Film Title Name ( Year)</strong></p><p><strong>Create a new column that is able to strip the year from the title strings and set this new column as YEAR</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/830/1*QRr11b8DQ8vQaRlQAmthPQ.jpeg" /><figcaption><strong><em>fandango[‘YEAR’] = fandango[‘FILM’].apply(lambda title:title.split(‘(‘)[-1])</em></strong></figcaption></figure><p><strong>TASK: How many movies are in the Fandango DataFrame per year?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/368/1*Ep6UwI0nxwNwwor31FO4VQ.jpeg" /><figcaption><strong>fandango[‘YEAR’].value_counts()</strong></figcaption></figure><p><strong>TASK: Visualize the count of movies per year with a plot:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/899/1*EWH54kfgRBUCVyNJNb21Mw.jpeg" /><figcaption><strong>sns.countplot(data=fandango,x=’YEAR’)</strong></figcaption></figure><p><strong>TASK: What are the 10 movies with the highest number of votes?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/851/1*wOHlQ6YXqfccW0kUqOr7Lw.jpeg" /><figcaption><strong>fandango.nlargest(10,’VOTES’)</strong></figcaption></figure><p><strong>TASK: Create DataFrame of only reviewed films by removing any films that have zero votes.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/1*I0cDB29nx9JS3vTMRLpIAA.jpeg" /><figcaption><strong>fan_reviewed = fandango[fandango[‘VOTES’]&gt;0]<br>fan_reviewed</strong></figcaption></figure><p><strong>As noted in the article, due to HTML and star rating displays, the true user rating may be slightly different than the rating shown to a user. Let’s visualize this difference in distributions.</strong></p><p><strong>TASK: Create a KDE plot (or multiple kdeplots) that displays the distribution of ratings that are displayed (STARS) versus what the true rating was from votes (RATING). Clip the KDEs to 0–5.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0oFqFE2Wm_XrhnUJOLhgdg.jpeg" /><figcaption><strong>plt.figure(figsize=(10,4),dpi=150)<br>sns.kdeplot(data=fan_reviewed,x=’RATING’,clip=[0,5],fill=True,label=’True Rating’)<br>sns.kdeplot(data=fan_reviewed,x=’STARS’,clip=[0,5],fill=True,label=’Stars Displayed’) plt.legend(loc=(1.05,0.5))</strong></figcaption></figure><p><strong>TASK: Create a count plot to display the number of times a certain difference occurs:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5387To7Dgb999j0z3pjUdA.jpeg" /><figcaption>plt.figure(figsize=(12,4),dpi=150)<br>sns.countplot(data=fan_reviewed,x=’STARS_DIFF’,palette=’magma’)</figcaption></figure><h3>Part Three: Comparison of Fandango Ratings to Other Sites</h3><p>Let’s now compare the scores from Fandango to other movies sites and see how they compare.</p><p><strong>TASK: Read in the “all_sites_scores.csv” file by running the cell below</strong></p><p><em>all_sites = pd.read_csv(“all_sites_scores.csv”)</em></p><p><strong>TASK: Explore the DataFrame columns, info, description.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2_M1IcflD19ExUUsM6b6ew.jpeg" /><figcaption>all_sites.head()</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qRZXDUnpB237wc5CEUrr4A.jpeg" /><figcaption>all_sites.describe()</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/1*Zz6YEYpKwL-Qrh-J1e13KA.jpeg" /><figcaption>all_sites.info()</figcaption></figure><h3>Rotten Tomatoes</h3><p>Let’s first take a look at Rotten Tomatoes. RT has two sets of reviews, their critics reviews (ratings published by official critics) and user reviews.</p><p><strong>TASK: Create a scatterplot exploring the relationship between RT Critic reviews and RT User reviews.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JoOxbeuEtJJIw0jWOCDnuQ.jpeg" /><figcaption><strong>plt.figure(figsize=(10,4),dpi=150)<br>sns.scatterplot(data=all_sites,x=’RottenTomatoes’,y=’RottenTomatoes_User’)<br>plt.xlim(0,100)<br>plt.ylim(0,100)</strong></figcaption></figure><p>Let’s quantify this difference by comparing the critics ratings and the RT User ratings. We will calculate this with RottenTomatoes-RottenTomatoes_User. Note: Rotten_Diff here is Critics — User Score. So values closer to 0 means aggrement between Critics and Users. Larger positive values means critics rated much higher than users. Larger negative values means users rated much higher than critics.</p><p><strong>TASK: Create a new column based off the difference between critics ratings and users ratings for Rotten Tomatoes. Calculate this with RottenTomatoes-RottenTomatoes_User:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GTK6v0AuBHs1yP7hbaY0Ow.jpeg" /><figcaption><strong>all_sites[‘Rotten_Diff’] = all_sites[‘RottenTomatoes’] — all_sites[‘RottenTomatoes_User’]<br>all_sites</strong></figcaption></figure><p>Let’s now compare the overall mean difference. Since we’re dealing with differences that could be negative or positive, first take the absolute value of all the differences, then take the mean. This would report back on average to absolute difference between the critics rating versus the user rating.</p><p><strong>TASK: Calculate the Mean Absolute Difference between RT scores and RT User scores as described above.</strong></p><p>code: <em>all_sites[‘Rotten_Diff’].apply(abs).mean()</em>: <strong>15.095890410958905</strong></p><p><strong>TASK: Plot the distribution of the differences between RT Critics Score and RT User Score. There should be negative values in this distribution plot. Feel free to use KDE or Histograms to display this distribution.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lxIHIfDjVQi2_J0ToYD75g.jpeg" /><figcaption><strong>plt.figure(figsize=(10,4),dpi=200)<br>sns.histplot(data=all_sites,x=’Rotten_Diff’,kde=True,bins=25)<br>plt.title(“RT Critics Score minus RT User Score”);</strong></figcaption></figure><p><strong>TASK: Now create a distribution showing the <em>absolute value</em> difference between Critics and Users on Rotten Tomatoes.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A4Y1KZxm4Zmy9M81KVRMZw.jpeg" /><figcaption><strong>plt.figure(figsize=(10,4),dpi=200)<br>sns.histplot(x=all_sites[‘Rotten_Diff’].apply(abs),bins=25,kde=True)<br>plt.title(“Abs Difference between RT Critics Score and RT User Score”);</strong></figcaption></figure><p><strong>Let’s find out which movies are causing the largest differences. First, show the top 5 movies with the largest <em>negative</em> difference between Users and RT critics. Since we calculated the difference as Critics Rating — Users Rating, then large negative values imply the users rated the movie much higher on average than the critics did.</strong></p><p><strong>TASK: What are the top 5 movies users rated higher than critics on average:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/471/1*uuwiA4B-GFa0d7QYnzZ4ag.jpeg" /><figcaption><strong>print(“Users Love but Critics Hate”)<br>all_sites.nsmallest(5,’Rotten_Diff’)[[‘FILM’,’Rotten_Diff’]]</strong></figcaption></figure><h3>MetaCritic</h3><p>Now let’s take a quick look at the ratings from MetaCritic. Metacritic also shows an average user rating versus their official displayed rating.</p><p><strong>TASK: Display a scatterplot of the Metacritic Rating versus the Metacritic User rating.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sQhoNkZiuKLJQ0rBJbJinw.jpeg" /><figcaption><strong>plt.figure(figsize=(10,4),dpi=150)<br>sns.scatterplot(data=all_sites,x=’Metacritic’,y=’Metacritic_User’)<br>plt.xlim(0,100)<br>plt.ylim(0,10)</strong></figcaption></figure><h3>IMBD</h3><p>Finally let’s explore IMDB. Notice that both Metacritic and IMDB report back vote counts. Let’s analyze the most popular movies.</p><p><strong>TASK: Create a scatterplot for the relationship between vote counts on MetaCritic versus vote counts on IMDB.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xnsB1EBTMW7WgSNU7FTLEg.jpeg" /><figcaption>plt.figure(figsize=(10,4),dpi=150)<br>sns.scatterplot(data=all_sites,x=’Metacritic_user_vote_count’,y=’IMDB_user_vote_count’)</figcaption></figure><p><strong>Notice there are two outliers here. The movie with the highest vote count on IMDB only has about 500 Metacritic ratings. What is this movie?</strong></p><p><strong>TASK: What movie has the highest IMDB user vote count?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RKuvMnw_ZRAj-bd87Aig0w.jpeg" /></figure><p><strong>TASK: What movie has the highest Metacritic User Vote count?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T9AwMindBCGPch5Eft7kQg.jpeg" /></figure><h3>Fandago Scores vs. All Sites</h3><p>Finally let’s begin to explore whether or not Fandango artificially displays higher ratings than warranted to boost ticket sales.</p><p><strong>TASK: Combine the Fandango Table with the All Sites table. Not every movie in the Fandango table is in the All Sites table, since some Fandango movies have very little or no reviews. We only want to compare movies that are in both DataFrames, so do an <em>inner</em> merge to merge together both DataFrames based on the FILM columns.</strong></p><p><em>df = pd.merge(fandango,all_sites,on=’FILM’,how=’inner’)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/729/1*bFKKNFx6-Jt8NYc8YtBjsw.jpeg" /><figcaption>df.info()</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4CyApXiyf2C-9FLBD8Os2g.jpeg" /><figcaption>df.head()</figcaption></figure><h3>Normalize columns to Fandango STARS and RATINGS 0–5</h3><p>Notice that RT,Metacritic, and IMDB don’t use a score between 0–5 stars like Fandango does. In order to do a fair comparison, we need to <em>normalize</em> these values so they all fall between 0–5 stars and the relationship between reviews stays the same.</p><p><strong>TASK: Create new normalized columns for all ratings so they match up within the 0–5 star range shown on Fandango. There are many ways to do this.</strong></p><p>Hint link: <a href="https://stackoverflow.com/questions/26414913/normalize-columns-of-pandas-data-frame">https://stackoverflow.com/questions/26414913/normalize-columns-of-pandas-data-frame</a></p><p>Easier Hint:</p><p>Keep in mind, a simple way to convert ratings:</p><ul><li>100/20 = 5</li><li>10/2 = 5</li></ul><p><em>df[‘RT_Norm’] = np.round(df[‘RottenTomatoes’]/20,1)<br>df[‘RTU_Norm’] = np.round(df[‘RottenTomatoes_User’]/20,1)</em></p><p><em>df[‘Meta_Norm’] = np.round(df[‘Metacritic’]/20,1)<br>df[‘Meta_U_Norm’] = np.round(df[‘Metacritic_User’]/2,1)</em></p><p><em>df[‘IMDB_Norm’] = np.round(df[‘IMDB’]/2,1)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tKl1VOvyTa6sz8BMaLZuUQ.jpeg" /><figcaption>df.head()</figcaption></figure><p><strong>TASK: Now create a norm_scores DataFrame that only contains the normalizes ratings. Include both STARS and RATING from the original Fandango table.</strong></p><p><em>norm_scores = df[[‘STARS’,’RATING’,’RT_Norm’,’RTU_Norm’,’Meta_Norm’,’Meta_U_Norm’,’IMDB_Norm’]]</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/822/1*AC7oZ8kFQqUDMXJm9MagDw.jpeg" /><figcaption>norm_scores.head()</figcaption></figure><h3>Comparing Distribution of Scores Across Sites</h3><p>Now the moment of truth! Does Fandango display abnormally high ratings? We already know it pushs displayed RATING higher than STARS, but are the ratings themselves higher than average?</p><p><strong>TASK: Create a plot comparing the distributions of normalized ratings across all sites. There are many ways to do this, but explore the Seaborn KDEplot docs for some simple ways to quickly show this. Don’t worry if your plot format does not look exactly the same as ours, as long as the differences in distribution are clear.</strong></p><p>Quick Note if you have issues moving the legend for a seaborn kdeplot: <a href="https://github.com/mwaskom/seaborn/issues/2280">https://github.com/mwaskom/seaborn/issues/2280</a></p><p><em>def move_legend(ax, new_loc, **kws):<br> old_legend = ax.legend_<br> handles = old_legend.legendHandles<br> labels = [t.get_text() for t in old_legend.get_texts()]<br> title = old_legend.get_title().get_text()<br> ax.legend(handles, labels, loc=new_loc, title=title, **kws)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OF2XxHsk2yHJyeszCFCdDg.jpeg" /><figcaption><strong>fig, ax = plt.subplots(figsize=(15,6),dpi=150)<br>sns.kdeplot(data=norm_scores,clip=[0,5],shade=True,palette=’Set1&#39;,ax=ax)<br>move_legend(ax, “upper left”)</strong></figcaption></figure><p><strong>Clearly Fandango has an uneven distribution. We can also see that RT critics have the most uniform distribution. Let’s directly compare these two.</strong></p><p><strong>TASK: Create a KDE plot that compare the distribution of RT critic ratings against the STARS displayed by Fandango.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SREPYmn4MqDM9umujPWyNA.jpeg" /><figcaption>fig, ax = plt.subplots(figsize=(15,6),dpi=150)<br>sns.kdeplot(data=norm_scores[[‘RT_Norm’,’STARS’]],clip=[0,5],shade=True,palette=’Set1&#39;,ax=ax)<br>move_legend(ax, “upper left”)</figcaption></figure><p><strong>OPTIONAL TASK: Create a histplot comparing all normalized scores.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QIMQ-FQ2C48w8xNIIDgYEA.jpeg" /><figcaption>plt.subplots(figsize=(15,6),dpi=150)<br>sns.histplot(norm_scores,bins=50)</figcaption></figure><h3>How are the worst movies rated across all platforms?</h3><p><strong>TASK: Create a clustermap visualization of all normalized scores. Note the differences in ratings, highly rated movies should be clustered together versus poorly rated movies. Note: This clustermap does not need to have the FILM titles as the index, feel free to drop it for the clustermap.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/1*9_ck_FNzPPLX4CeJngq0mw.png" /><figcaption>sns.clustermap(norm_scores,cmap=’magma’,col_cluster=False)</figcaption></figure><p><strong>TASK: Clearly Fandango is rating movies much higher than other sites, especially considering that it is then displaying a rounded up version of the rating. Let’s examine the top 10 worst movies. Based off the Rotten Tomatoes Critic Ratings, what are the top 10 lowest rated movies? What are the normalized scores across all platforms for these movies? You may need to add the FILM column back in to your DataFrame of normalized scores to see the results.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y3HLpLVmTgNroHu17s6myQ.jpeg" /></figure><p><strong>FINAL TASK: Visualize the distribution of ratings across all sites for the top 10 worst movies.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8TBogAbkm1RislrcUUMZ-g.jpeg" /><figcaption>print(‘\n\n’)<br>plt.figure(figsize=(15,6),dpi=150)<br>worst_films = norm_films.nsmallest(10,’RT_Norm’).drop(‘FILM’,axis=1)<br>sns.kdeplot(data=worst_films,clip=[0,5],shade=True,palette=’Set1&#39;)<br>plt.title(“Ratings for RT Critic’s 10 Worst Reviewed Films”);</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/905/1*6tPCuiDpuPg_4oKLg5YvaQ.jpeg" /></figure><p><strong>Final thoughts: Wow! Fandango is showing around 3–4 star ratings for films that are clearly bad! Notice the biggest offender, </strong><a href="https://www.youtube.com/watch?v=tJrfImRCHJ0"><strong>Taken 3!</strong></a><strong>. Fandango is displaying 4.5 stars on their site for a film with an </strong><a href="https://en.wikipedia.org/wiki/Taken_3#Critical_response"><strong>average rating of 1.86</strong></a><strong> across the other platforms!</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/367/1*90GfqAe0sxoqD-I7nituqg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iElTolDY9Z58RP0M" /><figcaption>Photo by Jakob Owens on Unsplash</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d617c0ecca02" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>