<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Segun Lawal on Medium]]></title>
        <description><![CDATA[Stories by Segun Lawal on Medium]]></description>
        <link>https://medium.com/@lawalsegun2025?source=rss-be557c5e285e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*nEiTf3ed85wQMhMNzDTfNw.png</url>
            <title>Stories by Segun Lawal on Medium</title>
            <link>https://medium.com/@lawalsegun2025?source=rss-be557c5e285e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 08:46:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@lawalsegun2025/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[IPL Match Win Probability Predictor]]></title>
            <link>https://medium.com/@lawalsegun2025/ipl-match-win-probability-predictor-bcfc261a8499?source=rss-be557c5e285e------2</link>
            <guid isPermaLink="false">https://medium.com/p/bcfc261a8499</guid>
            <dc:creator><![CDATA[Segun Lawal]]></dc:creator>
            <pubDate>Sun, 18 Jun 2023 09:23:41 GMT</pubDate>
            <atom:updated>2023-06-19T15:44:56.333Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data Science/Machine Learning Project</h3><p>The github repository for this project can be found <a href="https://github.com/lawalsegun2025/ipl_match_win_predictor">here.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p6ceTllAT4GxbWObGXpdng.png" /><figcaption>IPL logo</figcaption></figure><h4>Overview</h4><p>Indian Premier League (IPL) is a Twenty20 cricket format league in India. It is usually played in April and May every year. As of 2019, the title sponsor of the game is Vivo. The league was founded by Board of Control for Cricket India (BCCI) in 2008</p><p>The objective of this project is to create a model capable of predicting the winning probability percentage of each team. Based on the first inning performance the model takes in current data of the scores in the second innings and make the predictions.</p><h4>DataSet</h4><p>The dataset consist of data about IPL matches played from the year 2008 to 2019. The sources of the data sets are from;</p><ul><li>Data source from 2008–2017, <a href="https://cricsheet.org/">CricSheet.org</a> and Manas — Kaggle</li><li>Data source for 2018–2019 — <a href="https://www.iplt20.com/">IPL T20 — Official website</a></li></ul><h4>Importing dependencies</h4><p>Wee will first import all the dependencies we need for this project</p><pre># import dependencies<br>import pandas as pd <br>import numpy as np <br>import seaborn as sns<br>import matplotlib.pyplot as plt<br># import model building libriaries<br>from sklearn.compose import ColumnTransformer<br>from sklearn.preprocessing import OneHotEncoder<br>from sklearn.linear_model import LogisticRegression<br>from sklearn.ensemble import RandomForestClassifier<br>from sklearn.pipeline import Pipeline<br>from sklearn import metrics </pre><h4>Loading Dataset</h4><p>We will load in the two data sets for the project.</p><pre># load datasets<br>matches = pd.read_csv(&quot;matches.csv&quot;)<br>deliveries = pd.read_csv(&quot;deliveries.csv&quot;)<br>matches.head()</pre><h4>Exploratory Data Analysis</h4><pre># Check the shape of both datasets<br>matches.shape, deliveries.shape<br><br># get the name of the columns of the delivery dataset<br>deliveries.columns</pre><p>We are assuming that the first inning has taken place and the second inning is about to take place, so we want to see which team is winning.</p><pre><br># grouping the 1st innings,2nd innings score in a particular matchid<br># lets say match id = 1,so inning 1 score = 207,inning 2 score = 172,in that way<br><br>total_runs_df = deliveries.groupby([&quot;match_id&quot;, &quot;inning&quot;]).sum(numeric_only=True)[&quot;total_runs&quot;].reset_index()<br><br># Extracting only the first innings as we will be predicting for the second inning.<br>total_runs_df = total_runs_df[total_runs_df[&quot;inning&quot;] == 1]<br><br># we are adding 1 to total_runs, because for the second team to win they must get more runs than the first team<br>total_runs_df[&quot;total_runs&quot;] = total_runs_df[&quot;total_runs&quot;].apply(lambda x:x+1)<br></pre><p>We have gotten the total runs for the first inning of each match, we will now merge this data frame with the matches data frame. A left side merging will be done on the &quot;id&quot; column of the matches data frame and a right side merge will be done on the &quot;match_id&quot; of the total_run_df</p><pre># merge the match df and the total_run_df<br><br>match_df = matches.merge(total_runs_df[[&quot;match_id&quot;, &quot;total_runs&quot;]], <br>             left_on=&quot;id&quot;, right_on=&quot;match_id&quot;)<br><br># Let&#39;s have a look at the teams<br>match_df[&quot;team1&quot;].unique()</pre><pre>array([&#39;Sunrisers Hyderabad&#39;, &#39;Mumbai Indians&#39;, &#39;Gujarat Lions&#39;,<br>       &#39;Rising Pune Supergiant&#39;, &#39;Royal Challengers Bangalore&#39;,<br>       &#39;Kolkata Knight Riders&#39;, &#39;Delhi Daredevils&#39;, &#39;Kings XI Punjab&#39;,<br>       &#39;Chennai Super Kings&#39;, &#39;Rajasthan Royals&#39;, &#39;Deccan Chargers&#39;,<br>       &#39;Kochi Tuskers Kerala&#39;, &#39;Pune Warriors&#39;, &#39;Rising Pune Supergiants&#39;,<br>       &#39;Delhi Capitals&#39;], dtype=object)</pre><pre># we will be considering the most frequently participating teams<br>teams = [<br>    &#39;Sunrisers Hyderabad&#39;,<br>    &#39;Mumbai Indians&#39;,<br>    &#39;Royal Challengers Bangalore&#39;,<br>    &#39;Kolkata Knight Riders&#39;,<br>    &#39;Kings XI Punjab&#39;,<br>    &#39;Chennai Super Kings&#39;,<br>    &#39;Rajasthan Royals&#39;,<br>    &#39;Delhi Capitals&#39;<br>]</pre><pre># replacing the old name Delhi Daredevils with new name Delhi Capitals<br><br>match_df[&#39;team1&#39;] = match_df[&#39;team1&#39;].str.replace(&#39;Delhi Daredevils&#39;,&#39;Delhi Capitals&#39;)<br>match_df[&#39;team2&#39;] = match_df[&#39;team2&#39;].str.replace(&#39;Delhi Daredevils&#39;,&#39;Delhi Capitals&#39;)<br><br># replacing the old name Deccan Chargers with new name Sunrises Hyderabad<br><br>match_df[&#39;team1&#39;] = match_df[&#39;team1&#39;].str.replace(&#39;Deccan Chargers&#39;,&#39;Sunrisers Hyderabad&#39;)<br>match_df[&#39;team2&#39;] = match_df[&#39;team2&#39;].str.replace(&#39;Deccan Chargers&#39;,&#39;Sunrisers Hyderabad&#39;)</pre><pre># will consider only frequently occuring teams,<br># which are mentioned in the teams list<br><br>match_df = match_df[match_df[&#39;team1&#39;].isin(teams)]<br>match_df = match_df[match_df[&#39;team2&#39;].isin(teams)]<br><br>match_df[&#39;team1&#39;].unique()</pre><p>The Duckworth–Lewis–Stern method (DLS) is a mathematical formulation designed to calculate the target score for the team batting second in a limited overs cricket match <strong>interrupted by weather or other circumstances</strong>. It is generally accepted to be the most accurate method of setting a target score</p><pre># Check matches that resulted in Duckworth–Lewis–Stern method <br># (dl meethod)<br>match_df[match_df[&quot;dl_applied&quot;]==1].style.background_gradient(cmap=&quot;plasma&quot;)</pre><pre># We will drop every row of matches decided by DLS method<br># and considering only matches that were completed<br>match_df = match_df[match_df[&quot;dl_applied&quot;]==0]<br><br># for this analysis we will consider the match_id, city, winner <br># and total runs<br><br>match_df = match_df[[&quot;match_id&quot;, &quot;city&quot;, &quot;winner&quot;, &quot;total_runs&quot;]]</pre><p>The above features are what we need from the match_df. More important features will be extracted from the deliveries data frame</p><p>The two data frames have the match_id column. Therefore we will combine both of them.</p><pre># merging match_df with deliveries on the match_id column<br>delivery_df = match_df.merge(deliveries, on=&quot;match_id&quot;)</pre><p>Our initial data frame only contains total runs of first innings, this is because we want to predict the total runs of second innings. Now we will consider the second innings.</p><pre># considering 2nd innings<br>delivery_df = delivery_df[delivery_df[&quot;inning&quot;] == 2]</pre><p>By observing, we can see that in the match_df, we had taken first innings total_runs, and in the second case, that is the delivery_df, we considered the second innings runs because our main aim is to find the probability of either teams winning or loosing the match.</p><p>For this, we need current runs and run rate. For current runs, we apply groupby on match_id and take the cumulative sum with respect to total_runs_y. Total_runs was present match_df as well as deliveries dataframe, but as we merged the two dataframes, it resulted in total_runs_x for match_df and total_runs_y for deliveries dataframe. Therefore, total_run_x is the first innings runs and total_runs_y are the second innings runs, ball by ball. By applying cumulative sum, this runs becomes the current score.</p><p>total_runs_y gives the runs scored after each ball, therefore in the second innings, we want to get the total second inning runs.Therefore we will groupby match_id and will apply the cumulative sum.</p><pre># current score af a particular match<br>delivery_df[&quot;current_score&quot;] = delivery_df.groupby([&quot;match_id&quot;]).cumsum(numeric_only=True)[&quot;total_runs_y&quot;]<br><br># Calculate the runs left to chase<br>delivery_df[&quot;runs_left&quot;] = delivery_df[&quot;total_runs_x&quot;] - delivery_df[&quot;current_score&quot;]</pre><p>if one ball is played,then balls left = 120–1 = 119……..(1) if two balls are played,then balls left = 120–2 = 118</p><p>so similarly if over=1,over has 6 balls right,so 1*6 = 6 now,ball = 1,so 6+1 = 7,now 126–7 = 119,which is same as (1)</p><p>so we’ll use balls_left = 126-(over*6+current_ball)</p><pre># Calculate total balls left<br>delivery_df[&#39;balls_left&#39;] = 126-(delivery_df[&#39;over&#39;]*6+delivery_df[&#39;ball&#39;])</pre><p>Nan represents that the player was not dismissed and a name indicates that the player was dismissed. We want to replace Nan with 0 and a players name with 1</p><pre># fill NaN values with &quot;0&quot;<br>delivery_df[&quot;player_dismissed&quot;] = delivery_df[&quot;player_dismissed&quot;].fillna(&quot;0&quot;)<br><br># now we will convert this player_dismissed col into a boolean col<br># if the player is not dismissed then it&#39;s 0 else its 1<br><br>delivery_df[&quot;player_dismissed&quot;] = delivery_df[&quot;player_dismissed&quot;].apply(lambda x: x if x==&quot;0&quot; else &quot;1&quot;)<br><br># Convert string to integer<br>delivery_df[&quot;player_dismissed&quot;] = delivery_df[&quot;player_dismissed&quot;].astype(&quot;int&quot;)</pre><pre># wickets fallen<br>wickets = delivery_df.groupby(&#39;match_id&#39;).cumsum(numeric_only=True)[&#39;player_dismissed&#39;].values<br><br># a wicket = 10<br>delivery_df[&#39;wickets&#39;] = 10-wickets<br><br># current run rate<br><br>delivery_df[&#39;cur_run_rate&#39;] = (delivery_df[&#39;current_score&#39;]*6)/(120-delivery_df[&#39;balls_left&#39;])<br><br># required run rate<br><br>delivery_df[&#39;req_run_rate&#39;] = (delivery_df[&#39;runs_left&#39;]*6)/(delivery_df[&#39;balls_left&#39;])<br><br></pre><pre># function<br>def result_fun(row):<br>    return 1 if row[&quot;batting_team&quot;] == row[&quot;winner&quot;] else 0<br><br>delivery_df[&quot;result&quot;] = delivery_df.apply(result_fun, axis=1)</pre><pre># Select only specific columns<br>final_df = delivery_df[[&#39;batting_team&#39;,&#39;bowling_team&#39;,&#39;city&#39;,&#39;runs_left&#39;,<br>                        &#39;balls_left&#39;,&#39;wickets&#39;,&#39;total_runs_x&#39;,&#39;cur_run_rate&#39;,<br>                        &#39;req_run_rate&#39;,&#39;result&#39;]]</pre><h3>Model Building</h3><pre># Serach for null values<br>final_df.isnull().sum()<br><br># Droppin all null values<br><br>final_df = final_df.dropna()<br><br>final_df.isnull().sum()</pre><pre># we will selt rows that &quot;balls_left&quot; != 0<br>final_df = final_df[final_df[&quot;balls_left&quot;] != 0]</pre><pre># Create a copy of the dataset<br>data = final_df.copy()<br><br>y = data[&quot;result&quot;]<br><br>X = data.drop([&quot;result&quot;], axis=1)</pre><pre>from sklearn.model_selection import train_test_split<br><br>X_train, X_test, y_train, y_test = train_test_split(X, <br>                                                    y, <br>                                                    test_size=0.2, <br>                                                    random_state=1)</pre><pre># batting team,bowling team and city are categorical columns<br># they will be converted to numeric using onehot encoder<br>ct = ColumnTransformer([<br>    (&quot;trf&quot;, OneHotEncoder(sparse_output=False, drop=&quot;first&quot;), <br>     [&#39;batting_team&#39;,&#39;bowling_team&#39;,&#39;city&#39;])<br>], remainder=&quot;passthrough&quot;)</pre><pre># Create the Pipeline<br><br>pipe = Pipeline(steps=[<br>    (&quot;step1&quot;,ct), <br>    (&quot;step2&quot;,LogisticRegression(solver=&quot;liblinear&quot;))<br>])<br><br># fitting the training data<br>pipe.fit(X_train, y_train)</pre><pre># Make prediction<br>y_pred = pipe.predict(X_test)<br>print(metrics.accuracy_score(y_test, y_pred))</pre><pre># Lets train a Random forest model<br>pipe2 = Pipeline(steps=[<br>    (&quot;step1&quot;,ct), <br>    (&quot;step2&quot;,RandomForestClassifier())<br>])<br><br># fitting the training data<br>pipe2.fit(X_train, y_train)<br>print(metrics.accuracy_score(y_test, pipe2.predict(X_test)))</pre><p>We will go with the Logistic Regression Model because we need to showcase the probability. Though Random Forest Model is giving a much accurate result, bit it is being biased to one of the teams. Therefore it is better to use a model which gives “Equal Justice” towards both sides as we do not know which team will out perform and win the game in the second inning!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bcfc261a8499" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Udemy Course Recommendation System]]></title>
            <link>https://medium.com/@lawalsegun2025/udemy-course-recommendation-system-17b3a5882555?source=rss-be557c5e285e------2</link>
            <guid isPermaLink="false">https://medium.com/p/17b3a5882555</guid>
            <dc:creator><![CDATA[Segun Lawal]]></dc:creator>
            <pubDate>Sun, 11 Jun 2023 10:45:35 GMT</pubDate>
            <atom:updated>2023-06-11T10:45:35.911Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*INji6E56ayOxt85rLHyUCQ.png" /><figcaption>udemy courses</figcaption></figure><h4>Overview</h4><p>This project helps recommend similar courses on udemy for which the user is searching for. The user enters the name or phrase of a subject of interest and related courses will be recommended and displayed. There is also and analysis dashboard that gives all the analysis of the udemy courses available in the dataset.</p><p>Link to the project github repository can be found <a href="https://github.com/lawalsegun2025/udemy_course_recommendation_system">here</a></p><h4>Source of Dataset</h4><p>This dataset contains 3683 records of courses from 4 subjects (Business Finance, Graphic Design, Musical Instruments and Web Design) taken from Udemy.</p><p>Udemy is a massive online open course (MOOC) platform that offers both free and paid courses. Anybody can create a course, a business model by which allowed Udemy to have hundreds of thousands of courses. This version modifies column names, removes empty columns and aggregates everything into a single csv file for ease of use.</p><h4>Import dataset</h4><pre># Import dependencies<br>import pandas as pd <br>import numpy as np <br>import seaborn as sns <br>import matplotlib.pyplot as plt <br><br>%matplotlib inline<br><br># read in / load dataset<br>df = pd.read_csv(&quot;udemy_course_data.csv&quot;)<br><br># convert price from Rupee to Dollar<br>df[&quot;price&quot;] = df[&quot;price&quot;] * 0.0121<br><br>df.head()</pre><h4>Exploratory Data Analysis</h4><p>For the exploratory data analysis, we explored the dataset by trying to answer the following question to have a better understanding of the data;</p><p><strong>EDA on Subject/Category</strong></p><p>Subjects/Categories<br>What is the distribution of subjects?<br>- How many courses per subject?<br>- Distribution of subjects per year?<br>- How many people purchase a particular subject?<br>- Which subjects is the most popular?<br> <br><strong>What is the distribution of subjects?</strong></p><pre># Plot the grap of subject/category of courses<br>plt.figure(figsize=(10, 7))<br>sns.countplot(x=&quot;subject&quot;, data=df, palette=&quot;plasma&quot;);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/859/1*CE9ID6T7Q9gFXui7p2d7ng.png" /><figcaption>Numbers of subscribers per course subject/category</figcaption></figure><p><strong>EDA on Subscribers</strong></p><p><strong>Subscribers</strong></p><ul><li>Which course have the highest number of subscribers?</li><li>Average number of subscribers?</li><li>Number of subscribers per subject?</li><li>Number of subscribers per year?</li></ul><pre># Group number of subscribers by subject and get sum<br>df.groupby(&quot;subject&quot;)[&quot;num_subscribers&quot;].sum().plot(kind=&quot;barh&quot;);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/1*Iej9BBgc_F3XJ8PhkUkq2w.png" /></figure><p><strong>EDA on Levels</strong></p><p><strong>Levels</strong></p><ul><li>How many levels do we have?</li><li>What is the distribution of courses per level?</li><li>Which subject have the highest levels?</li><li>How many subscribers per level?</li><li>How many courses per level?</li></ul><pre># number of levels an subscribers count<br>plt.figure(figsize=(10, 7))<br>sns.barplot(x=&quot;level&quot;, y=&quot;num_subscribers&quot;, hue=&quot;subject&quot;, data=df)<br>plt.xticks(fontsize = 20,fontweight = &#39;bold&#39;)<br>plt.yticks(fontsize = 20,fontweight = &#39;bold&#39;)<br>plt.xlabel(&#39;Level&#39;,fontsize = 20,fontweight = &#39;bold&#39;,<br>           color = &#39;blue&#39;)<br>plt.ylabel(&#39;Subscribers Count&#39;,fontsize = 20,fontweight = &#39;bold&#39;,<br>           color = &#39;blue&#39;)<br>plt.xticks(rotation = &#39;vertical&#39;)<br>plt.legend(loc=&#39;upper right&#39;);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/915/1*MhLDCejUz5flCIozKy_kqA.png" /></figure><p><strong>EDA on Price</strong></p><p><strong>Price</strong></p><ul><li>What is the average price of a course?</li><li>What is the minimum and maximum price?</li><li>How much does Udemy earn?</li><li>The most profitable courses?</li></ul><pre># plot the count plot<br>plt.figure(figsize=(10, 7))<br>df[&quot;price&quot;].value_counts().plot(kind=&quot;bar&quot;);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/831/1*aGMg6V0bO1G7ahHI69N11w.png" /></figure><h3>Course Recommendation System</h3><p>For building the course recommendation system, we will be working with only the course_title column only. We start by cleaning the course_title using neattext.functions column as it is a text data.</p><p><strong>neattext</strong> is a simple Natural Language Processing package for cleaning text data and pre-processing text data. It can be used to clean sentences, extract emails, phone numbers, weblinks, and emojis from sentences.</p><pre># Import dependancies<br>import pandas as pd # data analysis and manipulation tool<br>import numpy as np # brings the computational power of languages like C and Fortran to Python<br>import neattext.functions as nfx # used to clean text data<br>from sklearn.feature_extraction.text import TfidfVectorizer # Convert a collection of raw documents to a matrix of TF-IDF features<br>from sklearn.feature_extraction.text import CountVectorizer # a method to convert text to numerical data<br>from sklearn.metrics.pairwise import cosine_similarity # measures the similarity between two vectors of an inner product space<br>from sklearn.metrics.pairwise import linear_kernel # Compute the linear kernel between X and Y</pre><pre># convert course title to lower cases<br>df[&quot;course_title&quot;] = df[&quot;course_title&quot;].apply(lambda x: x.lower())<br><br># Remove stopwords<br>df[&quot;clean_title&quot;] = df[&quot;course_title&quot;].apply(nfx.remove_stopwords)<br><br># Remove special characters<br>df[&quot;clean_title&quot;] = df[&quot;course_title&quot;].apply(nfx.remove_special_characters)<br><br>df.head()</pre><h3>Vectorize clean_title column</h3><p>Text Vectorization is the process of converting text into numerical representation.</p><p>We decided not to stem or lemmatize the titles because titles do not contain lots of texts.</p><pre># Vectorice the titles<br>count_vect = CountVectorizer()<br>cv_matrix = count_vect.fit_transform(df[&quot;clean_title&quot;])</pre><h3>Cosine Similarity</h3><p>Cosine similarity measures the similarity between two vectors of an inner product space. It is measured by the cosine of the angle between two vectors and determines whether two vectors are pointing in roughly the same direction. It is often used to measure document similarity in text analysis.</p><pre># Instanciate cosine_similarity<br>cos_sim = cosine_similarity(cv_matrix)</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=17b3a5882555" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[WhatsApp Chat Analyzer]]></title>
            <link>https://medium.com/@lawalsegun2025/whatsapp-chat-analyzer-8f47fb760a46?source=rss-be557c5e285e------2</link>
            <guid isPermaLink="false">https://medium.com/p/8f47fb760a46</guid>
            <dc:creator><![CDATA[Segun Lawal]]></dc:creator>
            <pubDate>Sun, 04 Jun 2023 06:06:32 GMT</pubDate>
            <atom:updated>2023-06-04T06:06:32.893Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7VVluJLFU6qPi9O0VAq6tg.png" /></figure><h4>Overview</h4><p>We often find ourselves joining or being added to numerous whatsapp chat groups and some time the number of unread chats can run up to hundreds or in some cases even more. This app can help a use to get the most relevant information(s) rather than manually skimming through a whole lot of chats.</p><p><strong>Note:</strong> The data used for this project does not include media files.</p><h4>Source of dataset</h4><p>The data for this project can be gotten from any whatsapp group chat. To get a whatsapp group chat data, follow the steps listed below;</p><ul><li>Move to the whatsapp chat group</li><li>At the top right click on the three vertical column</li><li>You will be presented with a drop down menu, click on more</li><li>Select export</li><li>Choose the option without media files (as this project work with data that did not include media files)</li></ul><h4>Data(Text) Preprocessing</h4><p>First we import the required modules and dependencies as shown below:</p><pre># Import modules and libriaries<br>import pandas as pd <br>import numpy as np <br>import seaborn as sns <br>import matplotlib.pyplot as plt <br>import re <br>import datetime as dt <br>from urlextract import URLExtract # used to extract url&#39;s in a string<br>from collections import Counter<br>from wordcloud import WordCloud<br>import string <br>import nltk<br>import emoji<br><br><br>pd.set_option(&#39;display.max_colwidth&#39;, 100)<br><br>stopwords = nltk.corpus.stopwords.words(&quot;english&quot;)<br>wn = nltk.WordNetLemmatizer()<br>punct = string.punctuation<br><br>%matplotlib inline# Import modules and libriaries<br>import pandas as pd <br>import numpy as np <br>import seaborn as sns <br>import matplotlib.pyplot as plt <br>import re <br>import datetime as dt <br><br>%matplotlib inline</pre><p>The following function was used to preprocess the raw text data into a pandas dataframe.</p><pre>def preprocess(file, key):<br>    split_formats = {<br>        &#39;12hr&#39; : &#39;\d{1,2}/\d{1,2}/\d{2,4},\s\d{1,2}:\d{2}\s[APap][mM]\s-\s&#39;,<br>        &#39;24hr&#39; : &#39;\d{1,2}/\d{1,2}/\d{2,4},\s\d{1,2}:\d{2}\s-\s&#39;,<br>        &#39;custom&#39; : &#39;&#39;<br>    }<br>    datetime_formats = {<br>        &#39;12hr&#39; : &#39;%m/%d/%y, %I:%M %p - &#39;,<br>        &#39;24hr&#39; : &#39;%m/%d/%y, %H:%M - &#39;,<br>        &#39;custom&#39;: &#39;&#39;<br>    }<br>    <br>    raw_data = open(file, &#39;r&#39;, encoding=&quot;utf-8&quot;)<br>    raw_data = raw_data.read()<br>    raw_string = &#39; &#39;.join(raw_data.split(&#39;\n&#39;)) # converting the list split by newline char. as one whole string as there can be multi-line messages<br>    user_msg = re.split(split_formats[key], raw_string) [1:] # splits at all the date-time pattern, resulting in list of all the messages with user names<br>    date_time = re.findall(split_formats[key], raw_string) # finds all the date-time patterns<br><br>    df = pd.DataFrame({&#39;Date&#39;: date_time, &#39;user_msg&#39;: user_msg}) # exporting it to a df<br>        <br>    # converting date-time pattern which is of type String to type datetime,<br>    # format is to be specified for the whole string where the placeholders are extracted by the method <br>    df[&#39;Date&#39;] = pd.to_datetime(df[&#39;Date&#39;], format=datetime_formats[key])<br>    <br>    # split user and msg <br>    usernames = []<br>    msgs = []<br>    for i in df[&#39;user_msg&#39;]:<br>        a = re.split(&#39;([\w\W]+?):\s&#39;, i) # lazy pattern match to first {user_name}: pattern and spliting it aka each msg from a user<br>        if(a[1:]): # user typed messages<br>            usernames.append(a[1])<br>            msgs.append(a[2])<br>        else: # other notifications in the group(eg: someone was added, some left ...)<br>            usernames.append(&quot;Group Notification&quot;)<br>            msgs.append(a[0])<br><br>    # creating new columns         <br>    df[&#39;User&#39;] = usernames<br>    df[&#39;Message&#39;] = msgs<br><br>    # dropping the old user_msg col.<br>    df.drop(&#39;user_msg&#39;, axis=1, inplace=True)<br>    <br>    df = df[[&quot;Message&quot;, &quot;Date&quot;, &quot;User&quot;]]<br>    <br>    df[&quot;Message&quot;] = df[&quot;Message&quot;].apply(lambda x: x.strip())<br>    <br>    # Extract the date without time<br>    df[&#39;Only date&#39;] = pd.to_datetime(df[&#39;Date&#39;]).dt.date<br><br>    # Get only the Year<br>    df[&#39;Year&#39;] = pd.to_datetime(df[&#39;Date&#39;]).dt.year<br><br>    # Get month number<br>    df[&#39;Month_num&#39;] = pd.to_datetime(df[&#39;Date&#39;]).dt.month<br><br>    # Get name of month<br>    df[&#39;Month&#39;] = pd.to_datetime(df[&#39;Date&#39;]).dt.month_name()<br><br>    # Get Day<br>    df[&#39;Day&#39;] = pd.to_datetime(df[&#39;Date&#39;]).dt.day<br><br>    # Get name of Day<br>    df[&#39;Day_name&#39;] = pd.to_datetime(df[&#39;Date&#39;]).dt.day_name()<br><br>    # Get hour<br>    df[&#39;Hour&#39;] = pd.to_datetime(df[&#39;Date&#39;]).dt.hour<br><br>    # Get minutes<br>    df[&#39;Minute&#39;] = pd.to_datetime(df[&#39;Date&#39;]).dt.minute<br><br>    <br>    <br>    return df</pre><h4>Data (text) Analysis</h4><p>We first started with numerical to calculate:</p><p><strong>Total number of messages in the chat</strong></p><pre># Get total number of messsages in the group<br>len(df[&quot;Message&quot;])</pre><p><strong>Total number of words in the group</strong></p><pre># Total number of words in the group chat<br>words = []<br>for message in df[&quot;Message&quot;]:<br>    words.extend(message.split())<br>    <br>print(len(words))</pre><p><strong>Total number of media files shared in the group</strong></p><pre># Remove spaces<br>df[&quot;Message&quot;][18830].strip()<br><br># Get number of media files shared in the group<br>len(df[df[&quot;Message&quot;] == &quot;&lt;Media omitted&gt;&quot;])</pre><p><strong>Total number of links in the group</strong></p><pre># Get number of links in the group<br>extract = URLExtract()<br><br>links = []<br>for message in df[&quot;Message&quot;]:<br>    links.extend(extract.find_urls(message))<br>    <br>len(links)</pre><h4>Word Cloud</h4><p>This displays the most common words in the chat</p><pre># create word cloud object<br>wc = WordCloud(width=1000, height=500, min_font_size=10, <br>              background_color=&quot;white&quot;)<br><br># generate wordcloud image<br>wc = wc.generate(wc_df[&quot;Message&quot;].str.cat(sep=&quot; &quot;))<br><br># plot word cloud<br>plt.figure(figsize=(10, 7))<br>plt.imshow(wc)<br>plt.axis(&quot;off&quot;);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/1*W6QVwuL-nVKFxD033tGqDw.png" /><figcaption>word cloud displaying the most common words.</figcaption></figure><h4>Word Count (Top 20 Words)</h4><p>A graph of the top 20 words in the chat was plotted against the number of occurrences of the words.</p><pre># Plot a graph of the most common words<br>plt.figure(figsize=(10, 7))<br>plt.barh(most_common[0], most_common[1])<br>plt.xticks(rotation=&quot;vertical&quot;)<br>plt.title(&quot;Most Common Words&quot;);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/1*Oe2K40yC7ibz2O5Zdww6Tg.png" /></figure><h4>Number of Emojis</h4><p>Emojis are usually used in chats. The following codes extract the count of emojis used in the chat;</p><pre># extract all the emojis in the &quot;Message&quot; column<br>emojis = []<br>for message in df[&quot;Message&quot;]:<br>    emojis.extend([c for c in message if c in emoji.EMOJI_DATA.keys()])<br><br>    <br># Create and emoji dataframe<br>emoji_df = pd.DataFrame(Counter(emojis).most_common())<br><br># rename column<br>emoji_df.columns = [&quot;Emoji&quot;, &quot;Count&quot;]<br><br># Create a percentage usage column<br>emoji_count = list(emoji_df[&quot;Count&quot;])<br>per_list = [(i/sum(emoji_count))*100 for i in emoji_count]<br>emoji_df[&quot;Percentage use&quot;] = np.array(per_list).round(2)<br><br>emoji_df</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9qEDwyz5wPkvBGAn9tchrg.png" /><figcaption>Count of emojis used and the percentage</figcaption></figure><h3>Monthly Timeline</h3><p>We want the count of the messages per-month, therefore we will apply .groupby() method and adding reset_index() will get the result in a dataframe format.</p><p>We will then add the Year and Month columns to get as an example &quot;January-2018&quot;. This is to enhance our analysis.</p><pre># Get monthly timeline<br>monthly_df = df.groupby([&quot;Year&quot;, &quot;Month_num&quot;, &quot;Month&quot;]).count()[&quot;Message&quot;].reset_index()<br><br>monthly_timeline = []<br>for i in range(monthly_df.shape[0]):<br>    monthly_timeline.append(monthly_df[&quot;Month&quot;][i]+&quot;-&quot;+str(monthly_df[&quot;Year&quot;][i]))<br><br>monthly_df[&quot;monthly_timeline&quot;] = monthly_timeline<br><br># Plot the monthy timeline of messages<br>plt.figure(figsize=(10, 7))<br>plt.plot(monthly_df[&quot;monthly_timeline&quot;], monthly_df[&quot;Message&quot;])<br>plt.xticks(rotation=&quot;vertical&quot;)<br>plt.tight_layout();</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/1*NnyYD59Kk4CjKG99fvwoqg.png" /><figcaption>Monthly timeline of number of messages received in the chat</figcaption></figure><h4>Activity Map</h4><p>This analyzes the count of messages on weekly and monthly basis.</p><p><strong>Most Active Days</strong></p><pre># Get a count of chats perday<br>active_days = df[&quot;Day_name&quot;].value_counts()<br><br># Plot the graph of messages per day<br>plt.bar(active_days.index, active_days.values, color=&quot;purple&quot;)<br>plt.xticks(rotation=&quot;vertical&quot;);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/561/1*EjXe5a_YSh8GFSViZcAVsw.png" /><figcaption>Days of the week and the number of messages</figcaption></figure><p><strong>Most Active Months</strong></p><pre>active_months = df[&quot;Month&quot;].value_counts()<br><br># plot a graph of the most active months<br>plt.bar(active_months.index, active_months.values, color=&quot;orange&quot;)<br>plt.xticks(rotation=&quot;vertical&quot;);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/561/1*njoqI_STd_o5ORlVuf0I2g.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f47fb760a46" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Laptop Price Predictor]]></title>
            <link>https://medium.com/@lawalsegun2025/laptop-price-predictor-7f94e9011c74?source=rss-be557c5e285e------2</link>
            <guid isPermaLink="false">https://medium.com/p/7f94e9011c74</guid>
            <dc:creator><![CDATA[Segun Lawal]]></dc:creator>
            <pubDate>Sun, 28 May 2023 10:51:19 GMT</pubDate>
            <atom:updated>2023-05-28T10:51:19.617Z</atom:updated>
            <content:encoded><![CDATA[<p>The github repository for this project can be found <a href="https://github.com/lawalsegun2025/laptop_price_predictor">here</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OXXp9QBCeSp4IAZuuFLGpw.jpeg" /></figure><p><strong>Project Overview</strong></p><p>This is a machine learning supervise learning project of building a model that predicts the price rang of a laptop based on specific software and hardware requirements.</p><p><strong>Dataset</strong></p><p>The dataset comprises of 1303 rows and 11 columns. Here is the link for the <a href="https://github.com/lawalsegun2025/laptop_price_predictor/blob/main/laptop_data.csv">dataset</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/1*rvP0GsrhwBXPeTQTemOjhg.png" /></figure><p><strong>Data Cleaning and Preprocessing</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AT_HBxJ0KPE8o_KFnbRmQA.png" /></figure><p>We can see above that the Ram and Weights columns are in object data form. Therefore we have to convert these columns to integers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4YLqdhWk4s2dRV4oNhlB5A.png" /></figure><p><strong>Exploratory Data Analysis</strong></p><p>Let’s take a look at the distribution of the target column “Price”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AheerO0HQ9hLOheyKyBd_A.png" /></figure><p>It can be observed that the price column distribution is skewed to the right.</p><p><strong>“ScreenResolution” Column</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2Vao2VTVs2Tr8onEaMjvcQ.png" /></figure><p>For the Screen Resolution column, we will create a new column called touch screen, that indicates if the laptop has touch screen ability or not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DM0_q372cRN8RW1mlQMcjg.png" /></figure><p>A new “IPS” column will be created, indicating if the laptop has IPS ability. This is also extracted from the Screen Resolution column.</p><p><strong>Extracting X and Y Resolutions</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XUJfqZBWIcKQ7c313v1KCA.png" /></figure><p>We will also be extracting the X and Y resolution and create a dataframe with the values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k7t68jsknBS2Fndzc58lkQ.png" /></figure><p>The extracted X and Y resolutions will be added to the the dataframe. We are extracting these values just to see if we can extract valuable information from the Screen Resolution text data. The Y resolution is just in integer form, bit he X Resolution has some text in it. We will need to get only the resolution number.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mdreNoSH_abmQC7ohqaPYw.png" /></figure><p>The code above extracts just the resolution number for the X resolution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/925/1*nX1TdW6OZ6mes78zxkBtkQ.png" /></figure><p>It can be observed from the correlation heat map above that X_res and Y_res are highly correlated, this is called collinearity. So we will have to handle that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bC1p2pym1Tive7BXga10jQ.png" /></figure><p>We will create a Pixels Per Inches formula using the inches, X_res and Y_res and then drop the X_res and Y_res columns since they are both highly correlated.</p><p><strong>CPU Column</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GmEirepJF_RXhx4-ziWWwA.png" /></figure><p>It can be observed that the most common processors are made by Intel. Therefore, we will cluster their processor into different categories like i3, i5, i7, and others. Here &quot;others&quot; means the processors of intel which do not have i3, i5 or i7 attached to it, it is completely different so this is the reason we will cluster them into other and the other category is inclusive of AMD which is a different Cpu category. This is because their presence is not much in the market.</p><p>If we observe closely, we will see that we need to extract the first 3 words of the Cpu column entries as the first 3 words of every row under the Cpu column is the type of Cpu used by the specific laptop.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gvqsw-CjtESBCZKgHNjnuQ.png" /></figure><p>As mentioned earlier, if we get any of the intel i3, i5, or i7 versions, we will return them as it is, but if we get any other processor, we will first check whether it is a variant of the intel or not. If it is a variant of intel, we will tag it as &quot;Other intel Processor&quot; else we will say it is and &quot;AMD Processor&quot;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ynjiTbn_Bk8HNMcymBaoSg.png" /></figure><p>Then we apply the above function to the CPU column</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BYJlwcqpQS5EN7IGRh_k1g.png" /></figure><p><strong>Memory Column</strong></p><p>We will separate the Type of memory and the value of the memory, just similar to the one which is done in the previous part.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*qePK20PxQ7fjoHhWC-B9lA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ULC3pdGrNcju6FSQ5ydgcw.png" /></figure><p><strong>GPU column</strong></p><p>For the GPU column we will just be extracting the brands name</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NRZc-hkwgYCwTU81IR3HPQ.png" /></figure><h3>Model Building</h3><p>As we saw when visualizing the distribution of the Price column, that it is skewed to the right, we will take the log of the Price column. The will improve the performance of the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gK64poTWNYZNExxnYNpoDg.png" /></figure><p>The above will import the required modules and dependencies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CgVaITyBHZH8RkDe-n-90A.png" /></figure><p>We will split the data, 85% for training and 15% for testing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*idHMDwanqY0FqRAN8WQIOg.png" /></figure><p>There is a Class which we imported named as ColumnTranformer. We use this widely while building our model using Pipelines so for this we have to get the index numbers for the columns which are having categorical variables.</p><pre>def run_model(model):<br>    &quot;&quot;&quot;<br>    This fuction takes in the instance of a model, then it transforms<br>    the data, create a pipeline, <br>    make predictions, and prints the R2_score and MAE<br>    &quot;&quot;&quot;<br>    # Transform the data<br>    cat_transform = ColumnTransformer(<br>        transformers=[<br>            (&quot;col_tnf&quot;, OneHotEncoder(sparse_output=False, drop=&quot;first&quot;), [0, 1, 3, 8, 11])<br>        ],remainder=&quot;passthrough&quot;<br>    )<br>    <br>    global pipeline<br>    # Create a Pipeline<br>    pipeline = Pipeline([<br>        (&quot;cat_transform&quot;, cat_transform), <br>        (&quot;model&quot;, model)<br>    ])<br>    <br>    # Fit the pipeline and train the model<br>    pipeline.fit(X_train, y_train)<br><br>    y_pred = pipeline.predict(X_test)<br><br>    print(&quot;R2 score: &quot;, metrics.r2_score(y_test, y_pred))<br>    print(&quot;MAE: &quot;, metrics.mean_absolute_error(y_test, y_pred))</pre><p>The function above will help us build a model and evaluate the model’s prediction.</p><pre># Instanciate a Random Forest Regression<br>rfr_model = RandomForestRegressor(n_estimators=100, <br>                                 random_state=3, <br>                                 max_samples=0.5, <br>                                 max_features=0.75, <br>                                 max_depth=15)<br><br>run_model(rfr_model)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3BZHnz3LCmj1I20woirEqg.png" /></figure><p>After trying several model, RandomForestRegression model gave us the best score.</p><p>We ran some Hyperparameter tuning for the Random Forest Model, but the result did not improve, we we decided to preprocess the data a little bit more using pandas .get_dummies method.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hgSEwDdJp15wz4YGPSB08g.png" /></figure><p>This method of handling categorical columns really improved the performance of the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HDkva9OQuw0juMRfdslJ_g.png" /></figure><p><strong>Finally</strong></p><p>We made prediction on the whole data set and compared it to the true values.</p><pre>sns.kdeplot(df[&#39;Price&#39;],color=&#39;orange&#39;,label=&#39;Actual&#39;)<br>sns.kdeplot(df[&#39;Predicted Price&#39;],color=&#39;blue&#39;,label=&#39;Predicted&#39;)<br>plt.legend()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*xVzVUY73h3_-t1Y0ShzUiA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7f94e9011c74" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>