<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Janice Senanu on Medium]]></title>
        <description><![CDATA[Stories by Janice Senanu on Medium]]></description>
        <link>https://medium.com/@janice.senanu?source=rss-80af274689c1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*0D_vMRxPfaXQNkvvygAbOQ@2x.jpeg</url>
            <title>Stories by Janice Senanu on Medium</title>
            <link>https://medium.com/@janice.senanu?source=rss-80af274689c1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 06:04:50 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@janice.senanu/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Identifying Countries In Need Of Aid Using K-Means Clustering Algorithm]]></title>
            <link>https://medium.com/@janice.senanu/identifying-countries-in-need-of-aid-using-k-means-clustering-algorithm-33df456288e4?source=rss-80af274689c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/33df456288e4</guid>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[k-means]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Janice Senanu]]></dc:creator>
            <pubDate>Tue, 01 Apr 2025 17:06:10 GMT</pubDate>
            <atom:updated>2025-04-07T22:54:43.292Z</atom:updated>
            <content:encoded><![CDATA[<h3>Aid Prioritization for an NGO using Machine Learning</h3><p><em>Implementing A K-Means Clustering Algorithm</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IqrDm8Mpz6Y8XcZZ" /><figcaption>Photo by <a href="https://unsplash.com/@micheile?utm_source=medium&amp;utm_medium=referral">micheile henderson</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>Background</strong></p><p><strong>HELP International</strong> is an international humanitarian NGO that is committed to fighting poverty and providing the people of backward countries with basic amenities and relief during the time of disasters and natural calamities. It runs a lot of operational projects from time to time along with advocacy drives to raise awareness as well as for funding purposes.</p><p><strong>Problem Statement</strong></p><p>After the recent funding programmes, they have been able to raise around $ 10 million. Now the CEO of the NGO needs to decide how to use this money strategically and effectively. The significant issues that come while making this decision are mostly related to choosing the countries that are in the direst need of aid.</p><p><strong>Objective</strong></p><p>To categorize countries using socio-economic and health factors to identify which countries need financial assistance the most.</p><p><strong>Data Dictionary</strong></p><p>The dataset used for this analysis can be found on <a href="https://www.kaggle.com/datasets/gauravduttakiit/help-international"><strong>Kaggle.com</strong></a></p><ul><li><strong>country</strong>: Name of the country</li><li><strong>child_mort:</strong> Death of children under 5 years of age per 1000 live births</li><li><strong>exports</strong>: Exports of goods and services per capita. Given as %age of the GDP per capita</li><li><strong>health:</strong> Total health spending per capita. Given as %age of GDP per capita</li><li><strong>imports:</strong> Imports of goods and services per capita. Given as %age of the GDP per capita</li><li><strong>Income:</strong> Net income per person Inflation The measurement of the annual growth rate of the Total GDP</li><li><strong>life_expec</strong>: The average number of years a new born child would live if the current mortality patterns are to remain the same</li><li><strong>total_fer:</strong> The number of children that would be born to each woman if the current age-fertility rates remain the same.</li><li><strong>gdpp:</strong> The GDP per capita. Calculated as the Total GDP divided by the total population.</li></ul><h3><strong>1. Understanding the data</strong></h3><p>To understand the nature of the dataset, we first need to import the necessary python libraries as well as the dataset.</p><pre># DATA ANALYSIS AND VISUALIZATION LIBRARIES<br>import pandas as pd<br>import numpy as np<br>import pandas as pd<br>from random import sample<br>from numpy.random import uniform<br>from math import isnan<br>import seaborn as sns<br>import matplotlib.pyplot as plt<br><br><br># MACHINE lEARNING LIBRARIES<br>import sklearn<br>from sklearn.preprocessing import StandardScaler<br>from sklearn.cluster import KMeans<br>from sklearn.metrics import silhouette_score<br>from sklearn.neighbors import NearestNeighbors<br>from kneed import KneeLocator<br><br>import warnings<br>warnings.filterwarnings(&#39;ignore&#39;)</pre><pre># IMPORTING EXCEL FILES<br>country_df = pd.read_excel(&quot;/Users/janice/Desktop/DATASETS/HELP INTERNATIONAL/Country-data.xls&quot;)<br>country_df.head(10)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tL-CKITFrA2tSJFaW5kMTA.png" /></figure><ul><li><strong>Data Inspection</strong></li></ul><p>The next step is to inspect the data for any missing data or duplicates</p><pre>country_df.info()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/1*ibJ-96DpCEp9BrKaj3xp1Q.png" /></figure><pre>country_df.duplicated().sum()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/242/1*Z1F_wTSG5l28oCo1n16fXg.png" /></figure><ul><li><strong>Feature Transformation</strong></li></ul><p>According to the data dictionary, imports, exports and health are represented as percentages of GDP per capita. Using these figures for further analysis can skew results. It can give the impression that certain countries spend similar amounts on health such as Australia and Afghanistan(8.73% and 7.58%). However, this is inaccurate especially when their respective GDP per capita are far apart, hence the need to convert them into their actual values.</p><pre>country_df[&#39;exports&#39;] = (country_df[&#39;exports&#39;]/100) * country_df[&#39;gdpp&#39;]<br>country_df[&#39;imports&#39;] = (country_df[&#39;imports&#39;]/100) * country_df[&#39;gdpp&#39;]<br>country_df[&#39;health&#39;] = (country_df[&#39;health&#39;]/100) * country_df[&#39;gdpp&#39;]<br>country_df.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KJfgelS_ApPfwnzgkowp6Q.png" /></figure><h3>2.<strong> Exploratory Data Analysis</strong></h3><p>After the data has been inspected and transformed, EDA is performed to discover patterns,to spot anomalies,to test hypothesis and to check assumptions with the help of summary statistics and graphical representations.</p><p><strong>2.1 Univariate Analysis:</strong> In Univariate Analysis, only one variable is analyzed at a time. This analysis is used to describe the data and find patterns that exist within it.</p><pre>features =[&#39;child_mort&#39;, &#39;exports&#39;, &#39;health&#39;, &#39;imports&#39;, &#39;income&#39;, &#39;inflation&#39;,<br>       &#39;life_expec&#39;, &#39;total_fer&#39;, &#39;gdpp&#39;]<br>plt.figure(figsize=(12,12))<br>for i in enumerate(features):<br>    ax = plt.subplot(3, 3, i[0]+1)<br>    sns.distplot(country_df[i[1]])<br>    plt.xticks(rotation=0)<br>    plt.tight_layout()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PzTeGEZ0CayyTEdvZHM2NQ.png" /></figure><pre>plt.figure(figsize=(12,12))<br>for i in enumerate(features):<br>    ax = plt.subplot(3, 3, i[0]+1)<br>    sns.boxplot(x=country_df[i[1]])<br>    plt.xticks(rotation=0)<br>    plt.tight_layout()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7RnWuIdTjEE0fuq_vJUgVQ.png" /></figure><p>OBSERVATIONS FROM UNIVARIATE ANALYSIS:</p><p>a. Child Mortality, Exports, Imports, Income, Inflation and GDP per capita are all highly skewed to the right (positively-skewed) and have several outliers</p><p>b. Health and Total fertility are also positively skewed</p><p>c. Life expectancy is negatively skewed (left-skewed) with a few outliers</p><p><strong>2.2 Multivariate Analysis:</strong> In Multivariate Analysis, more than two different variables are analyzed. This analysis deals with causes and relationships and the analysis is done to find out the relationship between the variables.</p><pre># Heatmap to determine the correlation between the features.<br>plt.figure(figsize=(8,5))<br>sns.heatmap(country_df.corr(), annot = True, cmap=&quot;YlGnBu&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B4rRO55Ku-LoMpkaB2kPfQ.png" /></figure><p>OBSERVATIONS FROM MULTIVARIATE ANALYSIS</p><p>a. Child mortality and life expectancy : strong negative correlation</p><p>b. Life expectancy and total fertility : strong negative correlation</p><p>c. Child mortality and total fertility : strong positive correlation</p><p>d. Exports, GDP Imports and Income. : moderate to strong positive. correlation</p><p>e. Health and GDP : Strong positive correlation</p><h3><strong>3. Strategy</strong></h3><p>The above analysis has provided some insight into the relationships of the variables. The next steps involve selecting the right features to perform clustering analysis after which the outliers for the selected features will be treated.</p><p><strong>3.1 Feature Selection</strong></p><p>Based on the objective, we are to select countries in need based on socio-economic and health factors. Therefore, we need to know what features/variables fall under these factors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/994/1*2FHszRHhvMWbm8JPzYPZuA.png" /></figure><p><strong>The Health Factor of choice is Child Mortality:</strong></p><p>The weak correlation between healthcare expenditure and the other health factors (Child Mortality, Total Fertility, and Life Expectancy) suggests that the healthcare funds may not be reaching the areas that have the greatest impact on child mortality and the other health factors.</p><p>Child mortality, influenced by various determinants of health and environmental factors such as sanitation, poverty and nutrition is a better indicator for financial assistance. By targeting child mortality, HELP International can address both healthcare needs and broader social determinants of health, aligning with the goal of fighting poverty. Moreover, child mortality correlates with other health factors such as total fertility and life expectancy, making it a comprehensive measure to improve population health.</p><p><strong>Socio-Economic Factor of Choice is Income:</strong></p><p>Inflation has a weak correlation with the other socio-economic factors because its influence is indirect and mediated by various economic and social factors such as monetary policy decisions, fiscal measures, and supply-demand dynamics within an economy.</p><p>Income, on the other hand, provides a more comprehensive understanding of individuals’ economic well-being by considering non-trade-related factors and domestic economic impact. It captures income disparities, poverty rates, and ability to access essential services, making it a more holistic indicator for financial assistance targeting socio-economic conditions.</p><p><strong>3.2 Outlier Treatment</strong></p><p>From the box plots in the univariate analysis, both child mortatlity and income have outliers and are skewed to the right. In treating the outliers for child mortality, we would refrain from the deletion method since this would exclude countries that are in the direst need of aid. Hence, there would be no outlier treatment for Child Mortality.</p><p>For Income, we are more focused on countries with low income per person so we would adopt the winorization method or percentile capping to treat the high values. Hence, we would cap at the 95th percentile which means that values that are greater than the value at 95th percentile are replaced by the value of the 95th percentile.</p><pre># Determine the 95th percentile (upper bound)<br>upper_bound = np.percentile(country_df[&#39;income&#39;], 95)<br><br># Apply winsorization: Cap only the upper extreme values<br>country_df[&#39;income_winsorized&#39;] = np.where(country_df[&#39;income&#39;] &gt; upper_bound, upper_bound, country_df[&#39;income&#39;])<br><br>#  Check the max value before and after winsorization<br>print(country_df[[&#39;income&#39;, &#39;income_winsorized&#39;]].describe())<br><br>## Checking for outliers after capping<br>sns.boxplot(data=country_df, x=&quot;income_winsorized&quot;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/1*KgfrqRpa0Mq1mtF5qVuWMw.png" /><figcaption>Values before and after winorization</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6E2YWl3NdbjczyMgctFP3Q.png" /><figcaption>boxplot after winorization (no outliers present)</figcaption></figure><h3>4. Data Preprocessing</h3><p>Now that the features have been selected and outliers treated, we can go ahead and prepare and test the data for clustering.</p><p><strong>4.1 Feature Scaling</strong></p><p>The scales for the selected features (i.e child mortality and income) are different hence the need to adjust the values and put them on a common scale.</p><pre># Create a new dataframe with the selected features<br>cluster_df = country_df[[&#39;country&#39;,&#39;child_mort&#39;,&#39;income_winsorized&#39;]].set_index(&#39;country&#39;)<br>features = cluster_df.columns <br>cluster_df </pre><pre>## Scale the features in the new dataframe<br>scale=StandardScaler()  # INITIALIZE<br>cluster_df_scaled= pd.DataFrame(scale.fit_transform(cluster_df))    ## fit the data to be studied by the algorithm<br>cluster_df_scaled.columns=features <br>cluster_df_scaled.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/1*aUgHcWVn_qoXNmfuBigWCg.png" /><figcaption>Scaled Features</figcaption></figure><p><strong>4.2 Hopkins Test</strong></p><p>The hopkins test is a way to measure the clusterability of a dataset to ascertain if there are any meaningful clusters.<br>NB: A uniformly distributed dataset can still be clustered but with no meaningful clusters hence the neccesity of hopkins test.</p><pre>def hopkins(X):<br>    d = X.shape[1] ## Length of colums<br>    n = X.shape[0] ## length of rows<br>    m = int(0.1 * n) # size of the randomly sampled dataset<br>    nbrs = NearestNeighbors(n_neighbors=1).fit(X.values)<br> <br>    rand_X = sample(range(0, n, 1), m)<br> <br>    ujd = []<br>    wjd = []<br>    for j in range(0, m):<br>        <br>        #draw uniformly from the space that is strechted from the least point to the maximum point &amp; Calculate their distance to the nearest neighbor<br>        u_dist, _ = nbrs.kneighbors(uniform(np.amin(X,axis=0),np.amax(X,axis=0),d).reshape(1, -1), 2, return_distance=True)<br>        ujd.append(u_dist[0][1])<br>        <br>        # generate another random sample from the sample itself &amp; Calculate the distance to the nearest neigbor<br>        w_dist, _ = nbrs.kneighbors(X.iloc[rand_X[j]].values.reshape(1, -1), 2, return_distance=True)<br>        wjd.append(w_dist[0][1])<br> <br>    H = sum(ujd) / (sum(ujd) + sum(wjd))<br>    if isnan(H):<br>        print(ujd, wjd)<br>        H = 0<br> <br>    return H</pre><pre>hopkins(cluster_df_scaled)</pre><p>The hopkins statistic value of 0.92 indicates that the data set has a high clustering tendency</p><h3>5. The K-Means Clustering Model</h3><p>K-Means Clustering is an Unsupervised Machine Learning algorithm, which groups the unlabeled dataset into different clusters.</p><p><strong>5.1 The Elbow Method</strong></p><p>Used to decide on the optimal number of clusters to use</p><pre>inertia_scores=[]    ## create an empty list to put all the inertia scores in once calculated<br>for i in range(1,11):   ## For each cluster in a range of 1 to 10,<br>    kmeans=KMeans(n_clusters=i, random_state=42) ### Initialize the algorithm<br>    kmeans.fit(cluster_df_scaled[[&#39;child_mort&#39;,&#39;income_winsorized&#39;]])  ## Fit the data to be studied by the algorithm<br>    inertia_scores.append(kmeans.inertia_)     ### append the calculated WCSS to inertia scores that was created<br>    <br>plt.plot(range(1,11), inertia_scores, marker=&#39;o&#39;)<br>plt.title(&#39;The Elbow Method&#39;)<br>plt.ylabel(&quot;WCSS&quot;)<br>plt.xlabel(&#39;Number of Clusters&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/1*HJIIJxTiiXpNkDuKyQgwng.png" /></figure><pre>k=KneeLocator(range(1,11), inertia_scores, curve=&#39;convex&#39;, direction=&#39;decreasing&#39;)<br>k.elbow</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/310/1*qsYNFkhDUPvqYPIejIywgg.png" /></figure><p><strong>5.2 The K Means Algorithm</strong></p><pre>## With our chosen cluster, we can input it into the KMeans algorithm<br>kmeans=KMeans(n_clusters=3, random_state=50) ## initialize algorithm with optimal clusters<br>kmeans.fit(cluster_df_scaled)<br><br># Assign clustering result to each country in the data frame<br>cluster_df[&#39;Cluster Label&#39;]=kmeans.labels_<br>cluster_df</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*2xBAA_-nTAHmELaEZzCR9Q.png" /></figure><pre>cluster_df[&#39;Cluster Label&#39;].value_counts(ascending=True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/470/1*aG8Ky_xq_0PY_QRjiEfwTg.png" /></figure><p><strong>5.3 Visualization of Clustering Results</strong></p><pre>### Visualize to better understand clustering result<br>plt.figure(figsize=(10,8))<br>sns.scatterplot(data=cluster_df, x=&#39;child_mort&#39;,y=&#39;income_winsorized&#39;,hue=&#39;Cluster Label&#39;, palette=&quot;Dark2&quot;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*dFP6b_N8AR9rFR1VIh9OXg.png" /></figure><p>On a development scale, the clusters can be identified as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/1*CaVD5sFtZFpLH3vGGj8aKw.png" /></figure><h3>6. Conclusion and Recommendations</h3><p><strong>6.1 Results</strong></p><p>According to the clusters above, the countries in the most need of financial assistance are those in Cluster 0 (Under-developed Countries).</p><pre>## Create a new dataframe for the underdeveloped countries <br>Underdeveloped = cluster_df[cluster_df[&#39;Cluster Label&#39;] == 0]<br>Underdeveloped.head(5)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/758/1*PlzgAD99OlppLoPPCetmOA.png" /></figure><p><strong>6.2 Recommendations</strong></p><p>Since Cluster 0 has 41 countries, we will narrow the selection down to the top 10 countries that need financial assistance the most. That is, countries with the highest child mortality rates and those with the lowest income levels.</p><pre>plt.figure(figsize=(18,6))<br>sns.barplot(x = &quot;country&quot;,<br>            y=&#39;child_mort&#39;,<br>            palette=&quot;rocket&quot;,<br>            data=Underdeveloped.reset_index().nlargest(20, &#39;child_mort&#39;))<br>plt.title(&quot;TOP 20 COUNTRIES WITH THE HIGHEST CHILD MORTALITY RATES&quot;)<br>plt.xticks(rotation = 45, horizontalalignment = &quot;right&quot;)<br>plt.show()       </pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pU_Mn9mvzeIF0PwUuSR1jw.png" /></figure><pre>plt.figure(figsize=(18,6))<br>sns.barplot(x=&#39;country&#39;, <br>            y=&#39;income_winsorized&#39;,<br>            palette=&quot;rocket&quot;,<br>            data=Underdeveloped.reset_index().nsmallest(20, &#39;income_winsorized&#39;))<br>plt.title(&quot;TOP 10 COUNTRIES WITH THE LOWEST INCOME LEVELS &quot;)<br>plt.xticks(rotation = 45, horizontalalignment = &quot;right&quot;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H7RKEdpwPNHI4YBAh6PHxg.png" /></figure><p>These under developed countries are ranked based on both income levels and child mortality rates to identify those that need financial assistance the most.</p><p>Income ranking is done in <strong>ascending order</strong> (lowest income gets rank 1). <strong>Lower income rank </strong>means <strong>higher priority</strong> for financial aid.</p><p>Child mortality ranking is done in <strong>descending order</strong> (highest mortality gets rank 1). <strong>Lower mortality rank </strong>means <strong>higher priority</strong> for financial aid.</p><p>The ranks from both features are summed and the countries with the <strong>lowest combined rank</strong> have the <strong>worst combination of low income and high child mortality</strong>, making them top candidates for financial assistance.</p><pre># Rank countries in cluster 0 based on child mortality and income levels combined<br>Underdeveloped[&#39;rank&#39;] = (Underdeveloped[&#39;income_winsorized&#39;].rank(ascending=True) + Underdeveloped[&#39;child_mort&#39;].rank(ascending=False)).rank()<br>Underdeveloped[&quot;rank&quot;].nsmallest(10)</pre><p><strong>THE TOP 10 COUNTRIES RANKED ACCORDING TO THOSE IN NEED OF THE MOST FINANCIAL ASSISTANCE ARE :</strong></p><ol><li><strong>CENTRAL AFRICAN REPUBLIC</strong></li><li><strong>CONGO DEM. REPUBLIC</strong></li><li><strong>NIGER</strong></li><li><strong>SIERRA LEONE</strong></li><li><strong>HAITI</strong></li><li><strong>BURUNDI</strong></li><li><strong>GUINEA</strong></li><li><strong>MOZAMBIQUE</strong></li><li><strong>GUINEA-BISSAU</strong></li><li><strong>BURKINA FASO</strong></li></ol><p>Link to <a href="https://github.com/Janice-Afi/Help-International-Clustering-Project/blob/941df18c6cb6fb153e908c4c68998b08e9b318af/HELP%20INTERNATIONAL.ipynb">Python Notebook</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=33df456288e4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An Analysis on Housing Affordability in Canada]]></title>
            <link>https://medium.com/@janice.senanu/an-analysis-on-housing-affordability-in-canada-2d07675d08de?source=rss-80af274689c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/2d07675d08de</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[housing]]></category>
            <category><![CDATA[housing-costs]]></category>
            <category><![CDATA[web-scraping]]></category>
            <dc:creator><![CDATA[Janice Senanu]]></dc:creator>
            <pubDate>Thu, 20 Oct 2022 20:27:26 GMT</pubDate>
            <atom:updated>2024-10-10T15:18:13.186Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>A settlement guide for aspiring immigrants</em></h4><h3>INTRODUCTION</h3><p>In 2015, Canada launched an immigration system known as the Express Entry for certain immigration pathways. This program has been the fastest and most popular way to communicate and select qualified applicants to move to Canada as permanent residents. Through this system, millions of foreigners have successfully immigrated to Canada to start new lives and more are expected in the coming years. Canada is expected to welcome at most <strong>445,000</strong> permanent residents by the end of 2022 as part of its Immigration Levels Plan for 2022–2024 to help foster economic growth [<a href="https://www.canada.ca/en/immigration-refugees-citizenship/corporate/publications-manuals/departmental-plan-2022-2023/departmental-plan.html">1</a>]. With its comparatively less rigorous immigration system , exceptional quality of life, and an appreciation for multiculturalism, Canada ranks first as the best country in the world — making it a top destination for immigrants [<a href="https://www.usnews.com/news/best-countries/rankings">2</a>]. Despite all this, Canada is also well known for its high housing prices [<a href="https://www.cbc.ca/news/business/crea-housing-february-1.6385274">3</a>], which poses a threat of financial burden on new and aspiring immigrants.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MqfNtbxLQy5XaTocv9Ik2g.jpeg" /></figure><h3><strong>PROBLEM STATEMENT</strong></h3><p>For many people looking to start a new life in Canada, housing prices are not the only major concern. Some Canadian employers, for various debatable reasons, fail to recognize foreign credentials, often leading immigrants to accept lower-income jobs <strong>[4]</strong>. Given the rise in housing prices and the likelihood of securing a low-paying job, prospective immigrants must ponder an essential question: <em>“Is it financially viable to move to Canada?”</em></p><h3><strong>OBJECTIVE AND SIGNIFICANCE OF ANALYSIS</strong></h3><p>The primary objective of this analysis is to ascertain the housing cost burden different family sizes face in the event where immigrants are compelled to take full-time low-income jobs upon arrival in Canada.</p><p>The findings from this analysis could offer valuable insights to help newcomers make informed decisions about settling in Canada and assist Canadian lawmakers in crafting housing policies that enhance the country’s appeal.</p><p><em>N:B: A housing cost burden is defined as a household that spends more than 30% of its before-tax income on housing/shelter.</em></p><h3><strong>SCOPE OF ANALYSIS</strong></h3><ol><li><strong>Listing Type</strong>: For the purposes of this project, <strong><em>only apartment listings</em></strong><em> </em>were scraped from multiple web pages followed by an exploratory analysis to find trends in the data.</li><li><strong>Data</strong>: A total of <strong><em>1,572 </em></strong>apartment listings were scraped for <strong><em>Alberta, British Columbia and Ontario</em></strong> because these provinces constitute the top 3 English speaking destinations for immigrants in Canada [<a href="https://www.immigrationdirect.ca/immigration-articles/canadas-top-five-immigration-destinations/index.html">5</a>].</li><li><strong>Household Type</strong>: This analysis is performed for a family of 1,2 and 3 representing an individual, a couple and a couple with a child respectively.</li><li><strong>Income Type</strong>: To estimate low income jobs, the <strong><em>minimum wage</em></strong> for each province as at August 2022 is used since it is the minimum rate an employer can <strong>legally</strong> pay its employees [<a href="https://www.canada.ca/en/employment-social-development/corporate/portfolio/labour/programs/labour-standards/reports/issue-paper-federal-minimum-wage.html">6</a>].</li><li>Below is the minimum wage per hour for each province [<a href="https://www.retailcouncil.org/resources/quick-facts/minimum-wage-by-province/">7</a>]:</li></ol><ul><li>Alberta — <strong>$15.00/hour</strong></li><li>British Columbia — <strong>$15.65/hour</strong></li><li>Ontario — <strong>$15.50/hour</strong></li></ul><p>6. The standard hours of work recognized in Canada is 8 hours in a day and 40 hours in a week. [<a href="https://www.canada.ca/en/services/jobs/workplace/federal-labour-standards/work-hours.html">8</a>]</p><h3><strong>METHODOLOGY</strong></h3><p>For this project, apartment listings were scraped online using the the <em>Beautiful Soup</em> and <em>Requests</em> python libraries. Data cleaning and manipulation were done using the <em>Pandas</em> library while the interactive visualization was achieved with <em>Plotly Express</em> and <em>Chart Studio</em> libraries.</p><p>The process for this project is summarized in the flowchart below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wR5J1cfI95kOyoxbkRSX4Q.jpeg" /></figure><p>The Github repository for the dataset can be found <a href="https://github.com/Janice-Afi/Housing-Affordability">here.</a></p><p>The complete Python codes for web scraping, data preparation and exploration on Jupyter Notebook can be found <a href="https://nbviewer.org/github/Janice-Afi/Housing-Affordability/blob/main/Housing%20Affordability%20in%20Canada.ipynb">here.</a></p><h3>DATA EXPLORATION</h3><p>The interactive graph below shows the comparison of average rent prices for 0–3 bedroom sized apartments in Alberta (AB), British Columbia (BC) and Ontario (ON).</p><p><em>N:B: A zero bedroom apartment represents a studio apartment.</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplotly.com%2F%7Ejanice.gbedemah%2F1.embed%3Fautosize%3Dtrue&amp;display_name=Plotly&amp;url=https%3A%2F%2Fchart-studio.plotly.com%2F%7Ejanice.gbedemah%2F1%2F&amp;image=https%3A%2F%2Fchart-studio.plotly.com%2Fstatic%2Fwebapp%2Fimages%2Fplotly-logo.8d56a320dbb8.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=plotly" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/4ae89e70ec2d5db81e7c3dddfb2a2a19/href">https://medium.com/media/4ae89e70ec2d5db81e7c3dddfb2a2a19/href</a></iframe><ol><li>From the graph above, Alberta has the cheapest average rent price for 0–3 bedroom apartments followed by British Columbia then Ontario.</li><li>To determine how financially reasonable it is to move to Canada by estimating the housing cost burden, the cheapest averaging province for rent which is Alberta, is used for further analysis.</li><li>For further analysis, monthly income for different family sizes is compared to the average rent prices in Alberta to determine the percentage of income spent on housing.</li></ol><h3>ANALYSIS</h3><ol><li>The table below compares rent prices of different bedroom sized apartments in Alberta for a family of 1, 2 ( a couple) and 3 (a couple with a child) earning minimum wage and working full time.</li><li>For an individual, the total working hours is 40 hours while that of a family of 2 and 3 is 80 hours/week.</li><li>The percentage of income spent on housing is calculated to determine Housing Cost Burden.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*-u1BtKFjSD0ebX8rnnvecQ.png" /></figure><p><em>N:B: A housing cost burden is defined as a household that spends more than 30% of its before-tax income on housing/shelter.</em></p><h3>RESULTS</h3><p>From the outcome of the analysis, it can be concluded that:</p><ol><li>Individuals hoping to move to Canada are likely to face a higher financial burden even in Alberta, a relatively cheaper province as they will be greatly burdened by housing cost.</li><li>However, a family of 2 (a couple) aspiring to move to Canada seems financially reasonable . This is because household income increases.</li><li>Housing affordability for a family of 3 is similar to that of a family of 2. However, the former is more likely to be burdened by other household costs since there is an additional financial responsibility of a dependant who can not bring in any additional income. Hence, making it less financially reasonable to move to Canada as a family of 3 for a low income job.</li></ol><h3>RECOMMENDATIONS</h3><p>To mitigate the risk of financial and housing cost burden, these measures can be employed:</p><ol><li>Before arriving in Canada, prepare and apply for jobs that match your qualifications and experiences. This gives one a better chance of landing higher paying jobs. In addition to this, use LinkedIn to showcase your potential and connect/network with people in your desired field of work.</li><li>Immigrate as a couple as this increases household income and reduces the risk of housing cost burden.</li><li>Enroll in Canadian courses to obtain any necessary qualifications, as this makes you better qualified and increases the chance of being recognized by Canadian employers for better job opportunities.</li><li>The government can assist with funding to build more housing units. As opposed to the current state where there is a shortage in housing units, a glut of housing units in the market may drive prices down due to less demand on the market.</li><li>The government can also offer rent assistance programs for low income households.</li></ol><h3>REFERENCES</h3><p>All references used for this project are hyperlinked in the write-up.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2d07675d08de" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[3 TOOLS NEEDED TO CONVERT AN EXCEL WORKBOOK INTO A DATABASE.]]></title>
            <link>https://medium.com/@janice.senanu/3-tools-needed-to-convert-an-excel-workbook-into-a-database-73abee66d6dc?source=rss-80af274689c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/73abee66d6dc</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[sqlite]]></category>
            <dc:creator><![CDATA[Janice Senanu]]></dc:creator>
            <pubDate>Tue, 28 Jun 2022 11:32:52 GMT</pubDate>
            <atom:updated>2025-04-04T12:38:14.342Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Practice placing data into a SQLite Database for easy querying</em></p><p>With the continuous rise in the need for data analysts/scientists, there is an abundance of resources, information and study materials for anyone beginning their journey into data. As much as this can be exciting, it can also be overwhelming and discouraging with all the technicalities and different tools at one’s disposal. Platforms such as datacamp and leetcode do a great job of providing demo spaces to practice languages such as python, R and SQL. However, gaining the hands on experience and real world application is paramount and certainly experiential. This article particularly focuses on the practicality of using SQL in querying databases. With the help of <a href="https://medium.com/u/d8312f4f39d7">Scott A. Adams</a>, in his article <strong><em>‘</em></strong><a href="https://medium.com/towards-data-science/turn-your-excel-workbook-into-a-sqlite-database-bc6d4fd206aa"><strong><em>Turn your Excel Workbook Into a SQLite Database,</em></strong></a>’ I was able to turn my excel workbook into a database as well as connecting to it for easy querying using DbVisualizer. In this article, I document my steps and understanding in achieving this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5hMXw0ePXQMNe-Ro" /><figcaption>Photo by <a href="https://unsplash.com/@pawel_czerwinski?utm_source=medium&amp;utm_medium=referral">Pawel Czerwinski</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3><strong>Tools Required</strong></h3><p>I will be using the Wikipedia dataset from my previous post for this illustration. The only tools required include:</p><ol><li>Excel workbook (The Dataset)</li><li>Python</li><li>DbVisualizer</li></ol><p><strong>EXCEL WORKBOOK</strong></p><p>As stated previously, this is the Wikipedia dataset used in my previous article. The dataset has only one sheet and originally comes in a wide format and hence needs to be transformed into a long format for efficient analysis. In my first article, I used python in unpivoting the dataset. However, the power query tool in excel can also be used to reshape the data. This youtube video serves as a guide in using the power query tool.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FVff2kRBM95o%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DVff2kRBM95o&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FVff2kRBM95o%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/46dbe3c061778232b6b6d0d8039b5357/href">https://medium.com/media/46dbe3c061778232b6b6d0d8039b5357/href</a></iframe><p><strong>PYTHON</strong></p><p>Once the excel file is ready, we set up our python environment and import <em>pandas </em>and<em> </em><em>sqlite3.</em> Before we start creating our database, we need to first read the Wikipedia excel file with the code below.</p><p>N:B — If there were other sheets in the workbook, then the data from each sheet would have to be read into a separate <em>pandas</em> dataframe using read_excel.</p><pre>import pandas as pd<br>import sqlite3 as sql<br>wikipedia = pd.read_excel(&#39;wikipedia_dataset_flat.xlsx&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*tNLNUOaAeV8yYZS1rEMbuQ.png" /><figcaption>(Screenshot from authors Notebook)</figcaption></figure><p>The next step is to create a SQLite database. SQLite is a SQL query engine that is a file-based, severless and portable relational database management system that can be accessed using any programming language. First, we need to <strong>create an empty SQLite database connection </strong>by assigning it to a variable called database using the code below.</p><pre>database = sql.connect(&quot;wikipedia_dataset.db&quot;)</pre><p>When the code above is executed, an empty database file is created in the same directory being worked in. The empty <em>wikipedia_dataset.db</em> file now has to be populated with the appropriate data. In order to do this, a variable(cur)is assigned to a <a href="https://docs.python.org/2/library/sqlite3.html#sqlite3.Cursor">cursor</a> object that allows us to first run a SQL code/statement needed to <strong>populate the data tables into the empty file created.</strong></p><pre>cur = database.cursor()</pre><p>Once the cursor object has been created, we can now execute the SQL code that will create the tables in the empty database file. Notice that <strong>the order in which the table is created is consistent with the structure of the <em>wikipedia </em>data frame created earlier. </strong>The code cur.execute() is used along with the SQL statements to create the table.</p><p>N:B— If there were other sheets in the excel workbook, then additional tables would have to be created for each of them. Here, cur.execute() would have to be run for each table that is to be created. Since the excel workbook has only one sheet, only one table is created.</p><pre>cur.execute(<br>    &quot;&quot;&quot;<br>    CREATE TABLE wikipedia(<br>        date INTEGER,<br>        page TEXT NOT NULL,<br>        visits INTEGER<br>        );<br>    &quot;&quot;&quot;<br>)</pre><p>The final step is to <strong>populate the data into the table in the empty database file </strong>and this can be done using to_sql to transfer data from the <em>wikipedia</em> data frame into the table in the database file. This is why the order in which the tables are created is important.</p><pre>wikipedia.to_sql(&#39;wikipedia&#39;, database, if_exists=&#39;append&#39;,    index=False)</pre><p>The code above transfers data from the <em>wikipedia</em> data frame to the database file and excludes the index column from the table. Now that the database has been created and populated with data, we can connect to a database software tool (DbVisualizer) to run queries.</p><p>N:B — For an excel workbook with multiple related sheets, similar codes would have to be run to transfer the data from their respective data frames to the tables you would create for each sheet in the database file.</p><p>Alternatively, queries can equally be run with python using read_sql. just like the code below.</p><pre>pd.read_sql(&quot;SELECT * FROM wikipedia&quot;, database)</pre><p>database.close()is used to close the connection after working on the database in python. To restore the connection, the code below is used.</p><pre>database = sql.connect(&quot;wikipedia_dataset.db&quot;)</pre><p><strong>DBVISUALIZER</strong></p><p>DbVisualizer is a universal database tool that connects to all major databases including SQLite. DbVisualizer can be downloaded and used for free with this <a href="https://www.dbvis.com/download/"><em>link</em>.</a> Once it is downloaded and set up, we can now connect to the SQLite database created in python, which is stored locally on our computer. The video below demonstrates the steps required to connect to our database file.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FChep2uK-5Wc%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DChep2uK-5Wc&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FChep2uK-5Wc%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/f05c9215164938fc662c406ee3ef08e1/href">https://medium.com/media/f05c9215164938fc662c406ee3ef08e1/href</a></iframe><p>Below are the steps for connecting our database to DbVisualizer</p><ul><li>Download and Launch DbVisualizer</li><li>Create a new database connection</li><li>In the dialogue box that pops up, we give a name to the database, say <em>“wikipedia database”</em> and select SQLite for both the database type and driver.</li><li>For the database file name, we copy and paste the file path where our wikipedia database is store.</li><li>Then we click on Connect. We can now see in the window on the left hand side that the name we gave to our database appears there.</li><li>After, we click on the new SQL commander to start writing queries as demonstrated in the video above.</li></ul><p>Kindly leave a comment and feedback would be appreciated.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=73abee66d6dc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[EXPLORATORY DATA ANALYSIS ON WIKIPEDIA PAGES]]></title>
            <link>https://medium.com/@janice.senanu/exploratory-data-analysis-on-wikipedia-pages-32d5fb7058a3?source=rss-80af274689c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/32d5fb7058a3</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[eda]]></category>
            <dc:creator><![CDATA[Janice Senanu]]></dc:creator>
            <pubDate>Tue, 05 Apr 2022 09:03:29 GMT</pubDate>
            <atom:updated>2025-03-14T11:59:50.054Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Thinking critically and generating insights on several wikipedia pages using python</em></p><h3><strong>Introduction</strong></h3><p>Having a background in Economics and Finance, I have been exposed to some degree of research. I found that I particularly enjoy the quantitative aspect where I get to analyze data and make meaning out of it. Even though I am familiar with working with data using Stata, it has its own limitations in relation to costs and flexibility among other things. And with my plans of venturing into machine learning, it was only right to explore other tools and programming languages. This analysis is my attempt at using python in exploring a Wikipedia dataset for my final project of the Afro Fem Coders Course.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TLiBT9TmifKvd742" /><figcaption>Photo by <a href="https://unsplash.com/@christinhumephoto?utm_source=medium&amp;utm_medium=referral">Christin Hume</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3><strong>Dataset description</strong></h3><p>The dataset contains the daily number of webpage visits for several Wikipedia webpages. The dataset includes daily page visit counts for 1,500 pages starting on 2016–01–01 until 2016–12–31.</p><p><em>Dataset source: Kaggle.com</em></p><h3><strong>Questions</strong></h3><p>1. How many languages are represented in this dataset? What proportion of the pages does each language represent?</p><p>2. Which day(s) of the week is/are the most popular for visiting wikipedia?</p><p>3.Which day(s) of the week is/are the least popular for visiting wikipedia?</p><p>4. Which device type is used more frequently for visiting wikipedia i.e. desktop or mobile devices?</p><p>5. What were some of the most trending search topics on Wikipedia on the following days?<br>a- New year’s day <br>b- November 8, 2016</p><h3>Lets get started!</h3><p>Before we start any kind of data manipulation and analysis, we need to first understand the data and the form in which it is presented. This helps prepare the mind on how to go about answering the questions and deriving insights. In using Python, the necessary libraries are needed before any work can be done.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tdd93NbsB-x3d4kdCWoFhw.png" /><figcaption>(Screenshot from authors Jupyter Notebook)</figcaption></figure><p>We first import <em>Pandas</em> for basic data manipulation, <em>date</em> for the <em>datetime </em>function and<em> re</em> for regular expression. The dataset which is stored as a csv is then imported for analysis . The shape of the dataframe tells us that there are 1500 rows and 367 columns. No real analysis can be done with the dataset in a pivoted manner thus the need to unpivot it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5PYwnnoHeZfg8qpJqIFLIg.png" /><figcaption>(Screenshot from authors Jupyter Notebook)</figcaption></figure><p>After changing the dataset from a wide to a long format, we can see from the above table that there are now three columns:Page, Date and Visits. We then check for any missing values using the <a href="https://pandas.pydata.org/docs/reference/api/pandas.isnull.html">.isnull() </a>function. From using the function, we see that the Visits column has 87 null values. There are various ways in treating missing values such as central tendencies, inputation, deletion etc…. However, for this project, we replace the missing values with a zero which indicates no page visits.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*30bhx9CDxsoX90DlPyAgdw.png" /><figcaption>(Screenshot from authors Jupyter Notebook)</figcaption></figure><p>The next thing will be to check for duplicates. So a unique id is created which combines the Page and the Date column. In order for the combination to work, the date feature will have to be converted to a string. The code below and its output indicates that no duplicates were found in the ID column that was created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5MmryIDevTkLrKxV-yeG0w.png" /><figcaption>(Screenshot from authors Jupyter Notebook)</figcaption></figure><p>After all this preliminary inspection and treatment, the data is prepared and wrangled to answer the questions. The first is to convert the Date column into days and months. The <a href="https://pandas.pydata.org/docs/reference/api/pandas.Series.dt.day_name.html">.dt.day_name()</a> and <a href="https://pandas.pydata.org/docs/reference/api/pandas.Series.dt.month_name.html">dt.month_name()</a> function is used to get the names of the day and month respectively as opposed to an integer format where Monday is indexed as 0 and Sunday is 6.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vyOVgjm4iTI5wF84L2ArNA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZZitAUWKfyb1-JNyfOx7vA.png" /><figcaption>(Screenshot from authors Jupyter Notebook)</figcaption></figure><p>From the Page column, one may infer which language a page is written in based on the page name.e.g. ‘Special:Search_fr.wikipedia.org_all-access_all-agents’ is written in French and ‘Special:Book_en.wikipedia.org_all-access_spider’ is written in English. So in order to answer the question about the languages represented in this dataset, we need to extract the language codes embedded in the page names. Regular Expression, popularly known as <a href="https://docs.python.org/3/library/re.html">regex</a>, is used for this feature extraction. Regex are sequence of special characters representing a pattern to be searched for in a text. After the language codes were extracted, they were then assigned to their respective language names using <a href="https://www.loc.gov/standards/iso639-2/php/code_list.php">ISO 639</a> which is a standardized nomenclature used to classify languages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TBZEodbB7gpY_oOdxYPHuA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xjO3T49Jl_puTuYsww977g.png" /><figcaption>(Screenshot from authors Jupyter Notebook)</figcaption></figure><p>For some of the pages in the dataset, it is possible to distinguish whether the visits to the page came from a desktop or a mobile device. The page names are structured in a way where the device type is indicated after ‘.org’, and the actual page name or title of the page comes before ‘.org.’Therefore, we can split the pages on ‘.org’ into titles and device types . After the split, the device types are grouped. Four unique device types are present in the dataset but since we want to focus on only mobile and desktop, we classify all the other device types (excluding mobile and desktop) as “all-access.” This is indicated in the image above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iJbboEXuX8rHrL-Fm6atKw.png" /><figcaption>(Screenshot from authors Jupyter Notebook)</figcaption></figure><p>Since new features have been created, it is only right to verify and make sure all the fields have been populated. We use the .isna() function again to check for any missing values. It can be seen that the language columns are missing 6222 values. Since the language codes were extracted from the pages, we filter out the missing values and investigate their respective pages further. We notice that other <strong>Wikimedia</strong> pages are also present in the dataset. To explain further, <a href="https://www.wikimedia.org/"><strong>Wikimedia</strong></a> is a global, free and multilingual internet encyclopedia whose mission is to bring free educational content to the world. Hence Wikipedia falls under Wikimedia along with other products/services such as Wikimedia Commons, MediaWiki, WikiData etc…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aTpsN76LNR7jg5R2skPsOQ.png" /><figcaption>(Screenshot from authors Jupyter Notebook)</figcaption></figure><p>From the investigation, we realize that the languages that didn’t populate came from non Wikipedia pages. Since our focus for this project is Wikipedia pages, we extract only those pages in order to begin our analysis. this new dataframe has 542778 rows and 11 columns compared to the initial dataset we started with which had 549000 rows and 3 columns. the difference between the rows is 6222 which is equivalent to the missing values form the Language column.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yle4OIO72ALoTOTzpO9EvA.png" /><figcaption>(Screenshot from authors Jupyter Notebook)</figcaption></figure><p>To answer the first question, we grouped Title and Language_Names to return its corresponding cumulative visits. It was grouped this way to help with easier visualization and to know what proportion of pages each language represents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tZcRd5zLg4A6gHA7ONjXNg.png" /><figcaption>(Screenshot from authors Jupyter Notebook)</figcaption></figure><p>For the second and third question, Monday is the most popular day for visiting Wikipedia while Friday is the least popular. This is rather unexpected, as one might assume that people are less inclined to use Wikipedia because of work but have more time to do so on the weekends. However, the dataset is limited as it does not provide the times these pages were visited. This would have provided more context and maybe explained the unexpected results.</p><p>NB : You can change the format of the output figures by using the <a href="https://pandas.pydata.org/docs/user_guide/options.html">‘set.option()</a>’ function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jKN6o2fNEEm9eaLmmd_vrg.png" /><figcaption>(Screenshot from authors Jupyter Notebook)</figcaption></figure><p>Another unexpected outcome noticed is that, Desktops were used more frequently in visiting wikipedia pages than that of mobile phones. With the easy access to mobile phones in this era, one might think that mobile phones will occupy the top spot. However, we can also argue that since wikipedia is mostly visited during working days, it makes sense why desktops (mostly found at the work place) would be used more frequently. The ‘whys’ can not be definitive since there is limited data to answer that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*voHhQ-Yk5mSEGgJ20VYdNA.png" /><figcaption>(Screenshot from authors Jupyter Notebook)</figcaption></figure><p>For the fifth question, I decided to focus only on English pages to easily identify and interpret the results. To easily follow the steps and have a clean code, we extract only the English pages and filter out new years day and November 8th.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RBiFCFCvtnmrIeKdOlIN9g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mxE_--dA8O7cNCd5X35G-A.png" /><figcaption>(Screenshot from authors Jupyter Notebook)</figcaption></figure><p>After filtering and sorting in descending order, we notice that for new years day, even though the ‘Main’ page and the ‘Special:Search’ page are the most <strong>popular pages</strong> visited, we can say that the top 2 most <strong>popular topics </strong>searched for are ‘Star Wars: The Force Awakens’ and ‘Star Wars.’ The same interpretation is applied to November 8th where the ‘United States Presidential Election 2016’ and ‘Donald Trump’ were the top 2 most searched topics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K3ngQVL72nTFppOmkhO45Q.png" /><figcaption>(Screenshot from authors Jupyter Notebook)</figcaption></figure><p>All the questions answered were assigned variables in order to export them for easy and simple visualization in Tableau.</p><p>Link to my dashboard can be found <a href="https://public.tableau.com/views/WikipediaPages/Dashboard1?:language=en-US&amp;:display_count=n&amp;:origin=viz_share_link">here.</a></p><p>Kindly comment what you think and how this analysis can be improved.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=32d5fb7058a3" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>