<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by su_sandy on Medium]]></title>
        <description><![CDATA[Stories by su_sandy on Medium]]></description>
        <link>https://medium.com/@susandyhtun1?source=rss-33aa80f87909------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*OimS4kJcPixUb2ijGXeBmQ.jpeg</url>
            <title>Stories by su_sandy on Medium</title>
            <link>https://medium.com/@susandyhtun1?source=rss-33aa80f87909------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 00:24:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@susandyhtun1/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Happiness by the Numbers, Part 3: Digging Deeper Into Numbers]]></title>
            <link>https://medium.com/@susandyhtun1/happiness-by-the-numbers-part-3-digging-deeper-into-numbers-86147bea4d2e?source=rss-33aa80f87909------2</link>
            <guid isPermaLink="false">https://medium.com/p/86147bea4d2e</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[power-bi]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[su_sandy]]></dc:creator>
            <pubDate>Fri, 20 Jun 2025 16:14:17 GMT</pubDate>
            <atom:updated>2025-06-20T16:14:17.684Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cGDcd12UgF0FvhaL" /><figcaption>Photo by <a href="https://unsplash.com/@christianlue?utm_source=medium&amp;utm_medium=referral">Christian Lue</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In <a href="https://medium.com/@susandyhtun1/happiness-by-the-numbers-part-2-when-the-data-comes-alive-5f16fc44f65c">Part 2</a>, we explored the happiest and least happy countries and did some regional trend comparisons. Now, let’s dive a little deeper into how countries are spread within each region and see if we can spot any interesting insights.</p><p>I wanted to use a scatter plot with each dot representing the happiness score of a country within its region, but I also wanted to compare regions side-by-side. I decided to go for a Python visual for this. Below is the code I used. The code calculates both the regional median and the global median, plots each country’s score, and color-codes them based on whether they’re above or below their regional median. I specifically used an accessible color palette.</p><pre>import pandas as pd<br>import matplotlib.pyplot as plt<br>import seaborn as sns<br><br># Accessible color palette<br>color_palette = [&#39;#364b59&#39;, &#39;#c480a7&#39;, &#39;#6e98b9&#39;, &#39;#67002e&#39;]<br><br># A mapping for the palette parameter<br>palette_mapping = {<br>    &#39;Above Regional Median&#39;: color_palette[0],  # &#39;#364b59&#39;<br>    &#39;Below Regional Median&#39;: color_palette[1],  # &#39;#c480a7&#39;<br>    &#39;At Median&#39;: color_palette[2]               # &#39;#6e98b9&#39;<br>}<br><br>df = dataset.copy()<br><br># Compute regional medians and global median for the current slicer selection<br>region_medians = df.groupby(&#39;Region&#39;)[&#39;Happiness score&#39;].median()<br>global_median = df[&#39;Happiness score&#39;].median()<br><br># Classify each country as above/below/at regional median<br>df[&#39;Above Regional Median&#39;] = df.apply(<br>    lambda row: &#39;Above Regional Median&#39; if row[&#39;Happiness score&#39;] &gt; region_medians[row[&#39;Region&#39;]] else<br>                (&#39;Below Regional Median&#39;  if row[&#39;Happiness score&#39;] &lt; region_medians[row[&#39;Region&#39;]] else &#39;At Median&#39;),<br>    axis=1<br>)<br><br>plt.figure(figsize=(12, 6))<br>ax = sns.stripplot(<br>    data=df,<br>    x=&#39;Region&#39;,<br>    y=&#39;Happiness score&#39;,<br>    hue=&#39;Above Regional Median&#39;,<br>    dodge=False,<br>    jitter=True,<br>    palette= palette_mapping, <br>    size=7<br>)<br>plt.axhline(global_median, color=color_palette[3], linestyle=&#39;--&#39;, label=f&#39;Global Median ({global_median:.2f})&#39;)<br><br>ax.spines[&#39;top&#39;].set_visible(False)<br>ax.spines[&#39;right&#39;].set_visible(False)<br>ax.spines[&#39;left&#39;].set_visible(False)<br>ax.spines[&#39;bottom&#39;].set_visible(False)<br><br>plt.legend(title=&#39;&#39;)<br>plt.tight_layout()<br>plt.show()</pre><p>This is the result visual:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*lpzIAuGUPJw77nhUgaXGww.png" /><figcaption><strong>Fig. 1.</strong> A strip plot showing the spread of countries’ happiness within each region for 2024.</figcaption></figure><p>Each dot represents a country’s happiness score, stacked up against both the regional and global median.</p><p><strong>Asia:</strong></p><ul><li>Huge spread! Some countries are way below both medians, and only a handful are near or above the global average. Afghanistan is a clear outlier, far below everyone else.</li></ul><p><strong>Europe:</strong></p><ul><li>Tight cluster above the global median, with a couple of pink dots below it. The outlier here is Egypt, with the lowest happiness score (3.817) which is even lower than the regional median in Asia (5.494).</li></ul><p><strong>Africa:</strong></p><ul><li>All clustered below the global median. The regional median is the lowest of all regions.</li></ul><p><strong>South America:</strong></p><ul><li>Almost all countries are above the global median, except for one pink dot (Venezuela). Surprisingly, there isn’t a huge outlier here.</li></ul><p><strong>Oceania:</strong></p><ul><li>As expected, both are well above the global median.</li></ul><p><strong>North America:</strong></p><ul><li>Almost all dots are above the global median, with the only exception being the Dominican Republic. No major outlier here either.</li></ul><p><strong>Key takeaways:</strong> North America, Oceania, and South America lead with close-knit clusters. Europe is trying to catch up, Asia is a runner-up with a big spread within the region, and Africa is sadly at the bottom.</p><blockquote>Now, let’s shift to something more dynamic: <strong>which countries fell or rose the fastest between two given years?</strong> I think this is a pretty interesting way to spot the impact of major global events.</blockquote><p>For this chart, I used the built-in Line chart, and set up two parameters: one for choosing top 5 fallers/risers and one for selecting two different years. Here’s the measure I used to calculate the average happiness score between any two selected years:</p><pre>Avg. ladder score between selected years = <br>VAR StartYear =<br>    MIN ( &#39;Year parameter&#39;[Year] )<br>VAR EndYear =<br>    MAX ( &#39;Year parameter&#39;[Year] )<br>RETURN<br>    CALCULATE (<br>        AVERAGE ( &#39;Happiness data&#39;[Happiness score] ),<br>        FILTER (<br>            &#39;Happiness data&#39;,<br>            &#39;Happiness data&#39;[Year] &gt;= StartYear<br>                &amp;&amp; &#39;Happiness data&#39;[Year] &lt;= EndYear<br>        )<br>    )</pre><p>This measure shows the average for all countries between two selected years. But I wanted to filter down to just the top 5 risers or bottom 5 fallers (as selected by the user). For this, I created another measure to be used as a filter in the visual. It calculates the difference in happiness scores for all countries between the two years, ranks them by that difference, and filters to show only the top or bottom 5 depending on user selection. I used this measure in my visual filter and set the value to 1.</p><pre>Show Top5 or Bottom5 = <br>VAR RiserorFaller = <br>    SELECTEDVALUE(&#39;Risers and fallers&#39;[Risers/fallers])<br>VAR StartYear =<br>    MIN ( &#39;Year parameter&#39;[Year] )<br>VAR EndYear =<br>    MAX ( &#39;Year parameter&#39;[Year] )<br>VAR ThisCountry =<br>    SELECTEDVALUE ( &#39;Region&#39;[CountryName] )<br>VAR AllCountryChanges =<br>    SUMMARIZE (<br>        ALL ( &#39;Region&#39;[CountryName] ),<br>        &#39;Region&#39;[CountryName],<br>        &quot;ChangeScore&quot;,<br>            VAR ThisCountry = &#39;Region&#39;[CountryName]<br>            VAR StartScore =<br>                CALCULATE (<br>                    AVERAGE ( &#39;Happiness data&#39;[Happiness score] ),<br>                    ALL ( &#39;Year&#39;[Year] ),<br>                    &#39;Happiness data&#39;[Year] = StartYear,<br>                    &#39;Happiness data&#39;[Country name] = ThisCountry<br>                )<br>            VAR EndScore =<br>                CALCULATE (<br>                    AVERAGE ( &#39;Happiness data&#39;[Happiness score] ),<br>                    ALL ( &#39;Year&#39;[Year] ),<br>                    &#39;Happiness data&#39;[Year] = EndYear,<br>                    &#39;Happiness data&#39;[Country name] = ThisCountry<br>                )<br>            RETURN<br>                IF (<br>                    NOT ISBLANK ( StartScore ) &amp;&amp; NOT ISBLANK ( EndScore ),<br>                    ROUND ( EndScore - StartScore, 2 ),<br>                    BLANK ()<br>                )<br>    )<br>VAR FilteredCountryChanges =<br>    FILTER ( AllCountryChanges, NOT ISBLANK ( [ChangeScore] ) )<br>VAR RankedCountries =<br>    ADDCOLUMNS (<br>        FilteredCountryChanges,<br>        &quot;RankDesc&quot;, RANKX ( FilteredCountryChanges, [ChangeScore],, DESC, SKIP ),<br>        &quot;RankAsc&quot;, RANKX ( FilteredCountryChanges, [ChangeScore],, ASC, SKIP )<br>    ) <br>VAR Top5Risers =<br>    FILTER(RankedCountries, [RankDesc] &lt;= 5)<br><br>VAR Bottom5Fallers =<br>    FILTER(RankedCountries, [RankAsc] &lt;= 5)<br><br>VAR IsTopRiser =<br>    CONTAINS(Top5Risers, [CountryName], ThisCountry)<br><br>VAR IsBottomFaller =<br>    CONTAINS(Bottom5Fallers, [CountryName], ThisCountry)<br><br>RETURN<br>    SWITCH(<br>        SELECTEDVALUE(&#39;Risers and fallers&#39;[Risers/fallers]),<br>        &quot;Top 5 Risers&quot;, IF(IsTopRiser, 1, 0),<br>        &quot;Bottom 5 Fallers&quot;, IF(IsBottomFaller, 1, 0),<br>        0 // Default case if no match is found<br>    )</pre><p>Here is the result visual. I also added a custom tooltip (see Fig. 4) so when you hover over a point, you get the breakdown of the happiness score by factors for all 5 countries and makes comparison a lot easier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*LEzSKKEWlp_wvtXi9CjHWg.png" /><figcaption><strong>Fig. 3.</strong> A line chart showing the happiness score of the bottom 5 fallers between 2019 and 2021.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*lQUxCm5zfX-fj2XpzBmsRA.png" /><figcaption><strong>Fig. 4.</strong> A custom tooltip for detailed breakdown of the happiness score.</figcaption></figure><p>I was curious about the Covid period, so I set the year selection to 2019 and 2021. From the results, Lebanon stands out with a very sharp drop, going from 4.77 to 2.95. This matches up with their economic and political crises on top of the pandemic. Costa Rica is another interesting one. It technically is a faller, but barely lost any points and was hardly shaken by Covid. The other countries showed more moderate declines.</p><p>So, what about we look at the overall top 5 risers and bottom 5 fallers?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h6b00eNYqO47vF8pSvpqXw.png" /><figcaption><strong>Fig. 5.</strong> Side-by-side comparison of Top 5 Risers and Bottom 5 Fallers over all years.</figcaption></figure><p>There’s much more turbulence in the fallers’ lines, which hints at external shocks like war, crisis, or political instability. Afghanistan and Lebanon were especially hard-hit. The risers, on the other hand, show steady improvement, even if they started from a tough spot.</p><blockquote>Now for another question that popped up: <strong>how do the happiness factors interact? </strong>For example, do countries with high GDP also tend to have high social support?</blockquote><p>I wanted to see this with a heat map showing the correlation between all the factors. For this, I used another Python visual with the code below:</p><pre>import pandas as pd<br>import seaborn as sns<br>import matplotlib.pyplot as plt<br><br>factor_cols = [&#39;GDP per capita&#39;, &#39;Social support&#39;, &#39;Generosity&#39;, &#39;Healthy life expectancy&#39;, &#39;Perceptions of corruption&#39;, &#39;Dystopia + residual&#39;, &#39;Freedom&#39;]<br><br>df = dataset[factor_cols]<br><br>df = df.dropna()<br><br>corr = df.corr(method = &#39;pearson&#39;)<br><br>plt.figure(figsize = (8,6))<br>ax = sns.heatmap(corr, annot = True, cmap = &#39;Blues&#39;, center = 0, vmin = -1, vmax = 1, linewidths = 0.5, linecolor = &#39;white&#39;)<br>ax.xaxis.set_ticks_position(&#39;top&#39;)<br>ax.xaxis.set_label_position(&#39;top&#39;)<br>plt.xticks(rotation = 45, ha = &#39;left&#39;, fontsize = 12)<br>plt.yticks(rotation = 0, fontsize = 12)<br>plt.tight_layout()<br>plt.show()</pre><p>And this is the result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*_Y3cSUuLwVW8FRe0XLETyQ.png" /><figcaption><strong>Fig. 6.</strong> A heat map showing the correlation between happiness factors for 2024.</figcaption></figure><p>Here’s what stands out:</p><ul><li>Strong positive relationships: GDP per capita &amp; Healthy life expectancy (0.72), GDP per capita &amp; Social support (0.69), Social support &amp; Healthy life expectancy (0.53). Countries with more wealth tend to have better healthcare and social support.</li><li>Generosity has weak (or even negative) relationships with other factors, especially GDP per capita (-0.18). So, you don’t need to be rich to be generous (and vice versa).</li><li>Freedom is also a bit of a wild card, with weak relationships to the other factors.</li></ul><blockquote>Now for a fun bit: <strong>what happens if we play around with the factor scores and see which country is the closest match?</strong></blockquote><p>For this, I created six parameters (one for each happiness factor) and a simple measure to get the calculated happiness score:</p><pre>Predicted happiness score = <br>VAR gdp = &#39;GDP per Capita&#39;[GDP per Capita Value]<br>VAR social = &#39;Social support&#39;[Social support Value]<br>VAR freedom = Freedom[Freedom Value]<br>VAR generosity = Generosity[Generosity Value]<br>VAR life = &#39;Healthy life expectancy&#39;[Healthy life expectancy Value]<br>VAR corruption = &#39;Perceptions of corruption&#39;[Perceptions of corruption Value]<br>VAR Residual = 1.5 // static value<br>RETURN<br>    ROUND(gdp + social + freedom + generosity + life + corruption + Residual, 3)</pre><p>Then, I created a second measure to find the country with the closest happiness score to our calculated result, and display it on a card:</p><pre>Closest country to the predicted score = <br>VAR predicted = [Predicted happiness score]<br>VAR year_ = SELECTEDVALUE(&#39;Year&#39;[Year])<br>VAR Countries =<br>    ADDCOLUMNS (<br>        FILTER (<br>            ALL ( &#39;Happiness data&#39; ),<br>            &#39;Happiness data&#39;[Year] = year_<br>                &amp;&amp; NOT ISBLANK ( &#39;Happiness data&#39;[Happiness score] )<br>        ),<br>        &quot;Difference&quot;, ABS( ABS ( &#39;Happiness data&#39;[Happiness score] ) - predicted)<br>    )<br>VAR closestCountry = <br>    SELECTCOLUMNS(<br>        TOPN(1, Countries, [Difference], ASC),<br>        &quot;Country&quot;, &#39;Happiness data&#39;[Country name], <br>        &quot;Score&quot;, &#39;Happiness data&#39;[Happiness score]<br>    )<br><br>VAR resultcountry = MAXX(closestCountry, [Country])<br>VAR resultscore = MAXX(closestCountry, [Score])<br><br>RETURN<br>    &quot;Closest match: &quot; &amp; resultcountry &amp;<br>    &quot; in &quot; &amp; year_ &amp;<br>    &quot; (Score : &quot; &amp; FORMAT(resultscore, &quot;0.000&quot;) &amp; &quot;)&quot;</pre><p>Here’s what that looks like in the report:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*9_Kf7FqxtDdiGA6qJoLpUA.png" /><figcaption><strong>Fig. 7.</strong> The user can slide the score for each factor, get the calculated happiness score, and then see which country is the closest match.</figcaption></figure><p>As I mentioned before, this is more about playing around and learning fun facts — no major insights here.</p><p>As my last visual, I added a Q&amp;A visual and populated it with some interesting questions so users can interact and get answers directly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/857/1*UpfC26k72ndrXAHKcMRkOA.png" /><figcaption><strong>Fig. 8.</strong> A Q&amp;A visual for user interaction.</figcaption></figure><p>So, that’s a wrap for all my visuals, code, and insights from the global happiness index data. Hope you enjoyed it!</p><p><strong>Keep learning. Enjoy the journey!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=86147bea4d2e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Happiness by the Numbers, Part 2: When the Data Comes Alive]]></title>
            <link>https://medium.com/@susandyhtun1/happiness-by-the-numbers-part-2-when-the-data-comes-alive-5f16fc44f65c?source=rss-33aa80f87909------2</link>
            <guid isPermaLink="false">https://medium.com/p/5f16fc44f65c</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[dax]]></category>
            <category><![CDATA[power-bi]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[su_sandy]]></dc:creator>
            <pubDate>Fri, 06 Jun 2025 11:03:13 GMT</pubDate>
            <atom:updated>2025-06-06T16:11:27.749Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Yd9Eue5hMi5TGuX4" /><figcaption>Photo by <a href="https://unsplash.com/@freestockpro?utm_source=medium&amp;utm_medium=referral">Alexandr Podvalny</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>If <a href="https://medium.com/@susandyhtun1/happiness-by-the-numbers-part-1-unpacking-the-data-c659cf4855a9">Part 1</a> was all about asking the right questions, this is where things start to get interesting. After building the basic data model and setting up my roadmap, now, in Part 2, we’ll go through some of the visuals and the DAX code behind them, along with a few of the key insights that popped up along the way. This is the fun part (I promise 😉).</p><p>First up, I really wanted to add a map (because, why not?). The data’s about countries, so it just made sense. Plus, a map is both visually pleasing and makes it easy to spot which regions are happiest at a glance. I decided to use the Shape Map visual in Power BI (still in preview, by the way). The first hurdle: the default map options don’t include a world map. So, I went to the <a href="https://www.naturalearthdata.com/downloads/50m-cultural-vectors/">Natural Earth website</a> and grabbed a map of all countries, then converted it to TopoJSON format using <a href="https://mapshaper.org/">mapshaper</a>. Once I uploaded that file into Power BI, voilà — a beautiful map with tooltips for country name, happiness score, and rank (see Fig. 1).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*sjzMCVUR3p6w47wv8LbotA.png" /><figcaption>Fig. 1. A shape map showing countries’ happiness scores and ranks for the year 2024</figcaption></figure><p>At a glance, you can already see the <strong>classic divide: Nordic countries are a darker shade than much of Africa and Asia</strong>, so there’s a pretty clear split across continents.<br>Oh, and I used the ‘Accessible Orchid’ theme which is specifically designed to improve accessibility for users with various visual impairments, including color vision deficiencies.</p><p>Looking at this, I got curious: which countries are at the very top (or bottom) of the happiness list, and what’s driving those scores? To dig into the factors, I wanted to plot them on the X-axis and use country as the legend. For this, I needed a specific DAX measure to calculate the average happiness score by factor:</p><pre>Score for each factor =<br>SWITCH (<br>    TRUE (),<br>    SELECTEDVALUE ( Factor[Factor] ) = &quot;Dystopia + residual&quot;, AVERAGE ( &#39;Happiness data&#39;[Dystopia + residual] ),<br>    SELECTEDVALUE ( Factor[Factor] ) = &quot;GDP per capita&quot;, AVERAGE ( &#39;Happiness data&#39;[GDP per capita] ),<br>    SELECTEDVALUE ( Factor[Factor] ) = &quot;Social support&quot;, AVERAGE ( &#39;Happiness data&#39;[Social support] ),<br>    SELECTEDVALUE ( Factor[Factor] ) = &quot;Freedom&quot;, AVERAGE ( &#39;Happiness data&#39;[Freedom] ),<br>    SELECTEDVALUE ( Factor[Factor] ) = &quot;Healthy life expectancy&quot;, AVERAGE ( &#39;Happiness data&#39;[Healthy life expectancy] ),<br>    SELECTEDVALUE ( Factor[Factor] ) = &quot;Perceptions of corruption&quot;, AVERAGE ( &#39;Happiness data&#39;[Perceptions of corruption] ),<br>    SELECTEDVALUE ( Factor[Factor] ) = &quot;Generosity&quot;, AVERAGE ( &#39;Happiness data&#39;[Generosity] )<br>)</pre><p>I also thought it’d be useful to compare each factor to the global average, so I created a second measure for that average line. Basically, this measure calculates the average for each factor, regardless of country or region filters:</p><pre>Global avg. score = <br>VAR SelectedFactor = SELECTEDVALUE(Factor[Factor])<br>RETURN<br>SWITCH (<br>    TRUE (),<br>    SelectedFactor = &quot;Dystopia + residual&quot;,<br>        CALCULATE (<br>            AVERAGE ( &#39;Happiness data&#39;[Dystopia + residual] ),<br>            REMOVEFILTERS ( &#39;Region&#39; ),<br>            REMOVEFILTERS ( &#39;Happiness data&#39;[Rank group] ),<br>            REMOVEFILTERS ( &#39;Happiness data&#39;[Country name] )<br>        ),<br>    SelectedFactor = &quot;GDP per capita&quot;,<br>        CALCULATE (<br>            AVERAGE ( &#39;Happiness data&#39;[GDP per capita] ),<br>            REMOVEFILTERS ( &#39;Region&#39; ),<br>            REMOVEFILTERS ( &#39;Happiness data&#39;[Rank group] ),<br>            REMOVEFILTERS ( &#39;Happiness data&#39;[Country name] )<br>        ),<br>    SelectedFactor = &quot;Social support&quot;,<br>        CALCULATE (<br>            AVERAGE ( &#39;Happiness data&#39;[Social support] ),<br>            REMOVEFILTERS ( &#39;Region&#39; ),<br>            REMOVEFILTERS ( &#39;Happiness data&#39;[Rank group] ),<br>            REMOVEFILTERS ( &#39;Happiness data&#39;[Country name] )<br>        ),<br>    SelectedFactor = &quot;Freedom&quot;,<br>        CALCULATE (<br>            AVERAGE ( &#39;Happiness data&#39;[Freedom] ),<br>            REMOVEFILTERS ( &#39;Region&#39; ),<br>            REMOVEFILTERS ( &#39;Happiness data&#39;[Rank group] ),<br>            REMOVEFILTERS ( &#39;Happiness data&#39;[Country name] )<br>        ),<br>    SelectedFactor = &quot;Healthy life expectancy&quot;,<br>        CALCULATE (<br>            AVERAGE ( &#39;Happiness data&#39;[Healthy life expectancy] ),<br>            REMOVEFILTERS ( &#39;Region&#39; ),<br>            REMOVEFILTERS ( &#39;Happiness data&#39;[Rank group] ),<br>            REMOVEFILTERS ( &#39;Happiness data&#39;[Country name] )<br>        ),<br>    SelectedFactor = &quot;Perceptions of corruption&quot;,<br>        CALCULATE (<br>            AVERAGE ( &#39;Happiness data&#39;[Perceptions of corruption] ),<br>            REMOVEFILTERS ( &#39;Region&#39; ),<br>            REMOVEFILTERS ( &#39;Happiness data&#39;[Rank group] ),<br>            REMOVEFILTERS ( &#39;Happiness data&#39;[Country name] )<br>        ),<br>    SelectedFactor = &quot;Generosity&quot;,<br>        CALCULATE (<br>            AVERAGE ( &#39;Happiness data&#39;[Generosity] ),<br>            REMOVEFILTERS ( &#39;Region&#39; ),<br>            REMOVEFILTERS ( &#39;Happiness data&#39;[Rank group] ),<br>            REMOVEFILTERS ( &#39;Happiness data&#39;[Country name] )<br>        )<br>)</pre><p>With those two measures, I picked a line and clustered column chart to tell the story (see Fig. 2 and 3). To make it more interactive, I added a slicer for Top 10 Happiest, Middle, and Bottom 10 Happiest.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*mX_nfh5JFqVXl-qvdLC1UA.png" /><figcaption>Fig. 2. Happiness score for each factor for top 10 happiest countries in 2024</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*AVUY0W83piFTIQ2HB6eIuQ.png" /><figcaption>Fig. 3. Happiness score for each factor for bottom 10 happiest countries in 2024</figcaption></figure><p><strong>What stands out from these two charts?</strong></p><p><strong>Top 10 Happiest Countries (Fig. 2):</strong></p><ul><li><em>Consistently above average</em>: Most of these countries are crushing it in GDP per capita, social support, and healthy life expectancy. Their bars are way above the global average.</li><li><em>Dystopia + residual</em>: Higher than average here too. This is possibly acting as a baseline or ‘catch-all’ for happiness not explained by the other factors. I am curious what they could be.</li><li><em>Outliers</em>:</li><li><strong>Luxembourg:</strong> No surprise that it leads in GDP per capita.</li><li><strong>Costa Rica &amp; Mexico:</strong> Stand out for “Dystopia + residual.” I wonder what their secret recipe is.</li></ul><p><strong>Bottom 10 Happiest Countries (Fig. 3):</strong></p><ul><li><em>Struggling on almost all factors</em>: Mostly below global averages.</li><li><em>Freedom, generosity, and perceptions of corruption</em>: These are particularly low. People in these countries generally report less freedom, low generosity, and high corruption which can make things even tougher.</li><li><em>Outliers</em>:</li><li><strong>Botswana:</strong> Has a relatively high GDP per capita compared to its peers.</li><li><strong>Comoros:</strong> Higher “Dystopia + residual” which is possibly reflecting some unique local factors.</li><li><strong>Yemen:</strong> Surprisingly high in social support for its overall score.</li></ul><blockquote>With data, it’s always the outliers and exceptions that make me the most curious as there’s always a story hiding there.</blockquote><p>All in all, not all happiness factors are created equal. Even among the happiest countries, some shine in specific areas (Luxembourg : money, Iceland : support, Iceland/Netherlands : generosity). The bottom 10 are struggling on almost every front, but countries like Botswana and Yemen do show surprising strengths even if it doesn’t translate to overall happiness.</p><p>So far, all this analysis is based on 2024 data by country. But what about trends over time? How do regions compare year by year?</p><p>For this, I used the Cycle Plot by Nova Silva (Fig. 4). Each curve represents the average happiness score trend for each region from 2011 to 2024, with the horizontal line marking the median score for each region.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*mGmMU6IeDRsUDNSkEMJvOA.png" /><figcaption>Fig. 4. Regional happiness scores from 2011 to 2024</figcaption></figure><p>At a glance, we could say:</p><ul><li><strong>Africa:</strong> Consistently lowest scores, hovering around 4.</li><li><strong>Asia:</strong> Pretty steady, just above 5. There are no dramatic changes, but a tiny rise at the end and this maybe a sign of positive changes?</li><li><strong>Europe:</strong> On the rise! Starts just above 6 and keeps climbing, crossing 6.5 by 2024.</li><li><strong>North America:</strong> A bit of a rollercoaster, but seems to bounce back at the end.</li><li><strong>Oceania:</strong> Starts out as the happiest region but shows a slow, steady decline, dropping below 7 by 2024.</li><li><strong>South America:</strong> All over the place. We can see lots of volatility but no clear improvement.</li></ul><p>In short, Europe is on the rise, Africa and Asia are stuck in neutral, Oceania is slipping, North America is bouncing back, and South America is… still figuring things out!</p><p>So far, we’ve uncovered some interesting patterns, outliers, and regional differences. In Part 3, I’ll look at how countries are spread within each region, which ones had the sharpest jumps or drops between two years, and whether any of the happiness factors are strongly correlated.</p><p>Keep exploring, and enjoy the journey!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5f16fc44f65c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Happiness by the Numbers, Part 1: Unpacking the Data]]></title>
            <link>https://medium.com/@susandyhtun1/happiness-by-the-numbers-part-1-unpacking-the-data-c659cf4855a9?source=rss-33aa80f87909------2</link>
            <guid isPermaLink="false">https://medium.com/p/c659cf4855a9</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[happiness]]></category>
            <category><![CDATA[power-bi]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[su_sandy]]></dc:creator>
            <pubDate>Wed, 04 Jun 2025 12:24:38 GMT</pubDate>
            <atom:updated>2025-06-04T12:24:38.511Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OuNEJilLoHUKPf00sNsgAQ.jpeg" /><figcaption>A sneak peek at my Power BI report on global happiness — just a glimpse of the insights inside!</figcaption></figure><p>Recently, I joined a Power BI challenge hosted by “<a href="https://www.linkedin.com/posts/su-sandy-htun_211-and-counting-the-power-bi-boosting-activity-7335251166534004736--V-i?utm_source=share&amp;utm_medium=member_desktop&amp;rcm=ACoAABsl7lsBfY6Yy1aS6V9FdU1XVETv7ZA3Q8c">The Power BI: Boosting Inspiration Summer Edition</a>”, where the provided dataset focused on happiness scores and the elements affecting happiness across different countries from 2011 to 2024. Here’s a quick peek at the dataset 👇</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*cMao63vuk2INOmTJShU7fA.png" /><figcaption>Screenshot of the Global Happiness Index_data</figcaption></figure><p>A quick rundown:</p><ul><li><strong>Rank</strong>: the country’s position in the happiness ranking (lower = happier)</li><li><strong>Ladder Score</strong>: citizens’ average happiness level (higher = happier)</li><li><strong>Upper/Lower Whiskers</strong>: the possible range for each happiness score</li><li>The other columns are the factors influencing the ladder score</li></ul><p>The first things that caught my eye while exploring:</p><ul><li>There are no factor scores for 2011–2018 — just rank and ladder scores for those years.</li><li>With data from 2011 to 2024, there’s potential to look at the Covid years.</li><li>It’s interesting to see how a country’s happiness fluctuates over time, especially in places with political, social, or economic challenges.</li><li>Since only country names are provided, mapping them to regions would allow for some cool regional comparisons.</li><li>There are multiple factors affecting happiness — so, how do they interact, and what’s the average effect of each factor by country, region, or year?</li></ul><p>To bring in region context, I pulled region data from Our World In Data (latest: 2023). Here’s the <a href="https://ourworldindata.org/grapher/continents-according-to-our-world-in-data?tab=table">link to the regions table</a>.</p><p><strong>Data Model:</strong><br>I combined the given dataset (‘Global Happiness Index_data.xlsx’) with the region data. The Factor table stands alone and contains the seven key happiness factors — perfect for cross-filtering the fact data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/775/1*5QDjVchkvAl8NpD4O0c1RA.png" /><figcaption>Data Model</figcaption></figure><p>I also added a calculated column in the <em>‘Happiness Data</em>’ table called ‘<em>Rank group</em>’ with the below DAX formula:</p><pre>Rank group = <br>SWITCH (<br>    TRUE(),<br>    &#39;Happiness data&#39;[Rank] &lt;= 10, &quot;Top 10 Happiest&quot;,<br>    RANKX (<br>        FILTER (<br>            ALL ( &#39;Happiness data&#39; ),<br>            &#39;Happiness data&#39;[Year] = EARLIER ( &#39;Happiness data&#39;[Year] )<br>        ),<br>        ROUND ( &#39;Happiness data&#39;[Happiness score], 3 ),<br>        ,<br>        ASC,<br>        DENSE<br>    ) &lt;= 10, &quot;Bottom 10 Happiest&quot;,<br>    &quot;Middle&quot;<br>)</pre><p>This lets me label each country per year as Top 10 Happiest, Middle, or Bottom 10 Happiest based on their rank and happiness score.</p><p>Now that I’m familiar with the data and have my base model, my next step is to decide what questions I want to answer.</p><blockquote>Since this is a challenge (not a stakeholder request), I get to be my own audience — which is actually pretty fun! So I asked myself:<br> <em>What insights would I want to learn from this report? What would make me curious to play with it?</em></blockquote><p>Here are a few questions that came to mind:</p><ul><li>What are the top factors influencing happiness across countries and regions?</li><li>Which countries improved or declined the most in happiness over time — and what drove these changes?</li><li>What does happiness inequality look like, regionally and globally?</li><li>Are there visible global patterns or clusters of happiness (e.g., do the Nordic countries always come out on top)?</li></ul><p>In my next post, I’ll dive into how I tackled these questions through specific visuals, walk through some of the DAX behind the scenes, and share my thoughts on making a Power BI report both insightful and visually engaging.</p><p>Stay tuned!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c659cf4855a9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Model behind my Maven Slopes Challenge Entry]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@susandyhtun1/data-model-behind-my-maven-slopes-challenge-entry-aa802d2c967b?source=rss-33aa80f87909------2"><img src="https://cdn-images-1.medium.com/max/1373/1*0Go_Si2oH4LV0voGBdt-RA.png" width="1373"></a></p><p class="medium-feed-link"><a href="https://medium.com/@susandyhtun1/data-model-behind-my-maven-slopes-challenge-entry-aa802d2c967b?source=rss-33aa80f87909------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@susandyhtun1/data-model-behind-my-maven-slopes-challenge-entry-aa802d2c967b?source=rss-33aa80f87909------2</link>
            <guid isPermaLink="false">https://medium.com/p/aa802d2c967b</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-modeling]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[power-bi]]></category>
            <category><![CDATA[visualization]]></category>
            <dc:creator><![CDATA[su_sandy]]></dc:creator>
            <pubDate>Sun, 05 Mar 2023 18:49:08 GMT</pubDate>
            <atom:updated>2025-06-06T11:19:23.788Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[‘Machine Learning’ course: Recoding with Python — Part6: Neural Network Learning]]></title>
            <link>https://medium.com/@susandyhtun1/machine-learning-course-recoding-with-python-part6-neural-network-learning-50c1652452fb?source=rss-33aa80f87909------2</link>
            <guid isPermaLink="false">https://medium.com/p/50c1652452fb</guid>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[python3]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[su_sandy]]></dc:creator>
            <pubDate>Sun, 26 Jun 2022 19:18:46 GMT</pubDate>
            <atom:updated>2022-06-26T19:18:46.351Z</atom:updated>
            <content:encoded><![CDATA[<h3>‘Machine Learning’ course: Recoding with Python — Part6: Neural Network Learning</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RklNlzx_UU4HCEAZNfKK1w.jpeg" /><figcaption>Image Ref: ”<a href="https://www.freepik.com/vectors/coding">https://www.freepik.com/vectors/coding</a>&quot; Coding vector created by storyset — <a href="http://www.freepik.com/">www.freepik.com</a></figcaption></figure><p>This is the 6th article in this series where I try to recode the exercises in the (old) Machine Learning course by Andrew Ng (where the programming exercises are done using Octave). My intention in writing these articles is to assist the learners of this course to use Python as an alternative while doing the exercises. Please also feel free to explore the previous parts in this series too:<br><a href="https://medium.com/@susandyhtun1/machine-learning-course-by-andrew-ang-recoding-with-python-part1-f076c247bb6e">Part1: Linear Regression model with one feature</a><br><a href="https://medium.com/@susandyhtun1/machine-learning-course-by-andrew-ng-recoding-with-python-part2-a852eee972bd">Part2: Linear Regression with multiple features</a><br><a href="https://medium.com/@susandyhtun1/machine-learning-course-by-andrew-ng-recoding-with-python-part3-bd4718ac13ea">Part3: (Unregularized) Logistic Regression model</a><br><a href="https://medium.com/@susandyhtun1/machine-learning-course-by-andrew-ng-recoding-with-python-part4-regularized-logistic-eda839579694">Part4: Regularized Logistic Regression</a><br><a href="https://medium.com/@susandyhtun1/machine-learning-course-by-andrew-ng-recoding-with-python-part5-one-vs-all-classification-6f4613fafe66">Part5: One-vs-All Classification &amp; Neural Network</a></p><p>In Part5, we have already implemented a feedforward propagation for our neural network using the given trained parameters (for theta1 and theta2). In this exercise, we will implement the backpropagation algorithm to learn the parameters for the neural network and use it to predict the handwritten digits (the same dataset that we used in the previous exercise).</p><p>The dataset contains 5000 training examples of handwritten digits where each training example is a 20pixel by 20pixel grayscale image of the digit. So, our training dataset is a 5000 by 400 matrix. The second part of the training set is a 5000-dimensional vector y that contains labels for the training set. There are a total of 10 classes (‘1’, ‘2’, ‘3’, …., ‘10’). (Please note that the 0 digit is labeled as ‘10’ while the digits 1 to 9 are labeled as ‘1’ to ‘9’ in their natural order.) Please refer to Part5 for loading the dataset and visualizing it.</p><p>Below are the main steps we will implement in this exercise:<br>a) We will build two functions: one to calculate the cost (with regularization) using a feedforward propagation and another to calculate the gradient of theta using the back propagation.<br>b) To make sure that our backpropagation is implemented correctly, we will build a function to calculate the numerical gradient (i.e., to compute the gradient using ‘finite differences’ which gives us numerical estimates of the gradients) and use it to double-check our back propagation algorithm results. <br>c) Lastly, we will use an advanced optimization method to learn a good set of parameters and check the accuracy of the model (based on the training set).</p><p>Let’s skip the visualization of the given dataset (as we have already done it in Part5) and go straight to our model representation.</p><h3>1. Model Representation</h3><p>Similar to our model in Part5, we will use a neural network of 3 layers — an input layer, a hidden layer, and an output layer. In our input layer, we have total of 400 units (excluding the extra bias unit). The hidden layer has 25 units (excluding the bias unit) and the output layer has 10 units (corresponding to 10-digit classes).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tEgM_BZJBebFBeD1TxqmCA.jpeg" /><figcaption>Model Representation</figcaption></figure><h3>2. Regularized Cost Function</h3><p>The cost function for our neural network (composed of 400 units in the input layer, 25 units in the hidden layer, and 10 units in the output layer) with regularization is given by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/915/1*nrvcRPdGmYpry2ieLwc3DQ.png" /></figure><p>, where m = 5000 (number of training examples), K = 10 (total number of possible labels).</p><p>Also, please note that:<br>1) We should not be regularizing the terms that correspond to the bias. For Theta1 and Theta2, this corresponds to the first column of each matrix.<br>2) The original labels (in the variable y) were 1,2,…,10. For the purpose of training a neural network, we need to recode the labels as vectors containing only values of 0 or 1. For example, if X(i) is an image of the digit 5, then the corresponding y(i) should be a 10-dimensional vector having 1 in the fifth row and other elements equal to 0. <br>3) In the below code, the input ‘para’ is the vector obtained from unrolling the weight matrices Theta1 and Theta2.</p><pre>def sigmoid(z):<br>    g = 1/(1 + np.exp(-z))<br>    return g</pre><pre># NNCostFunc implements the neural network cost function for a 2 layer neural network which performs classification.<br># This function will compute the cost.</pre><pre>def NNCostFunc(para, input_layer_size, hidden_layer_size, labels, x,y,lambdaa):<br>    Theta1 = para[0: hidden_layer_size * (input_layer_size+1)].reshape((hidden_layer_size, input_layer_size+1))<br>    <br>    Theta2 = para[hidden_layer_size*(input_layer_size+1)::].reshape((labels, hidden_layer_size+1))<br>    <br>#Theta1 will be (25x401) matrix and Theta2 will be (10x26) matrix<br>    <br>    m = y.size #no.of training examples<br>    <br>    # Forward Propagation<br>    y_recode = np.zeros((m, labels)) <br>    for i in range(m):               <br>        y_recode[i, y[i]-1] = 1      <br># recode the original y vector to a 5000x10 matrix with only 0s and 1s. <br>                                     <br># column index 0 will represent label 1, index 1 is label 2,..., index 9 is label 10 (which is typically 0)<br>        <br>    a1 = np.concatenate((np.ones((len(x),1)),x), axis=1) <br>    z2 = np.dot(a1, np.transpose(Theta1))<br>    a2 = sigmoid(z2)<br>    a2 = np.concatenate((np.ones((m,1)), a2), axis=1)<br>    z3 = np.dot(a2, np.transpose(Theta2))<br>    a3 = sigmoid(z3)<br>    <br>    term1 = np.multiply(y_recode, np.log(a3))<br>    term2 = np.multiply((1-y_recode), np.log(1-a3))<br>    <br>    # regualarization term<br>    # here, we will not ignore the first columns of Theta1 and Theta2 which correspond to the bias terms.<br>    term3 = 0.5*lambdaa*(np.sum(np.sum(np.power(Theta1[:,1:],2))) + np.sum(np.sum(np.power(Theta2[:,1:],2)))) <br>    J     = (np.sum((-term1 - term2)) + term3)/m<br>    return J</pre><h3>3. Back Propagation</h3><p>Now, we will implement the back propagation algorithm to compute the gradient for the neural network cost function.</p><p>Below are the steps for back propagation algorithm:</p><ol><li>Given a training example (X(i), y(i)), we will first run a forward pass to compute all the activations throughout the network, including the output value of the hypothesis h(x).</li><li>For each output unit k in layer 3 (the output layer), set δ(3) = a(3)-y.</li><li>Then, for each node in the hidden layer, we will compute an error term δ that measures how much that node was responsible for any errors in our output.</li><li>Accumulate the gradients calculated using the formula: (Note that we should skip or remove δ for the bias term in the hidden layer.) Repeat Steps 1 to 4 for all the training examples.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/337/1*W5QPygXDaxpAsxwUnMoW5Q.png" /></figure><p>5. Obtain the (regularized) gradient for the neural network cost function by using the formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*pzDglOgfwDe3F420rHkoqA.png" /></figure><pre># NNGradientFunc implements the neural network back propagation to compute the gradient of theta for a 2 layer neural network</pre><pre>def NNGradientFunc(para, input_layer_size, hidden_layer_size, labels, x,y,lambdaa):<br>    <br>    Theta1 = para[0: hidden_layer_size * (input_layer_size+1)].reshape((hidden_layer_size, input_layer_size+1))<br>    <br>    Theta2 = para[hidden_layer_size*(input_layer_size+1)::].reshape((labels, hidden_layer_size+1))<br>    <br>    Theta1_grad = np.zeros(Theta1.shape) <br>    Theta2_grad = np.zeros(Theta2.shape) <br>    <br>    m = y.size <br>    n = x.shape[1] </pre><pre>    y_recode = np.zeros((m, labels)) <br>    for i in range(m):               <br>        y_recode[i, y[i]-1] = 1      <br>    <br># Compute Gradient using Back Propagation<br>    for i in range(m):<br>        a_1 = np.transpose(x[i,:].reshape((1,n)))  <br>        a_1 = np.insert(a_1,0,1,axis=0)            <br>        z_2 = np.dot(Theta1, a_1)                  <br>        a_2 = sigmoid(z_2)<br>        a_2 = np.insert(a_2,0,1,axis=0)            <br>        z_3 = np.dot(Theta2, a_2)                  <br>        a_3 = sigmoid(z_3)<br>        <br>        y_i = np.transpose(y_recode[i,:].reshape(1,labels)) <br>        <br>        delta3 = a_3 - y_i    <br>        delta2 = np.multiply(np.dot(np.transpose(Theta2),delta3), sigmoidGradient(np.insert(z_2,0,1,axis=0))) <br>        delta2 = delta2[1:,:] # leave out the delta for the bias term. No need to calculate delta0.<br>        <br>        Theta2_grad += np.dot(delta3, np.transpose(a_2))   <br>        Theta1_grad += np.dot(delta2, np.transpose(a_1))   <br>        <br>        <br>    Theta1_grad = (1/m) * Theta1_grad <br>    Theta2_grad = (1/m) * Theta2_grad <br>    <br>    # Adding regularization term<br>    Theta1_grad[:,1:] += (lambdaa/m) * Theta1[:,1:] # for j&gt;=1<br>    Theta2_grad[:,1:] += (lambdaa/m) * Theta2[:,1:] # for j&gt;=1<br>    <br>    # unroll gradients<br>    Gradients = np.concatenate((Theta1_grad.ravel(), Theta2_grad.ravel()), axis=0)<br>        <br>    return Gradients</pre><h3>4. Gradient Checking</h3><p>Suppose we have a function f(ϴ) that purportedly computes partial derivatives of J(ϴ) w.r.t. ϴ; we’d like to check if f is outputting corret derivative values. We will numerically verify f(ϴ) correctedness by checking, for each i, by using the below formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/346/1*P9xxV6o39LDzTzs-P2o5qg.png" /></figure><p>where ϴ(i+) is where the i-th element has been incremented by ε, ϴ(i-) is where the i-th element decreased by ε. (By assuming ε = 0.0001, we’ll usually find that the left- and right-hand sides of the above equation will agree to at least 4-significant digits).</p><p>Below is the code to calculate the numerical estimate of the gradient:</p><pre>def NumericalGradient(para, input_layer_size, hidden_layer_size, labels, x,y,lambdaa):<br>    gradApprox = np.zeros((para.size))<br>    e = 1e-4<br>    l = lambdaa<br>    <br>    for i in range(para.size):<br>        thetaPlus = para.copy()<br>        thetaPlus[i] += e<br>        JPlus = NNCostFunc(thetaPlus, input_layer_size, hidden_layer_size, labels, x,y,l)<br>        <br>        thetaMinus = para.copy()<br>        thetaMinus[i] -= e<br>        JMinus = NNCostFunc(thetaMinus, input_layer_size, hidden_layer_size, labels, x,y,l)<br>        <br>        gradApprox[i] = (JPlus - JMinus)/ (2*e)  <br>         <br>    return gradApprox</pre><p>Let’s perform our gradient checking by comparing the outputs from our NNGradientFunc() and NumericalGradient() functions. Here, for better efficiency, we will use a small neural network taking a random subset of X and y and initializing the random weights for theta1 and theta2.</p><pre># This function randomly initializes the weights of a layer with L_in incoming connections and L_out outgoing connections.</pre><pre>def randInitialWeights(L_in, L_out):<br>    epsilon = 0.12<br>    weights = np.random.random((L_out, L_in+1)) * 2 * epsilon - epsilon<br>    return weights</pre><pre># This function creates a small neural network to check the backpropagation gradients.<br># It will output the analytical gradients produced by our backprop code and the numerical gradients (computed using NumericalGradient() function)<br># These two gradient computations should result in very similar values.</pre><pre>def checkNNGradients(lambdaa):<br>    input_layer_size = 3<br>    hidden_layer_size = 5<br>    labels = 3<br>    m = 5<br>    <br>    theta1 = randInitialWeights(input_layer_size, hidden_layer_size) <br>    theta2 = randInitialWeights(hidden_layer_size, labels)           <br>    <br>    # Random subset of data from X and y<br>    idx = np.random.randint(5000, size = 5)<br>    xx = X[idx, 0:3]<br>    yy = 1 + np.transpose(np.mod([i for i in range(m)], labels)) <br>    <br>    # unroll parameters<br>    parameters = np.concatenate((theta1.ravel(), theta2.ravel()), axis=0)<br>    <br>    grad       = NNGradientFunc(parameters, input_layer_size, hidden_layer_size, labels, xx,yy,lambdaa)<br>    numgrad    = NumericalGradient(parameters, input_layer_size, hidden_layer_size, labels, xx,yy,lambdaa)<br>    <br>    # The two columns printed should be very similar<br>    print(&quot;\n&quot;.join(&quot;{:.5f}  ,{:.5f}&quot;.format(x, y) for x, y in zip(numgrad, grad)))<br>    print(&quot;\nLeft- Your Numerical Gradient, Right- Analytical Gradient&quot;)<br>    <br>    diff = np.linalg.norm(grad-numgrad)/ (np.linalg.norm(numgrad) + np.linalg.norm(grad))<br>    print(&quot;\nIf the backpropagation implementation is correct, then\n the relative difference will be small (less than 1e-9).\n Relative Difference : {}\n&quot;.format(diff))<br></pre><pre># checking the two gradients using lambda = 1<br>checkNNGradients(1)</pre><p>As theta1 is a (5,4) matrix and theta2 is a (3,6) matrix, the output unrolled parameters will have a total of 38 values. A snapshot of the output is as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*vwAV8GZQn-Eb4w-vZFmznA.png" /></figure><p>Congrats! Our backpropagation is implemented correctly and now, we can use it for training our parameters with peace of mind.</p><h3>5. Learning Parameters</h3><p>We will use an advanced optimization method fmin_cg() to learn the parameters and use them to check our model accuracy using the training set. Here, I have limited the maxiter parameter to 50 (and I get a training accuracy of 95.3%). It is possible to get higher training accuracies by training the neural network with more iterations (if we use lambda = 1 and maxiter = 100, we can get a 98.6% accuracy).</p><pre>import scipy.optimize as opt</pre><pre>input_layer = 400<br>hidden_layer = 25<br>labels = 10<br>lambdaa = 1</pre><pre>theta_ = opt.fmin_cg(NNCostFunc, x0=initial_parameters, args = (input_layer,hidden_layer,labels,X,y,lambdaa), fprime = NNGradientFunc,maxiter = 50) # Taking so long</pre><pre># Reshaping<br>Theta1_ = theta_[0: hidden_layer * (input_layer+1)].reshape((hidden_layer, input_layer+1))<br>Theta2_ = theta_[hidden_layer*(input_layer+1)::].reshape((labels, hidden_layer+1))</pre><pre># Prediction<br>def predict(t1,t2,x):<br>    m = x.shape[0]<br>    labels = t2.shape[0]<br>    <br>    a1 = np.concatenate((np.ones((len(x),1)),x), axis=1)<br>    z2 = np.dot(a1, np.transpose(t1))<br>    a2 = sigmoid(z2)<br>    a2 = np.concatenate((np.ones((m,1)), a2), axis=1)<br>    z3 = np.dot(a2, np.transpose(t2))<br>    a3 = sigmoid(z3)<br>    <br>    hmax    = np.amax(a3, axis=1)   <br>    prediction = np.argmax(a3, axis =1 )+1  <br>    prediction = prediction.reshape((prediction.shape[0],1)) <br>    <br>    return prediction</pre><pre>pred = predict(Theta1_, Theta2_,X)<br>print(&quot;Training set accuracy : {}%&quot;.format(np.mean((pred == y).astype(float))*100))</pre><p>Congratulations again! We have successfully implemented a neural network model for labeling the handwritten digits and it gives us a good accuracy. Keep looking forward to Part7 where we will discuss the tradeoff between bias and variance using a linear regression model.</p><p>Keep Learning. Enjoy the journey!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=50c1652452fb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[‘Machine Learning’ course by Andrew Ng: Recoding with Python — Part5: One-vs-All Classification &…]]></title>
            <link>https://medium.com/@susandyhtun1/machine-learning-course-by-andrew-ng-recoding-with-python-part5-one-vs-all-classification-6f4613fafe66?source=rss-33aa80f87909------2</link>
            <guid isPermaLink="false">https://medium.com/p/6f4613fafe66</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[python3]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[classification]]></category>
            <dc:creator><![CDATA[su_sandy]]></dc:creator>
            <pubDate>Sun, 19 Jun 2022 09:45:08 GMT</pubDate>
            <atom:updated>2022-06-19T11:34:52.830Z</atom:updated>
            <content:encoded><![CDATA[<h3>‘Machine Learning’ course: Recoding with Python — Part5: One-vs-All Classification &amp; Neural Network</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rD0UZElTf1vJbghpqRbAww.jpeg" /><figcaption>Image Ref: ”<a href="https://www.freepik.com/vectors/python-language">https://www.freepik.com/vectors/python-language</a>&quot;&gt;Python language vector created by svstudioart — <a href="http://www.freepik.com/">www.freepik.com</a></figcaption></figure><p>This is the 5th article in this series where I try to recode the exercises in the (old) Machine Learning course by Andrew Ng (where the programming exercises are done using Octave). My intention in writing these articles is to assist the learners of this course to use Python as an alternative while doing the exercises. Please also feel free to explore the previous parts in this series too:<br><a href="https://medium.com/@susandyhtun1/machine-learning-course-by-andrew-ang-recoding-with-python-part1-f076c247bb6e">Part1: Linear Regression model with one feature</a><br><a href="https://medium.com/@susandyhtun1/machine-learning-course-by-andrew-ng-recoding-with-python-part2-a852eee972bd">Part2: Linear Regression with multiple features</a><br><a href="https://medium.com/@susandyhtun1/machine-learning-course-by-andrew-ng-recoding-with-python-part3-bd4718ac13ea">Part3: (Unregularized) Logistic Regression model</a><br><a href="https://medium.com/@susandyhtun1/machine-learning-course-by-andrew-ng-recoding-with-python-part4-regularized-logistic-eda839579694">Part4: Regularized Logistic Regression</a></p><p>For this exercise, we are given a dataset containing 5000 training examples of handwritten digits where each training example is a 20pixel by 20pixel grayscale image of the digit. Each pixel is represented by a floating-point number indicating the grayscale intensity at that location. So, our training dataset is a 5000 by 400 matrix. The second part of the training set is a 5000-dimensional vector y that contains labels for the training set. There are a total of 10 classes (‘1’, ‘2’, ‘3’, …., ‘10’). Please note that the ‘0’ digit is labeled as ‘10’ while the digits ‘1’ to ‘9’ are labeled as ‘1’ to ‘9’ in their natural order.</p><h3>1. Getting to know the data</h3><p>Let’s first take a look at the dataset given.</p><pre># load the data</pre><pre>import scipy.io<br>import pandas as pd<br>import numpy as np<br>import matplotlib.pyplot as plt</pre><pre>data = scipy.io.loadmat(&#39;ex3data1.mat&#39;)<br>data</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*INrUq-f6Z9UHaGDnKFu_bg.png" /><figcaption>‘X’ has all the feature columns. ‘y’ has the target column. Both contain 5000 training examples.</figcaption></figure><p>To visualize the data, I will create a function called displayData() which will take two parameters: the training dataset and the number of subplots (on one axis). P.S. For now, you can ignore the first part of the ‘if’ statement which is to display the predicted result in the last part of the article.</p><pre>def displayData(X, subplot):<br>    <br>    width = int(round(math.sqrt(X.shape[1])))<br>    m, n = X.shape<br>    height = int(n/width)<br>    <br>    # create subplots<br>    fig, axarr = plt.subplots (subplot,subplot, figsize=(6,6),<br>                              gridspec_kw = {&#39;wspace&#39;:0, &#39;hspace&#39;:0})<br>    <br>    # this part is for visualizing the training and prediciton in the last part.<br>    if subplot == 1:<br>        pixels = X<br>        pixels = pixels.reshape(width, height)<br>        axarr.imshow(pixels.T, cmap = &#39;gray_r&#39;)<br>        axarr.set_xticks([]) # remove the ticks<br>        axarr.set_yticks([])<br>    <br>    # this part is for showing random digits from X<br>    else:<br>        for i in range(subplot):<br>            for j in range(subplot):<br>                random_index = np.random.choice(len(X))<br>                pixels = X[random_index]   <br>                pixels = pixels.reshape(width, height)<br>                axarr[i,j].imshow(pixels.T, cmap =&#39;gray_r&#39;)<br>                axarr[i,j].set_xticks([]) # remove the ticks<br>                axarr[i,j].set_yticks([])<br>    plt.show()</pre><pre>X = data[&#39;X&#39;]<br>y = data[&#39;y&#39;]<br>displayData(X, 10)</pre><p>This will randomly select 100 rows from X and display them in a figure with 100 subplots as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/509/1*m7KbT7aQlGNT3R2_TtFiGQ.png" /></figure><h3>2. Cost Function and Gradient (with Regularization)</h3><p>The dataset contains more than two labels, and therefore, we will be using multiple one-vs-all logistic regression models to build a multi-class classifier. Since there are 10 classes (labels ‘1’ to ‘10’), we will train 10 separate logistic regression classifiers. Before implementing the multi-class classifier, let’s first define the necessary functions: the sigmoid function, Cost function, and gradient function.</p><p><strong>Regularized Cost Function</strong></p><p>Here, for both the cost function and gradient function, don’t forget to add a column of 1s to X for the bias term. While finding the gradient of the cost, please take note that we are not to regularize the theta value for the bias term. Please refer to <a href="https://medium.com/@susandyhtun1/machine-learning-course-by-andrew-ng-recoding-with-python-part4-regularized-logistic-eda839579694">Part4 </a>for the mathematical equations for both functions.</p><pre># this function will compute cost of using theta as the parameter for regularized logistic regression</pre><pre>def sigmoid(z):<br>    g = 1/(1 + np.exp(-z))<br>    return g</pre><pre>def J(theta, x, y, lambdaa):<br>    J = 0<br>    m = len(y)<br>    theta = theta.reshape((theta.shape[0],1))<br>    <br>    x = np.concatenate((np.ones((len(x),1)), x), axis = 1)  # add ones to the X: (5000x401) matrix<br>    <br>    y_prime = np.transpose(y)<br>    x_prime = np.transpose(x)<br>    h_theta = sigmoid(np.dot(x,theta))<br>    <br>    term_1 = (np.dot(y_prime, np.log(h_theta)))<br>    term_2 = (np.dot(np.transpose(1-y), np.log(1-h_theta)))<br>    term_3 = np.array(0.5*lambdaa*np.sum(np.power(theta[1::],2)))<br>    J     = (-term_1 -term_2 + term_3)/m<br>    J = np.sum(J)<br>    return J</pre><p><strong>Gradient of the Cost (Regularized)</strong></p><pre>def Gradient(theta, x, y, lambdaa):<br>    m = len(y)<br>    theta = theta.reshape((theta.shape[0],1))<br>    <br>    x = np.concatenate((np.ones((len(x),1)), x), axis = 1)  # add ones to the X: (5000x401) matrix<br>    <br>    j = np.ones((x.shape[1],1))<br>    j[0] = 0 #we will not regularize for j=0</pre><pre>    grad = (np.dot(np.transpose(x), sigmoid(np.dot(x,theta))-y ) +  lambdaa* np.multiply(j,theta))/m<br>    return grad</pre><h3>3. One-vs-All Classification</h3><p>We will implement one-vs-all classification by training 10 regularized logistic regression classifiers, one for each of the K classes (in our dataset, K = 10).<br>All the classifier parameters will be returned in a matrix as below where each row corresponds to the learned logistic regression parameters for one class.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/271/1*HDFs42RmF_4Rs7MXD8lyhA.png" /></figure><p>For our case, the below function will return a (10,401) matrix where each row represents theta values for each label (e.g., row with index0 represents theta values for label ‘1’, index1 for label ‘2’,…, index9 for label ‘10’).</p><pre>import scipy.optimize as opt<br>from scipy.optimize import minimize</pre><pre>def oneVsall(X,y,labels,lambdaa):<br>    m = X.shape[0]  # no. of training examples: 5000<br>    n = X.shape[1]  # no. of features: 400<br>    all_theta = np.zeros((labels,n+1)) <br>    <br>    for i in range(1, labels+1):<br>        y_ = (y==i).astype(int)<br>        theta_start = np.zeros((n+1,1)) <br>        <br>        theta_ = opt.fmin_tnc(func=J, x0=theta_start, fprime = Gradient, args = (X,y_,lambdaa)) #-&gt; result<br>        <br>        all_theta[i-1,:] = theta_[0]<br>    return all_theta</pre><pre>labels = 10 <br>lambdaa = 0.1<br>result = oneVsall(X,y,labels,lambdaa)</pre><h3>4. Prediction</h3><p>We can now use our one-vs-all classifier to predict the digit contained in a given image. For each input, we will compute the ‘probability’ that it belongs to each class using the trained classifiers. The one-vs-all prediction function will pick the class for which the corresponding logistic regression classifier outputs the highest probability and return the class label (1,2,…,10) as the prediction.</p><pre># the labels are in the range (1 to K) where K = 10.</pre><pre>def predict(all_theta, x):<br>    labels = all_theta.shape[0] <br>    <br>    h_theta = sigmoid(np.dot(x,np.transpose(all_theta)))  <br>    hmax    = np.amax(h_theta, axis=1)   <br>    prediction = np.argmax(h_theta, axis =1 )+1  <br>    prediction = prediction.reshape((prediction.shape[0],1)) # just reshaping into similar shape as y<br>    <br>    return prediction</pre><pre>X_ = np.concatenate((np.ones((len(X),1)), X), axis = 1)  # add ones to the X: (5000x401) matrix<br>pred = predict(result, X_)</pre><pre>print(&quot;Training set accuracy : {}%&quot;.format(np.mean((pred == y).astype(float))*100))</pre><p>The resulted ‘pred’ is a (5000,1) matrix containing all the predicted results for the 5000 training examples. The training set accuracy obtained is 96.46%.</p><h3>5. Multi-class Logistic Regression vs. Neural Network</h3><p>We have implemented multi-class logistic regression to recognize handwritten digits. However, logistic regression cannot form more complex hypotheses as it is only a linear classifier (we can add more features such as polynomial features to logistic regression, but that can be very expensive to train). So, we will now implement a neural network to recognize handwritten digits using the same training set before. The neural network will be able to represent complex models that form non-linear hypotheses.</p><p>For this part of the exercise, we are given the weights (already trained) and we just need to implement the feedforward propagation algorithm for prediction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tEgM_BZJBebFBeD1TxqmCA.jpeg" /><figcaption>Model Representation</figcaption></figure><p>Our neural network has 3 layers — an input layer, a hidden layer, and an output layer. We have 400 units in our input layer (excluding the extra bias unit which always outputs +1). We have a set of trained parameters for ϴ(1) and ϴ(2) (which are stored in ex3weights.mat). The parameters have dimensions that are sized for a neural network with 25 units in the second layer and 10 units in the output layer (corresponding to 10 classes).</p><pre>weights = scipy.io.loadmat(&#39;ex3weights.mat&#39;)<br>Theta1 = weights[&#39;Theta1&#39;]<br>Theta2 = weights[&#39;Theta2&#39;]</pre><p>Here, Theta1 is a (25,401) matrix and Theta2 is a (10,26) matrix.</p><h3>5.1 Feedforward Propagation and Prediction</h3><pre># This function will predict the label of an input given a trained neural network</pre><pre>def predict2(theta1, theta2, x):<br>    x = np.concatenate((np.ones((len(x),1)), x), axis = 1)  # add ones to the X: (5000x401) matrix<br>    <br>    m = x.shape[0]<br>    labels = theta2.shape[0] <br>    <br>    a1 = x  # we have already added bias colunm to x <br>    <br>    z2 = np.dot(a1, np.transpose(theta1))<br>    a2 = sigmoid(z2)<br>    a2 = np.concatenate((np.ones((m,1)), a2), axis=1)    <br>    z3 = np.dot(a2, np.transpose(theta2)) <br>    a3 = sigmoid(z3)<br>    <br>    hmax    = np.amax(a3, axis=1)   <br>    prediction = np.argmax(a3, axis =1 ) + 1 <br>    prediction = prediction.reshape((prediction.shape[0],1)) # just reshaping into similar shape as y<br>    <br>    return prediction</pre><pre>pred = predict2(Theta1, Theta2, X)<br>print(&quot;Training set accuracy : {}%&quot;.format(np.mean((pred == y).astype(float))*100))</pre><p>The training set accuracy obtained from this neural network is 97.52% (which is a little bit higher than our previous one-vs-all classifiers).</p><p>We can use the below code to display the image of the training set (one at a time) together with its predicted label.</p><pre># to display images from the training set one at a time</pre><pre>i = np.random.randint(5000)<br>train_eg = X[i,:].reshape((1,X[i,:].shape[0]))<br>pred = predict2(Theta1, Theta2, train_eg)<br>print(&quot;Neural Network Prediction : {} (digit {})\n&quot;.format(pred, pred%10))<br>displayData(train_eg,1)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/547/1*fzOucb688-IINL5KD2M6Zg.png" /><figcaption>Here, the predicted label is ‘10’ which represents the digit ‘0’. And the image corresponding to the particular training example is displayed.</figcaption></figure><p>This is all for this part. In the upcoming Part6, we will train our own parameters using the cost function and the gradients obtained from the neural network backpropagation (using the same dataset).</p><p>Keep Learning. Enjoy the journey!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6f4613fafe66" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[‘Machine Learning’ course by Andrew Ng: Recoding with Python — Part4: Regularized Logistic…]]></title>
            <link>https://medium.com/@susandyhtun1/machine-learning-course-by-andrew-ng-recoding-with-python-part4-regularized-logistic-eda839579694?source=rss-33aa80f87909------2</link>
            <guid isPermaLink="false">https://medium.com/p/eda839579694</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[python3]]></category>
            <category><![CDATA[logistic-regression]]></category>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[su_sandy]]></dc:creator>
            <pubDate>Sat, 18 Jun 2022 20:42:31 GMT</pubDate>
            <atom:updated>2022-06-18T20:42:31.639Z</atom:updated>
            <content:encoded><![CDATA[<h3>‘Machine Learning’ course by Andrew Ng: Recoding with Python — Part4: Regularized Logistic Regression</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h5t8bNTijLVc8qNFW6XOMQ.jpeg" /><figcaption>Image Ref: ”<a href="https://www.freepik.com/vectors/computer-cartoon">https://www.freepik.com/vectors/computer-cartoon</a>&quot; Computer cartoon vector created by svstudioart — <a href="http://www.freepik.com/">www.freepik.com</a></figcaption></figure><p>This is the fourth article in this series where I try to recode the exercises in the (old) Machine Learning course by Andrew Ng (where the programming exercises are done using Octave). My intention in writing these articles is to assist the learners of this course to use Python as an alternative while doing the exercises. Please also feel free to explore the previous parts in this series too:<br><a href="https://medium.com/@susandyhtun1/machine-learning-course-by-andrew-ang-recoding-with-python-part1-f076c247bb6e">Part1: Linear Regression model with one feature</a><br><a href="https://medium.com/@susandyhtun1/machine-learning-course-by-andrew-ng-recoding-with-python-part2-a852eee972bd">Part2: Linear Regression with multiple features</a><br><a href="https://medium.com/@susandyhtun1/machine-learning-course-by-andrew-ng-recoding-with-python-part3-bd4718ac13ea">Part3: (Unregularized) Logistic Regression model</a></p><p>In this Part4, we will build a regularized logistic regression model to predict whether microchips from a fabrication plant pass the quality assurance (QA) test. Suppose that we are the product managers of the factory and we have the test results for some microchips on two different tests. From these two tests, we would like to determine whether the microchips should be accepted or rejected.</p><h3>1. Getting to know the data</h3><p>The dataset given contains 118 records with two features (Test1 score and Test2 score) and 1 target column (here, 1 means the microchip passes the QA test and is accepted while 0 means it’s rejected). As usual, I will load the data and try to visualize the data to get an idea of what it looks like.</p><pre>import pandas as pd<br>import numpy as np<br>import matplotlib.pyplot as plt</pre><pre>data = pd.read_table(&#39;ex2data2.txt&#39;, header = None, sep=&#39;,&#39;)<br>data.columns =[&#39;Test1&#39;, &#39;Test2&#39;, &#39;Score&#39;]</pre><pre>def plotdata(data):<br>    one_index = data.index[data[&#39;Score&#39;] ==1].tolist()<br>    zero_index = data.index[data[&#39;Score&#39;] ==0].tolist()</pre><pre>    plt.plot(data.iloc[one_index, 0], data.iloc[one_index, 1], &#39;k+&#39;, linewidth = 2, markersize = 7)<br>    plt.plot(data.iloc[zero_index, 0], data.iloc[zero_index, 1], &#39;ko&#39;, markersize = 7)<br>    plt.xlabel(&#39;Test1&#39;)<br>    plt.ylabel(&#39;Test2&#39;)<br>    plt.legend([&#39;Accepted&#39;, &#39;Rejected&#39;])<br>    plt.show()</pre><pre>plotdata(data)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/619/1*cgkO9PkzuFnJ8VB9jDY_-w.png" /><figcaption>The accepted microchips are black dots and the rejected chips are plus symbols.</figcaption></figure><p>From the above plot, we can see the decision boundary cannot be represented by a linear line anymore.</p><h3>2. Feature Mapping</h3><p>One way to fit the data better is to create more features from each data point. We will map the features into all polynomial terms of x1 and x2 up to the sixth power. The result of the mapfeature() function will be something like the below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/463/1*nUD4oSntKgWlCDsAVWLrPg.png" /></figure><pre># This function will map the two input features to quadratic features used in the regularization exercise.<br># Will return a new feature array with more features.<br># input x1 and x2 must be of the same size.</pre><pre>def mapfeature(x1,x2,degree):<br>    <br>    m = x1.shape[0]<br>    # the bias term on the first column<br>    result = np.ones((x1.shape[0])).reshape(m,1)<br>    <br>    for i in range(1, degree+1):<br>        for j in range(i+1):<br>            out = np.multiply(np.power(x1,(i-j)),np.power(x2,j)).reshape(m,1)<br>            result = np.concatenate((result, out), axis =1)<br>            <br>    return result</pre><pre>m = len(data.iloc[:, 2])<br>y = data.iloc[:, 2].to_numpy().reshape(m,1)<br>x_ = data.iloc[:, :2].to_numpy()</pre><pre>x = mapfeature(x_[:,0], x_[:,1], 6)</pre><p>The final mapped result x is a (118,28) matrix now. A logistic regression classifier trained on this higher-dimension feature matrix will have a more complex decision boundary. With feature mapping, we can build a more expressive classifier but overfitting can become a problem.</p><h3>3. Cost Function and Gradient (with Regularization)</h3><p>The regularized cost function of logistic regression is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/1*qkMOVfPkfGifGAPRNJCJuA.png" /></figure><p>And the gradient of the cost function is defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/672/1*xnb8HDR-sgYcXAqDsOVcOA.png" /></figure><p>Below is the python code for the sigmoid function, regularized cost function, and gradient function.</p><pre># this function will compute cost of using theta as the parameter for regularized logistic regression</pre><pre>def sigmoid(z):<br>    g = 1/(1 + np.exp(-z))<br>    return g<br></pre><pre>def J_reg(theta, x, y, lambdaa):<br>    m = len(y)<br>    y_prime = np.transpose(y)<br>    x_prime = np.transpose(x)<br>    h_theta = sigmoid(np.dot(x,theta))<br>    <br>    J = (-np.dot(y_prime, np.log(h_theta)) -  np.dot(np.transpose(1-y), np.log(1-h_theta)) <br>        + 0.5*lambdaa*np.sum(np.power(theta[1::],2)))/ m<br>    J = np.sum(J)<br>    <br>    return J<br></pre><pre>def Gradient(theta, x, y, lambdaa):<br>    m = len(y)<br>       <br>    error = sigmoid(np.dot(x,theta)) - y<br>    grad = np.zeros((x.shape[1], 1))<br>    <br>    for i in range(x.shape[1]):<br>        if i==0:<br>            grad[i] = (np.dot(np.transpose(x[:,i]), error))/ m<br>        else:<br>            grad[i] = (np.dot(np.transpose(x[:,i]), error))/ m + theta[i]*lambdaa/m<br>   <br>    return grad.flatten()</pre><h3>4. Learning parameters using Newton-Conjugate Gradient</h3><p>I have used this fmin_ncg() function (an unconstrained minimization using the Newton-CG method) from scipy.optimize in my previous logistic regression model (without regularization) as well. The details of the function can be read <a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.fmin_ncg.html">here</a>.</p><p>The required parameters for this function are:</p><ol><li>An objective function to be minimized. (this is our cost function J(ϴ)).</li><li>Initial guess of the parameters to be optimized i.e., our initialized ϴ values.</li><li>Gradient of J(ϴ)</li><li>maxiter: maximum number of iterations to perform</li></ol><p>The function will return the optimized ϴ values (together with other information).</p><pre>import scipy.optimize as opt</pre><pre>theta_start = np.zeros((x.shape[1], 1))<br>lambdaa=1</pre><pre>result = opt.fmin_ncg(J_reg, x0=theta_start, fprime = Gradient, args = (x,y.flatten(), lambdaa), maxiter = 400)</pre><pre>print(&quot;Cost at theta found by fmin_ncg() : {}\n&quot;.format(J_reg(result,x,y,lambdaa)))<br>print(&#39;Theta : {}&#39;.format(result))</pre><p>The result is as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*rg9C6VvddKboNP60-UhweQ.png" /></figure><h3>5. Prediction Accuracy</h3><p>Let’s evaluate the quality of the parameters we have found by asking the model to predict our training set data.</p><pre># Function for predicting<br># this will predict whether the label is 1 or 0 using the leraned logistic regression parameters theta</pre><pre>def predict(theta, x):<br>    m = len(x)<br>    p = np.zeros((m,1))<br>    for i in range(m):<br>        sig = sigmoid(np.dot(x[i],theta))<br>        if sig &gt; 0.5:<br>            p[i] = 1<br>        else:<br>            p[i] = 0<br>    return p</pre><pre>p = predict(result,x)<br>print(&quot;Train Accuracy : {}%&quot;.format(int(sum(p==y))))</pre><p>From this, our train accuracy is 98%.</p><p>This is all for this part. In the upcoming Part5, we will build multiple one-vs-all logistic regression models to implement a multi-class classifier and compare this model performance with the performance of a simple neural network.</p><p>Keep Learning. Enjoy the journey!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eda839579694" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[‘Machine Learning’ course by Andrew Ng: Recoding with Python — Part3]]></title>
            <link>https://medium.com/@susandyhtun1/machine-learning-course-by-andrew-ng-recoding-with-python-part3-bd4718ac13ea?source=rss-33aa80f87909------2</link>
            <guid isPermaLink="false">https://medium.com/p/bd4718ac13ea</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[logistic-regression]]></category>
            <category><![CDATA[python3]]></category>
            <dc:creator><![CDATA[su_sandy]]></dc:creator>
            <pubDate>Tue, 07 Jun 2022 17:52:27 GMT</pubDate>
            <atom:updated>2022-06-07T17:52:27.934Z</atom:updated>
            <content:encoded><![CDATA[<h3>‘Machine Learning’ course by Andrew Ng: Recoding with Python — Part3</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ljbFji53JQfmkuAURSDGvw.jpeg" /><figcaption>Image by ”<a href="https://www.freepik.com/vectors/homework">https://www.freepik.com/vectors/homework</a>&quot;&gt;Homework vector created by svstudioart — <a href="http://www.freepik.com/">www.freepik.co</a>m</figcaption></figure><p>This is the third article in this series where I try to recode the exercises in the (old) Machine Learning course by Andrew Ng (where the programming exercises are done using Octave). My intention in writing these articles is to assist the learners of this course to use Python as an alternative while doing the exercises. Please also feel free to explore <a href="https://medium.com/@susandyhtun1/machine-learning-course-by-andrew-ang-recoding-with-python-part1-f076c247bb6e">Part1</a> and <a href="https://medium.com/@susandyhtun1/machine-learning-course-by-andrew-ng-recoding-with-python-part2-a852eee972bd">Part2</a>.</p><p>In this Part3, we will build a logistic regression model to predict whether a student gets admitted into a university or not. Suppose we are the administrator of a department at a university and we want to determine each applicant’s chance of admission based on their results on two exams. We already have the historical data from previous applicants’ scores on these exams and the result of the admission.</p><h3><strong>1. Getting to know the data</strong></h3><p>The dataset given contains 100 records with two features (exam1_score and exam2_score) and 1 target column (admission result). As usual, I will load the data and try to visualize the data to get an idea of what it looks like.</p><pre>import pandas as pd<br>import numpy as np<br>import matplotlib.pyplot as plt</pre><pre>data = pd.read_table(&#39;ex2data1.txt&#39;, header = None, sep=&#39;,&#39;)<br>data.columns =[&#39;exam1&#39;, &#39;exam2&#39;, &#39;admission&#39;]<br>data.head()</pre><p>Below is the snippet of the dataset:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/357/1*akZ5bmKMRsTvFcuDxfnTLA.png" /></figure><p>In the below code, ‘plotdata()’ function is written to draw a scatter plot of our data. From the plot, we can see that the admitted students have a total score (the sum of exam1 and exam2 scores) of more than 100. Those who have a total score lower than 100 are not admitted.</p><pre>def plotdata(data):<br>    one_index = data.index[data[&#39;admission&#39;] ==1].tolist()<br>    zero_index = data.index[data[&#39;admission&#39;] ==0].tolist()</pre><pre>    plt.plot(data.iloc[one_index, 0], data.iloc[one_index, 1], &#39;k+&#39;, linewidth = 2, markersize = 7)<br>    plt.plot(data.iloc[zero_index, 0], data.iloc[zero_index, 1], &#39;ko&#39;, markersize = 7)<br>    plt.xlabel(&#39;Exam 1 score&#39;)<br>    plt.ylabel(&#39;Exam 2 score&#39;)<br>    plt.legend([&#39;Admitted&#39;, &#39;Not admitted&#39;])<br>    plt.show()</pre><pre>plotdata(data)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/637/1*lasp0amlxbfLs67bLIlmXQ.png" /></figure><h3><strong>2. Sigmoid Function</strong></h3><p>The logistic regression hypothesis is defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/247/1*uGmI4rbIweM_lFyzCOOayw.png" /></figure><p>where the function g is the sigmoid function. It is defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/273/1*4sw_HepZNKhQ8BGiib-7DQ.png" /></figure><p>Below is the sigmoid function coded in python:</p><pre># the function &#39;sigmoid&#39; can compute the sigmoid of each value of z (z can be a matrix vector or a scalar)</pre><pre>def sigmoid(z):<br>    g = 1/(1 + np.exp(-z))<br>    return g</pre><h3>3. Cost Function and Gradient</h3><p>The cost function in logistic regression is defined as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*VRNYozLGjx1lzOHAl76wEA.png" /></figure><p>, where m is the number of training examples. And the gradient of the cost is defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/508/1*8y3f0HNgIQG_VsXx_ILp0g.png" /></figure><p>, for j = 0,1,2,…,n where n is the number of features.</p><p>Now, let’s define x and y from our dataset. For x, we will take out the first two columns from our data frame ‘data’ and add a column of ones for the bias term.</p><pre># getting total number of training examples<br>m = data.shape[0]</pre><pre># define x and y<br>y = data.iloc[:, 2].to_numpy().reshape(m,1)<br>x = np.concatenate((np.ones((m,1)), data.iloc[:, :2]), axis = 1)</pre><pre># initialize theta<br>theta_start = np.zeros((x.shape[1], 1))</pre><p>We will define two separate functions to calculate the cost and the gradient.</p><pre># this function will compute cost of using theta as the parameter for logistic regression</pre><pre>def J(theta, x, y):<br>    m = len(y)<br>    y_prime = np.transpose(y)<br>    x_prime = np.transpose(x)<br>    h_theta = sigmoid(np.dot(x,theta))<br>    <br>    J = (-np.dot(y_prime, np.log(h_theta)) -  np.dot(np.transpose(1-y), np.log(1-h_theta)))/ m<br>    J = np.sum(J)<br>    <br>    return J</pre><pre># Gradient of the cost</pre><pre>def Gradient(theta, x, y):<br>    m = len(y)<br>    x_prime = np.transpose(x)<br>    <br>    error = sigmoid(np.dot(x,theta)) - y<br><br>    grad = (np.dot(x_prime, error))/ m<br>    grad = grad.flatten()<br>    return grad</pre><h3>4. Learning Parameters using Newton-Conjugate-Gradient</h3><p>Instead of taking the gradient descent steps for a number of iterations, we will use the ‘fmin_ncg()’ function (an unconstrained minimization using the Newton-CG method) from scipy.optimize. Note: for logistic regression, there is no constraint on the theta values (i.e., theta can take any real value). You can read more about the function <a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.fmin_ncg.html">here</a>.</p><p>The required parameters for this function are:</p><ol><li>An objective function to be minimized. (this is our cost function J(ϴ)).</li><li>Initial guess of the parameters to be optimized i.e., our initialized ϴ values.</li><li>Gradient of J(ϴ)</li><li>maxiter: maximum number of iterations to perform</li></ol><p>The function will return the optimized ϴ values (together with other information).</p><pre>import scipy.optimize as opt</pre><pre>theta_start = np.zeros((x.shape[1], 1))</pre><pre>result = opt.fmin_ncg(J, x0=theta_start, fprime = Gradient, args = (x,y.flatten()), maxiter = 400)</pre><pre>print(&quot;Cost at theta found by fmin_ncg() : {}&quot;.format(J(result,x,y)))<br>print(&#39;Theta : {}&#39;.format(result))</pre><p>Below is the output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/672/1*ub0oqc7BnU0G7K_cYdx7QA.png" /></figure><h3>5. Plotting the Boundary Line</h3><p>I will write another function to call our previous ‘plotdata()’ function and add a boundary line to the plot.</p><pre>def plotDB(data, theta, x, y):<br>    plotdata(data)<br>    <br>    # we take out two values for x1<br>    x_values = [np.min(x[:, 1]), np.max(x[:, 1])]<br>    # use these x1 values to calculate x2 values<br>    y_values = - (theta[0] + np.dot(theta[1], x_values)) / theta[2]<br>    <br>    plt.plot(x_values, y_values, label=&#39;Decision Boundary&#39;)</pre><pre>plotDB(data, result, x, y)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/613/1*evXyYkGtsn7btnIAqeIe0g.png" /><figcaption>The final graph with the boundary line.</figcaption></figure><h3>6. Evaluating the model</h3><p>In this step, we can use our model to predict the probability of whether a particular student will be admitted or not. Note: don’t forget to add a bias term to the input X.</p><p>Furthermore, we can evaluate the quality of the parameters we have found by asking the model to predict our training set data.</p><pre># Function for predicting<br># this will predict whether the label is 1 or 0 using the learned logistic regression parameters theta</pre><pre>def predict(theta, x):<br>    m = len(x)<br>    p = np.zeros((m,1))<br>    for i in range(m):<br>        sig = sigmoid(np.dot(x[i],theta))<br>        if sig &gt; 0.5:<br>            p[i] = 1<br>        else:<br>            p[i] = 0<br>    return p</pre><pre>p = predict(result, x)<br>print(&quot;Train Accuracy : {}%&quot;.format(int(sum(p==y))))</pre><p>From this, our model accuracy based on the training set is 89%.</p><p>This is all for this part. The next part will be about regularized logistic regression where we will try to create a non-linear decision boundary and therefore, need to add regularization parameters to prevent overfitting.</p><p>Keep Learning. Enjoy the journey!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd4718ac13ea" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[‘Machine Learning’ course by Andrew Ng : Recoding with Python — Part2]]></title>
            <link>https://medium.com/@susandyhtun1/machine-learning-course-by-andrew-ng-recoding-with-python-part2-a852eee972bd?source=rss-33aa80f87909------2</link>
            <guid isPermaLink="false">https://medium.com/p/a852eee972bd</guid>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[python3]]></category>
            <dc:creator><![CDATA[su_sandy]]></dc:creator>
            <pubDate>Fri, 20 May 2022 20:48:22 GMT</pubDate>
            <atom:updated>2022-05-20T20:49:39.090Z</atom:updated>
            <content:encoded><![CDATA[<h3>‘Machine Learning’ course by Andrew Ng : Recoding with Python — Part2</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ODf4X51nKEMElimXA706gQ.jpeg" /><figcaption>Image by ”<a href="https://www.freepik.com/vectors/php">https://www.freepik.com/vectors/php</a>&quot;&gt;Php vector created by roserodionova — <a href="http://www.freepik.com/">www.freepik.com</a></figcaption></figure><p>This is the second article in this series where I try to recode the exercises in the Machine Learning course by Andrew Ng (Part1 can be found <a href="https://medium.com/@susandyhtun1/machine-learning-course-by-andrew-ang-recoding-with-python-part1-f076c247bb6e">here</a>). My intention in writing these articles is to assist the learners of this course to use Python as an alternative while doing the exercises.</p><p>In this Part2, we will implement linear regression with multiple variables to predict the prices of houses. We already have a dataset that contains the housing prices in Portland, Oregon. The dataset has only three columns: the first is the size of the house (in squared feet), the second is the number of bedrooms and the third is the price of the house. We will use this dataset to train our linear regression model.</p><h3><strong>Step1: Getting to know the data</strong></h3><p>As usual, to get an idea about the dataset we can either use describe() method or we can plot the data. I used box plots to get an overview of how each feature is distributed in my data.</p><pre>import pandas as pd<br>import numpy as np<br>import matplotlib.pyplot as plt</pre><pre>data = pd.read_table(&#39;ex1data2.txt&#39;, header = None, sep=&#39;,&#39;)</pre><pre>plotdata = [data.iloc[:,0], data.iloc[:,1] , data.iloc[:,2]]</pre><pre>fig = plt.figure()<br>titles = {221: &#39;House_Size&#39;, 222: &#39;Bedroom&#39;, 223: &#39;Price&#39;}</pre><pre>for i in range(3):<br>    ax = fig.add_subplot(list(titles.keys())[i])<br>    ax.boxplot(plotdata[i])<br>    ax.title.set_text(list(titles.values())[i])</pre><p>Below is the output. We can see the different ranges of each feature: the size of the house (in the range of 1000 to 3000) and no. of bedrooms (in the range of 2 to 5). The house sizes are about 1000 times the number of bedrooms. In such a case, we need to perform feature normalization to help the gradient descent converge much more quickly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/621/1*JvwBH1GJYBW_mpP9lWHDCg.png" /></figure><h3>Step2: Feature Normalization</h3><p>We will create a function that will take in the features of our dataset and return the normalized version (i.e., for each feature column, we will subtract the mean and divide it by standard deviation). The function is as below:</p><pre>def FeatureNorm(x):<br>    x_norm = x<br>    mu    = np.zeros((1,x.shape[1])) # 2D array of shape(1,2)<br>    sigma = np.zeros((1,x.shape[1])) # std.deviation</pre><pre># for each feature dimension, we will compute mean and std.dev.<br># next, we will subtract mean from each entry of that feature and divide by std. dev.</pre><pre>for i in range(x.shape[1]):<br>        mu[0][i] = x[:,i].mean()<br>        sigma[0][i] = x[:,i].std()<br>        x_norm[:,i] = (x[:,i] - mu[0][i])/sigma[0][i]<br>    <br>    return x_norm, mu, sigma</pre><p>Now, let’s normalize our two feature columns (house size and no. of bedrooms).</p><pre>m  = len(data.iloc[:,2])   # number of training examples<br>x_ = data.iloc[:,:2].to_numpy(dtype=float)<br>y  = data.iloc[:,2].to_numpy().reshape(m,1)</pre><pre>x_, mu, sigma = FeatureNorm(x_)</pre><pre># let&#39;s see first 5 entries of our normalized feature columns.<br>print(x_[:5])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/386/1*kGTcC1HBcVeLMxaP7x0nMw.png" /><figcaption>First 5 entries of our normalized House Size and Bedroom columns.</figcaption></figure><pre># add the bias term<br>x = np.concatenate((np.ones((m,1)), x_), axis = 1)</pre><h3>Step3: Cost and Gradient Descent Functions</h3><p>For cost and Gradient Descent functions, we can reuse the functions we have written for Part1 (as we code the functions to be able to support more than one variable situation).</p><pre># function to calculate the cost J while we perform the gradient descent</pre><pre>def J(x,y,theta):<br>    m = len(y)<br>    x_prime = np.transpose(x)<br>    theta_prime = np.transpose(theta)<br>    y_prime = np.transpose(y)<br>    <br>    value = np.sum(np.power(np.dot(theta_prime, x_prime) - y_prime,2))<br>    J = value/ (2 * m)<br>    return J</pre><p>The below Gradient Descent function will take in values of ϴ, x, y, learning rate α and the number of iterations and update ϴ by taking small steps. When it’s called, this function will print out each value of J and ϴ for each iteration (this is purely for checking purpose to make sure that J decreases over each iteration and can be omitted) and return the final ϴ values.</p><pre>def gradient_des(x,y, theta, alpha, num_iters):<br>    m = len(y) <br>    J_history = np.zeros((num_iters,1)) <br>    x_prime = np.transpose(x)<br>    y_prime = np.transpose(y)<br>    <br>    for i in range(num_iters):<br>        theta_p = np.transpose(theta)<br>        <br>        term1 = np.dot(theta_p, x_prime) - y_prime<br>        term2 = np.dot(term1, x)<br>        term3 = np.transpose(term2)<br>        theta = theta - (alpha/m) * term3<br>        <br>        J_history[i] = J(x,y,theta)<br>        print(&quot;Theta = {}, J = {}, for iteration {}&quot;.format(theta, J_history[i],i))<br>    return theta, J_history</pre><p>Let’s test our functions by setting initial thetas to be zeros with a learning rate of 0.1 and iterate it 1000 times.</p><pre>alpha = 0.1<br>iteration = 1000<br>theta_start = np.zeros((3,1))</pre><pre>theta, J_history = gradient_des(x,y,theta_start,alpha,iteration)<br>print(&quot;Theta computed from gradient descent: {}, {} and {}&quot;.format(theta[0], theta[1], theta[2]))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-dM8PbHk1uj9SSv4G4K4Mw.png" /><figcaption>Output from testing the functions</figcaption></figure><p>Now, we got our final theta values and could use them for our prediction. Do not forget to normalize the given features before using the regression model.</p><pre># Let&#39;s predict the price of a 1650 sq-ft, 3 br house</pre><pre>x1 = np.array([1, 1650, 3], dtype = float).reshape(1,3)</pre><pre># feature normalization<br>x1[0][1] = (x1[0][1] - mu[0][0])/sigma[0][0]<br>x1[0][2] = (x1[0][2] - mu[0][1])/sigma[0][1]</pre><pre>price = float(np.dot(x1, theta))<br>print(&quot;Predicted price of a 1650 sq-ft, 3 br house (using gradient descent): {}&quot;.format(price))</pre><h3>Choosing the Learning Rate α</h3><p>So far, we have used a fixed learning rate of 0.1 in our Gradient Descent function. Because of the nature of the gradient descent function, we don’t really need to update the learning rate throughout the iterations but the learning rate value we use has an effect on how quickly the gradient descent can converge to the minimum point.</p><p>The code below will call our gradient_des() function and run it for 50 iterations at the learning rate chosen by the user. At the end of the loop, it will plot the J values against the number of iterations. We can try values of alpha on a log-scale at multiplicative steps of about 3 times the previous value (i.e., 0.3, 1, 0.03, 0.01 and so on).</p><p><strong>NOTE</strong>: With a small learning rate, the gradient descent takes a very long time to converge to the optimal value. Conversely, if the learning rate is too large, J(theta) can diverge and blow up, resulting in values that are too large for computer calculations.</p><pre>alpha = float(input(&quot;Please enter the learning rate&quot;))<br>iteration = 50</pre><pre>theta_start = np.zeros((3,1))</pre><pre>theta, J_history = gradient_des(x,y,theta_start,alpha,iteration)</pre><pre>plt.plot(range(iteration), J_history, &#39;-b&#39;, linewidth = 2 )<br>plt.xlabel(&quot;Number of iterations&quot;)<br>plt.ylabel(&quot;Cost J&quot;)</pre><p>To get an idea, below are the different graphs using different values of alpha.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/977/1*Qrm2r_C57uTBbcQa91abaw.png" /><figcaption>We can see that 1 is our best learning rate.</figcaption></figure><p>So, this is how we can choose our learning rate to help the Gradient Descent converge quickly.</p><h3>Using Normal Equation</h3><p>An alternative way to find optimum theta values is to use the closed-form normal equation which does not require feature scaling and looping until convergence. The equation is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/198/1*GJcygLaPp3kZfaF-HWLNgA.png" /></figure><pre>def normalEqn(x,y):<br>    theta = np.zeros((x.shape[1], 1)) <br>    x_prime = np.transpose(x)<br>    term1 = np.linalg.inv(np.dot(x_prime,x))<br>    term2 = np.dot(term1, x_prime)<br>    theta = np.dot(term2, y)<br>    return theta</pre><pre>data = pd.read_table(&#39;ex1data2.txt&#39;, header = None, sep=&#39;,&#39;)<br>m = len(data)</pre><pre>x_ = data.iloc[:,:2].to_numpy(dtype=float)<br>x = np.concatenate((np.ones((m,1)), x_), axis = 1)<br>y  = data.iloc[:,2].to_numpy().reshape(m,1)</pre><pre>theta = normalEqn(x,y)</pre><pre>print(&quot;Theta computed from the normal equations: {}, {}, and {}&quot;.format(theta[0], theta[1], theta[2]))</pre><p>Yes, these are all the steps in building a linear regression model for data with more than one feature (here, we have tried both Gradient Descent and Normal Equation). An extra thing we have done is to get to know how to choose a suitable learning rate. Part3 will be about Logistic Regression Model.</p><p>Seeya. :)</p><p>Keep learning. Enjoy the Journey!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a852eee972bd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[‘Machine Learning’ course by Andrew Ng : Recoding with Python — Part1]]></title>
            <link>https://medium.com/@susandyhtun1/machine-learning-course-by-andrew-ang-recoding-with-python-part1-f076c247bb6e?source=rss-33aa80f87909------2</link>
            <guid isPermaLink="false">https://medium.com/p/f076c247bb6e</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python3]]></category>
            <dc:creator><![CDATA[su_sandy]]></dc:creator>
            <pubDate>Tue, 17 May 2022 18:40:09 GMT</pubDate>
            <atom:updated>2022-05-17T19:12:34.204Z</atom:updated>
            <content:encoded><![CDATA[<h3>‘Machine Learning’ course by Andrew Ng : Recoding with Python — Part1</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C9DiGk1kRDJMr1UlfUXBMg.jpeg" /><figcaption>image by ”<a href="https://www.freepik.com/vectors/work-illustration">https://www.freepik.com/vectors/work-illustration</a>&quot; Work illustration vector created by macrovector — <a href="http://www.freepik.com/">www.freepik.com</a></figcaption></figure><p>I have taken this amazing <a href="https://www.coursera.org/learn/machine-learning">“Machine Learning” course</a> by Andrew Ng offered by Standford on Coursera. This course was created about 10 years ago. The machine learning exercises in this course were designed to use Octave programming language (in Matlab).</p><p>After finishing the course, I wonder why not redo these exercises with Python (which is now among the top 3 programming languages). So, here in this series of articles, I will use Python to redo the programming exercises from this course. I intend to write this article so that it will be helpful for the learners of this course to use Python as an alternative while doing the exercises.</p><p>The first exercise is to train a Linear Regression model using one feature. The given dataset consists of two columns (one is the population of a city and another one is the profit of a food truck in that city). We have total 194 records. The purpose is to build a model that will help the CEO of the food truck company select which city to expand to next. Below is the step-by-step guide to it.</p><h3>Step1: Plotting the data</h3><p>Numpy, and pandas libraries are used to read the data, take out the feature column and target column separately and change both into arrays (for calculation purpose later). Then, matplotlib is used to draw a scatter plot to get an idea of the correlation between the population and the profit.</p><pre># importing the file<br>import pandas as pd<br>import numpy as np</pre><pre>data = pd.read_table(&#39;ex1data1.txt&#39;, header = None, sep=&#39;,&#39;)<br>data.reset_index(drop=True, inplace = True)<br>data.head()</pre><p>Below is a glimpse of the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/325/1*NiNg6qGTmoRHMjF413tpgQ.png" /></figure><pre>population = data.iloc[:,0]      # population column<br>profit = data.iloc[:,1]          # profit column<br>m = len(population)              # number of training examples</pre><pre>x_ = population.to_numpy().reshape(m,1)<br>y = profit.to_numpy().reshape(m,1)</pre><pre># plotting x and y<br>import matplotlib.pyplot as plt<br>plt.scatter(x_,y, marker =&#39;x&#39;, color =&#39;red&#39;, s=10)<br>plt.ylabel(&#39;Profit in $10,000&#39;)<br>plt.xlabel(&quot;Population of the City in 10,000&quot;)</pre><p>The resulting graph is as below. We can see that there is a strong, positive correlation between the population and the profit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*dWYExzFw13xvZK3yvfyWnQ.png" /></figure><h3>Step2: Gradient Descent Function</h3><p>Our hypothesis h(ϴ) is given by the linear model :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/259/1*h9q0-FjCFV7yfPWxm4eNSg.png" /></figure><p>We will use the batch gradient descent to adjust ϴ values to minimize the cost function J(ϴ). Each iteration will perform the update to our initial ϴ values and find the global minimum point where we will have the least error (or cost). The equation is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*FeaPZ--aKBmYMktvLNY7Lg.png" /></figure><p>First thing is to add a bias term to our array x_ and set some initial values:</p><pre># adding a column of 1 to array x for theta0.<br>x = np.concatenate((np.ones((m,1)), x_), axis = 1)</pre><pre># initialize the first fitting parameter: both 0<br>theta_start = np.zeros((2,1))</pre><pre>iteration = 1500</pre><pre># learning rate<br>alpha = 0.01</pre><p>Secondly, a function will be built to calculate J(ϴ). This function will take in x, y and ϴ arrays and return the cost of using the input ϴ as the parameters for linear regression.</p><pre>def J(x,y,theta):<br>    m = len(y)<br>    x_prime = np.transpose(x)<br>    theta_prime = np.transpose(theta)<br>    y_prime = np.transpose(y)<br>    <br>    value = np.sum(np.power(np.dot(theta_prime, x_prime) - y_prime,2))<br>    J = value/ (2 * m)<br>    return J</pre><p>Once we have a function for J(ϴ), we can build a function for Gradient Descent which will take in values of ϴ, x, y, learning rate α and the number of iterations and update ϴ by taking small steps. When it’s called, this function will print out each value of J and ϴ for each iteration (this is purely for checking purpose to make sure that J decreases over each iteration and can be omitted) and return the final ϴ values.</p><pre>def gradient_des(x,y, theta, alpha, num_iters):<br>    m = len(y) <br>    J_history = np.zeros((num_iters,1)) # a 2D 0 array<br>    x_prime = np.transpose(x)<br>    y_prime = np.transpose(y)<br>    <br>    for i in range(num_iters):<br>        # perform a single gradient step on the parameter vector theta<br>        theta_p = np.transpose(theta)<br>        <br>        term1 = np.dot(theta_p, x_prime) - y_prime<br>        term2 = np.dot(term1, x)<br>        term3 = np.transpose(term2)<br>        theta = theta - (alpha/m) * term3<br>        <br>        # record each cost for each iteration <br>        J_history[i] = J(x,y,theta)<br>        print(&quot;Theta = {}, J = {}, for iteration {}&quot;.format(theta, J_history[i],i))<br>    return theta</pre><p>Now is time to test our functions and see the results.</p><pre># let&#39;s run Gradient Descent now</pre><pre>theta = np.array(gradient_des(x,y,theta_start,alpha,iteration))<br>print(&quot;Theta computed from gradient descent: {}, {}&quot;.format(theta[0], theta[1]))</pre><p>Here is a glimpse of the output and the final values of ϴ:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/814/1*0unJCGJrWY2-jpRUMcQTOQ.png" /></figure><h3>Step3: Plotting the Linear Regression Line</h3><p>On the previous scatter plot, we can add a linear regression calculated from the dot product of our feature matrix x and ϴ.</p><pre>plt.scatter(x_,y, marker =&#39;x&#39;, color =&#39;red&#39;, s=10)<br>plt.ylabel(&#39;Profit in $10,000&#39;)<br>plt.xlabel(&quot;Population of the City in 10,000&quot;)</pre><pre>plt.plot(x[:,1], np.dot(x,theta), &#39;-&#39;)</pre><pre>plt.legend([&#39;Training Data&#39;, &#39;Linear Regression&#39;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/613/1*yROw6CwFqxK4lVsW_R41tQ.png" /></figure><h3>Step4: Visualizing J(ϴ) [optional]</h3><p>Let’s draw a surface plot and contour plots of J(ϴ) over a 2D grid of ϴ0 and ϴ1 values using matplotlib. This will help us get a better idea of the cost function J(ϴ) and the global minimum point.</p><p><strong>Surface Plot</strong>:</p><pre>import matplotlib.pyplot as plt<br>from matplotlib import cm<br>from matplotlib.ticker import LinearLocator<br>import numpy as np</pre><pre>fig, ax = plt.subplots(subplot_kw={&quot;projection&quot;: &quot;3d&quot;})</pre><pre># values of theta over which we will calculate J<br>theta0 = np.linspace(-10,10,100)<br>theta1 = np.linspace(-1,4,100)</pre><pre>X, Y = np.meshgrid(theta0, theta1)</pre><pre>Z = np.zeros((len(theta0), len(theta1)))<br>for i in range(len(theta0)):<br>    for j in range(len(theta1)):<br>        t = [theta0[i], theta1[j]]<br>        Z[i,j] = J(x,y,t)</pre><pre>ax.plot_surface(X, Y, Z, cmap=cm.coolwarm,linewidth=0, antialiased=False)<br>plt.xlabel(&#39;theta0&#39;)<br>plt.ylabel(&#39;theta1&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/417/1*wZflXJrykQGb20sY6KUs6w.png" /></figure><p><strong>Contour Plot</strong>:</p><pre>X, Y = np.meshgrid(theta0, theta1)<br>plt.contour(X,Y,np.transpose(Z), np.logspace(-2,3,20))<br>plt.xlabel(&#39;theta_0&#39;)<br>plt.ylabel(&#39;theta_1&#39;)</pre><pre>plt.plot(theta[0], theta[1], &#39;rx&#39;, markersize= 10, linewidth= 2)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*LJ0x7aM7v9HFT_GH2u4Vmw.png" /><figcaption>Here, the red cross is our final theta value.</figcaption></figure><p>Yes, these are all the steps in getting a linear regression model for data with only one feature!</p><p>Part2 will be on building a linear regression model (using Gradient Descent and Normal Equation as an alternative) for data with more than one feature.</p><p>Keep Learning. Enjoy the journey!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f076c247bb6e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>