<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Weng Seng on Medium]]></title>
        <description><![CDATA[Stories by Weng Seng on Medium]]></description>
        <link>https://medium.com/@wengsengh?source=rss-b037edb4f2a7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*79OYGkfoWXQvB2NLOt1K5A.png</url>
            <title>Stories by Weng Seng on Medium</title>
            <link>https://medium.com/@wengsengh?source=rss-b037edb4f2a7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 10:15:34 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@wengsengh/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Customer Segmentation with PCA]]></title>
            <link>https://medium.com/@wengsengh/customer-segmentation-with-pca-6ddf9681ce1d?source=rss-b037edb4f2a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/6ddf9681ce1d</guid>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Weng Seng]]></dc:creator>
            <pubDate>Tue, 08 Oct 2019 11:55:37 GMT</pubDate>
            <atom:updated>2019-10-08T11:55:37.095Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*-v6U3LhNVwoXoFOjMBLlzg.jpeg" /></figure><p>This is project of the Udacity Data Scientist Nanodegree . You can find the details for the project on <a href="https://github.com/wengsengh/Identify-Customer-Segments">github</a>.</p><p>Using unsupervised learning techniques of Principal Component Analysis (PCA) and KMeans to identify customer segments of the German population that were popular or less popular with a mail-order sales in Germany.</p><h4>The Dataset</h4><p>The data is provided by <a href="https://www.bertelsmann.com/divisions/arvato/">Bertelsmann</a> partners AZ Direct and Arvato Financial Solution. There are two dataset provided first was demographic data for the general population of Germany and the second dataset was demographic data for customers of a mail-order company. Both dataset consists of 85 different features.</p><h4>Data Wrangling</h4><p>The data was quite messy and some was irrelevant. We need converting and mapping the data to appropriate format for data analytics.</p><ul><li>First we deal with missing values in the dataset by dropping variables and rows with extremely high frequencies of missing values.</li><li>Next we need convert categorical variables to binary variables by using one-hot encoded features.</li><li>Then we identify mixed-type features and engineer new features because some variables that had more than two different piece of information.</li><li>We use Use StandardScaler to scale each feature to mean 0 and standard deviation 1.</li></ul><h4>PCA — Dimensionality Reduction</h4><p>The main idea of principal component analysis (PCA) is to reduce the dimensionality of a data set consisting of many variables correlated with each other while retaining the variation present in the dataset.</p><p>The results of the PCA, we can observe the relative variance in the data is captured along each component.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/1*DMa2xeEZARTOPTyq0xJCgw.jpeg" /></figure><p>The PCA shows that first component which is the first blue bar got 16.2% the highest variance explained. Then the next component which explain the next highest percentage of variance. Total 71 components to capture all of the variance in the data. However, we’ll not use all the 71 components for 100% of our variance but we’ll use 30 components. Let see the plot below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*FqBgMDKoJFy4hc_ggwyfSg.jpeg" /></figure><p>This plot tells us that selecting 30 components we can preserve something around 80% of the total variance of the data.</p><h4>Interpret Principal Components</h4><p>We’ll show the top 5 weights of the variable on the first component to interpret their relationship, what a positive or negative value indicates.</p><p>Here’s the result:</p><ul><li><strong>Top 5 Positive Feature<br></strong>CAMEO_DEUG_2015 (urban working class)<br>PLZ8_ANTG3 (Number of 6–10 family houses in the PLZ8 region)<br>CAMEO_INTL_2015_WEALTH ( High positive: more likely poor)<br>EWDICHTE (Density of households per square kilometer)<br>ORTSGR_KLS9 (Size of community)</li><li><strong>Top 5 Negative Feature<br></strong>FINANZ_MINIMALIST (Low financial interest)<br>KBA05_ANTG1 (Number of 1–2 family houses in the microcell)<br>MOBI_REGIO (High Movement patterns)<br>LP_STATUS_FEIN (Low social status, fine scale)<br>LP_STATUS_GROB (Low social status, rough scale)</li></ul><p>Positive means that these features are likely to be high where negative means that these features are likely to be low.</p><p>The result shows that the first component is urban people with lower financial interest and income.</p><h4>K-Means Analysis</h4><p>We apply k-means clustering to the dataset and use the average within-cluster distances from each point to their assigned cluster’s centroid to decide on a number of clusters to keep.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/504/1*fIaPtUUn2oYMPfL1g7lV-w.jpeg" /></figure><p>This plot shows us the amount k-mean scores associated with each number of clusters. We decided to choose 14 because after 14 clusters, the scores does not decrease much than before.</p><p>Then, we compared the relative proportions of each clusters between the demographics and the customer data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/613/1*pC3nPUgfsU2ENBC_wlAsNA.jpeg" /></figure><p>As we can see the cluster 1, 3 and 10 are significantly over-represented where cluster 2, 5 and 8 significantly underrepresented.</p><p>For cluster 1 (over-represented):</p><ul><li>Estimated age around 30-60 years of age</li><li>High for dutiful</li><li>High for traditional</li><li>Higher for sparer and investor</li><li>Low for ‘be prepared’</li><li>Low for sensual minded</li></ul><p>For cluster 2 (underrepresented):</p><ul><li>Between 20 and 30</li><li>High for financial minimalist</li><li>Living in big cities</li><li>More households in community</li><li>Low interest in money saving</li><li>Low household net income</li><li>Low wealth</li></ul><p>Cluster 1 is an over-represented segment, they seen more attracted in purchasing by mail order sales. It also seems reasonable that older generation are more likely to be customers in traditional way of shopping compare to youngster.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6ddf9681ce1d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Car Model Classification]]></title>
            <link>https://medium.com/data-science/car-model-classification-e1ff09573f4f?source=rss-b037edb4f2a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/e1ff09573f4f</guid>
            <category><![CDATA[image-classification]]></category>
            <dc:creator><![CDATA[Weng Seng]]></dc:creator>
            <pubDate>Tue, 10 Sep 2019 01:22:39 GMT</pubDate>
            <atom:updated>2021-05-28T15:35:21.982Z</atom:updated>
            <content:encoded><![CDATA[<p>In this post I will show the result for car model classification with ResNet ( Residual Neutral Network). I use Python and Pytorch to build the model. The steps were similar to my previous flower species classifier <a href="https://medium.com/@wengsengh/test-f4eddce38aff">project</a> so I will brief about some key steps and the result.</p><p>You can check my github repo <a href="https://github.com/wengsengh/Car-Models-Classifier">here</a>.</p><h4>The Dataset</h4><p>I download the Stanford car dataset from <a href="https://www.kaggle.com/jutrera/stanford-car-dataset-by-classes-folder">Kaggle</a> by using Kaggle API and save in my google drive.</p><pre># The Kaggle API client expects this file to be in ~/.kaggle,<br>!mkdir -p ~/.kaggle<br>!cp kaggle.json ~/.kaggle/</pre><pre># This permissions change avoids a warning on Kaggle tool startup.<br>!chmod 600 ~/.kaggle/kaggle.json</pre><pre>#mount my gdrive<br>from google.colab import drive<br>drive.mount(&#39;/content/drive&#39;)</pre><pre>#download the dataset<br>!kaggle datasets download -d jutrera/stanford-car-dataset-by-classes-folder</pre><pre>#unzip the file<br>!unzip stanford-car-dataset-by-classes-folder.zip</pre><p>The Cars dataset contains 16,185 images of 196 classes of cars. The data is split into 8,144 training images and 8,041 testing images.</p><h4>Transfer Learning</h4><p>Transfer learning make use of the knowledge gained while solving one problem and applying it to a different but related problem.<br>The pre-trained model here is ResNet( Residual Neutral Network).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/535/1*6M9zescbaSV-FEIMdhZzfQ.jpeg" /></figure><p>Artificial Neural Network predicts the outcome by learn through the examples and previous experiences. Artificial Neural Network computations carried out in parallel and create their own representation of information which they receive during the learning time. In the artificial Neural Network, just add in one more layer which is convolutional layer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nxkqCCme7hVzopZy9i9_Kw.jpeg" /><figcaption>Image Source: Towards Data Science</figcaption></figure><p>For further explanation about ResNet architectures you can refer <a href="https://towardsdatascience.com/understanding-and-visualizing-resnets-442284831be8">here</a>.</p><p>Back to our case here, we got 196 classes of cars as the total number of outputs so we need change the output layer into 196.</p><pre>num_ftrs = model.fc.in_features<br>model.fc = nn.Linear(num_ftrs, 196)</pre><h4>Validation Accuracy</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3BsIqbowz1m8MfZUojBmIw.jpeg" /><figcaption>The Epochs</figcaption></figure><p>I have try few parameter and keep running epochs for quite a few times. In the end with 10 epochs it give us 77.72% validation accuracy give me the best so far I tried. The training process really time consuming just need to be patient. Cheer!</p><h3>The Result</h3><p>Let see what it actually did!!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M3RO33jjosimecFLtZDp7w.jpeg" /></figure><p>For the Mercedes-Benz S-Class Sedan 2012 the model predicted Mercedes-Benz E-Class Sedan 2012 which is inaccurate. The Mercedes-Benz S-Class Sedan 2012 also didn’t appear in the top 5 predicted classes.<br>BMW X6 SUV 2012 predicted semi-correctly because it appeared in third top classes. It supposed be the top classes in the prediction result not the Ford Fiesta Sedan 2012.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O40IbcQ22KXyjUamTN1aOg.jpeg" /></figure><p>Hyundai Veloster Hatchback 2012 and Volkswagen Golf Hatchback 2012 were well predicted with more than 30%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ztmyC2ImlDbNOabLe66jIQ.jpeg" /></figure><p>Both are BMW 3 Series Sedan 2012 just the different is the angle view. The left image is front view and the right image is rear view. <br>Surprisingly both different view of BMW 3 predicted correctly. The top 5 prediction result in the left side was very good because it predicted the top 5 classes were all BMW. In the right side result also good, BMW appeared in the top 3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/534/1*5AvfMu4DyYFSZvSV_dcn8g.jpeg" /></figure><p>However, in this case BMW M5 Sedan 2010 does not performing, it got all top 5 prediction classes wrong. Any BMW car model did not show up in the result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AuTXPlttMVjtM_26QezJeQ.jpeg" /></figure><p>In this case both model were SUV models and predicted correctly.</p><p>That’s it!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e1ff09573f4f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/car-model-classification-e1ff09573f4f">Car Model Classification</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Create a Tableau Story: Baseball Players]]></title>
            <link>https://medium.com/@wengsengh/create-a-tableau-story-baseball-players-c3c31e2fa54d?source=rss-b037edb4f2a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/c3c31e2fa54d</guid>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Weng Seng]]></dc:creator>
            <pubDate>Thu, 05 Sep 2019 04:06:16 GMT</pubDate>
            <atom:updated>2019-09-05T04:06:16.226Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*JM9ZuDflhNdpGrElQpmQ1A.jpeg" /></figure><p>In this post I will tell a story and trends in a baseball data set by using data visualization tool Tableau.</p><p>However, medium not allow us to embeds Tableau codes so I unable to embed my Tableau Story here. So I will post some screenshots of my Tableau story overview here.</p><p>For the project details can be found in my <a href="https://github.com/wengsengh/Create_a_Tableau_Story">github</a>.<br>For the Tableau Story can be view <a href="https://public.tableau.com/profile/weng.seng#!/vizhome/basellballEDAfinal/PerformanceBaseballPlayers?publish=yes">here</a>.</p><h4>Overview:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/814/1*ZpdltjIAFa5NCTPbEOJ--w.jpeg" /><figcaption>Handedness</figcaption></figure><p>I starting with bar plot for Handedness for my first data visualization. I found out that most of the players are right handed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/813/1*X-MsCFy1eNS38dXBLg11hA.jpeg" /><figcaption>Height vs Weight</figcaption></figure><p>Second I plots players weight and height by using scatter plot. It shows most of the players weight are between 150 to 230 pounds; height are between 66 to 80 inches.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/1*D2j1n7edQBja65X659p0AQ.jpeg" /><figcaption>Home Run vs Battling Average</figcaption></figure><p>There is high density player in battling average range from 0.17 to 0.28 and 0 to 100 for home run.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/817/1*AFG4ljdDnYyhUjOYwyqpTQ.jpeg" /></figure><p>I create a dashboard by combined previous three different plots. <br>To summaries, most of the players are right handed, weight are between 150 to 230 pounds, height are between 66 to 80 inches. The battling average range from 0.17 to 0.28.</p><p>So, this is my brief exploration of baseball players with Tableau.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c3c31e2fa54d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Wine Quality Exploration with R]]></title>
            <link>https://medium.com/@wengsengh/wine-quality-exploration-with-r-dca52264dca8?source=rss-b037edb4f2a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/dca52264dca8</guid>
            <dc:creator><![CDATA[Weng Seng]]></dc:creator>
            <pubDate>Tue, 03 Sep 2019 15:00:15 GMT</pubDate>
            <atom:updated>2019-09-03T15:00:15.319Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*VLmRv0NdSg1PR6WFkKh2NQ.jpeg" /></figure><p>In this post, I will highlight the exploratory data analysis (EDA) with R to explore relationships in one variable to multiple variables and to discover for distributions, outliers, and anomalies.</p><p>The full details of the project can be found <a href="https://github.com/wengsengh/Exploratory_Data_Analysis">here</a>.</p><h3>Table of Contents</h3><p>1. Introduction<br>2. Uni-variate Exploration<br>3. Bi-variate Exploration<br>4. Multivariate Exploration<br>5. Final Plots and Summary<br>6. Reflection</p><h3>1. Introduction</h3><p>This white wine data set is public available for research. The details are described in <a href="https://www.sciencedirect.com/science/article/pii/S0167923609001377?via%3Dihub">[Cortez et al., 2009]</a>.</p><p>For more information about the characteristic of the chemical composition for the features can refer this <a href="http://www.wineskills.co.uk/winemaking/winemaking-knowledge-base/chemical-composition">website</a>. This website did give us the insight how the chemical composition work and how possible it affect the flavor of the wine.</p><p>The data set it consists of 12 variables:</p><p>1.fixed acidity (tartaric acid — g / dm³)<br>2. volatile acidity (acetic acid — g / dm³)<br>3. citric acid (g / dm³)<br>4. residual sugar (g / dm³)<br>5. chlorides (sodium chloride — g / dm³)<br>6. free sulfur dioxide (mg / dm³)<br>7. total sulfur dioxide (mg / dm³)<br>8. density (g / cm³)<br>9. pH<br>10. sulphates (potassium sulphate — g / dm3)<br>11. alcohol (% by volume)<br>12. quality (score between 0 and 10)</p><h3>2. Uni-variate Exploration</h3><p>I will selected some of the variables from the total 12 variables to show the histogram plots. This is because that is a lot if I shows all 12 variables but you can find all the plots in my <a href="https://github.com/wengsengh/Exploratory_Data_Analysis">github</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/1*L7M7j-hWAkyxfiyRHDnr-A.jpeg" /><figcaption>pH histogram</figcaption></figure><p>The pH value in between 2.72 and 3.82 which mean the white wine from the data set are pretty sour and most of the pH around 3.2</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/696/1*mDbPGEbqXGRPvpui7kX3rA.jpeg" /><figcaption>alcohol histogram</figcaption></figure><p>Overall the alcohol histogram do shows a decreasing trend in count along the alcohol level from 9g/L.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/1*D_B6zuCXwvQ1W52SFLZtnQ.jpeg" /><figcaption>quality histogram</figcaption></figure><p>The histogram shows normal distribution. From the quality plot there is no perfect quality wine and undrinkable wine. Most of the quality of white wine are average.</p><h3>3. Bi-variate Exploration</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YcfWkPIQOfqswzpZLEVHeg.png" /><figcaption>ggpairs plot</figcaption></figure><p>From the ggpairs plot density vs sugar and density vs alcohol show strong correlation which is 0.839 and -0.78 repectively. So I will shows scatter plot on density vs sugar and density vs alcohol next.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*mohrJYT4LeyMlOyFeXZUHQ.jpeg" /><figcaption>Scatter Plot: Density vs Sugar</figcaption></figure><p>Density vs sugar shows positve correlation which tell as sugar level increase density also increase.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*L2yjyqhUAecmQfW1ME6yMA.jpeg" /><figcaption>Scatter Plot: Density vs Alcohol</figcaption></figure><p>Density vs alcohol shows negative correlation which alcohol level decrease density decrease as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*Ly03oBsv6Icn3iA5FglIOg.jpeg" /><figcaption>Quality with pH boxplot</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/707/1*FTZNAjU5r3H5TXi6Yj8VEA.jpeg" /><figcaption>Quality with alcohol boxplot</figcaption></figure><p>From the boxplots above, higher alcohol level tend to give better white wine quality. The boxplot with pH feature shows higher pH value the better the quality in narrow range of pH value but not obvious as alcohol.</p><h3>4. Multivariate Exploration</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*X79kq5gsXfFv0r35VWnrag.jpeg" /></figure><p>The quality histogram plot with pH above shows normal distribution for quality on 5, 6 and 7.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/1*rtKbMmDuLIaW2xmEkU4obg.jpeg" /></figure><p>The quality histogram plot with alcohol above shows quality concentrated on 5, 6 and 7 and not clear how the distribution vary from quality.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*nAXQbx8zV_CHxtPOycHRPQ.jpeg" /></figure><p>Higher quality (quality = 7) wines are concentrated on the right bottom corner in the scatter plot which represent low density and high alcohol.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*NADkkoXg6vzaItpg5LycaQ.jpeg" /></figure><p>Higher quality (quality = 7) wines are concentrated on the below quality equal to 5 in density across the the sugar level in the scatter plot. Which mean low density and low sugar level give better quality.</p><p>Alcohol and sugar level that I observed both are strengthened each other to give better quality.</p><h3>5. Final Plots and Summary</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/679/1*6FXrtDbPURiPLlXPKrhZUw.jpeg" /></figure><p>The plot the density shows negative correlation with alcohol meaning that high quality of white wine are low in density and high in alcohol level.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/679/1*jGDHbfhduhIm9zl0daGMnQ.jpeg" /></figure><p>The plot shows positive correlation between residual.sugar and density. Better quality wines are low sugar level and density.</p><h3>6. Reflection</h3><p>In uni-variate plots I manage to plot the histogram without any difficulty. However I do face some problems when plotting the features with quality where quality format originally are integer. After I realized this issues then I changed to factor.</p><p>I believe multivariate plots are better to show the significant relationship between different features impact the quality.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dca52264dca8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A/B Testing in Web Page Version]]></title>
            <link>https://medium.com/@wengsengh/a-b-testing-in-web-page-version-d033276466c3?source=rss-b037edb4f2a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/d033276466c3</guid>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Weng Seng]]></dc:creator>
            <pubDate>Sat, 31 Aug 2019 12:54:52 GMT</pubDate>
            <atom:updated>2019-08-31T12:54:52.354Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ac_bljLixI8bslfprWp5QQ.jpeg" /></figure><p>A/B testing is a method of comparing two versions of a web page each other to determine which one performs better.</p><p>In this post I will explain how I analyze with A/B Testing.</p><p>For this project, whether we should implement the new page or keep the old page with AB testing. For the full details of the result can be found <a href="https://github.com/wengsengh/Analyze-AB-Test-Results"><strong>here</strong></a>.</p><h4>Null and Alternative Hypotheses</h4><p>We assume that the old page is better unless the new page proves to be definitely better at a Type I error rate of 5%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/296/1*ZfVY8b4r8UjUprgOe7ITVA.jpeg" /></figure><p>Assume under the null hypothesis, <em>p new and p old are </em>converted success rate regardless of page.</p><p>I will use two method to calculate the p-value, first method will be without using statsmodels library and second method will be using statsmodels.</p><h4>Calculate p-value without Statsmodels Library</h4><p>First we perform the sampling distribution for the difference in <strong>converted</strong> between the two pages over 10,000 iterations. Store all 10,000 values in <strong>p_diffs </strong>where <em>n_new </em>is<em> </em>total number of landing new pages, <em>n_old </em>is<em> </em>total number of landing old pages.</p><pre>p_diffs = []<br><br><strong>for</strong> _ <strong>in</strong> range(10000):<br>    new_page_converted = np.random.binomial(1, p_new, n_new)<br>    new_page_converted_mean = new_page_converted.mean()<br>    old_page_converted = np.random.binomial(1, p_old, n_old)<br>    old_page_converted_mean = old_page_converted.mean()<br>    p_diffs.append(new_page_converted_mean - old_page_converted_mean)</pre><p>The we plot the <strong>p_diffs </strong>histogram<strong>:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/422/1*1rh36P7exLlNf1AH9FXH2A.jpeg" /></figure><p>The red line is the <strong>Actual difference. </strong>Then we calculate the proportion of the <strong>p_diffs</strong> are greater than the <strong>actual difference </strong>and we got 0.90349</p><p><strong>Note</strong>:<br><strong>p-diffs</strong> represents the simulated difference between converted rates of new page and old page, based on 10000 simulated samples.<br><strong>Actual difference</strong> represents the difference between converted rates of new page and old page, based on our data.</p><h4><strong>Explanation:</strong></h4><p>The value 0.90349 calculated here is called p-value and suggest that we failed to reject the null hypothesis.</p><h4>Calculate p-value with Statsmodels Library</h4><p>Using the built-in might be easier to code, the above section are a walkthrough of the ideas that are critical to correctly thinking about statistical significance.</p><pre><strong>import statsmodels.api as sm<br></strong>z_score, p_value = sm.stats.proportions_ztest([convert_old, convert_new], [n_old, n_new], alternative=&#39;smaller&#39;)<br>z_score, p_value</pre><pre>Output: (1.3109241984234394, 0.90505831275902449)</pre><p><strong>Note</strong>:<br><em>convert_old </em>and <em>convert_new </em>are the number of conversions for each page.</p><h4>Explanation:</h4><p>p-value 0.90505831275902449 similar to previous result we failed to reject the null hypothesis.</p><h3>A regression approach</h3><p>We will use logistic regression to predict because only two possible outcomes.</p><p>First we need to create a column for the intercept, and create a dummy variable column for which page each user received. Our goal is to use <strong>statsmodels</strong> to fit the regression model.</p><pre>df2[[&#39;control&#39;, &#39;treatment&#39;]] = pd.get_dummies(df2[&#39;group&#39;])<br>  <br>df2[&#39;intercept&#39;] = 1 log_mod = sm.Logit(df2[&#39;converted&#39;], df2[[&#39;intercept&#39;,&#39;treatment&#39;]]) result = log_mod.fit()</pre><pre>result.summary()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/1*d-lag1O4IwhYdshLs5bkYQ.jpeg" /><figcaption>Result</figcaption></figure><h4>Explanation:</h4><p>The logistic regression determines only two possible outcomes. The new page is equal to the old page or not equal:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/165/1*3pUtwZE9uVrjhh5x-ejz9A.jpeg" /></figure><p>The p-value associated with ab_page column is 0.19 which is lower than the p-value calculated previously using the z-score, p_value function. The reason why is different is due to the intercept added.</p><h4><strong>Add New Variable — Country</strong></h4><p>We<strong> </strong>merge together the datasets and <strong>countries.csv</strong> dataset on the appropriate rows.</p><pre>df_country = pd.read_csv(&#39;countries.csv&#39;)</pre><pre>df_country[[&#39;CA&#39;, &#39;UK&#39;, &#39;US&#39;]] = pd.get_dummies(df_country[&#39;country&#39;])</pre><pre>df2_country = df2.join(df_country.set_index(&#39;user_id&#39;), on=&#39;user_id&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/1*i9xU6SnEgt_63Ulzs91euQ.jpeg" /><figcaption>df2_country.head()</figcaption></figure><pre>log_mod2 = sm.Logit(df2_country[&#39;converted&#39;], df2_country[[&#39;intercept&#39;, &#39;UK&#39;,&#39;US&#39;]])</pre><pre>result2 = log_mod2.fit()</pre><pre>result2.summary()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/482/1*eCPzlpZCn2-p6ePIby1kiw.jpeg" /><figcaption>result 2</figcaption></figure><h3>Conclusion:</h3><p>In this case, we found that the old page was better than the new page, because we fail to reject the null hypothesis.</p><p>From the regression above we see that the p-value is lower in UK, which means that users in the UK are more likely to convert, but still not enough evidence to reject the null hypothesis with alpha level 0.05 but if the alpha level 0.10 then we can reject the null hypothesis.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d033276466c3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Investigate a Dataset]]></title>
            <link>https://medium.com/@wengsengh/investigate-a-dataset-688c5db7851e?source=rss-b037edb4f2a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/688c5db7851e</guid>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Weng Seng]]></dc:creator>
            <pubDate>Fri, 30 Aug 2019 01:39:51 GMT</pubDate>
            <atom:updated>2019-08-30T02:07:24.389Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*_cwS28WtYMA44U9uKVCTHQ.jpeg" /></figure><p>In this post, I will provide a brief introduction to the data set that I selected for analysis and the key findings. The full details of this project can be found <a href="https://github.com/wengsengh/Investigate-a-Dataset"><strong>here</strong></a><strong>.</strong></p><h3>No Show Appointment Data Set</h3><p>I selected the no show appointment data set from <a href="https://www.kaggle.com/joniarroba/noshowappointments"><strong>kaggle</strong></a>. The data set consists of 300k medical appointments and its 15 variables (characteristics) of each. The data recorded patient show-up or no-show the appointment.</p><h3>Exploratory Data Analysis</h3><p>After we done with the data wrangling we are ready for the data exploration visualizations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*xeH8jgwOcBlp_yrW-OZ79w.jpeg" /><figcaption>Bar Chart and Pie Chart</figcaption></figure><p>The bar chart and pie chart shows total 88199 (79.8%)who were show up and 22316 (20.2%) were absent in the appointment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/1*7q4g6BJ6kcfd8LUQaTwVaQ.jpeg" /><figcaption><em>Data Distribution</em></figcaption></figure><p>In this data distribution we can see quite high number of people who show up in the appointment were not alcoholism, not diabetes, low in handcap, no hypertension, no scholarship and no sms received. All these are reasonable because at first our data consists of 79.8% who are show up.</p><h4>Balance the Responses</h4><p>79.8% mean the data is imbalance, it can be balanced out by dropping some of the “show up” entries become 50:50. However, in this analysis we did not include this balance response step probable in future we will revisit and proceed with balance step. <a href="https://www.analyticsvidhya.com/blog/2017/03/imbalanced-classification-problem/"><strong>Here</strong></a><strong> </strong>explained how to handle imbalanced classes.</p><h4>So What Next?</h4><p>Now let start with two serious question:</p><ol><li>What factors are important for us to know in order to predict if a patient will show up for their scheduled appointment?</li><li>Which day are the most appointment made? How was the attendance rate for each day?</li></ol><h4>Question 1</h4><p>To answer the question we using groupby function to select each of the variables with the no_show result, then compare each of groupby data with pie chart.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/835/1*uF7h3TXQDoOGhRUm3XFF1A.jpeg" /><figcaption>Pie Chart for hancap and alcoholism</figcaption></figure><p>Zero handcap variable give highest attendance rate with 78.2%. <br>Follow by not alcoholism with 77.4% attendance rate.<br>This two are the significant factors to predict if a patient will show up for their scheduled appointment.</p><h4>Question 2</h4><p>We using groupby function to select people who show up with the day of the week , then presented in bar chart and pie chart:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/824/1*JNZI01e2v2zheiNsZa8F_w.jpeg" /></figure><p>Wednesday was the highest total amount of appointment made which is 25867 (23%). Then followed by Tuesday with 25640 (23.2%).</p><h3>Conclusions</h3><p>Handcap and alcoholism are the significant factors to predict if a patient will show up for their scheduled appointment.</p><p>Wednesday was the most popular day for people to attend their appointment.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=688c5db7851e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Let’s Begin]]></title>
            <link>https://medium.com/@wengsengh/lets-begin-4923c7f8a448?source=rss-b037edb4f2a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/4923c7f8a448</guid>
            <dc:creator><![CDATA[Weng Seng]]></dc:creator>
            <pubDate>Thu, 29 Aug 2019 11:31:34 GMT</pubDate>
            <atom:updated>2019-08-29T11:31:34.396Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*etOj4JpafrumiXaKBc-2dw.jpeg" /></figure><p>This was my first project that required to complete for the Udacity Data Analyst Nanodegree (DAND). We been asked us to analyze local and global temperatures trends. This project give us basic ideas to handle the data.</p><p>In this post I will highlights the finding from the project.</p><p>First, we extract the data from SQL database with SQL query and saved in Excel CSV. Next, we need show the data in graph with Excel. Then show the results and analysis in a PDF report.</p><p>The details of the project report can be found <a href="https://github.com/wengsengh/Exploring_Weather_Trends">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*2n-D3SCysdSKm52whZgbMQ.png" /></figure><p><strong>Conclusion</strong>:</p><p>Malaysia and Thailand are tropical country, so the temperature is hotter compare to average global temperature</p><p>The change of temperature in Malaysia, Thailand and global is gradually increasing in past 20 decades result in global warming.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4923c7f8a448" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Airbnb in Singapore: Descriptive and Predictive Analytics]]></title>
            <link>https://medium.com/analytics-vidhya/airbnb-in-singapore-descriptive-and-predictive-analytics-58802975e181?source=rss-b037edb4f2a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/58802975e181</guid>
            <category><![CDATA[airbnb]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Weng Seng]]></dc:creator>
            <pubDate>Sun, 25 Aug 2019 04:41:40 GMT</pubDate>
            <atom:updated>2019-10-10T12:31:05.946Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cqOuQ5HZog_-b0_l49HJ7w.jpeg" /><figcaption>Airbnb® | Singapore</figcaption></figure><h3>Introduction</h3><p>In this post, I will be analyzing the <a href="http://insideairbnb.com/get-the-data.html">Airbnb Dataset in Singapore</a> with <strong>CRISP-DM</strong> process. There were two main datasets used in the analysis: listings data and calendar data for Singapore.</p><h3>CRISP-DM (Cross-Industry Standard Process for Data Mining)</h3><ul><li>Business Understanding</li><li>Data Understanding</li><li>Data Preparation</li><li>Modeling</li><li>Evaluation</li><li>Deployment</li></ul><h3>Business Understanding</h3><p>A‌i‌r‌b‌n‌b‌ is an online marketplace connects people looking for lodging, primarily homestays, or tourism experiences to hosts of spaces available for rent. In this exploration, I try find out the patterns of listings through Airbnb.</p><p>I considered 3 questions to give better view in business understanding:</p><ol><li><strong>When in the year 2019 give the higher revenues in Singapore with Airbnb?</strong></li><li><strong>Which locations give the higher revenue in Singapore with Airbnb?</strong></li><li><strong>What are the property characteristics give higher revenues in Singapore with Airbnb?</strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RoLQqwOj768X9UmjDL9atw.jpeg" /></figure><h3>Data Understanding</h3><p>There were only two dataset which is Listing and calendar. Understand the data will give us some intuitive about data information and what potential feature should include to build the model.</p><h4>Listing Information</h4><p>Here some example of the data variables:</p><ul><li><em>id</em></li><li><em>listing_url</em></li><li><em>host_picture_url</em></li><li><em>neighbourhood_cleansed</em></li><li><em>country</em></li><li><em>property_type</em></li></ul><p>From here we can see what features going to include what features going to drop for further analysis. <br>‘<em>listing_url</em>’, ‘<em>host_picture_url</em>’, ‘<em>country</em>’ to be dropped and the rest remain because ‘<em>listing_url</em>’, ‘<em>host_picture_url</em>’ just url does not provide much information and ‘<em>country</em>’ we know just only Singapore.</p><p><strong>Calendar Information</strong></p><p>Calendar information is important that provide the date information. This is because we can know day of the week, month of the year, season of the year, and festival events. These all are relevant to the current business opportunities. However the date is start from 2019 June to 2020 June, I cannot find explanation from Airbnb open data regarding this time period. I believe is Airbnb privacy protection policy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/907/1*E-cAucZZkSjHs1Swa-MKjg.png" /><figcaption>Source: The Broke Backpacker</figcaption></figure><h3>Data Preparation</h3><p>First is data cleaning like drop unnecessary columns and specified rows with missing value, convert columns to ‘str’, convert columns to ‘1’, ‘0’, convert price columns to ‘float’, convert date columns to datetime and etc.</p><p>Then create two dataset to answers my three questions in business understanding:</p><ol><li>with date dataset</li><li>without date dataset</li></ol><p>Last is feature engineering like convert categorical columns to dummies, create data dummies and train-test split.</p><p>After the data preprocessing let get some sight about the data with some charts:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/957/1*cKZvNG8xFNyveJZTdqXgfA.jpeg" /><figcaption>Average Listing Price Chart</figcaption></figure><p>The average listing price trend were gradually increasing and spike in September and end of December.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*gwLF5857DtdmmgA_5A8Xpg.jpeg" /><figcaption>Total Revenue Chart</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*4nszJso6VOofLu9tpO9IiQ.jpeg" /><figcaption>Total Properties Rented Chart</figcaption></figure><p>From the two chart above, both total revenue chart and total properties rented chart were correlated. This is because higher properties rented the higher amount of revenue is.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/899/1*dyB2ZeuPjpi6lQngC9pnMA.jpeg" /><figcaption>Summer Holiday</figcaption></figure><h3>Question 1: When in the year 2019 give the higher revenues in Singapore with Airbnb?</h3><h3>Modeling</h3><ul><li>We using the dataset with date preproccessing and saved earlier.</li><li>Then we select some features in the dataset we want to focused which is price, month and day of week.</li><li>We calculate the total revenue and total properties rented.</li><li>The total revenue and total properties rented with two different level of date will be plot in bar chart.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*lTX0Gcn6xKZ4GY1n0SniAA.jpeg" /><figcaption><strong>Monthly </strong>Total Revenue and Total Properties Rented</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/776/1*nDTV7kbpaTFKAk9tXTm8Rw.jpeg" /><figcaption><strong>Daily </strong>Total Revenue and Total Properties Rented</figcaption></figure><h3>Evaluation</h3><p>Month as criteria: <br>- Highest total revenue is July and the first 7 months in the year can see uptrend of total revenue.<br>- Hence, property owners can focus more on they property listing in first 7 months of the year.</p><p>Day as criteria:<br>- Tuesday give the highest total revenue, then followed by Friday and Saturday.<br>- Property owners can focus more on they property listing on Tuesday, Friday and Saturday.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nXims455dMyDGvxdGeRvQA.jpeg" /></figure><h3><strong>Question 2: Which locations give the higher revenue in Singapore with Airbnb?</strong></h3><h3>Modeling</h3><ul><li>We using dataset without date</li><li>Then we select some features in the dataset we want to focused which is neighbourhood cleansed, and revenue.’</li><li>We calculate the total earning, number of listing, average earning per listing</li><li>The top 5 highest revenue listing location will be presented in table</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/677/1*GdAuWxWFuYemTeYzKADjfw.jpeg" /><figcaption>Top 5 Highest Revenue Listing Location</figcaption></figure><h3>Evaluation</h3><ul><li>Tuas give the highest average revenue which is $ 2,750,000 but only 1 rented out listing in Tuas.</li><li>hen followed by Southern Islands, Bukit Panjang, Jurong East, and Marina South with $ 347,906; $ 118,588; $ 43,318; $ 40,320 respectively.</li><li>Jurong East recorded highest number of listing which mean existing of greater competition, resulting in more competitive pricing.</li><li>Checking out the Tuas listing actually is <a href="https://www.airbnb.com/rooms/20791161">luxury penthouse</a>. For this case shows the higher investment the higher in return.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DfOwBFxPWZ87NA6U4Tqc7Q.png" /><figcaption>Apartments in Bukit Panjang</figcaption></figure><h3>Question 3: What are the property characteristics give higher revenues in Singapore with Airbnb?</h3><h3>Modeling</h3><ul><li>We using dataset without date.</li><li>Then we select features in the dataset we want to analyze and store in new dataframe.</li><li>We encode the categorical features, and standardize the data values.</li><li>Then we split the data into train and test set.</li><li>Instantiate the prediction model and measuring the error with model scoring.</li><li>We using LinearRegression, DecisionTreeRegressor, Lasso, RidgeCV, RandomForestClassifier for the prediction.</li><li>Lasso give the best result with Training score:0.4875 Test score:0.5161 among the models.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*68Zl44E-2feBzdQLANmF4Q.jpeg" /><figcaption>Features against Coefficients</figcaption></figure><h3>Evaluation</h3><p>Five important property characteristics give higher revenues:</p><ul><li>Price</li><li>neighbourhood_cleansed_Bukit Panjang</li><li>neighbourhood_cleansed_Southern Islands</li><li>property_type_Serviced apartment</li><li>property_type_Apartment</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=58802975e181" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/airbnb-in-singapore-descriptive-and-predictive-analytics-58802975e181">Airbnb in Singapore: Descriptive and Predictive Analytics</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[test]]></title>
            <link>https://medium.com/@wengsengh/test-f4eddce38aff?source=rss-b037edb4f2a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/f4eddce38aff</guid>
            <dc:creator><![CDATA[Weng Seng]]></dc:creator>
            <pubDate>Tue, 06 Aug 2019 11:37:55 GMT</pubDate>
            <atom:updated>2019-09-14T03:36:24.470Z</atom:updated>
            <content:encoded><![CDATA[<h3>Flower Species Classifier With PyTorch</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*DicQEFTpV3pbrgqA2__HpQ.jpeg" /></figure><h3>Introduction to the Project</h3><p>In this article I will show how to create an image classifier with PyTorch. I going to use transfer learning to train pre-trained neural networks to classify 102 different species of flowers.</p><p>The flower data and information can be found <a href="http://www.robots.ox.ac.uk/~vgg/data/flowers/102/index.html">here</a>. The dataset consists of 102 folders each species per folder. The codes details of this project can be found at my github <a href="https://github.com/wengsengh/Flowers-Image-Classifier">here</a>.</p><p>Overview of the flower data within a species:</p><figure><img alt="English Marigold" src="https://cdn-images-1.medium.com/max/1024/1*fpIUpPiiSg4zFzWMDVh8xw.jpeg" /><figcaption>English Marigold</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h4uiQq6IR_avPqYc-2D0Ug.jpeg" /><figcaption>Snapdragon</figcaption></figure><p>In this project, the folder structure will be 3 folders which contain all the flower species for each folder:</p><ol><li>Train</li><li>Validation</li><li>Test</li></ol><p>Since my laptop is kinda old which currently run on i5 and NVIDIA Cuda 540 is not easy to handle larger dataset with very deep neutral network. It like taking few hours to finished the task which very time consuming. So I decided to use Google Colab because got free GPU.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HngJw5gCmTKn1zjZcgUfew.jpeg" /></figure><h4>Thanks Google!!!</h4><p>First the flower dataset I will uploads into my google drive then from there I mount my drive to my colab:</p><pre>from google.colab import drive<br>drive.mount(&#39;/content/drive&#39;)</pre><h3>Steps:</h3><p>Step 1: Load Dataset<br>Step 2: Transform the Dataset<br>Step 3: Create Model<br>Step 4: Train Model<br>Step 5: Save the Model<br>Step 6: Load the Model<br>Step 7: Predict the Image<br>Step 8: Show the result</p><h3>Step 1: Load Dataset</h3><p>It simple basically just load in the data and assign it.</p><pre>data_dir = &#39;/content/drive/My Drive/Colab Notebooks/flowers&#39;<br>train_dir = data_dir + &#39;/train&#39;<br>valid_dir = data_dir + &#39;/valid&#39;<br>test_dir = data_dir + &#39;/test&#39;</pre><p>Udacity provided label mapping in JSON file so just load it for later use.</p><pre>with open(&#39;/content/drive/My Drive/name.json&#39;, &#39;r&#39;) as f:<br>    name = json.load(f)</pre><h3>Step 2: Transform the Dataset</h3><p>In the training dataset I will transform the original image in resize, crop, rotate, and flip. Training data set cropped, rotated, and flipped and not in validation and test dataset because I want create more robust model in training dataset. For validation and test dataset just resized center cropped.</p><p>Then the image values also normalize with the means and standard deviations</p><pre>train_transforms = transforms.Compose([transforms.RandomRotation(30),<br>transforms.RandomResizedCrop(224),<br>transforms.RandomHorizontalFlip(),<br>transforms.ToTensor(),<br>transforms.Normalize([0.485, 0.456, 0.406],[0.229, 0.224, 0.225])])</pre><pre>test_transforms = <br>transforms.Compose([transforms.Resize(256),<br>transforms.CenterCrop(224),<br>transforms.ToTensor(),<br>transforms.Normalize([0.485, 0.456, 0.406],[0.229, 0.224, 0.225])])</pre><pre>validation_transforms = <br>transforms.Compose([transforms.Resize(256),<br>transforms.CenterCrop(224),<br>transforms.ToTensor(),<br>transforms.Normalize([0.485, 0.456, 0.406],[0.229, 0.224, 0.225])])</pre><p>ImageFolder and DataLoader is used to pass the image that went through the transformation and then stored it for training, validation or prediction later.</p><h3>Step 3: Create Model</h3><p>For the pretrained model I use VGG19 architecture. There still got some other popular pre-trained models like ResNet, AlexNet and densenet121.</p><pre>model = models.vgg19(pretrained=True)</pre><p>For VGG19 the final layer output is 25088. In our case, I want predict 102 flower classes so I use ordered dictionary to create the final layers of my network with 102 output. So I define new untrained feed-forward network as a classifier, using ReLU activations and dropout. LogSoftmax is used for the output values from the model for each image prediction.</p><pre>from collections import OrderedDict<br>        classifier = nn.Sequential(OrderedDict([<br>                          (&#39;fc1&#39;, nn.Linear(25088, 500)),<br>                          (&#39;relu&#39;, nn.ReLU()),<br>                          (&#39;dropout1&#39;, nn.Dropout(dropout)),<br>                          (&#39;fc2&#39;, nn.Linear(500, 102)),<br>                          (&#39;output&#39;, nn.LogSoftmax(dim=1))<br>                          ]))</pre><p>Replace the classifier of VGG19:</p><pre>model.classifier = classifier</pre><p>The criterion method used to evaluate the model, optimizer method used to update the weights.</p><pre>criterion = nn.NLLLoss()<br>optimizer = optim.Adam(model.classifier.parameters(), lr)</pre><h3>Step 4: Train Model</h3><p>Here I want to train the final layers. The epochs is the run of feedforward and backpropagation through the neural network. The device is set to cuda which mean only can run in GPU if want to run in CPU need to set to ‘cpu’.</p><p>The validation accuracy result give me around 83% for all the flower species. Higher accuracy result can achieved by model tuning, the parameters need to reconsider will be learning rate, unit in classifier, number of epochs and type of optimizer.</p><pre>epochs = 10<br>steps = 0<br>print_every = 40</pre><pre># change to gpu mode<br>model.to(&#39;cuda&#39;)</pre><pre>for e in range(epochs):</pre><pre>running_loss = 0<br>    <br>    # Iterating over data to carry out training step<br>    for ii, (inputs, labels) in enumerate(trainloader):<br>        steps += 1<br>        <br>        inputs, labels = inputs.to(&#39;cuda&#39;), labels.to(&#39;cuda&#39;)<br>        <br>        # zero parameter gradients<br>        optimizer.zero_grad()<br>        <br>        # Forward and backward passes<br>        outputs = model.forward(inputs)<br>        loss = criterion(outputs, labels)<br>        loss.backward()<br>        optimizer.step()<br>        <br>        running_loss += loss.item()<br>        <br>        # Carrying out validation step<br>        if steps % print_every == 0:<br>            # setting model to evaluation mode during validation<br>            model.eval()<br>            <br>            # Gradients are turned off as no longer in training<br>            with torch.no_grad():<br>                valid_loss, accuracy = validation(model, validloader, criterion)<br>                <br>            print(f&quot;No. epochs: {e+1}, \<br>            Training Loss: {round(running_loss/print_every,3)} \<br>            Valid Loss: {round(valid_loss/len(validloader),3)} \<br>            Valid Accuracy: {round(float(accuracy/len(validloader)),3)}&quot;)<br>            <br>            running_loss = 0<br>            <br>            # Turning training back on<br>            model.train()</pre><h3>Step 5: Save the Model</h3><p>Now the network is trained, I need to save the model so I can load it later for making predictions.<br>In PyTorch it allows me to save checkpoint of my models with torch.save function. This checkpoint here saves the model with classifier, optimizer, epochs,model.state_dict, and class_to_idx.</p><p>model.state_dict keep all of the weights and biases of our model for each layer in a dictionary</p><p>class_to_idx keeps track of our mapping of flower class values to the flower indices.</p><pre>checkpoint = {&#39;state_dict&#39;: model.state_dict(),<br>              &#39;classifier&#39;: model.classifier,<br>              &#39;class_to_idx&#39;: train_data.class_to_idx,<br>              &#39;opt_state&#39;: optimizer.state_dict,<br>              &#39;num_epochs&#39;: epochs}</pre><pre>torch.save(checkpoint, &#39;/content/drive/My Drive/Colab Notebooks/my_checkpoint1.pth&#39;)</pre><h3>Step 6: Load the Model</h3><p>After I saved the model in pth file so I should able to load back the model. In order to load I can simply use load function in PyTorch by providing the filepath.</p><pre>checkpoint = torch.load(filepath)<br>    <br>    # Checkpoint for when using CPU<br>    #checkpoint = torch.load(filepath, map_location=lambda storage, loc: storage)<br>    <br>    model.load_state_dict(checkpoint[&#39;state_dict&#39;])<br>    model.classifier = checkpoint[&#39;classifier&#39;] <br>    model.class_to_idx = checkpoint[&#39;class_to_idx&#39;]</pre><h3>Step 7: Predict the Image</h3><p>Before predict the image, I need to process the input image so that it can be used in the network.<br>I will use use <a href="https://pillow.readthedocs.io/en/latest/reference/Image.html">PIL </a>to load the image.</p><p>First, resize the images to 256 pixels. This can be done with <a href="http://pillow.readthedocs.io/en/3.1.x/reference/Image.html#PIL.Image.Image.thumbnail">resize</a> method. Second, I need to crop out the center 224x224 portion of the image with CenterCrop method. <br>Then I normalized the image with Normalize function. The idea is simply subtract the means from each color channel, then divide by the standard deviation.<br>Lastly the model expected floats 0–1 but color channels of images are encoded as integers 0–255, so I need convert it with Numpy array.</p><pre>def process_image(image):</pre><pre>&#39;&#39;&#39; Scales, crops, and normalizes a PIL image for a PyTorch model,</pre><pre>returns an Numpy array</pre><pre>&#39;&#39;&#39;</pre><pre># Converting image to PIL image using image file path<br>pil_im = Image.open(f&#39;{image}&#39; + &#39;.jpg&#39;)<br></pre><pre># Building image transform</pre><pre>transform = transforms.Compose([transforms.Resize(256),</pre><pre>transforms.CenterCrop(224),</pre><pre>transforms.ToTensor(),</pre><pre>transforms.Normalize([0.485, 0.456, 0.406],</pre><pre>[0.229, 0.224, 0.225])])<br></pre><pre>## Transforming image for use with network<br>pil_tfd = transform(pil_im)<br></pre><pre># Converting to Numpy array<br>array_im_tfd = np.array(pil_tfd)</pre><pre>return array_im_tfd</pre><p>To check process_image function works, the function below imshowconverts a PyTorch tensor and display the image as a result.</p><p>I reorder dimensions using <a href="https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.ndarray.transpose.html">ndarray.transpose</a>. This is because PyTorch expects the color channel (RGB) to be first dimension where in PIL image is in third dimension.</p><pre>def imshow(image, ax=None, title=None):</pre><pre>if ax is None:</pre><pre>fig, ax = plt.subplots()</pre><pre># PyTorch tensors assume the color channel is the first dimension</pre><pre># but matplotlib assumes is the third dimension</pre><pre>image = image.transpose((1, 2, 0))</pre><pre># Undo preprocessing</pre><pre>mean = np.array([0.485, 0.456, 0.406])</pre><pre>std = np.array([0.229, 0.224, 0.225])</pre><pre>image = std * image + mean<br></pre><pre># Image needs to be clipped between 0 and 1 or it looks like noise when displayed</pre><pre>image = np.clip(image, 0, 1)</pre><pre>ax.imshow(image)</pre><pre>return ax</pre><p>I try run the function to see image pre-processing successful or not</p><pre>image_name = data_dir + &#39;/test&#39; + &#39;/1/&#39; + &#39;image_06743&#39;<br>img_test =  process_image(image_name)<br>imshow(img_test)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/288/1*H9vcAK7Dm0Dlf4ZEqp2VBA.jpeg" /><figcaption>The Result</figcaption></figure><h3>Class Prediction</h3><p>Once the image in the correct format, it’s time making predictions with the model. A common practice is to predict the top 5 (usually called top-K) most probable classes. I need calculate the class probabilities then find the highest values.</p><pre>img = process_image(image_path)   <br>img_tensor = torch.from_numpy(img).type(torch.FloatTensor)<br>img_add_dim = img_tensor.unsqueeze_(0)</pre><pre>probs = torch.exp(output)<br>    probs_top = probs.topk(5)[0]<br>    index_top = probs.topk(5)[1]</pre><p>I changed the tensors to numpy arrays:</p><pre>probs_top_list = np.array(probs_top)[0]</pre><pre>index_top_list = np.array(index_top[0])</pre><p>Then I loading the index and class mapping then switch the direction of x, y index</p><pre>class_to_idx = loaded_model.class_to_idx<br><em># Inverting index-class dictionary</em><br>indx_to_class = {x: y <strong>for</strong> y, x <strong>in</strong> class_to_idx.items()}</pre><p>Converting index list to class list by index the idx_to_class then pull the top 5 classes.</p><pre>classes_top_list = []<br>    <strong>for</strong> index <strong>in</strong> index_top_list:<br>        classes_top_list += [indx_to_class[index]]</pre><h3>Step 8: Show the result</h3><p>Use matplotlib to plot the probabilities for the top 5 classes as a bar graph, along with the input image.</p><pre># Plotting test image and predicted probabilites</pre><pre>f, ax = plt.subplots(2,figsize = (6,10))</pre><pre>ax[0].imshow(image)</pre><pre>ax[0].set_title(names[0])</pre><pre>y_names = np.arange(len(names))</pre><pre>ax[1].barh(y_names, probs, color=&#39;darkblue&#39;)</pre><pre>ax[1].set_yticks(y_names)</pre><pre>ax[1].set_yticklabels(names)</pre><pre>ax[1].invert_yaxis()</pre><pre>plt.show()</pre><p>Here the example result that the model predicted correctly.</p><pre>image_path = data_dir + &#39;/test&#39; + &#39;/1/&#39;+ &#39;image_06743&#39;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/459/1*NXpON84m-npIut0JvS8mPQ.jpeg" /><figcaption>The Top 5 Classes Prediction</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f4eddce38aff" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>