<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jiew Wan Tan on Medium]]></title>
        <description><![CDATA[Stories by Jiew Wan Tan on Medium]]></description>
        <link>https://medium.com/@jiewwantan?source=rss-c3e3acabee1c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*0aK0WxOMPuJx29n062IgXA.jpeg</url>
            <title>Stories by Jiew Wan Tan on Medium</title>
            <link>https://medium.com/@jiewwantan?source=rss-c3e3acabee1c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 04:19:33 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jiewwantan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Sparkify user churn prediction using PySpark]]></title>
            <link>https://medium.com/data-science/sparkify-user-churn-prediction-using-pyspark-32be364e8296?source=rss-c3e3acabee1c------2</link>
            <guid isPermaLink="false">https://medium.com/p/32be364e8296</guid>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[aws-emr]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[churn-prediction]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Jiew Wan Tan]]></dc:creator>
            <pubDate>Sun, 25 Aug 2019 14:06:50 GMT</pubDate>
            <atom:updated>2019-08-30T13:46:36.225Z</atom:updated>
            <content:encoded><![CDATA[<h3>Sparkify user churn prediction using Pyspark</h3><h4>Predicting music streaming service user churn on local machine and AWS EMR.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CfIKuvdC0dTaN8LEBwtTcA.jpeg" /><figcaption><a href="http://www.freepik.com/">Designed by Freepik</a></figcaption></figure><h3>Overview</h3><p>User churn (cancellation) prediction is an imperative predictive tool. This project sets to solve this problem for a music streaming service: Sparkify. By exploring Sparkify usage data, the project identifies features for model learning. For computation efficiency reason, a tiny dataset (240Mb), a sample of the full dataset (12Gb) is used for initial data exploration, feature engineering and modelling experimentation on a local machine.</p><p>The initial work on the tiny dataset shall identify the most suitable model and hyperparameters for the full dataset to train the final model . Once features and model are identified, they will be used for modelling the full dataset on AWS EMR. Assuming the sample dataset represents the population dataset, hyperparameters tuned by the sample dataset shall generalise well and applicable to modelling on the full dataset too. We will see if this assumption can help save some computation resource without having to grid search on big data.</p><p>The actionable insight gained from churn prediction will identify users who are likely to churn and send them offers that hopefully keep them from clicking cancellation confirmation.</p><h3>Exploratory Data Analysis (EDA)</h3><p>The dataset schema below shows the dataset structure and the columns available for feature engineering. From EDA, we shall identify what features to engineer.</p><pre>root<br> |-- artist: string (nullable = true)<br> |-- auth: string (nullable = true)<br> |-- firstName: string (nullable = true)<br> |-- gender: string (nullable = true)<br> |-- itemInSession: long (nullable = true)<br> |-- lastName: string (nullable = true)<br> |-- length: double (nullable = true)<br> |-- level: string (nullable = true)<br> |-- location: string (nullable = true)<br> |-- method: string (nullable = true)<br> |-- page: string (nullable = true)<br> |-- registration: long (nullable = true)<br> |-- sessionId: long (nullable = true)<br> |-- song: string (nullable = true)<br> |-- status: long (nullable = true)<br> |-- ts: long (nullable = true)<br> |-- userAgent: string (nullable = true)<br> |-- userId: string (nullable = true)</pre><p>“Churn” label is generated from the dataset by identifying users who confirm their subscription cancellation. Once churned users are identified, we can see how it behaves with other columns in the dataset:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*SvUC_p88BdB1LHlVFarSwA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*UYXch_MZPnIr4TeeVd8v-Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*JM-GYZmwGwts8WGHjc89Wg.png" /><figcaption>Figure 1a, 1b and 1c</figcaption></figure><p>From the bar charts above, with 3 main data columns (Churn status, subscription level and gender) sorted differently, we can see that data are skewed towards one end and as such distributed rather unevenly. Non-churn users are much more that churn user, so are gender and subscription level. This is an important point to note for model learning and metric selection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*f97PChARdWqIxcgq_4pO-g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*rL8RmUA4VSKclWfi53se_Q.png" /><figcaption>Figure 2a and 2c</figcaption></figure><p>Figure 2a and 2c show that user upgrades are distributed more evenly than user downgrades.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GpF6jbm_Fx9SgflSYPfeag.png" /><figcaption>Figure 3</figcaption></figure><p>User locations scattered widely and are rather sparse in almost all locations. Unless locations can be grouped into categories of geographical locations, this may not be helpful for modelling. Intuitively, we can extract state codes from the locations’ last two characters to create a categorical feature for state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*3N_ahAAoNMixXvipr9V8zg.png" /><figcaption>Figure 4a</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*JHdbuPjRdJzSFsTIWFDjUw.png" /><figcaption>Figure 4b</figcaption></figure><p>Page visits are also heavily skewed away from “NextSong” where most interactions concentrate. With this removed, user page interaction distribution could have been more obvious. The pages that capture users’ behaviour are likely “Add Friend”, “Add to playlist”, “Cancellation confirmation” (adopted for “Churn” label generation), “Downgrade”, “Roll advert” (helpful for ads display insight), “Submit Downgrade”, “Submit Upgrade”, “Thumbs Down” and “Thumbs Up”. Among these, “Add Friend” and “Add to playlist” shall be be adopted for feature engineering. The other columns are likely helpful in predictions other than churn.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*Gt-6IuAbfRM-HUaDRytNqg.png" /><figcaption>Figure 5</figcaption></figure><p>Aggregating user counts by hour of the day, the streaming service looks to have more users from late evenings towards after mid night. However, user churn do not look to have an obvious trend with regards to hour of the day.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*B7uIwiu2L2GDQOqXKqP09w.png" /><figcaption>Figure 6</figcaption></figure><p>Page visits arranged by hour of the day share similar insight as its user count counterpart with a more prominent trend.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*93smIPbnrXlSRhC9-dTWbw.png" /><figcaption>Figure 7</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*SAAXzFANchW2J1SJVnsDGA.png" /><figcaption>Figure 8</figcaption></figure><p>The day of the week plots show more user engagement during weekdays.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RUUm3bj-eXVV4p0r2sMYSw.png" /><figcaption>Figure 9</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oLcsac1hZoT4nMLjxlXDXw.png" /><figcaption>Figure 10</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*68aTKcVidkhImdYVZ3_dsw.png" /><figcaption>Figure 11</figcaption></figure><p>Instead of aggregating data by day or hour, we can also look at how trend evolves over the period of the dataset. Although the tiny dataset has only two months of data to explore, the insights revealed here are rather interesting. Its periodical trend agrees with weekday plots in which users engaged more towards midweek. It also shows how user behaviour changes over the two-months period: less churn and more upgrades. We need more info to know how many upgrades are from existing and new users as well as its causal factor.</p><p>If computational resource allows, it will be even more interesting to see how this trend evolves in the full dataset.</p><h3>Feature Engineering</h3><p>After exploratory data analysis, 10 features hypothetically assumed to play a part in determining user churn were engineered. Following that, feature importance as a result of model training will determine what model and features to be adopted for full dataset modelling.</p><p>As a result, a Spark dataframe is created with 10 features and 1 label:</p><ol><li>Gender (binary)</li><li>Paid or Free (binary)</li><li>Total number of songs listened (numerical)</li><li>Total number of artist listened (numerical)</li><li>Number of songs in playlist(s) added by user (numerical)</li><li>Number of friends added by user (numerical)</li><li>Total length of listening time (numerical)</li><li>Average number of songs per session (numerical)</li><li>Average time per session (numerical)</li><li>Number of session per user (numerical)</li><li>Label: Number of session per user (binary)</li></ol><p>The dataframe is split into 80% for training, 20% for testing.</p><h3>Modeling</h3><p>Modelling is done with 3-fold cross-validation of the training set and grid searching of best hyperparameters for four classifiers. Best models are identified with evaluation results and feature importance for each classifier:</p><ol><li>Logistic regression</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EoTt13Qsdy7PGOGCEqx5vw.png" /><figcaption>Figure 12</figcaption></figure><pre>Evaluation result (With GridSearch):<br>+---------+------+------+--------+<br>|precision|recall|    f1|accuracy|<br>+---------+------+------+--------+<br>|   0.6684|0.7606|0.6753|  0.7606|<br>+---------+------+------+--------+<br>Training time 19.93 minutes</pre><pre>Evaluation result (Without GridSearch):<br>+---------+------+------+--------+<br>|precision|recall|    f1|accuracy|<br>+---------+------+------+--------+<br>|   0.6684|0.7606|0.6753|  0.7606|<br>+---------+------+------+--------+<br>Training time 10.31 minutes</pre><p>2. Decision tree</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0L_oL3fl9JP_1VKT0Ojdfw.png" /><figcaption>Figure 13</figcaption></figure><pre>Evaluation result (With GridSearch):<br>+---------+------+------+--------+<br>|precision|recall|    f1|accuracy|<br>+---------+------+------+--------+<br>|    0.738|0.7746|0.7351|  0.7746|<br>+---------+------+------+--------+<br>Training time 11.05 minutes</pre><pre>Evaluation result (Without GridSearch):<br>+---------+------+------+--------+<br>|precision|recall|    f1|accuracy|<br>+---------+------+------+--------+<br>|   0.7355|0.7746|0.7134|  0.7746|<br>+---------+------+------+--------+<br>Training time 8.49 minutes</pre><p>3. Gradient boosted tree</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TTJ2Y4vRExphQ62Mbb-rIw.png" /><figcaption>Figure 14</figcaption></figure><pre>Evaluation result: (With GridSearch)<br>+---------+------+------+--------+<br>|precision|recall|    f1|accuracy|<br>+---------+------+------+--------+<br>|   0.8611|0.8662|0.8622|  0.8662|<br>+---------+------+------+--------+<br>Training time 187.47 minutes</pre><pre>Evaluation result (Without GridSearch):<br>+---------+------+------+--------+<br>|precision|recall|    f1|accuracy|<br>+---------+------+------+--------+<br>|   0.8346| 0.838|0.8158|   0.838|<br>+---------+------+------+--------+<br>Training time 11.74 minutes</pre><p>4. Random forest</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fu_eVg1kgTR-9lV6qe9bhg.png" /><figcaption>Figure 15</figcaption></figure><pre>Evaluation result: (With GridSearch)<br>+---------+------+------+--------+<br>|precision|recall|    f1|accuracy|<br>+---------+------+------+--------+<br>|   0.8387|0.7958|0.7277|  0.7958|<br>+---------+------+------+--------+<br>Training time 11.36 minutes</pre><pre>Evaluation result (Without GridSearch):<br>+---------+------+-----+--------+<br>|precision|recall|   f1|accuracy|<br>+---------+------+-----+--------+<br>|   0.8258|0.7746|0.683|  0.7746|<br>+---------+------+-----+--------+<br>Training time 9.23 minutes</pre><p>F1 score is adopted as the criteria for model selection as it considers both recall and precision. The dataset has a disproportionate distribution of label and binary features. F1 score will ensure that our model is not confused by a mis-classified <a href="https://towardsdatascience.com/understanding-confusion-matrix-a9ad42dcfd62">confusion matrix</a>: classifying 100 % True for a dataset with 95% True label to achieve 95% accuracy rate, doesn’t matter the 5% False positives.</p><p>With an F1 score of 0.8622, Gradient boosting tree is selected to be used for modelling the large dataset with the following hyperparameters:</p><blockquote>best_model._java_obj.getMaxDepth() = 8</blockquote><blockquote>best_model._java_obj.getMaxIter() =30</blockquote><blockquote>best_model._java_obj.getMaxBins() = 40</blockquote><p>Training the large dataset by 3 x AWS EMR m3.xlarge instances took about 160 mins and produced the following evaluation result on its test set:</p><pre>Evaluation result:<br>+------+------+---------+--------+<br>|    f1|recall|precision|accuracy|<br>+------+------+---------+--------+<br>|0.7254|0.7868|   0.7444|  0.7908|<br>+------+------+---------+--------+</pre><h3>Tiny Data vs Big Data</h3><h4>Computation complexity</h4><p>Due to difference in <a href="https://en.wikipedia.org/wiki/Computational_complexity">computation complexity</a> for different learning, optimisation or even data wrangling algorithms, performing tasks on a tiny dataset and a large dataset can impose significant difference in time and resources. As such it is imperative to have a good sense of varied algorithms’ computation complexity to have a good judgement in task allocation (local vs distributed computing).</p><p>It is expensive to perform exploratory data analysis (EDA) in the cloud. A comprehensive EDA on a large dataset like this will easily costs US$25 of EC2 and EMR charges per round. As such it is important to have a good understanding of the dataset by performing local EDA, deciding what should be hypothesised and what shouldn’t of the large dataset before engaging cloud service to perform modelling.</p><h4>Hyperparameter tuning</h4><p>Grid-search is a computational expensive task to perform on a large dataset. To tune the right hyperparameters on a large dataset, it is easily an <em>O(</em>n-square) if not <em>O</em>(n-power) <a href="https://en.wikipedia.org/wiki/Time_complexity">time complexity</a> problem. Randomised search can be a faster and cheaper alternative for some performance trade-off. One smarter implementation of hyperparameter tuning is to combine random search and grid search:</p><ul><li>Use random search with a large hyperparameter grid</li><li>Use the results of random search to build a focused hyperparameter grid around the best performing hyperparameter values.</li><li>Run grid search on the reduced hyperparameter grid.</li><li>Repeat grid search on more focused grids until maximum computational/time budget is exceeded.</li></ul><p>A comparison of hyperparameter tuning with and without grid search comparison, the modelling evaluation results above shows that the trade-off between model performance and computational resources is mixed for the four classifiers experimented. We save almost half of the Logistic regression training time without grid search for no difference in model performance improvement. For GradientBoosting Tree (GBT)the performance improvement with grid search is substantial: F1 score of 0.8622 (with grid search) vs 0.8158 (without grid search). However its computation took almost 3 hours more with grid search. GBT is certainly a high computational complexity algorithm that increases exponentially as dataset increase in size. The grid search was done on a 240Mb dataset, imagine if it was done on a 12Gb dataset!</p><p>In comparison, Decision Tree and Random Forest’s trade off is more linear, a minor performance improvement with minor increase in computation demand. Therefore by knowing a learning algorithm’s computation complexity, we can make wiser decision to achieve an optimum performance vs computation trade off.</p><h4>Statistical comparison of features and label</h4><p>There are certainly some statistical differences (see below for statistical comparison) between the features generated by the tiny and large dataset and as such the evaluation result are considerably different: 0.862 (tiny dataset) vs 0.725 (full dataset). The following provides a glimpse of the two datasets that could probably explain the model performance difference:</p><ol><li>Gender (binary)</li></ol><pre>Tiny dataset &gt;<br>+-------+-----------------+------------------+<br>|summary|           userId|            gender|<br>+-------+-----------------+------------------+<br>|  count|              448|               448|<br>|   mean|67520.34821428571|0.4419642857142857|<br>| stddev|105964.5842848519|0.4971756279079038|<br>|    min|               10|                 0|<br>|    max|               99|                 1|<br>+-------+-----------------+------------------+<br>+-------------------+-------------------+<br>|   skewness(gender)|   kurtosis(gender)|<br>+-------------------+-------------------+<br>|0.23372261898725685|-1.9453737373737383|<br>+-------------------+-------------------+</pre><pre>Full dataset &gt;<br>+-------+------------------+-------------------+<br>|summary|            userId|             gender|<br>+-------+------------------+-------------------+<br>|  count|             22278|              22278|<br>|   mean|1498782.9615764432|0.47697279827632644|<br>| stddev| 288851.8472659188| 0.4994806768184825|<br>|    min|           1000025|                  0|<br>|    max|           1999996|                  1|<br>+-------+------------------+-------------------+<br>+-------------------+-------------------+<br>|   skewness(gender)|   kurtosis(gender)|<br>+-------------------+-------------------+<br>|0.09220664431939639|-1.9914979347433575|<br>+-------------------+-------------------+</pre><p>2. Paid or Free (binary)</p><pre>Tiny dataset &gt;<br>+-------+------------------+------------------+<br>|summary|            userId|             level|<br>+-------+------------------+------------------+<br>|  count|               691|               691|<br>|   mean| 67259.47033285093|0.4645441389290883|<br>| stddev|106161.02506630082|0.4991025734716588|<br>|    min|                10|                 0|<br>|    max|                99|                 1|<br>+-------+------------------+------------------+<br>+-------------------+-------------------+<br>|    skewness(level)|    kurtosis(level)|<br>+-------------------+-------------------+<br>|0.14218137235745473|-1.9797844573545489|<br>+-------------------+-------------------+</pre><pre>Full dataset &gt;<br>+-------+------------------+-------------------+<br>|summary|            userId|              level|<br>+-------+------------------+-------------------+<br>|  count|             22278|              22278|<br>|   mean|1498782.9615764432| 0.5992010054762547|<br>| stddev| 288851.8472659188|0.49007136327332323|<br>|    min|           1000025|                  0|<br>|    max|           1999996|                  1|<br>+-------+------------------+-------------------+<br>+-------------------+-------------------+<br>|    skewness(level)|    kurtosis(level)|<br>+-------------------+-------------------+<br>|-0.4025645802550173|-1.8379417587241018|<br>+-------------------+-------------------+</pre><p>3. Total number of songs listened</p><pre>Tiny dataset &gt;<br>+-------+-----------------+------------------+<br>|summary|           userID|          num_song|<br>+-------+-----------------+------------------+<br>|  count|              448|               448|<br>|   mean|67520.34821428571|1178.5825892857142|<br>| stddev|105964.5842848519|1380.6279647524054|<br>|    min|               10|                 3|<br>|    max|               99|              9767|<br>+-------+-----------------+------------------+<br>+------------------+------------------+<br>|skewness(num_song)|kurtosis(num_song)|<br>+------------------+------------------+<br>|2.2993135439060457| 6.814549363048505|<br>+------------------+------------------+</pre><pre>Full dataset &gt;<br>+-------+------------------+------------------+<br>|summary|            userID|          num_song|<br>+-------+------------------+------------------+<br>|  count|             22278|             22278|<br>|   mean|1498782.9615764432|1178.7054044348686|<br>| stddev|288851.84726591856|  5372.95993988227|<br>|    min|           1000025|                 1|<br>|    max|           1999996|            778479|<br>+-------+------------------+------------------+<br>+------------------+------------------+<br>|skewness(num_song)|kurtosis(num_song)|<br>+------------------+------------------+<br>|135.95349045633083|19660.671563405533|<br>+------------------+------------------+</pre><p>4. Total number of artist listened</p><pre>Tiny dataset &gt;<br>+-------+-----------------+-----------------+<br>|summary|           userId|       num_artist|<br>+-------+-----------------+-----------------+<br>|  count|              448|              448|<br>|   mean|67520.34821428571|658.9799107142857|<br>| stddev|105964.5842848519| 625.882698051957|<br>|    min|               10|                1|<br>|    max|               99|             3507|<br>+-------+-----------------+-----------------+<br>+--------------------+--------------------+<br>|skewness(num_artist)|kurtosis(num_artist)|<br>+--------------------+--------------------+<br>|  1.4675037203781365|  2.2190317071776393|<br>+--------------------+--------------------+</pre><pre>Full dataset &gt;<br>+-------+------------------+-----------------+<br>|summary|            userId|       num_artist|<br>+-------+------------------+-----------------+<br>|  count|             22261|            22261|<br>|   mean|1498833.2082116706|645.0307263824626|<br>| stddev| 288882.1163228876|602.2479741901458|<br>|    min|           1000025|                1|<br>|    max|           1999996|             4368|<br>+-------+------------------+-----------------+<br>+--------------------+--------------------+<br>|skewness(num_artist)|kurtosis(num_artist)|<br>+--------------------+--------------------+<br>|  1.5260667285754526|   2.656182841474317|<br>+--------------------+--------------------+</pre><p>5. Number of songs in playlist(s) added by user</p><pre>Tiny dataset &gt;<br>+-------+------------------+------------------+<br>|summary|            userID| num_playlist_song|<br>+-------+------------------+------------------+<br>|  count|               428|               428|<br>|   mean| 65764.93457943926|28.852803738317757|<br>| stddev|105363.38578382804|33.913090694566286|<br>|    min|                10|                 1|<br>|    max|                99|               248|<br>+-------+------------------+------------------+<br>+---------------------------+---------------------------+<br>|skewness(num_playlist_song)|kurtosis(num_playlist_song)|<br>+---------------------------+---------------------------+<br>|         2.4048605508981393|          7.736784413479519|<br>+---------------------------+---------------------------+</pre><pre>Full dataset &gt;<br>+-------+------------------+-----------------+<br>|summary|            userID|num_playlist_song|<br>+-------+------------------+-----------------+<br>|  count|             21260|            21260|<br>|   mean|1498898.9698494826|28.12422389463782|<br>| stddev|289180.40429718536|32.27499039023108|<br>|    min|           1000025|                1|<br>|    max|           1999996|              340|<br>+-------+------------------+-----------------+<br>+---------------------------+---------------------------+<br>|skewness(num_playlist_song)|kurtosis(num_playlist_song)|<br>+---------------------------+---------------------------+<br>|         2.3914875986095625|          8.073009618134558|<br>+---------------------------+---------------------------+</pre><p>6. Number of friends added by user</p><pre>Tiny dataset &gt;<br>+-------+------------------+------------------+<br>|summary|            userID|        num_friend|<br>+-------+------------------+------------------+<br>|  count|               409|               409|<br>|   mean|  66368.3716381418|19.772616136919314|<br>| stddev|106064.01609030597| 22.49443576627283|<br>|    min|                10|                 1|<br>|    max|                99|               158|<br>+-------+------------------+------------------+<br>+--------------------+--------------------+<br>|skewness(num_friend)|kurtosis(num_friend)|<br>+--------------------+--------------------+<br>|  2.4002649609428586|   7.678714779071942|<br>+--------------------+--------------------+</pre><pre>Full dataset &gt;<br>+-------+------------------+------------------+<br>|summary|            userID|        num_friend|<br>+-------+------------------+------------------+<br>|  count|             20305|             20305|<br>|   mean| 1499371.503718296| 18.79655257325782|<br>| stddev|288830.59626148926|20.747704116295065|<br>|    min|           1000025|                 1|<br>|    max|           1999996|               222|<br>+-------+------------------+------------------+<br>+--------------------+--------------------+<br>|skewness(num_friend)|kurtosis(num_friend)|<br>+--------------------+--------------------+<br>|  2.3834675795984976|   8.182711524378096|<br>+--------------------+--------------------+</pre><p>7. Total length of listening time</p><pre>Tiny dataset &gt;<br>+-------+-----------------+------------------+<br>|summary|           userID|       time_listen|<br>+-------+-----------------+------------------+<br>|  count|              448|               448|<br>|   mean|67520.34821428571|240270.49760906256|<br>| stddev|105964.5842848519|  286257.952604531|<br>|    min|               10|         131.00363|<br>|    max|               99|     2019435.10394|<br>+-------+-----------------+------------------+<br>+---------------------+---------------------+<br>|skewness(time_listen)|kurtosis(time_listen)|<br>+---------------------+---------------------+<br>|    2.303088337172893|    6.791766280616427|<br>+---------------------+---------------------+</pre><pre>Full dataset &gt;<br>+-------+------------------+------------------+<br>|summary|            userID|       time_listen|<br>+-------+------------------+------------------+<br>|  count|             22278|             22261|<br>|   mean|1498782.9615764432|232963.16116480672|<br>| stddev| 288851.8472659186|273559.41985437507|<br>|    min|           1000025|          78.49751|<br>|    max|           1999996|     2807182.33115|<br>+-------+------------------+------------------+<br>+---------------------+---------------------+<br>|skewness(time_listen)|kurtosis(time_listen)|<br>+---------------------+---------------------+<br>|    2.439989611449916|    8.466901311178267|<br>+---------------------+---------------------+</pre><p>8. Average number of songs per session</p><pre>Tiny dataset &gt;<br>+-------+-----------------+-----------------+<br>|summary|           userId|        avg_songs|<br>+-------+-----------------+-----------------+<br>|  count|              448|              448|<br>|   mean|67520.34821428571|65.61168409976814|<br>| stddev|105964.5842848519|39.49496566617267|<br>|    min|               10|              1.0|<br>|    max|               99|            360.0|<br>+-------+-----------------+-----------------+<br>+-------------------+-------------------+<br>|skewness(avg_songs)|kurtosis(avg_songs)|<br>+-------------------+-------------------+<br>| 1.5731533253340253|  6.713093146603137|<br>+-------------------+-------------------+</pre><pre>Full dataset &gt;<br>+-------+------------------+-----------------+<br>|summary|            userId|        avg_songs|<br>+-------+------------------+-----------------+<br>|  count|             22261|            22261|<br>|   mean|1498833.2082116706|67.28930119633611|<br>| stddev| 288882.1163228875|42.00146132153544|<br>|    min|           1000025|              1.0|<br>|    max|           1999996|            579.0|<br>+-------+------------------+-----------------+<br>+-------------------+-------------------+<br>|skewness(avg_songs)|kurtosis(avg_songs)|<br>+-------------------+-------------------+<br>|  1.736381217684329|  8.096277081017185|<br>+-------------------+-------------------+</pre><p>9. Average time per session</p><pre>Tiny dataset &gt;<br>+-------+-----------------+------------------+<br>|summary|           userId|       avgSessTime|<br>+-------+-----------------+------------------+<br>|  count|              448|               448|<br>|   mean|67520.34821428571|   267.78884543676|<br>| stddev|105964.5842848519| 164.3624010530248|<br>|    min|               10|13.166666666666666|<br>|    max|               99|            1502.4|<br>+-------+-----------------+------------------+<br>+---------------------+---------------------+<br>|skewness(avgSessTime)|kurtosis(avgSessTime)|<br>+---------------------+---------------------+<br>|   1.6431669833905285|    7.153986718436954|<br>+---------------------+---------------------+</pre><pre>Full dataset &gt;<br>+-------+------------------+------------------+<br>|summary|            userId|       avgSessTime|<br>+-------+------------------+------------------+<br>|  count|             22278|             22278|<br>|   mean|1498782.9615764432| 276.5377760334103|<br>| stddev| 288851.8472659185|180.68117321920786|<br>|    min|           1000025|               0.0|<br>|    max|           1999996| 5453.363730301772|<br>+-------+------------------+------------------+<br>+---------------------+---------------------+<br>|skewness(avgSessTime)|kurtosis(avgSessTime)|<br>+---------------------+---------------------+<br>|   2.8166509927927756|   38.509616923162206|<br>+---------------------+---------------------+</pre><p>10. Number of session per user</p><pre>Tiny dataset &gt;<br>+-------+-----------------+------------------+<br>|summary|           userId|           session|<br>+-------+-----------------+------------------+<br>|  count|              448|               448|<br>|   mean|67520.34821428571|13.571428571428571|<br>| stddev|105964.5842848519| 13.17102391180226|<br>|    min|               10|                 1|<br>|    max|               99|                92|<br>+-------+-----------------+------------------+<br>+------------------+-----------------+<br>| skewness(session)|kurtosis(session)|<br>+------------------+-----------------+<br>|2.3492597105561734|7.726622405284306|<br>+------------------+-----------------+</pre><pre>Full dataset &gt;<br>+-------+------------------+------------------+<br>|summary|            userId|           session|<br>+-------+------------------+------------------+<br>|  count|             22278|             22278|<br>|   mean|1498782.9615764432|20.431726366819284|<br>| stddev|288851.84726591856|1059.3297847404108|<br>|    min|           1000025|                 1|<br>|    max|           1999996|            158115|<br>+-------+------------------+------------------+<br>+------------------+-----------------+<br>| skewness(session)|kurtosis(session)|<br>+------------------+-----------------+<br>|149.21426452603328|22266.26341038374|<br>+------------------+-----------------+</pre><p>11. Label (churn)</p><pre>Tiny dataset &gt;<br>+-------+-----------------+-------------------+<br>|summary|           userId|              label|<br>+-------+-----------------+-------------------+<br>|  count|              448|                448|<br>|   mean|67520.34821428571|0.22098214285714285|<br>| stddev|105964.5842848519| 0.4153723104396363|<br>|    min|               10|                  0|<br>|    max|               99|                  1|<br>+-------+-----------------+-------------------+<br>+------------------+--------------------+<br>|   skewness(label)|     kurtosis(label)|<br>+------------------+--------------------+<br>|1.3449610206355533|-0.19107985297097096|<br>+------------------+--------------------+</pre><pre>Full dataset &gt;<br>+-------+------------------+-------------------+<br>|summary|            userId|              label|<br>+-------+------------------+-------------------+<br>|  count|             22278|              22278|<br>|   mean|1498782.9615764432|0.22457132597181076|<br>| stddev| 288851.8472659188| 0.4173090731235619|<br>|    min|           1000025|                  0|<br>|    max|           1999996|                  1|<br>+-------+------------------+-------------------+<br>+------------------+--------------------+<br>|   skewness(label)|     kurtosis(label)|<br>+------------------+--------------------+<br>|1.3200520841972045|-0.25746249500661467|<br>+------------------+--------------------+</pre><h3>Conclusion</h3><p>On hindsight, I could have made a mixed of right and wrong decisions when it comes to assumption made on the large dataset: what features to engineer, what model and hyperparameter to tune in training. Retrospectively, there should be rooms for improvement for a better trained big data predictive model that yield more promising evaluation result with minimum computation time and resources. Among these are:</p><ol><li>More data exploration and experimentation and on local machine to get a more refined modelling approach on the full dataset.</li><li>There are considerably statistical differences in the large dataset, hyperparameter tuning on the full dataset is still necessary. Adopt smarter tuning with balanced grid and random search.</li><li>Test for collinearity among the engineered features, remove the collinear features to save on computational resources</li></ol><p>For research beyond this project, aside from churn prediction, the dataset can be useful to identify more factors that cause and if not improve the promising trends shown in Figure 9 and 10.</p><p>The source of this project is available from my <a href="https://github.com/jiewwantan/Churn_prediction_Sparkify">Github repository.</a></p><h4>Further readings on customer churn:</h4><ol><li><a href="https://www.kdnuggets.com/2019/05/churn-prediction-machine-learning.html">Customer Churn Prediction Using Machine Learning: Main Approaches and Models</a> by Altexsoft.</li><li><a href="https://blog.markgrowth.com/eliminating-churn-is-growth-hacking-2-0-47a380194a06">Eliminating Churn Is Growth Hacking 2.0</a> by flo.tausend</li><li><a href="https://neilpatel.com/blog/improve-by-predicting-churn/">How to Improve Your Subscription Based Business by Predicting Churn</a> by Neil Patel</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=32be364e8296" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/sparkify-user-churn-prediction-using-pyspark-32be364e8296">Sparkify user churn prediction using PySpark</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring real estate investment opportunity in Boston and Seattle]]></title>
            <link>https://medium.com/data-science/exploring-real-estate-investment-opportunity-in-boston-and-seattle-9d89d0c9bed2?source=rss-c3e3acabee1c------2</link>
            <guid isPermaLink="false">https://medium.com/p/9d89d0c9bed2</guid>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <category><![CDATA[investment]]></category>
            <category><![CDATA[use-cases]]></category>
            <category><![CDATA[airbnb]]></category>
            <category><![CDATA[crisp-dm]]></category>
            <dc:creator><![CDATA[Jiew Wan Tan]]></dc:creator>
            <pubDate>Sat, 29 Jun 2019 15:15:03 GMT</pubDate>
            <atom:updated>2019-09-03T13:07:29.872Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jEtYC5lhwZH9JZsy5B2EyQ.jpeg" /><figcaption>Airbnb’s treehouse</figcaption></figure><h4>Let Airbnb data tells your next property for investment.</h4><p>This project sets to design a framework for property market analysis in a city with the objective of identifying property that generates rental income in Airbnb marketplace. As Airbnb becomes the de facto market to go for short-term accommodation, even as substitute for hotel, its user activities reflect the pulse of a city’s real estate and hospitality landscape. Property owners are now able to operate like a mini hotel chain to provide short to medium term accommodation space. Boston and Seattle datasets are used to develop the framework but the objective is really for it to be usable for other cities covered by Airbnb, assuming other cities dataset share the same feature data.</p><h3>Business understanding</h3><p>Boston and Seattle are two cities situated on two sides of the United States, supposedly with distinctive real estate characteristics: the demand and supply of properties and the factors that play a part in determining prices. The result of the analysis will be used for real estate investors, who intend to refurbish the invested property to generate rental income from Airbnb marketplace. The analysis seeks to answer questions on:</p><ol><li>Which city rental market is more robust and as such is more appealing for investment?</li><li>Is demand steady and increasing?</li><li>Which neighbourhood of this city provides better ROI?</li><li>Which type of property to invest in?</li><li>What factors determine rental yield/price?</li></ol><p>The following technical questions shall answer the above business questions:</p><ol><li>What are the price distribution for Boston and Seattle?</li><li>How has rental demand change over the years for both cities ? Is there any underlying factor that influences them? Has the market saturated?</li><li>What are the features that are highly correlate with price?</li></ol><h3>Data understanding</h3><p>Three dataset in CSV files are used for the analysis:</p><ol><li>Listings — Listings data showing 92 (Seattle) and 95 (Boston) attributes for each of the listings.</li><li>Reviews — Reviews given by the guests with 5attributes. Key attributes include date (datetime), listing_id (discrete), reviewer_id (discrete) and comment (textual).</li><li>Calendar — Booking data for the next year by listing. Four attributes in total including listing_id (discrete), date(datetime), available (categorical) and price (continuous).</li></ol><p>It is important to note the Airbnb dataset has its limitation and as such it is only part of the picture for the short-term rental market in both cities. For example, it doesn’t show unmet demand. It also doesn’t provide the price data for purchasing the properties for investment in order to get a good sense of return on investment (ROI):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cOhwVKuW4OFKGOcV.png" /></figure><p>The Airbnb dataset is the numerator. Property price dataset is still required for denominator, which is to be explored in future post of the same topic.</p><h3>Data preparation</h3><p>To prepare the data for analysis, several steps are taken to clean the dataset:</p><ol><li>Handling missing values — Columns with more than 1000 missing values are dropped</li><li>Low value columns — Columns with URL links, not usable records such as host-name, street, and with only one unique values are dropped.</li><li>Boolean columns — Columns with “t” and “f” are converted to Boolean “True” and “False”.</li><li>Numerical columns — Columns such as price, rates and percentage are cleaned such that its numeric values are extracted and saved as float.</li><li>Categorical data — Columns with categorical data such as zipcode, property type, neighbourhood, room type, bed type are encoded so that they can be used for machine learning algorithm.</li><li>Outliers — Outliers are identified and taken note it may affect analysis later. They however remains in the dataset to be observed for visualisation. Steps are taken later to handle outliers.</li><li>Collinear features — Since machine learning have not been adopted at this stage, collinear features have not been identified. A note is made here to handle collinear features when need comes in future work.</li></ol><h3>Q1. Which city rental market is more robust and as such is more appealing for investment?</h3><p>Looking at short-term stay business, both cities are eventful with loads of cultural festivals, concerts, conferences and exhibitions and sports events throughout the year. Boston is also a favourite locations for major events such as Boston Marathon that draw large crowd to the city. The vibrancy bring about by these events is great business opportunities for short-term rental market.</p><ul><li><a href="https://www.events12.com/seattle/">Seattle - calendar of events</a></li><li><a href="https://www.bostonusa.com/events/events-calendar/">Boston Events | Concerts, Festivals, Marathons, Exhibits</a></li></ul><p>Apart from leisure and cultural, we can also look at the city as a business hub that attracts business travellers. These travellers will be able to fill gaps during the quiet period when no event is happening in the city.</p><p>College graduation period is another seasonal happenings that attract big crowd into the city when parent and relatives attend graduation ceremonies. In this regard, Metropolitan Boston has a long list of colleges:</p><p><a href="https://en.wikipedia.org/wiki/List_of_colleges_and_universities_in_metropolitan_Boston">List of colleges and universities in metropolitan Boston - Wikipedia</a></p><p>while Seattle is a little behind as an education hub but still warrants graduation season:</p><p><a href="https://www.collegesimply.com/colleges/washington/seattle/four-year-colleges/">List of the Four Year Colleges near Seattle, Washington - Seattle, Washington Four Year Colleges</a></p><p>Tourist attractions in the city is another point to look to fill up vacant rooms during non-seasonal period.</p><p>To get a good sense of accommodation demand, we will likely need traveller and transportation data from both cities for a more comprehensive quantitative analysis. Meanwhile, booking data from Airbnb shed some light on demand as reflected on prices.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_QXxetk5KuaNvM-OYrXHqQ.png" /><figcaption>Figure 1</figcaption></figure><p>The price distribution chart generated from Airbnb booking dataset provides some hint on demand. Boston distribution has a fatter tail and shows a more volatile price range. Whereas Seattle price range mostly within $500.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J9bIwcIPLs4TdeXuy7JqRg.png" /><figcaption>Figure 2</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HL8Gt2MvLh8Up6oweKdpow.png" /><figcaption>Figure 3</figcaption></figure><p>When looking at the same booking dataset as trend, averaging prices for each day, we see how demands play its part in setting prices. When supply can not meet flash demand in big events of the city, price hikes are seen. The plot shows weekly fluctuation as well as seasonal dynamics. Boston price trend reflects a more eventful and robust rental market that are limited by supply. As such the volatile price fluctuation caused by unmet demand. The large spike in April 2017 in Figure 2 is very likely as a result of Boston marathon. The regular and mini fluctuations are weekend effect as gridlines are drawn on Saturday.</p><h3>Q2. Is demand steady and increasing?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/813/1*xUZLuj16VAhdOyTNXwKy4Q.png" /><figcaption>Figure 4</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/1*1OVdjah7uF7YVkT-R7AMOQ.png" /><figcaption>Figure 5</figcaption></figure><p>Reviews are provided by users after having to stay at the property and as such a good proxy for fulfilled demand. From both cities’ demand trend, we can say that demand is indeed climbing steadily over the years. Boston experienced steeper climb in demand in the last two years. The price fluctuation shows seasonal change in every year as partially revealed in Q1. Figure 4 and 5 provide the bigger picture here with demand usually peaking in the late summer to early fall.</p><p>Nevertheless, price dynamics for Boston is still in the upward trajectory whereas Seattle looks saturating after 4 years of growth.</p><h3>Q3. Which neighbourhood of this city provides better ROI?</h3><p>Looking at Figure 2 and 3, We observer that Boston has the higher price range, higher than the high of Seattle even during its low season. This demonstrates its stronger demand for short-term accommodation constrained by limited supply.</p><p>However, this is only a part of the picture when it comes to ROI. We will also need to look at prices for owning properties to verify if the higher rental income justify the overall yield. In another words, if Boston property prices are much higher to result in a lower yield from rental income, it will not be the market to provide better ROI.</p><h3>Q4. Which type of property to invest in?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TH7vMxMX5E_PCbALcLWVGA.png" /><figcaption>Figure 6</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AHhDyeenoPxhI18DmCpL1w.png" /><figcaption>Figure 7</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L-a4olVK4oTpR8queAzA6w.png" /><figcaption>Figure 8</figcaption></figure><h3>Boston</h3><p>For Boston market, we can see that, apartment, condominium and house are the most popular offering here in most neighbourhoods. Some of the property types that fetch higher rental income are house, townhouse and condominium, likely due to limited offerings looking at the Listing count bar chart.</p><p>According to Figure 8, Downtown, Leather District and South Boston Waterfront generally fetch higher rents than other neighbourhoods. Back Bay, Dorchester, Fen Way, Jamaica Plain and South End are the neighbourhoods with plenty of supplies, these should be the neighbourhood with more competitive marketplace as such below average rental price. Looking at this perspective, Bay Village, South Boston Waterfront and West End are the much less competitive marketplace on the supply side with good demand from the tenants as reflected in their willingness to pay higher price.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5KxtSN7tZxqBPpREwi2TQw.png" /><figcaption>Figure 9</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kE4OzRMt6TeAgMUFnouFZA.png" /><figcaption>Figure 10</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jtcCWWISCyT8LoOjEIz6aA.png" /><figcaption>Figure 11</figcaption></figure><h3>Seattle</h3><p>Condominium, house and apartment are the popular accommodation offering in Seattle. Most neighbourhoods rentals are quite consistent in staying within a range with the exception of Wallingford, which could be an outlier. The property type that fetches the high price is boat, an exotic property. Belltown, Fremont Minor and Wallingford rental properties are more competitively price with plenty of supplies. Briarcliff, Industrial District, Pioneer Square and Southeast Magnolia are neighbourhood with much lesser supplies and as such fetching higher rental price. If the cost of buying the property is comparative to other neighbourhood, these area could be worth investing.</p><p>In short, location plays an important part in price settings, along with seasonality.</p><h3>Q5. What factors determine rental yield/price?</h3><p>The correlation matrix composed of numeric features reveals that the number of people the property can accommodate, number of beds, bedrooms and bathroom correlate the most with rental price, which make perfect sense as that implies a bigger property. So it doesn’t help reveal much insight here.</p><p>Other factors including review scores play little part in setting rental prices.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L_PEjBywPKcK4KlX-nfT6A.png" /><figcaption>Figure 12</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QU8jazgXjv7r_GRt31xrKw.png" /><figcaption>Figure 13</figcaption></figure><p>From the above analysis, Bay Village, South Boston Waterfront and West End neighbourhoods in Boston looks to be the better investment choice. Perhaps we need property price data to agree on their ROI.</p><p>In conclusion, as Airbnb grows and expands its marketplace to more cities, the city-based dataset are becoming very handy in analysing real estate marketplace. However, as they only form part of the picture, it is necessary to couple the dataset with other real estate marketplace such as Trulia, Zillow as well as travellers and transportation tracking data, to facilitate real-estate investment analysis.</p><p>The framework is coded in a Jupyter notebook and is available from my <a href="https://github.com/jiewwantan/Real_estate_Investment_with_Airbnb">Github repository</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d89d0c9bed2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/exploring-real-estate-investment-opportunity-in-boston-and-seattle-9d89d0c9bed2">Exploring real estate investment opportunity in Boston and Seattle</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>