<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Cliff Chew on Medium]]></title>
        <description><![CDATA[Stories by Cliff Chew on Medium]]></description>
        <link>https://medium.com/@cliffy-gardens?source=rss-38a9686a70b7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*L_5rYNynWGn8gsuuavL4aw.jpeg</url>
            <title>Stories by Cliff Chew on Medium</title>
            <link>https://medium.com/@cliffy-gardens?source=rss-38a9686a70b7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 15:53:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@cliffy-gardens/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[What started my Analytics Career]]></title>
            <link>https://cliffy-gardens.medium.com/what-started-my-analytics-career-d7b98d1577a1?source=rss-38a9686a70b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/d7b98d1577a1</guid>
            <category><![CDATA[life]]></category>
            <dc:creator><![CDATA[Cliff Chew]]></dc:creator>
            <pubDate>Tue, 17 Nov 2026 07:40:03 GMT</pubDate>
            <atom:updated>2026-11-17T07:40:03.600Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qnY0PrATRZYzkUgu" /><figcaption>Photo by <a href="https://unsplash.com/@gorillaroi?utm_source=medium&amp;utm_medium=referral">Gorilla ROI Data Connector</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>It’s been a while since I wrote something, and there have been some serious life changes these few years. Will be starting a series with some very serious tone relating to people. Maybe I am much older now, or that the recent life experiences have reshaped me quite a bit.</p><h4>My NUS thesis supervisor</h4><p>Under him, my Master thesis was the impact of a subway line on neighbouring public housing prices. For those that know me, I have been milking this topic for years. It was through my Masters thesis that I gained a strong appreciation about the importance of Singapore housing policies, and how it was a huge enabler of our Singapore economy and the Singapore story. And while I failed to level up into a career in academic research, my empirical thesis paper pushed me towards my analytics career. <em>“Falling in love with the process of turning raw data into insights”</em> was what I would always say in every interview in my early analytics days.</p><p>As a social science major, I always felt that I was fundamentally flawed to succeed in an analytics role. This was before the advent of data science degrees or joint degrees of various sorts. Hence, the strong imposter syndrome of feeling that I am the least technical person in technical analytics discussion.</p><p>My last conversation with him was him advising me to work towards a managerial role, rather than focusing on being technical. He told me a specialist’s career progression will always be limited. As someone with no clear sense of what he wants to do with his life after his studies ( I was in typical study hard, get good grades, get a good job kind of kid ), this was the very first career advice I received.</p><p>Sadly, just before I completed my career conversion programme into becoming a data analyst, he passed away from pancreatic cancer. I never got to update him about my decision to be more technical. It would also have been interesting to discuss with him about all the developments in recent years, all the way up to the current Generative AI is eating up the entire world and our energy resources.</p><h4>My passion project</h4><p>Fittingly, my on-going main passion project still focuses on the Singapore public housing. It is one that combines my first intellectual undertaking with the technical skills I gathered throughout my analytics career, plus a whole lot more Generative AI of course. I do have many more research and analytics ideas relating to this geospatial data set. Interesting times we live in!</p><p>And in some small ways, I do feel that every piece of work I do that is related to this theme is my small little tribute to my late thesis supervisor, the one who unintentionally inspired me towards a technical path, despite his warnings against it. Such is the strange and beautiful ways of life…</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d7b98d1577a1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Thanks to Agoda, I almost got scammed 179.5 CHF !]]></title>
            <link>https://cliffy-gardens.medium.com/thanks-to-agoda-i-almost-got-scammed-179-5-chf-ac6901a848d4?source=rss-38a9686a70b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/ac6901a848d4</guid>
            <category><![CDATA[travel]]></category>
            <category><![CDATA[fraud-detection]]></category>
            <dc:creator><![CDATA[Cliff Chew]]></dc:creator>
            <pubDate>Fri, 31 Jan 2025 09:16:06 GMT</pubDate>
            <atom:updated>2025-06-16T08:50:38.121Z</atom:updated>
            <content:encoded><![CDATA[<h3>Thanks to Agoda, I almost got scammed 179.5 CHF !</h3><p><strong>But thanks to Agoda, I was not scammed in the end either</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tBDtZOhouxF6V0do" /><figcaption>Photo by <a href="https://unsplash.com/@morgi_corgi?utm_source=medium&amp;utm_medium=referral">Morgan Thompson</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>What happened?</h4><p>I booked a hotel on Agoda in early January, 2025. Not long after, I received a seemingly innocent message through my Agoda app’s hotel booking. The first message was a short sentence with a [linked removed] line, followed by a long, seemingly computer generated message warning me about the risk of clicking external links on their app. As I was busy with major Lunar New Year festivals, I didn’t immediately follow up with these messages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ieUQhyf9VcDN3JLUJQF58A.png" /><figcaption>Image 1 — A seemingly innocent message from the “hotel operator”, followed by a long warning message from Agoda</figcaption></figure><p>In the subsequent days, I continued to receive instructional messages and Agoda-blocked links from the “hotel operator”. I even asked my wife if she could understand what the “hotel operator” was trying to get us to do, as it doesn’t make sense for us to <strong><em>“fill in some online form for a booking fee that isn’t a deposit, and that we wouldn’t get charged for it”. </em></strong>I was<strong><em> </em></strong>starting to think if this is some real-life online “Squid-Game” test that I need to complete something to compete for some murderous prize money.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WjOuZ4XJHYmUx1IwyWWJkQ.png" /><figcaption>Image 2 — I wasn’t biting, so the “hotel operator” wanted to nudge me into action. Look at my hasty reply with all my spelling errors!</figcaption></figure><p>Finally, after all the mad Lunar New Year rush was done, I reached back to the “hotel operator” about the blocked links, only to get a message that says those messages were not from the hotel operator. It finally made sense that I was receiving messages from a scammer through my Agoda hotel booking. <strong>Fortunately, because Agoda was blocking those external links, I couldn’t get scammed through their app, even if I wanted to.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*55BWR9p9JNGfp-E5NZRGKA.png" /><figcaption>Image 3 — Finally, the real hotel operator replied, despite all my spelling errors, haha</figcaption></figure><h4>What I inferred from my experience ( which could be wrong )</h4><p>What I faced feels very similar to what I read about <a href="https://www.bangkokpost.com/business/general/2775646/japanese-hotels-fall-prey-to-phishing-scams">a few months ago in 2024</a>.</p><p><strong>In fraud detection 101, we have this concept of either user level detection and activity level detection. The general idea is where we either detect bad actors ( not those in C-grade chick-flick movies ) from entering our ecosystem to execute fraud, or to prevent bad actions from happening within our platforms.</strong></p><p>Even when such problems were flagged in early 2024, Agoda seems to still have issues detecting bad actors from entering their ecosystem. And interestingly, the official hotel point of contact on Agoda did not even seem to know the scammers were messaging me through Agoda. Either they do not actively monitor their Agoda messages ( they do not contact hotel guests directly, therefore did not see a need to monitor them ), or Agoda doesn’t inform them of any messages that they were sending out to guests. Fortunately, the hotel operator did respond to my question, although even at that point, they didn’t seem to know what messages were sent before I reached out to them.</p><h4>What Agoda did well</h4><p><strong>Without a doubt, by blocking the external links, Agoda prevented me from getting scammed, and I am very grateful for that. </strong>Singaporeans may realise that this is also done by another marketplace platform in Singapore, called Carousell, where the platform blocks things like phone numbers and external links to prevent scams from happening. <em>*Disclaimer, I used to work in Carousell.</em></p><h4>False positives — Calling non-fraudulent behaviour fraud</h4><p>The tricky thing about what Agoda did was that non-fraudulent behaviour could also be blocked. For example, if for some reason the hotel operator really wanted to share some relevant information to me, they can only do it through Agoda, and not any other external link.</p><p>Bringing this back to our Singapore context, I have heard complains about Carousell blocking the sharing of phone numbers that allow buyers and sellers to communicate directly with each other. In certain situations, this is creating friction for genuine users who are more comfortable chatting on their preferred communication apps. To a large extent, Carousell is sacrificing the user experience of a large non-scammer user base to protect them from a small pool of scammers.</p><p>Nonetheless, be it from my professional experience dealing with fraud at Grab, or how Agoda helped save me from some scam, I am somewhat agreeable to what both platforms are trying to do to prevent scams / fraud.</p><h4>Some suggestions to Agoda</h4><ul><li>Get hotels to receive all messages from their message boards with their guests, including their own messages. Hotel account managers will then have better visibility on any dubious activity that is using their accounts to reach out to their guests, and hopefully help prevent any scam attempts on their guests. However, I do understand that this may be passing the load of “fraud moderation” to hotel operators, who may already be very swamped with their existing work.</li><li>Boost their user level authentication process, such as allowing hotel account managers to reset all logins. Agoda may still be in the process of setting such things up, so I can understand that blocking links may be the best short term alternative.</li></ul><h4>To Agoda hotel guests</h4><p>As long as scammers have a chance to reach users through Agoda, if these scammers eventually find a gap in Agoda’s action level fraud detection ( maybe emails, phone numbers, something? ), Agoda users may be prone to getting scammed. Be mindful that it is the full-time job of many of these scammers to identify loopholes in these platforms, so them taking one step closer to us through Agoda is boosting their probability of potentially scamming us. <strong>The first step if you want to be an Agoda hotel user is to be mindful that scammers can message us directly through the Agoda app, at least for now.</strong></p><p><strong>Second, when in doubt, either try to speak to someone from the hotel directly, or speak to someone from Agoda ( they have a dedicated customer support team ). While the blocked links already prevent me from getting scammed, checking in through the app helped me realise this was indeed a scam.</strong></p><h4>Conclusion</h4><p>The title of this post is quite click-baity, but I do want Agoda users to be mindful that they could be speaking to scammers on the Agoda app. <strong>Agoda did also prevent me from eventually getting scammed, so kudos to Agoda. </strong>Even as an ex-fraud data analyst, I did not suspect anything until I was informed by the hotel operator that I wasn&#39;t receiving messages from someone else.</p><p>The negative impact of such activities on their branding is hard to measure, but definitely there. My <a href="https://medium.com/design-bootcamp/one-thing-i-discovered-about-agodas-hotel-booking-system-that-shocked-me-e5c8a1def24b">previous negative experience </a>with Agoda caused me to write an extremely lengthy article, and it took me about 6 months before I started using them again ( <em>since then, I did become too strong a user of Agoda, so maybe hey, actions speak louder than lengthy essays, and in the end, it doesn’t even matter ? </em>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HKPWiCydeA287S3N8mKpxw.png" /><figcaption>I need to be honest, I have since became a very strong user of Agoda, haha…</figcaption></figure><p>Beyond branding, if these scammers found another loophole through their system, both users and hotels may lose confidence in Agoda and decide to go to other platforms instead ( Although one should also read <a href="https://www.cloudbeds.com/online-travel-agencies/market/">here</a> to know which travel apps are owned by the same companies, in case you really want to jump ship ).</p><p>And if anyone is asking, yes, I kept my booking with Agoda, although I may take up the free cancellation offer if my itinerary changes. And yes, CHF stands for Swiss Francs, and I am planning a trip to Switzerland / Italy with my wife for our European honeymoon this year :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ac6901a848d4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Python Polars — My (Late) 2024 Review]]></title>
            <link>https://cliffy-gardens.medium.com/using-python-polars-my-late-2024-review-da806618a102?source=rss-38a9686a70b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/da806618a102</guid>
            <category><![CDATA[polars-dataframe]]></category>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[analytics-tool]]></category>
            <category><![CDATA[pandas-dataframe]]></category>
            <dc:creator><![CDATA[Cliff Chew]]></dc:creator>
            <pubDate>Wed, 25 Sep 2024 07:19:12 GMT</pubDate>
            <atom:updated>2024-09-25T07:19:12.634Z</atom:updated>
            <content:encoded><![CDATA[<h3>Using Python Polars — My (Late) 2024 Review</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ENCVsmb4VWXU9u_m" /><figcaption>Photo by <a href="https://unsplash.com/@hansjurgen007?utm_source=medium&amp;utm_medium=referral">Hans-Jurgen Mager</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>About a year ago, I heard about Polars, a new data processing Python package written in Rust that is much faster than Pandas. Back then, Polars still felt like it was in beta ( this was before version 1.x ), so I waited. Finally, I started using Polars a couple of weeks ago, and here are some reflections done on late Sept 2024</p><h4>Different syntax</h4><p>Polars feel both similar and different to Pandas. Polars definitely forces me to frame my processes differently from Pandas, and in some ways do slow me down for now. Polars does seem to want to explicitly write many things out, while I have seen some who find Panda’s syntax confusing. Switching from Pandas to Polars requires some syntax switching.</p><h4>Faster runtime</h4><p>I feel Polars code execution runtime is faster than Pandas. Maybe from a mix of being unfamiliar with Polars and its faster speed, I am doing more “unit tests” that re-runs my entire process on every change I make. For example, for a set of housing price scripts, I am always running <strong><em>“max resale price”</em></strong> after any code change to ensure that I am not adding unknown side effects to my Polar’s code. <strong>Because I am able to run faster with Polars, I am more willing to run more with Polars, and this may allow me to write more robust analytics code in general.</strong></p><p>I am starting to think about splitting my data extraction and processing steps, especially since I make API calls to get the latest data for my work, and making unnecessary API calls do slow down my workflow. <strong>I am also starting to shift my working IDE from Jupyterlabs to Quarto-Neovim, which helps with my continued investments into vim-motions !</strong></p><p><strong><em>PS: I did have a use case where I was dealing with an API and performing some database backfill, and I decided to use Jupyter-Notebooks because it was just easier to check if my backfills were correct while keeping my connection state up.</em></strong></p><h4>More differences with Pandas</h4><p><strong>Polars cannot mix data types in a single column, while Pandas can.</strong> This may be a bad habit, but this allows simple logic treatments. For example, in Pandas, I can combine “lease_left” and “freehold” into a single column, thereby combining a numeric and string type together. This cannot be done in Polars, and it forces me to have to separate my filters for freehold and lease_left.</p><p>To my understanding, <strong>I also cannot use list comprehension in Polars to process columns. I fell in love them in Pandas. </strong>I felt list comprehensions were much more readable than lambda functions and .apply() methods.</p><p><strong>Polars has been in rapid development up till version 1.4.0, so quite a bit of Stackoverflow information is outdated, </strong>and I have to constantly go to the Polars’ official documentation, which at times, isn’t as clear.</p><p>I just started using LazyFrames, so I am still evaluating its speed. However, <strong>LazyFrames definitely add more programming overheads to me,</strong> <strong>and its errors harder to debug and entire chunk of code.</strong> I am using chatgpt to optimise my Polar code, but this has also been a hit-and-miss experience so far. Knowing the right questions to ask is definitely important with chatgpt.</p><h4>Installation issues</h4><p>While I managed to solve this, I had some initial issues installing Polars on my MacBook M1 chip laptop. The solution was to install “polars-lts-cpu” instead. This took me quite a bit of time, and I almost gave up trying to use Polars.</p><h4>My Situational Takes</h4><ol><li><strong>Would I use Polars ? Strong YES ! As mentioned, </strong>I am using it with my new workflow that uses Neovim-Quarto. The runtime speed is really amazing, and I cannot wait to deal with really large GB amounts of data when they come.</li><li><strong>Would I revamp a large Pandas codebase into Polars? Soft No, especially if Pandas has been working well. </strong>But if Pandas is reaching its limit for you, I can see why one would suggest to shift from Pandas to Polars. The migration may not be that intuitive, and it will take some time. There are options to convert Pandas dataframes to Polars dataframes, and back, so piecewise refactoring is possible. However, I am not sure if it makes sense to have two packages run concurrently in a production codebase.</li><li><strong>If I am a newbie, should I learn Pandas or Polars first? Depends! </strong>With tools like GenAI, I feel the importance of analytics code may diminish, but important concepts about creating impact from data will remain. I reckon there is still more analytics codebases in companies that exist in Pandas, so learning Pandas first may help you land a job ( very strong maybe, because I still feel SQL is the more important factor ). <strong>At the end of the day, I feel that Pandas and Polars are both useful tools in an analytics professional’s tool box, and knowing both of them, even at a rudimentary level, would be helpful in many ways.</strong></li></ol><p>Thanks to everyone who has read this post. If you are interested in analytics side projects with a social science spin, follow me on Medium or <a href="https://www.linkedin.com/in/cliff-chew-kt/">Linkedin</a>. Some topics I have explored include (1) <a href="https://cliffy-gardens.medium.com/singapore-million-dollar-public-homes-2024-analysis-update-d8c22d4eca20">Singapore housing prices</a> and <a href="https://cliffy-gardens.medium.com/singapore-million-dollar-public-homes-2024-analysis-update-d8c22d4eca20">the updated Singapore million dollar public home analysis</a>, (2) <a href="https://cliffy-gardens.medium.com/identifying-the-most-accessible-singapore-hotel-with-geospatial-data-26a83d3ded7b">accessibility of Singapore hotels</a>, (3) <a href="https://cliffy-gardens.medium.com/exploring-taiwan-housing-prices-beyond-taipei-284d48744f51">Taiwan housing prices</a>, and (4) I <a href="https://cliffy-gardens.medium.com/my-singapore-library-app-is-live-b63fc9f5ae9a">even built a small web app for Singaporeans to track the library books they want to borrow</a>. I also share less technical topics, like (5) <a href="https://cliffy-gardens.medium.com/two-books-on-making-decisions-under-uncertainty-e68d3888e728">how I learned to deal with uncertainty</a> and (6) <a href="https://medium.com/@cliffy-gardens/how-i-learned-to-deal-with-uncertainty-and-became-a-freelance-data-analyst-b3fef6734f2a">how I end up being a freelance analytics consultant</a>.</p><p>Lastly, I have a <a href="https://cliffdatastrategies.substack.com/">Substack</a> (<em>it is still alive</em>) as well, where I share ideas on data concepts and strategies for targeted at busy business people.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da806618a102" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Comparing Public and Private Home Transactions in Singapore]]></title>
            <link>https://cliffy-gardens.medium.com/comparing-public-and-private-home-transactions-in-singapore-6d149c3fa34d?source=rss-38a9686a70b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/6d149c3fa34d</guid>
            <category><![CDATA[singapore]]></category>
            <category><![CDATA[housing-market]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Cliff Chew]]></dc:creator>
            <pubDate>Tue, 27 Aug 2024 00:24:36 GMT</pubDate>
            <atom:updated>2024-08-27T00:24:36.230Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LWBq8l_wRdK9V1k2" /><figcaption>Photo by <a href="https://unsplash.com/@mikeenerio?utm_source=medium&amp;utm_medium=referral">Mike Enerio</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>It has been a long while since I wrote a new Medium or SubStack post, because I have been busy writing code that enables every Singaporean to look at our Singapore public housing market through an analytics lens. While editing the dashboards for my HDB data set from data.gov.sg, I have also been playing with the private property market data set provided by <a href="https://www.ura.gov.sg/maps/api/#private-residential-property-transactions">URA</a>.</p><p><em>*While I am still publishing this post on Medium, I am also posting property-related posts on my property blog </em><a href="https://sg-housing.onrender.com/blog/posts/private-vs-public"><em>here</em></a><em> to allow readers to play with my interactive charts. The content on both sites will be the same. This link for this blog post is here.*</em></p><p>After several weeks of on-off data exploration, here are some of patterns I found between our private and public housing markets, or at least how URA differs with HDB:</p><h4><strong>Bulk sales exist for private residential homes</strong></h4><p><strong>… but never for public homes.</strong> While I was initially confused, speaking to some domain experts got me to realise this could be some tycoon buying a few private homes at one go ( yes, such cases exist ! ) or developers buying up an old estate to re-develop the entire estate into a new private housing development. This is commonly known in Singapore as en-bloc sales. Unfortunately, URA classifies such transactions into a single record, where <strong>“noOfUnit &gt; 1”, while also having only a single “nettPrice” value. </strong>While this may be an administrative commonplace, this does affect how I perform my analyses down the road. <strong>Firstly, I would want to build a dashboard that allows people ( <em>who are not private property developers </em>) to filter for private home sales in Singapore. </strong>These bulk sales would not provide a good representation of private home transactions to an individual household, and the easier way to reflect such bulk sales is to exclude them from my dashboard search. <strong>I also could just divide the nettPrice by the noOfUnits to reflect the average price of per unit sold for that en-bloc transaction.</strong> Having the price of a single unit sold, even in such en-bloc situations, and may still provide a better comparison of prices across private and potentially, public residential homes. That said, I will have to indicate any average prices derived from on-bloc sales that are shown on my dashboard, or include an option to drop bulk sale transactions from the search results.</p><h4><strong>Both URA and HDB provide home sizes in square metres.</strong></h4><p>However, the property marketing collateral I remember seeing shows home sizes in square feet instead. Largely, I do feel Singaporeans prefer to think in price per square feet, which means I need to convert square metres to square feet myself.</p><h4><strong>Zoning used by URA and HDB are different</strong></h4><p>URA uses a district code and a broader <a href="https://www.propertyguru.com.sg/property-guides/ccr-ocr-rcr-region-singapore-ura-map-21045">CCR, RCR and OCR zoning</a>, while HDB uses a residential zoning system. Just a quick glance at these zoning systems suggests that they don’t match up fully. If true ( I have yet to confirm this thoroughly ), this will require more work if I want a dashboard that combines both private and public home transactions, and to allow comparing between private and public home prices by regions. I will definitely need to investigate this further.</p><h4>Public Housing is regulated; Private Housing is not ( so ) regulated</h4><p><strong>The regulated nature of our public housing market shows up in its price distributions, especially when comparing with our private property prices. </strong>While our Singapore public resale homes are getting expensive in recent years, their box plot price distributions still exhibit visible 25th to 75th percentile ranges with some outliers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FkWKMXcHwZH1UgF5Q0yfag.png" /><figcaption>Fig 1 — Distribution of Singapore public home resale transactions; Taken from <a href="https://sg-housing.onrender.com/sg-public-home-trends">https://sg-housing.onrender.com/sg-public-home-trends</a></figcaption></figure><p><strong>On the other hand, plotting private home price transactions into their respective box plot distributions shows that their 25th to 75th percentile boxes are barely visible, while many more outlier price transactions exist too.</strong> This suggests the very diverse nature of our private housing market, which does span across leasehold and freehold units, and across high-rise apartments, penthouses and huge landed multi-storey bungalows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PysOPI1KgBTQmDU95Q7Tag.jpeg" /><figcaption>Fig 2— Box Plot Distributions of all SG Private Housing Transactions in the last 5 years</figcaption></figure><h4>Private Property Price Clusters Exist!</h4><p><strong>Creating box plot charts by private property house types do allow us to identify some price clustering within private housing types! </strong>However, apartments and condominiums still exhibit price distributions that have very short 25th to 75th percentile with many outliers. <strong>This suggests the wide variety of apartments and condominiums that exist in Singapore.</strong> This may also suggest the complicated nature of public home upgraders in choosing the right kind of condominium or apartment that they hope to upgrade too. Do note that the axes across different housing types are quite different, so for example, detached homes ( larger landed homes ) have price ranges that are much higher than Executive Condominiums.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SX4M-8lJsIa4C9pdH7sxCg.jpeg" /><figcaption>Fig 3 — Apartment Price Distributions with their 25th to 75th percentile being barely visible</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lf2_mynD2dupjzJDpe4o5g.jpeg" /><figcaption>Fig 4— Condo Price Distributions with their 25th to 75th percentile being barely visible</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eOKta7QINrjTX-Ina4m_Hg.jpeg" /><figcaption>Fig 5— Detached Home Distributions</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YuJHMEXhUZOAjGUoPTLy4w.jpeg" /><figcaption>Fig 6— Strata Detached Home Distributions</figcaption></figure><p>The Strata Detached Home distributions are most probably affected by the their low amount of transactions, as I would not expect too many of them to be transacted. <strong>I am also not too sure what does “Strata” mean, but quite a few housing type had “Strata” in them.</strong> I would need to research a bit more to see if I can just ignore “Strata” and combine several housing types together.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uq5_vsFvM1jfbXhfLXbDPQ.jpeg" /><figcaption>Fig 7 — Strata Semi-Detached Home Distributions</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7JDIEPlUdOnaLd8LfV5fNA.jpeg" /><figcaption>Fig 8 — Semi-Terrace Home Distributions</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l2CLgZWbdrNACSxKZOZgDQ.jpeg" /><figcaption>Fig 9— Semi-Detached Home Distributions</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cgZOjpS4jFWttYQ6rD3BdQ.jpeg" /><figcaption>Fig 10— Terrace Home Distributions</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7X_-cwpD5DmTTXvQkOLKhw.jpeg" /><figcaption>Fig 11 — Executive Condo Home Price Distributions — Don’t they look similar to the public housing price distributions shown earlier?</figcaption></figure><h4><strong>About Executive Condominiums ( ECs )</strong></h4><p>These are public home projects that get to transit to private housing after 10 years from their launch. These units also have a starting tenure of 99 years, like most other leasehold residential units in Singapore. And maybe because of their “humble” origins, their price distributions and movements do seem to follow closely to our public housing markets, and are less wide-ranging than their apartment and condominium counterparts.</p><h4>Next Steps</h4><p>This analysis that I am doing here is definitely very basic. <strong>Building up with this private housing data set, I would love to see if I can find any trends with our private housing data, and in particular, how it relates to our public resale housing market.</strong> However, my first cut would most probably be building a separate private housing market dashboard that caters to the quirks of our private housing market in Singapore.</p><p>Here are some links for those interested to explore more about our Singapore housing market :</p><ol><li><a href="https://sg-housing.onrender.com/public_homes">Singapore Public Housing Past Transaction Search</a></li><li><a href="https://sg-housing.onrender.com/sg-public-home-trends">Singapore Public Home Trends</a></li><li><a href="https://sg-housing.onrender.com/blog/">Singapore Public Home Analytics Deep Dives</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6d149c3fa34d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What I learned from my Full-Stack Engineering Side Project as an Analytics Consultant]]></title>
            <link>https://cliffy-gardens.medium.com/what-i-learned-from-my-full-stack-engineering-side-project-as-an-analytics-consultant-3d9c60e8b118?source=rss-38a9686a70b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/3d9c60e8b118</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[lifelong-learning]]></category>
            <category><![CDATA[upskilling]]></category>
            <category><![CDATA[side-project]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Cliff Chew]]></dc:creator>
            <pubDate>Mon, 06 May 2024 08:51:05 GMT</pubDate>
            <atom:updated>2024-05-06T08:51:05.919Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bSatqXPOykXlbXMo" /><figcaption>Photo by <a href="https://unsplash.com/@sunburned_surveyor?utm_source=medium&amp;utm_medium=referral">Scott Blake</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Previously, I wrote about <a href="https://medium.com/@cliffy-gardens/the-project-that-kickstarted-my-analytics-career-d489fe016879">my analytics side project</a> that I felt got me into the analytics industry. Now, I want to talk about another software engineering side project that I did, where I developed a web app from scratch and deployed it online for anyone interested to use. I will cover my motivations, how I went about doing it, what I learned, and what I achieved so far, as this project is still ongoing.</p><h4><strong>Motivations</strong></h4><p>I started this side project after five years as a full-time data analyst in two Singapore tech companies, and one year of experience as an freelance analytics trainer and consultant. I had two reasons for starting this project.</p><p><strong>Firstly, I wanted to improve my backend and frontend engineering skills through hands-on experience.</strong> A stretch goal would be to use this project to transit into a full-stack engineer from the skills I learned from it. But I felt that having direct experience dealing with data on backend servers and frontend applications will make me a more all-rounded analytics consultant when I work on analytics projects with other engineers. This was why I didn’t consider a no- / low-code platform for my web app, as implementing the actual engineering tasks was more important to me than just having the web app up.</p><p><strong>Secondly, I wanted to solve a personal real pain point.</strong> I am an avid Singapore library user who borrows physical library books. Unfortunately, our Singapore library web and mobile apps only show book availability at a book level. So when I am at a library looking for the books I had saved to borrow physically, I have to annoyingly click on every single saved book to know if it is available at the library that I am at. <strong>I would prefer if I could click on a library filter, and have the web app show me all my saved books that are physically available for borrowing from that library.</strong> Fortunately, I realised our Singapore National Library Board ( NLB ) has an API that I can use to build a simple web app to show books availability by libraries. And knowing how slow changes may happen within government agencies, I took it upon myself to make the change that I want.</p><p><strong>With these two rather compelling reasons, I rationalised my worst case scenario was spending a few months building a web app for no one else but my own personal use, while learning some full-stack engineering skills to aid my analytics consultant role. </strong>While the opportunity cost of <em>“a few months”</em> may feel high for an working adult professional, I felt the potential upside was worth a shot to see how far I can go with this project ( <em>I wrote a past post on how I think and plan about events with uncertain payoffs </em><a href="https://medium.com/@cliffy-gardens/how-i-learned-to-deal-with-uncertainty-and-became-a-freelance-data-analyst-b3fef6734f2a"><em>here</em></a><em> </em>).</p><p>Like my analytics side project post, I shall just gloss over the boring parts:</p><ol><li>I went with Python <a href="https://fastapi.tiangolo.com/">FastAPI</a> as my backend and <a href="https://jinja.palletsprojects.com/en/3.1.x/">Jinja2</a> templates to serve my frontend, because I am more familiar with Python than JavaScript.</li><li>As I had an Udemy Business account then ( from our NLB ), I took a course on using FastAPI to build web apps, and this really helped a lot.</li><li>I also supplement my FastAPI learnings from books from NLB.</li><li>I used <a href="https://www.mongodb.com/">MongoDB</a> ( freemium ) for cloud database storage.</li><li>I used <a href="https://render.com/">Render</a> ( $7 USD / month ) to host my web app.</li><li>Lastly, I added some <a href="https://htmx.org/">HTMX</a> on my frontend for some user interactivity.</li></ol><p>And now, the interesting stuff!</p><h4><strong>The working web app</strong></h4><p>After six months of work ( two months learning, four months implementation ), I successfully deployed my web app on Render. As of April 2024, the web app is still alive <a href="https://sg-nlb-available-books.onrender.com/">here</a>. <strong>Through word-of-mouth, I got a single highly active user who really loves to read physical books like me, while I also continue to use the app myself. </strong>I had some initial users churn too.</p><h4><strong>Learnings as an analytics consultant</strong></h4><p>Being my own product team, I witness firsthand perspectives from many different roles:</p><ol><li><strong>I am not a proficient UX designer or full-stack engineer </strong>— My app is somewhat functional, but my design and functionalities can be much improved.</li><li><strong>Operational versus analytics engineering — </strong>As an engineer, I only cared if my web app works. However, to do certain analytics, I needed to engineer things like daily snapshots or capture specific events into my database. I realised my demands as a data analyst adds more work to the engineer, and at times, the engineer me got a bit frustrated by the event tracking requirements from the analyst me.</li><li><strong>Server side event tracking — </strong>When I started to implement some event tracking, I initially just littered Mixpanel Python functions across my backend codebase, only to have my Mixpanel project suddenly go missing after a few weeks. I didn’t really understand why, but I ended up settling on sending event data into my MongoDB database when they are triggered on my backend. What were once JIRA tickets I make to engineers for event tracking are now code that I implemented to capture these data. Event tracking never felt so real to me until now.</li><li><strong>Client side event tracking — </strong>Unfortunately,<strong> </strong>I couldn’t<strong> </strong>figure out client side event tracking. It did help that I had both HTMX and JavaScript code on my frontend, and I was not sure how to send client side data for both types of interactions. This meant I couldn’t track some of the events that I wanted. This was another example of how my frontend engineering work, but more work would be needed if I need to meet the requirements of the data analyst me.</li></ol><h4><strong>Dealing with NLB</strong></h4><p>When using the NLB API to get library information, I had to engage the NLB tech team about their APIs. Honestly, I expected better from them. Those interested can read my past posts <a href="https://cliffy-gardens.medium.com/how-good-is-our-latest-singapore-library-apis-an-honest-review-c32b03e8299b">here</a> and <a href="https://cliffy-gardens.medium.com/improving-our-singapore-government-tech-culture-one-api-at-a-time-6c5322fa9794">here</a>. It was quite a learning experience that was unfortunately painful at times. Eventually, I was glad that I managed to deploy my web app despite the odds.</p><p>My engineering side project also triggered another meeting with the NLB analytics team, which is a different team from the API team. This team had interviewed me previously about my scraping efforts to get book availability data from the NLB website, and this was before I knew the NLB API existed. This second meeting was more about my ideas on tech and data, and in relation to my experience using the NLB API.</p><h4><strong>Concluding thoughts</strong></h4><p><strong>My engineering side project made me realise I was more passionate and competent in analytics than software engineering, which honestly isn’t a bad thing.</strong> For now, I most probably wouldn’t double down my efforts to become a full-stack engineer, as I think it would involve too much work and pull me too far away from my core analytics competencies.</p><p>That said, I did also gain some interesting perspectives on what a product team needs to do to build a workable tech solution, and it gives me a bit more confident to work with engineers, designers and product managers for my future analytics projects.</p><p><strong>As for the web app, I do intend to make my app more user-friendly and visually pleasing once I have more time.</strong> In the meantime, Singapore residents who borrow physical library books, consider supporting a fellow Singaporean who hopes to make our society a better place with data and technology. This is its <a href="https://sg-nlb-available-books.onrender.com/">web link</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fto2y2DhtEGuEjt-lR5-ww.png" /><figcaption>Fig 1 — My web app’s main page after login and saving some books</figcaption></figure><p>Thanks to everyone who has read this post. If you are interested in analytics side projects with a social science spin, follow me on Medium or <a href="https://www.linkedin.com/in/cliff-chew-kt/">Linkedin</a>. Some topics I have explored include (1) <a href="https://cliffy-gardens.medium.com/singapore-million-dollar-public-homes-2024-analysis-update-d8c22d4eca20">Singapore housing prices</a> and <a href="https://cliffy-gardens.medium.com/singapore-million-dollar-public-homes-2024-analysis-update-d8c22d4eca20">the updated Singapore million dollar public home analysis</a>, (2) <a href="https://cliffy-gardens.medium.com/identifying-the-most-accessible-singapore-hotel-with-geospatial-data-26a83d3ded7b">accessibility of Singapore hotels</a>, (3) <a href="https://cliffy-gardens.medium.com/exploring-taiwan-housing-prices-beyond-taipei-284d48744f51">Taiwan housing prices</a>, and (4) I <a href="https://cliffy-gardens.medium.com/my-singapore-library-app-is-live-b63fc9f5ae9a">even built a small web app for Singaporeans to track the library books they want to borrow</a>. I also share less technical topics, like (5) <a href="https://cliffy-gardens.medium.com/two-books-on-making-decisions-under-uncertainty-e68d3888e728">how I learned to deal with uncertainty</a>and (6) <a href="https://medium.com/@cliffy-gardens/how-i-learned-to-deal-with-uncertainty-and-became-a-freelance-data-analyst-b3fef6734f2a">how I end up being a freelance analytics consultant</a>.</p><p>Lastly, I have a <a href="https://cliffdatastrategies.substack.com/">Substack</a> (<em>it is still alive</em>) as well, where I share ideas on data concepts and strategies for targeted at busy business people.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3d9c60e8b118" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The project that kickstarted my analytics career]]></title>
            <link>https://cliffy-gardens.medium.com/the-project-that-kickstarted-my-analytics-career-d489fe016879?source=rss-38a9686a70b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/d489fe016879</guid>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[nba]]></category>
            <category><![CDATA[side-project]]></category>
            <category><![CDATA[data-analytics]]></category>
            <dc:creator><![CDATA[Cliff Chew]]></dc:creator>
            <pubDate>Mon, 08 Apr 2024 07:37:51 GMT</pubDate>
            <atom:updated>2024-04-08T10:40:12.744Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1bs8YpqOcdOp5S7q" /><figcaption>Photo by <a href="https://unsplash.com/@sickhews?utm_source=medium&amp;utm_medium=referral">Wes Hicks</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>My Python programming journey started in 2012, where I took two online courses on it, one from Coursera and one from MIT. While both courses gave me my programming fundamentals, I wanted deeper programming skills specifically for data analytics. I thought an analytics side project would be a great way to boost my Python data analytics skills, and keep me motivated at it.</p><h4><strong>My Side Project</strong></h4><p>The first thing I had to do was to decide on a topic. I wanted something that can keep me motivated and engaged, while also having enough data to analyse. <strong>As an avid NBA basketball fan, I felt predicting NBA games using their in-game data would be a really interesting project for me to explore.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mZmUqfmwj1ouWU-xdOpSLg.jpeg" /><figcaption>Fig 1 — I once flew all the way to Miami to watch Ray Allen play!</figcaption></figure><p>Skipping the boring parts, these were the highlights of what went down with my analytics side project:</p><ol><li>I learned to web scrape two websites to get the NBA in-game data I needed. This took almost two years of learning and multiple web scraping iterations, all while holding a full-time job. I had to change the website I was scraping midway because the first website implemented some impressive web scraping blocker that I couldn’t overcome.</li><li>After scraping the data, I had to clean up some weird issues, such as players changing names and changing number of playoff games.</li><li>The “fun” modelling part had me try different scikit-learn prediction models, different feature engineering techniques and applied the train-test-validate framework.</li><li>Finally, with my prediction model, I redid my workflow to scrape data of newly completed games to predict next day game results.</li></ol><h4><strong>Project Outputs</strong></h4><p><strong>This side project took three years to complete, as I had various life commitments that made me have to stop my side project for a while. </strong>I also didn’t manage to automate the flow fully as my life got even busier and I could not spend more time to further maintain or improve on the project. The scraping and prediction flow was manually triggered by me on a few Jupyter notebooks.</p><p>Along the way, I documented my work on Github, which I decided to remove a few years ago because I was too ashamed of my code and couldn’t maintain it further. I also presented my three years of learnings from this project to a Singapore tech community, going from learning basic Python to “deploying” a machine learning model that could give me daily NBA game predictions during the season and playoffs.</p><p><strong>While I didn’t become a machine learning data scientist, my side project did come up as a topic with my founders’ interview for an analytics role with a Singapore tech company. </strong>As I ran the entire project myself, I was able to use different learnings about dealing with real-world data from my project to substantiate my interest in analytics, and my willingness to learn the skills needed to get analytics done. I did all these, without having prior professional data analyst experience. And while I never confirmed this, I felt my side project did contribute to me landing my first analytics role in a Singapore tech company and forever changing my career trajectory.</p><h4><strong>Restarting the project? Potential extensions</strong></h4><p>Nowadays, NBA analytics has evolved greatly. There are now many open-source NBA analytics projects that can be found online. The rise of analytics has also been criticised for making the NBA boring, <strong><em>aka three-points galore</em></strong>. The rise of NBA analytics is also somewhat complicated by the legalisation of sports betting in the US. Although five years have passed since I stopped this project, I am constantly tempted to see what else I can do to restart my this side project, and what other things I can learn from it. <strong>Set up a more structured data engineering workflow? Figure a basic MLOps structure? Learn cloud deployments? What about having a full frontend to share my model results?</strong> As always, I have more ideas and desire than I have time for, especially with more life commitments adding up. Nonetheless, I also learned to never say never on many things in life. Lebron is still playing at age 39, while Jordan came out of retirement, twice.</p><h4><strong>Concluding thoughts</strong></h4><p>Competition for analytics roles is intense nowadays in 2024. An analytics side project cannot guarantee a job, but it can potentially boost your odds in such a crowded job marketplace. The right analytics side project can expose you to real-world data challenges that gives you a flavour of what real-world analytics may be. And because such side projects can take a lot of time and effort ( mine took three years ), having such side projects under your belt can help make your resume and even interviews more unique.</p><p>A positive side effect of my side project was it made me realise how much I love the investigative nature of analytics, which further fuelled my desire to make analytics my career, and not just a side hobby. <strong>So if you are still wondering if an analytics career is suitable for you, such a side project may help you find out more about your preferences about this field.</strong></p><p>All in all, if you are willing to commit some time and effort into an analytics side project, it can potentially give you some interesting learnings, insights and opportunities in your analytics journey!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HTFuun4aAmGVBylV" /><figcaption>Photo by <a href="https://unsplash.com/@deanbennett?utm_source=medium&amp;utm_medium=referral">Dean Bennett</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><em>I once applied to the Houston Rockets as their data analyst, to be under the basketball analytics pioneer, Daryl Mory. I was obviously rejected by an email bot, but hey, we miss all shots we don’t try, right? ( puns intended )</em></p><p>Thanks to everyone who has read this post. If you are interested in analytics side projects with a social science spin, follow me on Medium or <a href="https://www.linkedin.com/in/cliff-chew-kt/">Linkedin</a>. Some topics I have explored include (1) <a href="https://cliffy-gardens.medium.com/singapore-million-dollar-public-homes-2024-analysis-update-d8c22d4eca20">Singapore housing prices</a> and <a href="https://cliffy-gardens.medium.com/singapore-million-dollar-public-homes-2024-analysis-update-d8c22d4eca20">the updated Singapore million dollar public home analysis</a>, (2) <a href="https://cliffy-gardens.medium.com/identifying-the-most-accessible-singapore-hotel-with-geospatial-data-26a83d3ded7b">accessibility of Singapore hotels</a>, (3) <a href="https://cliffy-gardens.medium.com/exploring-taiwan-housing-prices-beyond-taipei-284d48744f51">Taiwan housing prices</a>, and (4) I <a href="https://cliffy-gardens.medium.com/my-singapore-library-app-is-live-b63fc9f5ae9a">even built a small web app for Singaporeans to track the library books they want to borrow</a>. I also share less technical topics, like (5) <a href="https://cliffy-gardens.medium.com/two-books-on-making-decisions-under-uncertainty-e68d3888e728">how I learned to deal with uncertainty</a>and (6) <a href="https://medium.com/@cliffy-gardens/how-i-learned-to-deal-with-uncertainty-and-became-a-freelance-data-analyst-b3fef6734f2a">how I end up being a freelance analytics consultant</a>.</p><p>Lastly, I have a <a href="https://cliffdatastrategies.substack.com/">Substack</a> (<em>it is still alive</em>) as well, where I share ideas on data concepts and strategies for targeted at busy business people.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d489fe016879" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reflecting on Skills from a life of learning]]></title>
            <link>https://cliffy-gardens.medium.com/reflecting-on-skills-from-a-life-of-learning-ffea71c80462?source=rss-38a9686a70b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/ffea71c80462</guid>
            <category><![CDATA[skills]]></category>
            <category><![CDATA[lifelong-learning]]></category>
            <category><![CDATA[thoughts-and-ideas]]></category>
            <dc:creator><![CDATA[Cliff Chew]]></dc:creator>
            <pubDate>Wed, 20 Mar 2024 03:47:20 GMT</pubDate>
            <atom:updated>2024-03-20T04:02:08.875Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JnS-K-OXhOB4Uy_Q" /><figcaption>Photo by <a href="https://unsplash.com/@linkedinsalesnavigator?utm_source=medium&amp;utm_medium=referral">LinkedIn Sales Solutions</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>There has been so much discussion on skills in Singapore for the past few years that I decided to <a href="https://medium.com/@cliffy-gardens/i-analysed-all-29-000-skill-futures-courses-in-singapore-and-this-is-what-i-found-2789ff6c869c"><strong>analyse all 29,000+ Skill Futures courses</strong></a><strong> from our Singapore government to try to understand what they are about. </strong>These discussions and my analysis got me interested to write more about my experience with skills. As a social science major turned technical analytics consultant, thinking about and acquiring skills was, and still is, something I do often enough.</p><p>For this piece, I am just loosely defining skills as <strong>abilities one acquire to perform a particular task well</strong>. Hence, my definition of skills includes activities like cooking, hairstyling to software engineering. In this post, I want to share some views on skills that I gained based on my own anecdotal experiences.</p><h4>Nature v.s Nurture</h4><p>First and foremost, despite our best efforts, we can still suck at certain skills. If I followed my passion, I would have pursued a career as a professional athlete. Be it upbringing or genes, I believe some are more gifted in certain tasks. While people have the right to chase their dreams, I also believe there is too much survivorship bias causing people to be overly confident in what they can achieve. Don’t chase the latest hype, because most hype dies off. Gaining some self-awareness on what you can be good at can help you decide what skills to best focus on.</p><p>It took me years to realise I can be a data analyst, not a great machine learning data scientist, and at best a lousy full-stack engineer. I see many aspiring stories of non-technical career-switchers landing full-time technical roles, but I have come to peace that I am most comfortable ( and valuable ) as a freelance analytics consultant focusing on actionable analytics.</p><h4>Deep v.s Broad</h4><p>Certain deep skills require more effort to acquire. Getting good in certain specialised deep skills can allow you entry into selected industries. Medical doctors, architects and lawyers are specialist professionals. However, having deep specialised skills also means you live and die by your industry’s boom and bust. As we have seen, tech workers who benefited from years of the tech boom are now facing more harsh times with the tech layoffs of recent months.</p><p>On the other hand, broad skills, like communication, management, ability to empathise and critical thinking are skills that apply well across different industries. In fact, many tech leaders frequently advise their tech workers to improve their communication and writing skills, because such soft skills can amplify the impact of their work and allow them to progress better at their technical roles.</p><p>This is why after years of learning technical skills, I decided to also work on broad skills in writing and communication to be more all-rounded. This was how I became an analytics trainer, where I teach applied analytics concepts to non-technical learners. This is also why in recent years I have been writing more analytics and non-technical content.</p><h4>Independency</h4><p>Even if you cannot reach full mastery for certain skills, it doesn’t mean that certain skills are not worth learning. I don’t need to be a Michelin chef, but knowing how to cook can help me in many ways. Similarly, in the tech world, a product manager with basic SQL skills will not always be dependent on your data analyst to extract data. A data analyst with basic task orchestration tools like Airflow ( <em>and basic access rights</em> ) will not be dependent on his data engineer for simple data transformation jobs. Even learning the basics of certain complementary skills that combine well with your core competencies can make you valuable to your organisation in a different way.</p><p>As a data analyst myself, the frontend work I learned have allowed me to better programmatically navigate websites to extract data. On the other hand, the backend work I learned have allowed me to better deal with APIs to get data too. Despite being a subpar, not production-ready, full-stack engineer, those skills translate me into a more well-rounded technical data analyst.</p><h4>Complementary v.s Orthogonal</h4><p>Loosely adapted from the concept of <a href="https://medium.com/neuro-genesis-project/orthogonal-learning-52dcd23a7ea1">orthogonal learning</a>, I see orthogonal skills as skills that are not related to each other. Picking up orthogonal skills may go against learning complementary skills, but one professional benefit of orthogonal skills is that they can potentially hedge against the woes of the industry that you are in. <strong>This could be a forced retrenchment or a pivot into a new industry due to personal reasons.</strong> There are stories of bankers becoming bakers or programmers becoming yoga instructors. Orthogonal skills may not directly add value to your current role, but it can potentially protect you from the downfall of your own industry, and set you up into a new one.</p><h4>Putting Them Together</h4><p><strong>Generally speaking, my ideas on deep, core, broad, complementary and orthogonal skills are fluid concepts.</strong> Communication skills could be a broad skill to me, but a deep skill for a TV news anchor. Is SQL a core or complementary skill for software engineers? Shouldn’t all business analysts ( and management people! ) know how to use Tableau as their core skill?</p><p><strong>More importantly, with how technology changes nowadays, these skill categories can change.</strong> Core deep skills may become redundant, while orthogonal skills may become complementary. A vet with a side hobby in video creation may utilise her video making skills to market her professional services on TikTok. A real estate agent with basic website design skills may design his own website to market his services using a no-code website builder.</p><p>I have no grand framework on the topic of skills, nor a roadmap on the skills people need to acquire to enter the next lucrative industry. What I shared are just interesting points about skills relating to our era of lifelong learning. You are your own creation, and it is definitely worth some of your time and effort to think about the combination of skills that you want to have to tackle our messy world moving forward.</p><p>Thanks to everyone who has read this post. If you are interested in analytics side projects with a social science spin, follow me on Medium or <a href="https://www.linkedin.com/in/cliff-chew-kt/">Linkedin</a>. Some topics I have explored include (1) <a href="https://cliffy-gardens.medium.com/singapore-million-dollar-public-homes-2024-analysis-update-d8c22d4eca20">Singapore housing prices</a> and <a href="https://cliffy-gardens.medium.com/singapore-million-dollar-public-homes-2024-analysis-update-d8c22d4eca20">the updated Singapore million dollar public home analysis</a>, (2) <a href="https://cliffy-gardens.medium.com/identifying-the-most-accessible-singapore-hotel-with-geospatial-data-26a83d3ded7b">accessibility of Singapore hotels</a>, (3) <a href="https://cliffy-gardens.medium.com/exploring-taiwan-housing-prices-beyond-taipei-284d48744f51">Taiwan housing prices</a>, and (4) I <a href="https://cliffy-gardens.medium.com/my-singapore-library-app-is-live-b63fc9f5ae9a">even built a small web app for Singaporeans to track the library books they want to borrow</a>. I also share less technical topics, like (5) <a href="https://cliffy-gardens.medium.com/two-books-on-making-decisions-under-uncertainty-e68d3888e728">how I learned to deal with uncertainty</a>and (6) <a href="https://medium.com/@cliffy-gardens/how-i-learned-to-deal-with-uncertainty-and-became-a-freelance-data-analyst-b3fef6734f2a">how I end up being a freelance analytics consultant</a>.</p><p>Lastly, I have a <a href="https://cliffdatastrategies.substack.com/">Substack</a> (<em>it is still alive</em>) as well, where I share ideas on data concepts and strategies for targeted at busy business people.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ffea71c80462" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Are Singapore public homes still cheap? I crunched the data to find out!]]></title>
            <link>https://cliffy-gardens.medium.com/are-singapore-public-homes-still-cheap-i-crunched-the-data-to-find-out-00d7c9e65b0e?source=rss-38a9686a70b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/00d7c9e65b0e</guid>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[singapore]]></category>
            <category><![CDATA[data-visualisation]]></category>
            <category><![CDATA[public-housing]]></category>
            <dc:creator><![CDATA[Cliff Chew]]></dc:creator>
            <pubDate>Tue, 12 Mar 2024 08:35:50 GMT</pubDate>
            <atom:updated>2024-03-29T10:22:48.746Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3Eujrvw5J2zuYjsM" /><figcaption>Photo by <a href="https://unsplash.com/@daniel_wong?utm_source=medium&amp;utm_medium=referral">Daniel Wong</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>A few weeks ago, I updated my analysis on the <a href="https://cliffy-gardens.medium.com/singapore-million-dollar-public-homes-2024-analysis-update-d8c22d4eca20">rising number of million dollar public homes in Singapore in 2024</a>. Not only did we continue to see a rise in Singapore million dollar public homes since 2012, these million dollar homes were continually spreading from the central areas of Singapore to her periphery regions. Monthly million dollar public home sales were starting to up to 1–2% of overall public home sales in Singapore, which is a stark rise from the less than 1% a few years ago.</p><p>Continuing with this analysis, I wanted to update with the latest data set (<a href="https://beta.data.gov.sg/datasets/d_8b84c4ee58e3cfc0ece0d773c8ca6abc/view"><em>downloaded</em></a><em> on 12th March, 2024</em>) and go beyond million dollar public homes to better understand our overall Singapore public housing market. <strong>Maybe our public homes are still cheap, and these million dollar public homes are just a minority of outlier public home sales? </strong>They only make up at most 2+% of all public home sales in any given period, right?</p><h4>The Analysis</h4><p>To refresh our memories, I plotted a box plot distribution of the public home prices from 2003 to present day. Public home prices were stable from 2003 to 2007, before its <strong>(1) steady increase in median prices</strong>,<strong> (2) increase in the number of positive outlier home prices</strong> and <strong>(3) an upwards shift of its price distribution from 2007 to 2013</strong>. Then, the years 2013 to 2019 saw a period of relative stability, before another spike from 2019 to our present day. The general trend seems to indicate an overall rise in public home prices in Singapore.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*30EjD49x6HwQrDOjLNeLkA.png" /><figcaption>Fig 1 — All public home sales from 2003 to 2024 (March)</figcaption></figure><h4>Heat Maps</h4><p>Next, I plotted a heat map by regions (y-axis), years (x-axis) and regional median prices (cell) to identify any regional public home trends. The different accessibility and amenity provisions of different regions should impact their public home prices.</p><p>The heat map (Fig 2) shows that median public home prices rose across all regions in Singapore since 2003, with Bishan, Bukit Timah and Bukit Merah <strong>reaching median prices of 800K and above in recent years (2024 is incomplete).</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tfgub_SPvgTh5H3eYWcoTQ.png" /><figcaption>Fig 2 — Heat Map of median public home prices by regions (Best viewed on desktop)</figcaption></figure><p>Next, I do the same heat map using median prices per square feet to account for size effects, and we see similar price patterns across the regions across the years. There is an indication that the rise in median home prices is not due to the recent year sales of bigger size flats.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x9NR3RtsVdiO6TgZ5SySrQ.png" /><figcaption>Fig 3 — Median prices per square feet</figcaption></figure><p>Lastly, I divide the median prices per square feet by their remaining lease to account for their effects on home prices. Public homes in Singapore have a 99-year lease when purchased from the government, and when the lease is up, these public homes theoretically have zero value. Hence, the lesser the remaining lease, in theory, the cheaper these homes should be.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pu3xNb49ZZ2GbXblNy7rxw.png" /><figcaption>Fig 4 — Median price per square foot of remaining lease</figcaption></figure><p>In this updated heat map, we now see that Central Area public homes have one of the higher price ‘scores’ (<em>these data transformations can make their interpretations confusing</em>). <strong>In simpler terms, Central Areas public homes of the same size and same remaining lease will largely cost more than their counterparts in other areas. In short, Singaporeans are willing to pay a premium for “central” (<em>accessible</em>) housing.</strong></p><h4>The Central Million Dollar Question</h4><p>From my million dollar analysis I found out that the Central Area has one of the most number of million dollar public homes. Yet, this current analysis shows that the Central Area doesn’t have a high median public home price (Fig 2 &amp; 3). To investigate this further, I plotted the box plot distributions between the Central Area and Bishan (an area with high median public home prices).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4eXWEOlBzNrsvgAkW0c94g.png" /><figcaption>Fig 5 — Interesting price distributions of the Central Area. <em>The dot represents the a public home sale</em></figcaption></figure><p><strong>This is where I found that the Central Area has had two distinct public home price groups in recent years with no outlier public home prices, whereas the Bishan area has a consistent spread of public home sale prices. </strong>This group of million dollar public homes and lower priced public homes caused the Central Area median home prices to be lower than the Bishan median prices. This could be due the presence of older public housing programmes with the newer programmes within the Central Area that is causing these two price clusters to form. <strong>In other words, our Central Area is not just all about expensive public homes after all!</strong> The existence of bimodal distributions (histograms with two distinct peaks) also reflects the trickiness of using a single metric to identify or represent a region or country’s housing market trend.</p><h4>Public Home Price Groups</h4><p>Moving away from median and outlier (million dollar) prices, I wanted to understand the proportion of public home prices by specific price groups instead. I am looking at just home prices, and not price per square feet or price per square feet of the remaining lease, because this would most directly show the financial burden of owning a public home in Singapore. I grouped the prices into the following: (1) &lt;=300K (2) 300–500K, (3) 500–800K, (4) 800K-1m and (5) &gt;1m.</p><p>I then plot a bar chart stacking the public home sales by their price groups across the years. From this bar chart, we can see the decline in public homes sold that are &lt; SGD 300K, and a small rise in those between SGD 500K to a million dollars.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*osZMJvK1W9a7Sl60jK_kKw.png" /><figcaption>Fig 6 — Number of public homes sold by their price groups</figcaption></figure><p>To study these changes more clearly, I plotted these price groups as a percentage of total quarterly public home sales (Fig 7). I found that <strong>in 2006, slightly more than 80% of public homes sold were &lt; SGD 300K, and none sold were &gt; SGD 500K ! Contrast this to 2023, where public home sales of &lt; SGD 300K make up less than 2%, and more than 60% of public homes sold are &gt; SGD 500K .</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nm3yRqsKGctZPmGi1lqlYA.png" /><figcaption>Fig 8 — % of public homes by their pricing groups</figcaption></figure><p>So, if you want to buy a public home in Singapore in 2024 from the resale market, there is a low chance you can get it at &lt; SGD 300K. You still have a chance of getting a public home in Singapore ranging from SGD 300–500K if you are not picky, but looking at SGD 500–800K public home prices will give you better odds of securing a public home in Singapore. <strong>The relatively good news is that public homes at &gt; SGD 800K are still only 10% of total public homes sold in Singapore, even though this number has slowly been increasing in recent few years, while overall public home sales have remained relatively constant. </strong>While this data set spans across 17 years, so it is natural to see inflation affecting our public home prices, it is still interesting to see the extent of this price shift.</p><h4>Housing Price Groups by Housing Type</h4><p>Lastly, I want to do a quick look at the same stacked bar chart across different flat types: (1) 5-Rooms, ECs and Multi-Gen flats together, (2) 4-Room flats, (3) 3-Room flats and (4) 2-Room flats.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T8dKrdtXP9jATm2onZMnKw.png" /><figcaption>Fig 9 — Price group % for 5-Room, ECs and Multi-Gen public homes</figcaption></figure><p>(Fig 9) Classifying 5-Room, ECs and Multi-Gen flats together, slightly more than 50% of them were sold at &lt; SGD 300K in the early 2000s. Now, about 70% of them are &gt; SGD 500K, and slightly more than 20% of them were sold at &gt; SGD 800K.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*31SrXq3PpWb_Rt_Z21qgnA.png" /><figcaption>Fig 10 — Price group % for 4-Room public homes</figcaption></figure><p>(Fig 10) For 4-room public homes, more than 90% of them used to be sold at &lt; SGD 300K in the early 2000s. By 2023, more than 60% of them are selling between SGD 500K to 800K. In fact, you cannot find any 4-room public homes that are sold at &lt; SGD 300K from 2022.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LkpadQsWUQc9JN8TG-XGHQ.png" /><figcaption>Fig 11 — Price group % for 3-Room public homes</figcaption></figure><p>(Fig 11) For 3-room public homes, more than 99% of them used to be sold at &lt; SGD 300K in the early 2000s. Now, more than 85% of them are selling between SGD 300K to 500K.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*slZNI5LLB0HOfiouPCNlrw.png" /><figcaption>Fig 12 — Price group % for 2-Room public homes</figcaption></figure><p>(Fig 12) For 2-room public homes, more than 90% of them used to be sold at &lt; SGD 300K in the early 2000s. However, since the early 2020s, this proportion has been shifting massively, to the point that in 2023, more than 70% of 2-room public homes are selling between SGD 300K to 500K.</p><h4>Concluding thoughts</h4><p>Even in 2024, there are public homes that cost &lt; SGD 300K, even though they are only now just 2-Room and 3-Room flats. 4-Room flats can still cost between SGD 300–500K, even though a larger proportion are between SGD 500–800K. Lastly, for the larger 5-Room, EC and Multi-Gen flats, only less than 1% of them can cost &lt; SGD 500K.</p><p>Adding on from the rise in million dollar public homes in Singapore, this analysis shows that more proportions of Singapore public homes are becoming expensive nowadays as compared to around two decades ago. There are still pockets of cheaper public homes, but they are more rare nowadays.</p><p>There are many factors that affect general public housing prices, from new infrastructure projects, population growth, growth in median household incomes, changing interest rates and housing supply crunch. All these factors contribute to the overall demand and supply of public homes in Singapore, which in turn affect the overall prices of our public housing market.</p><p>While the data does show an increase in overall public home prices, I am not saying whether these are good or bad changes. Public homes downgraders should be happy that they can cash out their public home to boost their retirement funds. Some may argue these price shifts indicate that our strong economy and high quality public homes are continually fuelling their demand and boosting public housing prices. Naturally, public home-buyers may look at this analysis and justify their displeasure on the red-hot Singapore public housing market of recent years. However, true public housing affordability studies also need to be aligned with general household incomes, and this is something beyond the scope of this analysis.</p><p>This said, since 2023, the Singapore government has worked on a series of policy interventions and public home launches to cool down our public housing market. I will follow this development and update my analysis accordingly.</p><p>All in all, I hope my analysis has provided some simple but data-driven view of our Singapore public housing market. A huge caveat for this analysis is that my analysis and interpretations are very much an art rather than science. Nonetheless, I still hope that everyone who has read till here was able to take away something from this article.</p><p>Lastly, those interested in seeing some of these interactivity charts can go to my <a href="https://public.tableau.com/app/profile/cliff.chew/viz/CliffsSingaporeHDBResaleMarketAnalysis/HDBAnalysis">Tableau page</a> here, where I replicated these charts on Tableau.</p><p>Thanks to everyone who has read this post. If you are interested in analytics side projects with a social science spin, follow me on Medium or <a href="https://www.linkedin.com/in/cliff-chew-kt/">Linkedin</a>. Some topics I have explored include (1) <a href="https://cliffy-gardens.medium.com/singapore-million-dollar-public-homes-2024-analysis-update-d8c22d4eca20">Singapore housing prices</a> and <a href="https://cliffy-gardens.medium.com/singapore-million-dollar-public-homes-2024-analysis-update-d8c22d4eca20">the updated Singapore million dollar public home analysis</a>, (2) <a href="https://cliffy-gardens.medium.com/identifying-the-most-accessible-singapore-hotel-with-geospatial-data-26a83d3ded7b">accessibility of Singapore hotels</a>, (3) <a href="https://cliffy-gardens.medium.com/exploring-taiwan-housing-prices-beyond-taipei-284d48744f51">Taiwan housing prices</a>, and (4) I <a href="https://cliffy-gardens.medium.com/my-singapore-library-app-is-live-b63fc9f5ae9a">even built a small web app for Singaporeans to track the library books they want to borrow</a>. I also share less technical topics, like (5) <a href="https://cliffy-gardens.medium.com/two-books-on-making-decisions-under-uncertainty-e68d3888e728">how I learned to deal with uncertainty</a> and (6) <a href="https://medium.com/@cliffy-gardens/how-i-learned-to-deal-with-uncertainty-and-became-a-freelance-data-analyst-b3fef6734f2a">how I end up being a freelance analytics consultant</a>.</p><p>Lastly, I have a <a href="https://cliffdatastrategies.substack.com/">Substack</a> (<em>trying to keep that going</em>) as well, where I share ideas on data concepts and strategies for targeted at busy business people.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=00d7c9e65b0e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I analysed all 29,000+ Skill Futures courses in Singapore, and this is what I found]]></title>
            <link>https://cliffy-gardens.medium.com/i-analysed-all-29-000-skill-futures-courses-in-singapore-and-this-is-what-i-found-2789ff6c869c?source=rss-38a9686a70b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/2789ff6c869c</guid>
            <category><![CDATA[lifelong-learning]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[skillsfuture-courses]]></category>
            <category><![CDATA[singapore]]></category>
            <dc:creator><![CDATA[Cliff Chew]]></dc:creator>
            <pubDate>Fri, 08 Mar 2024 02:54:45 GMT</pubDate>
            <atom:updated>2024-03-08T08:01:30.647Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aMVQarGi3CrTikQQ" /><figcaption>Photo by <a href="https://unsplash.com/@brett_jordan?utm_source=medium&amp;utm_medium=referral">Brett Jordan</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Background</h3><p>Growing up in the first-world country Singapore, I was told to fear for my livelihood. We are too small. We have no natural resources. Developing our human capital is our option for survival. In 2015, the Singapore government launched the “Skills Future” ( SF ) programme to provide subsidies for selected courses, as well as providing $500 credits for Singaporeans to sign up for selected courses. In March 2024, the Singapore government is updating this programme to give Singaporeans aged 40 and above $4,000 more credits on selected courses geared towards employable skills. More details are set to be released in the coming months in 2024. With much discussion on the low usage of these SF credits to whether these courses provide any employable skills, I want to add some analytics inputs to this discussion.</p><h4>Course Data</h4><p>Doing what I do, I got the basic details of all 29,000+ SF courses from their <a href="https://www.myskillsfuture.gov.sg/content/portal/en/index.html">website</a>. These were the data that I was able to get : <strong>“provider_name”, “course_name”, “(course) ratings”, “(course) duration”</strong> and <strong>“(course) costs”.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*vHYlRT8JeFDFYo2uk3y3Vw.png" /><figcaption>Fig 1 — Data from SF site</figcaption></figure><p>After removing duplicates, <strong>I have 25,000+ unique courses that my analysis will focus on</strong>. A quick check also showed <strong>782 unique course providers</strong>, which is an insane number of providers for the government to onboard onto the SF programme.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/383/1*5emfPmlmPeGOJVfDKb14gw.png" /><figcaption>Fig 2— Total unique SF courses</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/539/1*93bh_XfZMGN0SWjn01bVZg.png" /><figcaption>Fig 3— Total unique course providers</figcaption></figure><p>Next, I did some summary statistics on the full course prices ( <strong>I could not get</strong> <strong>the subsidised rates as they could only be found on the detailed course page<em> ). </em></strong>The cheapest course is a $18 short-course titled “Unmanned Aircraft Basic Training” from the Singapore Polytechnic, while the most expensive course is titled “Bachelor of Medicine and Bachelor of Surgery” from NUS, at $623,364. Median cost is at $1,000 and mean costs is at $3,696.36.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NmQv7p_06gqFSxhEIsaptA.png" /><figcaption>Fig 4- Cheapest SF course at $18</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qsSvFUFU1JJCpsJxVsgotA.png" /><figcaption>Fig 5- Most expensive SF course at $623,364.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/1*kv3OPh-fN5fl9D5Rkd_Pxg.png" /><figcaption>Fig 6— Overall median and mean SF course costs</figcaption></figure><p>I find it puzzling that the $500 SF credits can be used to offset a $600K surgery programme. In fact, there were many courses that cost more than $100K. Personally, the $500 credit feels so inconsequential to a $100K programme that it wouldn’t affect my decision to take up a course or not. I guess one may argue that every penny counts, or that the subsidy is a matter of principle in encouraging lifelong learning.</p><h4>Top 10 course providers by courses they provide</h4><p>Next, I looked at the top 10 course providers with the most SF courses</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sQ7e95zs0c-wAi6V6ngEow.png" /><figcaption>Fig 7- Top 10 Course Providers by number of courses</figcaption></figure><p>It is not surprising that the top 10 SF course providers are well-established Singapore educational institutions. Not only do they have many existing courses to qualify for the subsidies, they also have the necessary administrative support to clear the paperwork.</p><p>I then looked at their course cost distributions and course durations of these top 10 course providers. To make the patterns of these cost distributions more visible, I applied a logarithm function to reduce the effects that stems from the large number of outliers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k4vRT97O69c8jrgkpzxw6A.png" /><figcaption>Fig 8- Box plot distributions of course prices, by top 10 course providers</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1vv-H_d0lUMOKMRapU_VKA.png" /><figcaption>Fig 9- Heat map of top 10 course providers, by course duration and number of courses</figcaption></figure><p>It was visibly clear that median university course prices are higher than those from polytechnic and ITE. While I thought this could be a “university premium”, I realise polytechnics also have a greater proportion of shorter courses, which may make polytechnic courses generally cheaper than longer courses from their counterparts ( <em>For brevity, I am only analysing these three top public universities among all existing universities </em>). However, the ITE has both a lower median course price and larger proportion of longer duration courses, which could then suggest the price premium exists between universities and polytechnics over ITE. One possibility is that because ITE courses focus more on vocational, blue-collar skills, they are less attractive than the white-collar skill courses provided by the other top 10 course providers.</p><p><strong>SMU is an interesting course provider.</strong> They have a low distribution of prices among their courses, but then a large number of price outliers. SMU has a larger proportion of short courses, with no courses that are more than one year long. Yet, their median course prices are not too far off from NUS and NTU. Compared to the other top 10 providers, only NUS provides a large number of less than 1 week courses with a sizable number of longer than one year courses, while the NTUC Learning Lab is the only institution with courses that last up to only three months.</p><h4>Keywords Analysis</h4><p>To have a better understanding the course characteristics that I found above, I wanted to analyse the keywords of courses that qualify for our SF subsidies. To do this, I cleaned up the course names to remove generic English words, remove punctuations, remove domain-less words like “certificate” and “diploma”, and made all keywords lower cases. Then, I did a simple keyword count on these processed course titles to get this list of top 10 keywords found in our current SF course titles : ‘business’, ‘digital’, ‘design’, ‘data’, ‘workplace’, ‘safety’, ‘wsq’, ‘analytics’, ‘health’ and ‘marketing’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/438/1*2EqC7CW-_3kBPEwnqywCoA.png" /><figcaption>Fig 10- Top Keywords in Course Titles</figcaption></figure><p>A word cloud generated from the course names also identified keywords such as <strong>‘business’, ‘design’, ‘digital’, ‘safety’, ‘workplace safety’</strong> and <strong>‘data analytics’</strong>. Interestingly, the top keywords don’t draw out more technical roles or jobs like engineering or programming. Maybe such technical courses have more specific keywords in their titles ( such as the keywords “technology” and “machine learning” that are found in the word cloud ) that don’t get aggregated together, so I cannot say for sure if our SF courses are targeting more technical roles or not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iBrzxVqSsAk6jvECNjUtNw.png" /><figcaption>Fig 11 — General Course Title Keyword Word cloud</figcaption></figure><p>Doing the same analysis for universities ( 4,500 ) , polytechnics ( 6,270 ) and ITE courses ( 1,140 ), these are the list of keywords and word clouds that I was able to build. These are the keywords that are interesting to me:</p><ul><li><strong>Universities</strong> — “business”, “digital”, “design”, “innovation”, “system”, “analytics”, “artificial intelligence”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HDcYmQOrCqu9jaqBrmL5WQ.png" /><figcaption>Fig 12 — University Course Title Word Cloud</figcaption></figure><ul><li><strong>Polytechnics</strong> — “business”, “digital”, “application”, “security”, “analytics”, “engineering”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/1*ZIzN7iQAiacWBoBGIMGKZQ.png" /><figcaption>Fig 13 — Polytechnic Course Title Word Cloud</figcaption></figure><ul><li><strong>ITE</strong> — “business”, “technology”, “design”, “services”, “electrical”, “engineering”, “maintenance”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YlvL7CwHC2R4a0_RxCffDQ.png" /><figcaption>Fig 14 — ITE Course Title Word Cloud</figcaption></figure><p>This keyword analysis is still very superficial, as there are many nuances to consider. For example, “engineering” could have popped up as a keyword because it aggregates from a variety of different engineering courses, from software engineering to civil engineering, and some of these “engineering” courses could be very different from each other. “Security” as a keyword could potentially be for “cybersecurity”. Interestingly, the ITE courses do pop up more keywords that are more commonly linked to vocational skills, such as “electrical”, “maintenance” and “services”. Interestingly, “workplace safety” and “safety” are not found in any of these provider course title names, and this could suggest that such courses are provided by other course providers instead.</p><h4>An important caveat</h4><p>There is so much data and so many ways to to slice, and I could only cover a tip of the iceberg. However, I want to share something that I found about the SF programme. People interested can read the entire SF <a href="https://www.myskillsfuture.gov.sg/content/portal/en/footer/terms-of-use.html">Terms of Service</a> ( ToS ), but clause 19 of its ToS states that our government isn’t vouching for any of the course qualities that are found in this SF programme.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VbjJeKoN6siAi5eu7Z61qg.png" /><figcaption>Fig 15 — Clause 19 from the Skills Future Site</figcaption></figure><p>To be fair, it is impossible for any government to vet through such a large and diverse set of courses that they do not have expertise in. I do feel at fault for realising this only after nine years into the SF programme. However, for quite some time, I did get the impression that some of these course providers’ seem to use the SF subsidies as a way to legitimise their course by mere association, although I think none explicitly mention they had any government stamp of approval. While clause 19 makes things clear now, I feel stupid for falling for such old marketing gimmicks. I only hoped that this stand was placed on the first page of the SF site and not hidden on its ToS page.</p><h4>Concluding thoughts</h4><p>All in all, this was a stressful side project. The SF site is slow and difficult to navigate programmatically, making it an arduous task to get their data. Secondly, while the data felt rather sensitive to analyse, with all the muse around upskilling and unemployment, I also felt compelled to give a data-driven spin to this topic.</p><p>Most importantly, as I am qualifying for the updated SF scheme soon, I have vested interest to better understand the current system, so as to prepare for its update. Currently, I am an analytics freelancer, I would be interested to look at more technical or business centric courses to compliment my existing work, rather than to help me pivot into a new role.</p><p>In addition, I may also consider vocational skills like plumbing and electrical works courses, so as to ensure that my skills are not too anchored into a specific industry ( tech ). However, rest assured, when the updated SF scheme is up, I will be reading their updated ToS carefully this time round!</p><p>Thanks to everyone who has read this post. If you are interested in analytics side projects with a social science narrative, follow me on Medium or <a href="https://www.linkedin.com/in/cliff-chew-kt/">Linkedin</a>. Some topics I have explored include (1) <a href="https://cliffy-gardens.medium.com/singapore-million-dollar-public-homes-2024-analysis-update-d8c22d4eca20">Singapore housing prices</a>, the (2) <a href="https://cliffy-gardens.medium.com/exploring-taiwan-housing-prices-beyond-taipei-284d48744f51">Taiwan housing prices</a>, and (3) I <a href="https://cliffy-gardens.medium.com/my-singapore-library-app-is-live-b63fc9f5ae9a">even built a small web app for Singapore residents to track the library books that they want to borrow</a>. I also share less technical stuff, like (4) <a href="https://cliffy-gardens.medium.com/two-books-on-making-decisions-under-uncertainty-e68d3888e728">how I learned to deal with uncertainty</a> and (5) <a href="https://medium.com/@cliffy-gardens/how-i-learned-to-deal-with-uncertainty-and-became-a-freelance-data-analyst-b3fef6734f2a">how I ended up being a freelance analytics consultant</a>.</p><p>Lastly, I have a <a href="https://cliffdatastrategies.substack.com/">Substack</a> (<em>trying to keep that going</em>) as well, where I share ideas on data concepts and strategies for targeted at busy business people.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2789ff6c869c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data exploration on the most accessible Singapore hotels by public transport — Part 2]]></title>
            <link>https://cliffy-gardens.medium.com/data-exploration-on-the-most-accessible-singapore-hotels-by-public-transport-part-2-382ed1869756?source=rss-38a9686a70b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/382ed1869756</guid>
            <category><![CDATA[hotel]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[singapore]]></category>
            <category><![CDATA[tourism]]></category>
            <dc:creator><![CDATA[Cliff Chew]]></dc:creator>
            <pubDate>Wed, 28 Feb 2024 04:56:30 GMT</pubDate>
            <atom:updated>2024-02-28T04:56:30.127Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data exploration on the most accessible Singapore hotels by public transport — Part 2</h3><p>In this post, I share my attempt to solve the missing public transport routing issue from the OneMap Routing API, so as to calculate a Singapore hotel public transport accessibility metric. <a href="https://medium.com/@cliffy-gardens/identifying-the-most-accessible-singapore-hotel-with-geospatial-data-26a83d3ded7b"><strong>Previously</strong></a><strong>, I shared how I identified the most accessible Singapore hotels by their linear and vehicle distances to 15 selected tourist attractions.</strong> I wanted to use public transport distances as an accessibility measure too, as I feel tourists are more likely to use public transport to travel around Singapore. Missing data from the OneMap Routing API from my first attempt didn’t allow me to complete my public transport accessibility measure.</p><p>In that first attempt, I used the OneMap Routing API to get 6,435 public transport routes (429 hotels to 15 attractions), out of which 581 ( 9% ) of the responses had the response <strong>“404 NOT FOUND : Unable to get route”</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SM02Jzu6HMBFLFEb" /><figcaption>Fig 1 — The unexpected 404 error.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/0*2xs-J1h1TdWESMlL" /><figcaption>Fig 2 — I replaced the “404” missing routes with “0” to make them easier to find</figcaption></figure><p>After letting the problem sit for awhile, this is my proposed workaround:</p><ol><li>Get all existing train station locations using the OneMap Geocoding API.</li><li>Link hotels to their nearest train station, either by public transport (priority) or foot.</li><li>Link attractions to their nearest train station, either by public transport (priority) or foot.</li><li>Link the stations by rail (public transport) to connect the hotels and the attractions.</li></ol><p><strong>I hope that breaking down hotel-attraction routes into separate components and stitching them together will help me fix my current issue.</strong> While this approach may not give the shortest public transport route, I felt it was a reasonable compromise as train stations are easier to spot and navigate than public buses in general, which commonly make train networks the preferred mode of public transport for tourists.</p><h4><strong>Step 1 — Hotels to nearest train stations</strong></h4><p>I was able to easily link up most Singapore hotels to their top three train stations. I chose top three to give the route stitching more flexibility, in case certain attractions are nearer to specific train lines by certain train stations. At this point, an interesting and intuitive (obvious) pattern surfaced from the data — <strong><em>Sentosa hotels are not very accessible via public transport!</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/917/0*wpVjsI1USImxCY5q" /><figcaption>Fig 3 — All Sentosa hotels having 0 (missing) public distance. The OneMap API couldn’t find any public transport or walking route for these hotels</figcaption></figure><p>This is expected as Sentosa is an offshore island outside of mainland Singapore. Travelling into Sentosa hotels require changing to either monorail, cable car and shuttle buses, all of which are more inconvenient than commuting within mainland Singapore itself. That said, I am not saying to stay away from Sentosa hotels, <strong>but if having good accessibility to tourist attractions in mainland Singapore is your priority, don’t choose hotels in Sentosa!</strong></p><h4><strong>Step 2 — Attractions to nearest train stations</strong></h4><p>I was also able to link the 15 attractions to their top three nearest train stations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/916/0*45yAAsfvwR3OL6sz" /><figcaption>Fig 4 — Successfully matching attractions to their nearest train stations</figcaption></figure><h4><strong>Station to Station connections</strong></h4><p>With the hotel-mrt and attraction-mrt links done, the last part is to link up the train stations! To make this matching efficient, I only matched stations to fill the missing routes. <strong>Unfortunately, there still were a number of train stations that could not link up using public transport.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iqgK4y2uYImMN0zv_6PQRg.png" /><figcaption>Fig 5 – 93 Train station-pairs could not be linked</figcaption></figure><p>This feels a bit ironic, as I assume connectivity between train stations should easily be matched through our train network. <strong>However, on hindsight, this seems like an unrealistic assumption. While I am not very familiar with the provision of routing APIs, </strong>I think that there are an infinite number of data points that may need to be linked by routes on our Earth’s surface, and it might be too difficult for the OneMap team to say to prioritise connectivity between train stations to be through our train networks.</p><p>Unfortunately, because of these missing station-to-station routes, I am back at my initial problem with calculating the public transport accessibility measure, and I need to think about my next steps on this project.</p><h4><strong>Conclusion</strong></h4><p><strong>My stitching attempt using the OneMap Routing API failed, as I couldn’t complete my public transport distance measure.</strong> But I didn’t see it as a failure because I did learn a few interesting lessons. <strong>I wanted to share my analysis as well as I wanted to highlight the thought process that goes into a seemingly simple real-world analytics project, where unforeseen issues with data availability and data quality may arise and cause certain roadblocks.</strong> When such situations happen, the analyst will then have to figure out what additional resources he might need, and possible issues and workarounds, all while trying to fit the project’s timelines and budgets, as well as ultimately what type of value proposition the analytics project can still provide. Dealing with such problems require experience and domain knowledge to make informed and strategic bets on how the project’s next steps.</p><p>Nonetheless, I am glad to have gotten my vehicle distance measure using the OneMap Routing API. <strong>If the OneMap team is reading this, I am happy to link up to understand if I am using the OneMap Routing API wrongly. And if the “404” error is an expected response for the routing API, please add it into your official documentation.</strong></p><p>I am thinking how I can “fix” my public distance accessibility measure. I could consider paid routing APIs like <a href="https://mapsplatform.google.com/pricing/">Google Maps Routing API</a> or <a href="https://www.graphhopper.com/pricing/">Graphhopper Routing API</a>, which should provide much better data. Both APIs also have a freemium tier that should be sufficient for my use case. I am also considering the OpenStreetMap (OSM) routing API, but so far, it seems like the OSM routing API does not have public transport data as well. I will want to see how much I can do with open-source tools and datasets before going down the paid route (pun intended). Open to suggestions if anyone has any!</p><p>Thanks to everyone who has read this post. If you are interested in analytics side projects with a social science spin, follow me on Medium or <a href="https://www.linkedin.com/in/cliff-chew-kt/">Linkedin</a>. Some topics I have explored include (1) <a href="https://cliffy-gardens.medium.com/singapore-million-dollar-public-homes-2024-analysis-update-d8c22d4eca20">Singapore housing prices</a>, the (2) <a href="https://cliffy-gardens.medium.com/exploring-taiwan-housing-prices-beyond-taipei-284d48744f51">Taiwan housing prices</a>, and (3) I <a href="https://cliffy-gardens.medium.com/my-singapore-library-app-is-live-b63fc9f5ae9a">even built a small web app for Singapore residents to track the library books that they want to borrow</a>. I also share less technical stuff, like (4) <a href="https://cliffy-gardens.medium.com/two-books-on-making-decisions-under-uncertainty-e68d3888e728">how I learned to deal with uncertainty</a> and (5) <a href="https://medium.com/@cliffy-gardens/how-i-learned-to-deal-with-uncertainty-and-became-a-freelance-data-analyst-b3fef6734f2a">how I ended up being a freelance analytics consultant</a>.</p><p>Lastly, I have a <a href="https://cliffdatastrategies.substack.com/">Substack</a> (<em>trying to keep that going</em>) as well, where I share ideas on data concepts and strategies for targeted at busy business people.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=382ed1869756" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>