<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by SingTat on Medium]]></title>
        <description><![CDATA[Stories by SingTat on Medium]]></description>
        <link>https://medium.com/@st_sg?source=rss-7eef572e9eb1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*sQcRkm5173_rUSW7FjuWPw.jpeg</url>
            <title>Stories by SingTat on Medium</title>
            <link>https://medium.com/@st_sg?source=rss-7eef572e9eb1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 09:04:58 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@st_sg/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A Graph on COVID-19 cases in Singapore]]></title>
            <link>https://medium.com/swlh/a-graph-on-covid-19-cases-in-singapore-904a17c29e2f?source=rss-7eef572e9eb1------2</link>
            <guid isPermaLink="false">https://medium.com/p/904a17c29e2f</guid>
            <category><![CDATA[neo4j]]></category>
            <category><![CDATA[visualization]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[covid19]]></category>
            <category><![CDATA[singapore]]></category>
            <dc:creator><![CDATA[SingTat]]></dc:creator>
            <pubDate>Sat, 28 Mar 2020 09:33:59 GMT</pubDate>
            <atom:updated>2020-04-24T11:44:41.144Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yk68l0aKgXqlpVr4fmKvyg.jpeg" /></figure><p>It started when I was trying to keep up with the daily covid-19 updates from the Ministry of Health (MOH). At some point, it became pretty confusing “who is linked to who and where” so I created a simple graph to help me visualise the connections. Over time, I added filter and search functions to make it easier and quicker to find information about the cases and clusters.</p><p>You can explore the graph at <a href="https://www.maventechnologies.com.sg/covid-19/">https://www.maventechnologies.com.sg/covid-19/</a>.</p><p>Here’s a simple video to illustrate some of the functions in the web page.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.loom.com%2Fembed%2F209ed1ee21a647f299f866ccba10d511&amp;display_name=Loom&amp;url=https%3A%2F%2Fwww.loom.com%2Fshare%2F209ed1ee21a647f299f866ccba10d511&amp;image=https%3A%2F%2Fcdn.loom.com%2Fsessions%2Fthumbnails%2F209ed1ee21a647f299f866ccba10d511-00001.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=loom" width="1152" height="864" frameborder="0" scrolling="no"><a href="https://medium.com/media/fd01158563d5fcfce1bfa431dfd59530/href">https://medium.com/media/fd01158563d5fcfce1bfa431dfd59530/href</a></iframe><h3><strong>Behind the Scenes</strong></h3><blockquote><em>I feel I have to say this. I am not targeting any case or cluster. I bring them up only to help explain some of the concepts.</em></blockquote><p>These are the processes and tools I have used to build the page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AniJd3IpA-9Tvl3A7Lig-Q.jpeg" /><figcaption>Processes and tools for the project</figcaption></figure><p>The landscape has changed a lot since I started the project in early Feb. Looking back, I can actually categorise the work done under 3 main tasks:</p><ol><li>processing the daily source data; affects statistics, data integrity, search capability</li><li>implementing the graph; route nodes, reduce load time, maintain User Experience (UX)</li><li>miscellaneous; resolve bugs, add features, analyse graph for insights</li></ol><p>Visually, it looks like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KMlHtJI880OUhINhxgFsTA.jpeg" /><figcaption>Timeline and scope of project</figcaption></figure><p>Let’s look at the evolution of the first two items in more details.</p><h3><strong>Processing Source Data</strong></h3><p>The source data is easily the most critical component of the project. It affects the graph, statistics, filter, search, everything! There have been 3 revisions so far.</p><p><strong>1.0</strong></p><p>MOH has usually been the primary source of information but in the early days, I have also analysed content from reputable news agencies like Channel News Asia (CNA) and the Straits Times (ST). Occasionally, as shown in the following example, ST has slightly more details (i.e. specific relationships).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SQXczDf3FG2f_jx_hKrbRw.jpeg" /><figcaption>Extracting nodes and relationships from the best source</figcaption></figure><p>This might sound trivial but every bit of extra content helps to indirectly boost the search capability. Hence for each case, I would try to select the source with the most information to build the most comprehensive graph and data.</p><p>Over time, however, updates from CNA and ST became more consistent with MOH.</p><p><strong>2.0</strong></p><p>Having focused solely on MOH updates, I started analysing the source data and adding minor tweaks. It could be as simple as changing this:</p><ul><li>Case X is a 40 year-old male Singapore Citizen…</li></ul><p>to this:</p><ul><li>Case X is a 40-year-old Singaporean man… (notice the hyphen after 40)</li></ul><p>This allows me to check for some form of data consistency before I publish the latest updates for the day.</p><p>An example is verifying the total number of Singaporeans across summary, filter and search.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KO6k_a6kDzqPpz5LvejWXA.png" /><figcaption>Verifying the total for “singaporean” across summary, filter and search</figcaption></figure><p>Say, if you were to search for “-year-old”, you will get one record less than the total because case #28 is a 6-<em>month-old</em> baby.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J1gdD7PcnStSsout-3Xs1A.png" /><figcaption>Search for “-year-old” returns 1 record less than total</figcaption></figure><p>Another example is searching for a cluster name. The number of cases should tally with the number shown in the cluster menu.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*abyTnwWD6mRMs-f6wFfE7A.png" /><figcaption>Search is consistent with total in cluster menu</figcaption></figure><p>I believe some form of checks are necessary because users will start to doubt the page once they detect inconsistencies. These checks are especially critical when there is a change in the source data, which brings me to 3.0.</p><p><strong>3.0</strong></p><p>Since the surge in cases from 18 March onwards, MOH has, understandably, been sharing a new format with considerably less information. Here is a sample of the update and the corresponding description generated using a macro.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AIaD_jAmZp1g_uIzrpHybg.jpeg" /><figcaption>Extracting nodes and relationships from one random case on 18 March 2020</figcaption></figure><p>As seen, the locations where a case stays, works or travelled, are no longer shared; only the cluster location is provided. This resulted in a drop of 80% of Place-Nodes.</p><p>While having less nodes reduces the complexity of the graph and speeds up page load time, less information also means less discoveries. In fact, I was doing a bit of investigation into inter-cluster nodes but I had to put that on hold now.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OO6P-P-RQX_697147msHaw.png" /><figcaption>Using Cypher (Graph Query Language) to identify inter-cluster nodes</figcaption></figure><p>In addition, those checks I was doing in 2.0 have also expanded with the new format. It seems like given the time, I can always find something to improve on, for example:</p><ul><li>formatting date to ISO 8601 format</li><li>replacing nationality with country name</li><li>checking for different styles of country name (e.g. USA vs US)</li><li>verifying lookup of country name to ISO 3166 format</li><li>handling “pending” data</li></ul><h3><strong>Implementing The Graph</strong></h3><p>Since the beginning, I have been using the Compound Spring Embedder (cose) layout algorithm (in Cytoscape) to route the nodes. Everyday, new cases are reported and gradually, the number of nodes grew to the hundreds. I have made 3 major revisions in order to keep the graph loading time short and maintain the UX.</p><p><strong>1.0</strong></p><p>Early on, the positions of the nodes were generated at run-time using CPU resources in users’ browsers. This means every time a user loads the page, the graph layout would be different. At one point, it took 60s just to load the graph with only about 100 nodes and I knew I had to improve.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AMus3e3Pgt-1b_V-VrZ_8w.jpeg" /><figcaption>It took 60s to load the graph on an old piece of hardware</figcaption></figure><p><strong>2.0</strong></p><p>In 2.0, I generated the positions of the nodes at run-time using the CPU resource in <strong>my</strong> development laptop. Once done, I uploaded the positions of those nodes to the server. Every user now loads the same graph layout no matter how many times they refresh the page. Without the need to use their CPU for any intensive calculations, users can now load the graph quickly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0PQH05jv3OU2ml1ix_FnjA.jpeg" /><figcaption>Graph loads faster when positions of nodes are pre-determined</figcaption></figure><p>This went well for a couple of weeks until routing the ever-increasing nodes became an issue. I was not able to tweak the cose algorithm further so I had to manually drag the nodes to declutter them. That was really time consuming and frankly, unsustainable.</p><p><strong>3.0</strong></p><p>In 3.0, I removed Place-Nodes associated with single cases to simplify routing. Should these nodes be associated with another case in future, they will be reinstated (e.g. see Changi Airport, which is connected to two cases). Locations not shown in the graph can still be found under details or search.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cQqRk1Fluc9HZ9iDT94oPQ.jpeg" /><figcaption>Removing Place-Nodes that are connected to only single cases</figcaption></figure><p>I also started grouping nodes together (called compound nodes in Cytoscape) in my development version to help with the routing. With the ability to drag many nodes at once, it speeds up the process slightly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RKHj9_w82X7ssUOzwsR2AA.png" /><figcaption>Using compound nodes (rectangular boxes) to aid manual routing</figcaption></figure><p>As you know, since 18 March, there has been considerably less Place-Nodes to route but the Person-Nodes have increased significantly. One approach I am exploring to optimise routing further is a combination of static and dynamic positioning.</p><p>Hopefully I have given you a little insight on what goes into building and maintaining the graph. If you are keen to obtain the daily source data for your reference or research, visit <a href="https://www.maventechnologies.com.sg/covid-19/source/api.php">https://www.maventechnologies.com.sg/covid-19/source/api.php</a>.</p><p>Going forward, the situation is likely to remain fluid. Furthermore, it will be just a matter of time before the addition of nodes starts affecting the UX again. However, I will try to keep the information in the page as updated as possible and adapt to new developments along the way.</p><blockquote><em>Let us stay vigilant in this battle against the virus. Stay safe and well.</em></blockquote><h4>Update (21 Apr 2020)</h4><p>There has been a gigantic surge in the number of new cases in mid April. Here’s a comparison.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5j4aM3uz6Zqvb86bK2q2GQ.jpeg" /><figcaption>Graph on 30 Mar 2020 (appx. 1000 nodes; image size 1.5MB)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sBFCELln_D3O7DGjm1OK_w.jpeg" /><figcaption>Graph on 15 Apr 2020 (appx. 4000 nodes; image size 6.3MB)</figcaption></figure><p>Unfortunately, there are many problems with the graph:</p><ul><li>huge number of nodes leading to high page loading time; it again takes 60s to load this graph on my iPad Mini</li><li>it takes me half a day to complete the MOH updates in spreadsheets (including adding relevant search phrases), route the nodes and publish the changes for 400+ new cases; going forward, it is not sustainable (new cases hit 1,426 on 20 Apr)</li><li>it is cluttered; looking for information is cumbersome</li><li>the details about each case remain sparse and generic; imagine taking 60s to load the graph, tapping on a node and seeing “Unknown Gender, Unknown Age, Unknown Nationality”</li></ul><p>After much considerations, I made the tough decision to rework my processes and adapt the graph to focus on the clusters instead of cases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0-cxXqD9sbEx-TpW10hrjQ.jpeg" /><figcaption>New Graph on 19 Apr 2020 (Appx. 100 nodes; images size 0.4MB)</figcaption></figure><p>The new graph looks clearer now. You can see the number of cases linking clusters instantly. Hopefully, the number of clusters and cases will start coming down in the next couple of days.</p><p>Be sure to “give a medal” to the healthcare warriors when you visit the page.</p><blockquote>Stay safe and well.</blockquote><h3><strong>References</strong></h3><ul><li>Old Web Page; <a href="https://www.maventechnologies.com.sg/covid-19/old/">https://www.maventechnologies.com.sg/covid-19/old/</a></li><li>Ministry of Health (MOH) — Official source of data for COVID-19 cases in Singapore; <a href="https://www.moh.gov.sg/covid-19">https://www.moh.gov.sg/covid-19</a></li><li>UIkit — lightweight front end framework; <a href="https://getuikit.com/">https://getuikit.com</a></li><li>Neo4j — graph database platform; <a href="https://neo4j.com/">https://neo4j.com</a></li><li>Cytoscape.js — graph theory library for visualisation and analysis; <a href="https://js.cytoscape.org/">https://js.cytoscape.org</a></li><li>Loom — screen and video recording software; <a href="https://www.loom.com/">https://www.loom.com</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=904a17c29e2f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/a-graph-on-covid-19-cases-in-singapore-904a17c29e2f">A Graph on COVID-19 cases in Singapore</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Project Traffico: a piece of the puzzle]]></title>
            <link>https://medium.com/@st_sg/project-traffico-a-piece-of-the-puzzle-5f92877c739f?source=rss-7eef572e9eb1------2</link>
            <guid isPermaLink="false">https://medium.com/p/5f92877c739f</guid>
            <category><![CDATA[bus]]></category>
            <category><![CDATA[singapore]]></category>
            <category><![CDATA[transportation]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[visualization]]></category>
            <dc:creator><![CDATA[SingTat]]></dc:creator>
            <pubDate>Tue, 26 Nov 2019 01:06:16 GMT</pubDate>
            <atom:updated>2019-11-26T01:06:16.771Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><em>Project Traffico (Traffic Observer) is a creative use of </em><a href="https://www.mytransport.sg/content/mytransport/home/dataMall/dynamic-data.html"><em>bus arrival times</em></a><em> to detect traffic congestions. Every month, data for certain hotspots in Singapore are compiled and presented in the form of tables, charts and infographics under </em><a href="https://www.maventechnologies.com.sg/traffico/stats"><em>Statistics</em></a><em>. Once a while, I also share some interesting observations under </em><a href="https://www.maventechnologies.com.sg/traffico/stories"><em>Stories</em></a><em>.</em></blockquote><p>I was looking at the infographics for last month and I saw something weird.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KdM29FSYo2nd03jET9EWwg.jpeg" /><figcaption>Oct 2019 Snail Champion: Loyang Ave — TPE (towards Tampines)</figcaption></figure><p>Unless it occurs regularly (e.g. every Sat afternoon along Punggol Road towards Sengkang at TPE junction), any congestion accumulated above 200 mins in a single day is unusual.</p><p>Time to investigate.</p><p>Let’s start by analysing Traffico data along Loyang Ave on (Thu) 3 Oct 2019.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*hFRxCzxDJBtVJIB-zMT0sQ.png" /><figcaption>Congestion Chart for Loyang Ave — TPE (towards Tampines)</figcaption></figure><p>As shown above, 2 congestions were detected around noon:</p><ul><li>#1 at 11:31am for 125 min</li><li>#2 at 1:40pm for 95 mins</li></ul><p>Here’s a brief recap on how Traffico detects a congestion (ref: <a href="https://medium.com/@st_sg/analysing-bus-arrival-times-part-1-c9989835aff6">Analysing Bus Arrival Times — Findings</a>):</p><ul><li>journey times between bus stops are derived from bus arrivals times and stored</li><li>a threshold is computed using weeks of stored data</li><li>if a current journey time exceeds the threshold for consecutive samples, a congestion is reported</li><li>once a new journey time falls below the threshold, the congestion is deemed over and a congestion duration is calculated (represented by the red dots in the chart)</li></ul><p>In short, the Congestion Chart is generated using journey times derived from bus arrival times. A Journey-Time chart is shown below as a reference.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Le4S6jL5JZ49VlyW9XhMgQ.jpeg" /><figcaption>Journey-Time Chart for Loyang Ave — TPE (towards Tampines)</figcaption></figure><pre>Note: journey times above 1500s (25mins) are omitted; bus bunching causes huge journey times that are inaccurate</pre><p>From our understanding of Loyang Ave (towards Tampines) — morning 8 to 9am and evening 6 to 7pm peak hour traffic on weekdays — we can further conclude these 2 congestions are abnormal.</p><p>At this point, it would be up to our imagination to guess what caused the congestion:</p><ul><li>there was an accident</li><li>there was a broken down vehicle</li><li>there were roadworks</li><li>there were some obstructions on the road</li><li>there were some chickens crossing the road</li><li>there was a UFO sighting</li><li>there was an elephant in a prefabricated room on a trailer heading to a construction site but nobody wants to talk about it</li></ul><p>Oops, I digressed.</p><p>Fortunately, I started scraping LTA traffic news some time ago and I was able to retrieve all traffic incidents relating to “Loyang Ave” on 3 Oct 2019. After filtering the irrelevant ones, the incident in bold stands out.</p><pre>07:42 Heavy Traffic on TPE (towards PIE) between Pasir Ris Dr 12 Exit and Loyang Ave Exit.</pre><pre>07:42 Heavy Traffic on TPE (towards PIE) between Punggol Rd Exit and Loyang Ave Exit.</pre><pre>08:39 Roadworks on Loyang Avenue (towards TPE) after Old Tampines Road.</pre><pre>08:39 Roadworks on Loyang Avenue towards TPE. Avoid left lane.</pre><pre>08:51 Vehicle breakdown on Loyang Avenue (towards Pasir Ris Drive 1) after Old Tampines Road.</pre><pre><strong>10:20 Roadworks on TPE (towards PIE) at Loyang Ave Entrance.</strong></pre><pre>17:57 Accident on Loyang Avenue (towards Old Tampines Road) after Pasir Ris Drive 1.</pre><p>Putting everything we know on a map, we arrive at this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uAMecOOqOYD4DJ1BkP7C-Q.jpeg" /><figcaption>Map of Loyang Ave — TPE (towards Tampines) with data from Traffico &amp; LTA</figcaption></figure><p>Indeed, it is very likely that the congestions were caused by the roadworks and they probably caught the lunch crowd by surprise.</p><h3>How is this useful?</h3><p>For drivers, if you saw the <a href="https://www.maventechnologies.com.sg/traffico">Traffico homepage</a> at that time, you would have seen the congestion status and you could have tried an alternative route. Yes, sometimes it might be a false alarm but a prolonged status usually means something.</p><p>Traffico can monitor and store the status of any road between any 2 bus stops, without the need for new infrastructure or hardware. Using the concepts I have shared, you can also deploy your own version of Traffico to monitor your routes and explore your datasets.</p><p>Traffico alone was only good enough to detect the occurrence of a congestion. However, by collating news from LTA, we are able to explain the likely cause as well. Would adding more sources of information help us improve our decision making process at the strategic rather than operational level? For example, what type of maintenance work was being carried out and how long did it take? Is the congestion duration a good estimate of the total time taken to complete the work? Is there a way to quantify certain maintenance routines and simulate the “costs” if we were to perform the same work on another road? Would it cause a jam? If it will, would it be better to carry out the work at midnight? If there is a sudden surge in these midnight jobs, could a system request for resources ahead of schedule?</p><p>In the future, I see Artificial Intelligence (AI) taking up roles in detecting such intricacies and suggesting alternatives as we pursue our Smart Nation endeavours.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5f92877c739f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Metis Bootcamp Passion Project: Facial recognition to identify missing people]]></title>
            <link>https://medium.com/@st_sg/metis-bootcamp-passion-project-facial-recognition-to-identify-missing-people-9aa202fc619d?source=rss-7eef572e9eb1------2</link>
            <guid isPermaLink="false">https://medium.com/p/9aa202fc619d</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[time]]></category>
            <category><![CDATA[facial-recognition]]></category>
            <category><![CDATA[siamese-networks]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[SingTat]]></dc:creator>
            <pubDate>Wed, 13 Nov 2019 04:33:37 GMT</pubDate>
            <atom:updated>2019-11-14T00:54:28.353Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>As a student, I wanted to understand how facial recognition is done in deep learning so for my final project in the Metis Data Science Bootcamp, I built a siamese network to learn the fundamentals. I also discovered that there are many open sourced projects and libraries that could be used to expedite the development of facial recognition applications.</blockquote><h3><strong>Background</strong></h3><p>Facial recognition is used in many applications from taking attendance to <a href="https://www.straitstimes.com/singapore/speedier-check-in-process-for-hotels-possible-with-new-automated-facial-recognition-system">checking in at hotels</a>. The benefits of using facial recognition to identify missing people, in particular elderly with dementia, will be tremendous as our population ages and the number of <a href="https://alz.org.sg/dementia/singapore">dementia cases is expected to rise</a> significantly in Singapore.</p><p>The challenge of locating missing seniors can actually be tackled at the family, organisation or government levels.</p><p>The simplest and most cost effective way is for affected seniors to wear a tag (with contact number, address, etc.) so that the public can step in to help if necessary but this is really not a sensitive approach (see survey <a href="https://www.channelnewsasia.com/news/singapore/nearly-3-in-4-persons-with-dementia-in-singapore-feel-ashamed-11487380">3 in 4 persons with dementia feel ashamed</a>).</p><p>A better solution can be established through the use of mobile apps to engage help from the community. See “<a href="https://www.straitstimes.com/singapore/app-helps-families-find-lost-seniors-with-dementia">App helps families find lost seniors with dementia</a>” for an example of such initiatives.</p><p>Moving up the chain, the government has the infrastructure and resources to create the ideal system. I like to think of this project as a baby step towards that direction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ahw9TC_3T6qs9EKrF0jFFw.jpeg" /><figcaption>Inching towards the ideal system</figcaption></figure><h3>Dataset</h3><p>I could not find the ideal dataset for old people with asian origins so I chose the Asian Face Age Dataset (<a href="https://afad-dataset.github.io/">AFAD</a>). It has images of asian faces, which is suitable for our local context. More specifically, I am using the AFAD-Lite, which is a subset of AFAD.</p><p>After a largely manual process to select images of individuals with enough variations, I did some simple manipulations and ultimately arrived at sets of 96 x 96 images for my model. These are some samples.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kGrMQ9y1IlhOeXhCRK-Ttw.jpeg" /><figcaption>Dataset from AFAD Lite</figcaption></figure><p>I generated about 400 images for 40 individuals (myself included). It is not necessary to have a huge dataset as discussed in the next section.</p><h3>Model</h3><p>Siamese networks are highly effective for performing facial recognition. Unlike traditional Convolutional Neural Network (CNN), they require less images (i.e. faces) to train because they are designed to learn if 2 images are similar or not, rather than learn which output class an image belongs to.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9UFFG1Dpk7F_i7YZq5Qa1Q.jpeg" /><figcaption>Training the Siamese Network</figcaption></figure><p>However, training the model was challenging on 2 fronts:</p><ul><li>hardware constraints; my Macbook Air is pretty old and it only has 4GB of RAM. I was only able to train the model with grayscale images (i.e. 1 channel) because it ran out of memory when I used jpeg images.</li><li>time constraints; tweaking hyper-parameters like number of filters, kernel size, kernel_regularizer, dimension of dense neural network (after flattened), max pooling size, etc. is time consuming. Each iteration could take up to an hour or so to complete.</li></ul><p>To work with jpeg images and to speed things up, I turned to <a href="https://colab.research.google.com/">Google Colaboratory</a>, which is pretty awesome. Using the GPU available, each iteration was completed about 10 times faster and I ended up consuming about 18GB of RAM!</p><blockquote>Side note: Right, I broke the initial limit of 12GB of RAM offered by Google. Say yes to temporarily increase your RAM to 25GB.</blockquote><p>I was able to achieve about 98% training accuracy and 95% test accuracy for this set of hyper-parameters.</p><pre>Number of input samples    10,000 genuine, 10,000 imposite pairs  <br>Train; Validation; Test    11,250; 3750; 5000                           <br>Conv2D layers              64 (3 x 3); 128 (3 x 3)                      <br>Max pooling                (2 x 2); (2 x 2)                                     <br>Dense                      256</pre><h3>Results</h3><p>The idea is to feed a new image of myself into the network and see which are the top 10 images with the highest score.</p><p>This is the best result I have gotten after many iterations, with my face appearing in the top 10, twice.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DmPilsOgbDI0DTJTRENOCw.jpeg" /><figcaption>We are not pulling a long face; we just have long faces</figcaption></figure><p>Yes, I had a good laugh when I saw the images for the first time.</p><p>The model still needs a lot of work but at this point, with only 4 individuals (myself included) out of 40 in the top 10 images, I guess it is better than a random coin-flip.</p><h3>Notable Projects</h3><p>I also did a little read-up into these 2 well known projects with pre-trained models that could shorten the development time for any facial recognition applications.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hU-U0BweHKQ6XUdDoBC7zg.jpeg" /><figcaption>Open sourced facial recognition projects</figcaption></figure><p>In fact, I managed to try out one related package. Following the steps in this tutorial (<a href="https://www.pyimagesearch.com/2018/06/18/face-recognition-with-opencv-python-and-deep-learning">Face recognition with OpenCV, Python, and deep learning</a>), I was able to dive quickly into the world of facial recognition and be acquainted with libraries like OpenCV (image processing) and dlib (face detection). After resolving some installation issues in OpenCV due to Anaconda, I was able to get some results using a couple of training images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*--7X3oJ3NbWE2rYP-9fqXw.jpeg" /><figcaption>Experimenting with face_recognition package by Adam Geitgey (<a href="https://github.com/ageitgey/face_recognition">https://github.com/ageitgey/face_recognition</a>)</figcaption></figure><p>Accuracy aside, face detection using dlib is indeed powerful and all 9 faces (even the stranger in orange) in the photo were correctly highlighted. With more training images and given enough time to tweak the parameters, I believe the accuracy could be improved.</p><h3>Conclusion</h3><p>In this project, I built a siamese network for facial recognition and I achieved decent results. Some improvements include:</p><ul><li>getting access to and using more relevant datasets</li><li>using N-way one shot learning to better reflect the accuracy of the model</li><li>improving the accuracy further by using image augmentation (e.g. scale, skew, rotate, flip, etc.) to prevent overfitting</li><li>tweaking the layers, optimizers and regularizers within the networks</li><li>incorporating the dlib library to mark out multiple faces for recognition</li></ul><p>I also discovered numerous projects, packages and libraries out there which can help developers build facial recognition applications quickly. Just a caveat; there are discussions on whether the pre-trained models work well with asian faces so I would definitely want to explore that further to prevent any showstopper.</p><p>It has been an unreal journey. I have learnt so much in the bootcamp and my classmates and trainers have been incredible and accommodating to my alien comments.</p><p>Before we were accepted into the bootcamp, we had to complete a 48-hour quiz. In the last question, I had to briefly describe any ideas about my passion project. I had no clue what I wanted to do so I just drew something random and uploaded it. Days later, our trainer did a video interview and I told him I had no clue about my passion project. Passion project commences and I still had no inkling. A week later, the facial recognition use case suddenly pops in my head and I knew instantly that was it.</p><p>This is what I have submitted.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X1D_0o_Cylgi_e7EH6UMYQ.jpeg" /><figcaption>Not so random after all</figcaption></figure><p>I can’t help but think of the laws of attraction. That said, I hope to contribute in those 3 sectors. There, I said it :)</p><h3>References</h3><ul><li><a href="https://towardsdatascience.com/one-shot-learning-with-siamese-networks-using-keras-17f34e75bb3d">One Shot Learning with Siamese Networks using Keras</a></li><li><a href="https://hub.packtpub.com/face-recognition-using-siamese-networks-tutorial">Face recognition using siamese networks</a></li><li><a href="https://medium.com/predict/face-recognition-from-scratch-using-siamese-networks-and-tensorflow-df03e32f8cd0">Face Recognition from Scratch using Siamese Networks and TensorFlow</a></li><li><a href="https://becominghuman.ai/siamese-networks-algorithm-applications-and-pytorch-implementation-4ffa3304c18">Siamese Networks: Algorithm, Applications And PyTorch Implementation</a></li><li><a href="https://cmusatyalab.github.io/openface">OpenFace</a></li><li><a href="https://github.com/rcmalli/keras-vggface">VGGFace</a></li><li><a href="https://github.com/davidsandberg/facenet/issues/830">Pre-trained model on ASIAN faces dataset</a></li><li><a href="https://github.com/davidsandberg/facenet/issues/591">Poor performance on Asian faces</a></li><li><a href="https://machinelearningmastery.com/how-to-develop-a-face-recognition-system-using-facenet-in-keras-and-an-svm-classifier">How to Develop a Face Recognition System Using FaceNet in Keras</a></li><li><a href="https://medium.com/@ageitgey/machine-learning-is-fun-part-4-modern-face-recognition-with-deep-learning-c3cffc121d78">Machine Learning is Fun! Part 4: Modern Face Recognition with Deep Learning</a></li><li><a href="https://machinelearningmastery.com/introduction-to-deep-learning-for-face-recognition">A Gentle Introduction to Deep Learning for Face Recognition</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9aa202fc619d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Topic Modeling and Sentiment Analysis on Singapore National Day Rally 2019]]></title>
            <link>https://medium.com/swlh/topic-modeling-and-sentiment-analysis-on-singapore-national-day-rally-2019-94aadd2ab559?source=rss-7eef572e9eb1------2</link>
            <guid isPermaLink="false">https://medium.com/p/94aadd2ab559</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[sentiment-analysis]]></category>
            <category><![CDATA[singapore]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[topic-modeling]]></category>
            <dc:creator><![CDATA[SingTat]]></dc:creator>
            <pubDate>Tue, 15 Oct 2019 03:20:31 GMT</pubDate>
            <atom:updated>2019-10-16T01:41:49.803Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Why I chose this topic for Project 4 in the Metis Data Science Bootcamp? This concerns every Singaporean. I am curious to know what people say about national issues. Before this bootcamp, I had no idea what NLP was but I had the impression that sentiment analysis is useful in predicting election outcomes; this is going to be very relevant as we head to the polls, soon.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mv6rzsS8xJdxD-PJUIio9w.png" /><figcaption>Source: TODAY Online</figcaption></figure><h3><strong>Background</strong></h3><p>Every year, about 2 weeks after we celebrate our National Day in Singapore, the Prime Minister gives a National Day Rally (NDR) speech to talk about the challenges the country faces and how the government plans to address them. This year, the headlines are:</p><ol><li>Helping young people to succeed</li><li>Supporting people to work longer</li><li>Protecting the country from the effects of climate change</li></ol><p>My objective is to use Natural Language Processing (NLP) to uncover hidden topics and to find out how people are feeling about NDR2019.</p><h3><strong>Dataset</strong></h3><p>I knew NLP has its limitations especially when it comes to abbreviated text, slangs, etc. so I left out Twitter for now. There is a dedicated subreddit for NDR2019 but it wasn’t really active. Hence I decided to scrape data from the following Facebook sources:</p><ol><li>The Straits Times</li><li>Channel News Asia (CNA)</li><li>Today Online</li></ol><p>In fact, these news agencies have somewhat categorised the headlines into the following:</p><ol><li>Climate Change</li><li>Retirement Age</li><li>Preschool Fees</li><li>Subsidy for Tertiary Education</li><li>Greater Southern Waterfront</li><li>China vs US Trade War</li><li>Slowing Economy</li><li>CPF for Older Workers</li></ol><p>That means if all goes well, at the very least, I should be able to “discover” some of these topics and perhaps uncover others.</p><p><strong>Scraping Facebook Comments</strong></p><p>Facebook has tightened its security controls since 2018 so the old way of using <a href="https://medium.com/google-cloud/sentiment-analysis-of-comments-on-lhls-facebook-page-9db8b3a60eb3">access token no longer works</a>.</p><p>I came up with a manual method to circumvent the problem. It is tedious but it requires no third party libraries whatsoever.</p><p>Let’s go through the steps for this example <a href="https://www.facebook.com/ChannelNewsAsia/videos/national-day-rally-2019/470967700407311">https://www.facebook.com/ChannelNewsAsia/videos/national-day-rally-2019/470967700407311</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IEbu9GK90Ufr3N-TlQRbNw.jpeg" /><figcaption>Step 1 — Load the Facebook page</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*88qp1vAj-eZ9wGjWuqls-g.jpeg" /><figcaption>Step 2 — Click Newest</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Px6xCXfwW3jctOQmr97rbw.jpeg" /><figcaption>Step 3 — Find “view more” links at bottom of page and click them to load all comments</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XnrIVtwMnXr_1rtBVUa8jQ.jpeg" /><figcaption>Step 4 — Find “repl” and click every link to expand all reply and replies</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vnSeW28tLFy3_GQKc8R6YQ.jpeg" /><figcaption>Step 5 — Find “see more” and click those links to show full text</figcaption></figure><p>At this point, all comments and replies are fully loaded in the page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F2GGG3Uueu7KWYzSQEFQyg.jpeg" /><figcaption>Step 6 — In Chrome, right click inspect, go to the body tag and click “Copy element”</figcaption></figure><p>Paste the content into a text file. Having investigated the HTML tags, I derived a simple method to extract the usernames, timestamps and comments using Beautiful Soup. There are some minor changes between pages but in general the codes look like this.</p><pre># code snippet</pre><pre>def get_fields(div):<br>    ...<br>    name = div.find(&#39;img&#39;)[&#39;alt&#39;]<br>    ts = div.find(&#39;abbr&#39;)[&#39;title&#39;]<br>    spans = div.find_all(&#39;span&#39;, class_=&#39;<strong>UFICommentBody</strong>&#39;)<br>    for span in spans:<br>        print(span.text)<br>    ...</pre><pre># main<br>file = &#39;fb_cna_470967700407311.txt&#39;<br>with open(file) as f:<br>    content = f.readlines()</pre><pre>soup = BeautifulSoup(str(content), ‘lxml&#39;)</pre><pre>comments = soup.find_all(&#39;div&#39;, {&#39;aria-label&#39;: ‘<strong>Comment</strong>&#39;})<br>for comment in comments:<br>    get_fields(comment)</pre><pre>replies = soup.find_all(&#39;div&#39;, {&#39;aria-label&#39;: &#39;<strong>Comment reply&#39;</strong>})<br>for reply in replies:<br>    get_fields(reply)</pre><p>After processing the pages, I collected and stored about 3,700 comments in MongoDB.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pg1R8P_txciUY4cbszOzEQ.jpeg" /><figcaption>Comments stored in MongoDB</figcaption></figure><h3><strong>Topic Modeling</strong></h3><p>I tried topic modeling using these techniques:</p><ul><li>Latent Semantic Analysis (LSA)</li><li>Non-negative Matrix Factorisation (NMF)</li><li>Latent Dirichlet Allocation (LDA)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iU7L4haDf8HJhVQUUbFo7Q.jpeg" /></figure><p>It started out fun but it got out of hand pretty quickly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JsRTm0DMcJfz6ib83GoFNg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9Uw98fM2Lkd4-wePTB5L8g.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gX7bSLBBv1M2_a94hxN9Mw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xiyfcsc2cNL29boZJiK70w.jpeg" /></figure><p>By the fifth iteration, my sentiment was pretty much affected :|</p><p>I actually spent most of my time generating many of the combinations and interpreting the results from the different techniques. Here are some examples.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AoeeqJRMKyBfCMgt4sZ1Ag.jpeg" /><figcaption>Comments &gt; Remove — Lemmatisation — Bigram, Trigram — CV, TF-IDF — LSA &gt; Topics, Keywords</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y5QYDCYMlU1_l45QfcEA2w.jpeg" /><figcaption>Comments &gt; Remove — Bigram, Trigram — CV, TF-IDF — NMF &gt; Topics, Keywords</figcaption></figure><p>For LDA, I used a tool called pyLDAvis to choose the number of topics. You can find more in the Annex. In summary, I reduced the number of topics to minimise the overlap between the keywords in each topic. For 5 topics and 40 keywords, I got the following.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-7iue_mqnjUVaKlzSGYfAw.jpeg" /><figcaption>Comments &gt; Remove — Lemmatisation — Unigram, Bigram, Trigram — CV — LDA &gt; Topics, Keywords</figcaption></figure><p>I was expecting 8 topics but only these 3 are more prominent across the various techniques:</p><ul><li>Retirement Age</li><li>Climate Change</li><li>Central Provident Fund (CPF)</li></ul><p>It could be due to the lack of comments on the other 5 topics. Some other notable topics include:</p><ul><li>Endless work</li><li>Work, jobs and earnings</li><li>People’s Action Party (PAP); political party governing Singapore</li><li>Government issues</li><li>Love for Singapore</li><li>Mr Lee Kuan Yew; first Prime Minister of Singapore</li><li>Mr Lee Hsien Loong; current Prime Minister of Singapore</li><li>Civic mindedness</li><li>Fake news</li></ul><h3><strong>Sentiment Analysis</strong></h3><p>I wanted to see how people are feeling towards “Climate Change” and “Retirement Age”. Using cosine similarity, I extracted comments relating to those 2 topics and I then used VADER to check the sentiments of those comments. The mapping logic between a score and sentiment is as follows:</p><pre>+---------------------------------------------+<br>| positive sentiment: compound score &gt;= 0.05  |<br>| negative sentiment: compound score &lt;= -0.05 |<br>| neutral sentiment : otherwise               |<br>+---------------------------------------------+</pre><p>Some comments are shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1jiD2lPNHuhf13dTWO-vSw.jpeg" /><figcaption>Sentiments on Retirement Age</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t0V4bzJ2EbFx50vUq9tabw.jpeg" /><figcaption>Sentiments on Climate Change</figcaption></figure><p>Looking at the comments, we can see that some of the sentiments are not quite correct but it does give us something to work on for a start. I believe organisations like <a href="https://www.reach.gov.sg/">REACH</a> could complement its reach by seeking out proactive individuals and engaging them further in the online space.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AoyiXMuPyijAPKrmZDazMA.jpeg" /><figcaption>A slide in my presentation on engaging users</figcaption></figure><p>In this project, I experimented with NLP for topic modeling and sentiment analysis.</p><p>I started out looking for 8 topics and I found 3 of them consistently across LSA, NMF and LDA, though I did uncover a handful of other topics.</p><p>I also found out that while VADER is not perfect, being privy to sentiments on the ground can help government agencies better understand and engage public users.</p><p>Some possible enhancements include:</p><ul><li>acquiring comments from more sources (e.g. yahoo news, reddit, etc.)</li><li>performing topic modeling and sentiment analysis on data in 2018 or even 2017 to look out for notable trends</li><li>improving sentiment analysis by tuning VADER or by trying out other libraries</li></ul><p>Reach out if you have any comments :)</p><h3>Annex</h3><p><strong>1. Tools</strong></p><pre>Beautiful Soup    Web Scraping               <br>MongoDB           Document-oriented Database          <br>SpaCy             Lemmatisation              <br>LSA, NMF, LDA     Topic Modeling             <br>pyLDAvis          Visualisation tool for LDA  <br>VADER             Sentiment Analysis</pre><p><strong>2. Source</strong></p><p><strong>The Straits Times</strong></p><p>https://www.facebook.com/TheStraitsTimes/videos/1060769760795404</p><p>https://www.facebook.com/TheStraitsTimes/posts/10156285000372115</p><p>https://www.facebook.com/TheStraitsTimes/posts/10156285418647115</p><p>https://www.facebook.com/TheStraitsTimes/posts/10156285203707115</p><p>https://www.facebook.com/TheStraitsTimes/posts/10156285183632115</p><p>https://www.facebook.com/TheStraitsTimes/posts/10156285244462115</p><p><strong>CNA</strong></p><p>https://www.facebook.com/ChannelNewsAsia/videos/national-day-rally-2019/470967700407311</p><p>https://www.facebook.com/ChannelNewsAsia/videos/492829441144310</p><p>https://www.facebook.com/ChannelNewsAsia/videos/vl.356508028593190/365306094361151</p><p>https://www.facebook.com/ChannelNewsAsia/videos/vl.356508028593190/399891210656636</p><p>https://www.facebook.com/ChannelNewsAsia/videos/vl.356508028593190/2302129810047887</p><p>https://www.facebook.com/ChannelNewsAsia/videos/vl.356508028593190/409046133068432</p><p>https://www.facebook.com/ChannelNewsAsia/videos/vl.356508028593190/487043431841537</p><p>https://www.facebook.com/ChannelNewsAsia/videos/vl.356508028593190/465451444035196</p><p>https://www.facebook.com/ChannelNewsAsia/videos/vl.356508028593190/2447062985387473</p><p>https://www.facebook.com/ChannelNewsAsia/videos/vl.356508028593190/651435332019167</p><p>https://www.facebook.com/ChannelNewsAsia/videos/vl.356508028593190/2362467970486825</p><p><strong>TODAY Online</strong></p><p>https://www.facebook.com/todayonline/videos/379980786025462</p><p>https://www.facebook.com/todayonline/videos/359130755014882</p><p>https://www.facebook.com/todayonline/videos/2426746067559253</p><p>https://www.facebook.com/todayonline/videos/669430183576477</p><p>https://www.facebook.com/todayonline/videos/892886227760286</p><p>https://www.facebook.com/todayonline/videos/2140535006247456</p><p>https://www.facebook.com/todayonline/videos/307952643338725</p><p><strong>3. pyLDAvis</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*si_RnofGsNm-S_9ISRileA.png" /><figcaption>10 topics</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BIV7c9Us-Tvf_cAOt_KMZw.png" /><figcaption>5 topics</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=94aadd2ab559" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/topic-modeling-and-sentiment-analysis-on-singapore-national-day-rally-2019-94aadd2ab559">Topic Modeling and Sentiment Analysis on Singapore National Day Rally 2019</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Supervised Learning to detect Phishing URLs]]></title>
            <link>https://medium.com/swlh/supervised-learning-to-detect-phishing-urls-d0779d360dc8?source=rss-7eef572e9eb1------2</link>
            <guid isPermaLink="false">https://medium.com/p/d0779d360dc8</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[supervised-learning]]></category>
            <category><![CDATA[phishing]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[SingTat]]></dc:creator>
            <pubDate>Sun, 08 Sep 2019 16:42:46 GMT</pubDate>
            <atom:updated>2020-08-18T10:28:53.700Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>I recently enrolled in the Metis Data Science Bootcamp (Singapore) offered through <a href="https://metis.kaplan.com.sg/">Kaplan</a>. Together with my ten classmates of varied backgrounds, we are the first cohort and we have survived the midway mark! It has been a pretty intensive but awesome experience so far. This is one of the projects I have completed.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i-g1LBVP95qzOrS0JFSp6g.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@heresjohnny19?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">John Sekutowski</a> on <a href="https://unsplash.com/search/photos/fishing?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>Background</h3><p>For my third (of five) project, I chose to use supervised learning to detect if a URL is phished or legit.</p><p>How is this important? I have a client who nearly had his email account hijacked through phishing. In fact, his supplier’s email was compromised and he received an invoice asking for payment to be deposited to another bank account number. Fortunately, he called his supplier for verification and discovered the fraud. The threat of phishing is very real and not to be under-estimated.</p><p>Here are some examples of phishing websites. In general, they want your credentials and passwords.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dxQJ1yM8IAT0uDMUgWhyHg.jpeg" /><figcaption>Phishing page requesting for Paypal credentials</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KX_uf_-Uz74ldc6llfTp4g.jpeg" /><figcaption>Phishing page requesting for gamer credentials</figcaption></figure><h3>Preliminary Analysis</h3><p>There are some phishing datasets on Kaggle but I wanted to try generating my own datasets for this project. I rely on these 2 sources for my list of URLs:</p><ul><li>Legit URLs: Ebubekir Büber (github.com/ebubekirbbr)</li><li>Phish URLs: phishtank.com</li></ul><p>With a bit of domain knowledge and analysis done on some phished and legit websites, I was able to shortlist the following categories to extract information:</p><figure><img alt="5 Categories associated with web pages" src="https://cdn-images-1.medium.com/max/1024/1*vYpRoARKrcwFvY3mfBNKZA.jpeg" /><figcaption>Parameters to extract for a web page fall under 5 categories</figcaption></figure><p>Some useful information:</p><ul><li>“Phishers” usually hack into legit websites to insert phishing web pages, rather than set up a domain exclusively for phishing. While this could make it difficult to identify a phishing website through its domain, I understand registrars and hosting companies act swiftly to get website owners to remove those phishing pages in order to prevent negative impact to their ranking. This means we could possibly see empty registrar or name servers for compromised domains over time.</li><li>One precaution to take. Some phished websites might contain malware so instead of loading those URLs directly in my browser, I did the following: (a) view screenshot of these pages using tools like https://web-capture.net (b) analyse the HTML content using text editors</li></ul><h3>Data Acquisition</h3><p>Conceptually, my scraper process looks like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2NT9Z7Yr8sherBhjVNh2-w.jpeg" /></figure><p>The idea is to keep the codes modular so that I can keep adding new categories when necessary. Every page that I scraped is saved as a file in my local drive for reference in case it no longer exists in future.</p><p>I used <a href="https://www.crummy.com/software/BeautifulSoup">BeautifulSoup</a> to extract attributes in tags. Although it is not foolproof, it should help to set a (random) valid “user-agent” so that servers are less likely to reject your requests thinking you are a bot.</p><p>Basic preprocessing on the URL (i.e. remove www, trailing slash) is done to ensure consistency.</p><p>Refer to Annex for a brief description of the 22 features extracted after processing the JSON files for each URL.</p><h3>Exploratory Data Analysis</h3><p>Since scraping data is really time consuming, I decided to start my EDA concurrently to get some sensing. After analysing about 1817 URLs (930 phish, 887 legit) and their features, I chose to drop some generally empty or nominal features to work on these 15.</p><pre>  URL            Domain          Network      Page       Whois    <br> -------------- --------------- ------------ ---------- --------- <br>  length         len_subdomain   len_cookie   length     w_score  <br>  special_char   is_https                     anchors             <br>  depth                                       form                <br>                                              email               <br>                                              password            <br>                                              signin              <br>                                              hidden              <br>                                              popup</pre><h3><strong>Feature Selection</strong></h3><p>I used LASSO Regularization to identify the important features. Even with a small value of alpha, I can already see the 5 dominant features.</p><pre>[(<strong>&#39;len&#39;, 0.0006821926601753635</strong>),<br>(&#39;count_s&#39;, 0.0),<br>(&#39;depth&#39;, 0.0),<br>(&#39;len_subdomain&#39;, 0.0),<br>(&#39;is_https&#39;, 0.0),<br><strong>(&#39;len_cookie&#39;, -0.0002472539769316538),<br>(&#39;page_length&#39;, -2.4074484401619206e-07),<br>(&#39;page_num_anchor&#39;, -0.0006943876695101922),<br></strong>(&#39;page_num_form&#39;, -0.0),<br>(&#39;page_num_email&#39;, -0.0),<br>(&#39;page_num_password&#39;, 0.0),<br>(&#39;page_num_signin&#39;, 0.0),<br>(<strong>&#39;page_num_hidden&#39;, -0.00041105959874092535</strong>),<br>(&#39;page_num_popup&#39;, -0.0),<br>(&#39;w_score&#39;, -0.0)]</pre><p>Frankly, I am a little surprised that the w_score had little significance. I believe given enough time I might be able to engineer one or two features but with deadline drawing near, I decided to drop the whois category (i.e. suspend ExtractorWhois module) to focus on these 5.</p><pre>  URL      Domain   Network      Page      Whois  <br> -------- -------- ------------ --------- ------- <br>  length            len_cookie   length           <br>                                 anchors          <br>                                 hidden</pre><p>I then built a simple classifier with KNN to get a baseline. The optimal K is 3 with a decent accuracy score of 0.793.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/482/1*AkceKUIdVgy_gRkPyEqxMw.png" /></figure><h3>Models</h3><p>I ended up with 6906 (3501 legit, 3455 phished) observations after scraping. Not surprisingly, many of the phished pages no longer exist.</p><pre> Type    #URL processed   #Pages available  <br>------- ---------------- ------------------ <br> Legit            4,000              3,501  <br> Phish            6,000              3,455</pre><p>There could be some servers or Content Management Systems (CMS, e.g. WordPress, Joomla) that are configured improperly and they returned HTTP response 200 (instead of 404) even though the page is not found. I assumed these pages are in the minority and they have a minimum impact on the eventual accuracy.</p><p>I repeated the feature selection process with the 6906 observations and once again, the same 5 dominant features surfaced. The optimum K for KNN is still 3. Good!</p><p>Here are the results for the following models.</p><pre>Model               Accuracy  <br>------------------- ---------- <br>Naive Bayes            0.757  <br>SVC                    0.760  <br>KNN (K=3)              0.791  <br>Log. Reg.              0.822  <br>Decision Tree          0.836  <br>KNN (K=3, scaled)      0.845  <br>Random Forest          0.885</pre><blockquote>Side Note:</blockquote><blockquote>Since I have already scraped the data for 14 features, I fitted them to the different models as well. I was able to get the best accuracy from Random Forest (92.1%) but I would later find out using live data that it generally performs worse, likely due to overfitting.</blockquote><h3>Demo</h3><p>I built a website using <a href="https://flask.palletsprojects.com/">Flask</a>, <a href="https://picturepan2.github.io/spectre">Spectre CSS Framework</a> and native JavaScript on an Amazon EC2.</p><p>Let’s try some URLs not found in the dataset. How about “https://metis.kaplan.com.sg”?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/667/1*bkvHc9wlStXHIsYemtEsCQ.png" /><figcaption>Classifying a legit URL — Oops, could be better</figcaption></figure><p>How about a phished link? Visit <a href="http://phishtank.com/phish_archive.php">http://phishtank.com/phish_archive.php</a> to find a “valid phish”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TJoO9jFj8NraWskp9AdI0g.png" /><figcaption>Find a phished URL from phishtank.com</figcaption></figure><p>Copy and paste the URL and viola!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/699/1*at6XQq8VYr6gCRkkeH0Vbg.png" /><figcaption>Classifying a phished URL — not too bad</figcaption></figure><h3>Conclusion</h3><p>What a journey! I started with analysing phished and legit web pages, scraping them, doing feature selection, building the classification models and finished with a website to show the results.</p><p>Although the accuracy of 88.5% for Random Forest is pretty high, I find that when it comes to live data, the results still fall short of expectations.</p><p>There is definitely room for more improvements. I believe further efforts on the following would make the classifier more robust and accurate:</p><ul><li>more observations; if I were to increase the number of pages to scrape to say 50,000, I would improve the scraper by implementing some form of state (done or not) in Postgres SQL for each URL so that I will be able to resume scraping anytime it gets interrupted for whatever reasons (e.g. network error)</li><li>check for presence of url shorteners</li><li>implement feature engineering with whois or even SSL parameters</li><li>use ensemble models in sklearn to get a single prediction based on different forms of “averaging” the models</li></ul><h3>References</h3><p>[1] <a href="https://tools.ietf.org/html/rfc2616">https://tools.ietf.org/html/rfc2616</a></p><p>[2] <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers">https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers</a></p><p>[3] <a href="https://www.icann.org/resources/pages/approved-with-specs-2013-09-17-en#whois.1.4">https://www.icann.org/resources/pages/approved-with-specs-2013-09-17-en#whois.1.4</a></p><h3>Annex</h3><figure><img alt="Initial 22 fields during Preliminary Analysis" src="https://cdn-images-1.medium.com/max/904/1*KskgSPWfh9svXcBS9xoerA.png" /><figcaption>Description of the initial 22 features</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d0779d360dc8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/supervised-learning-to-detect-phishing-urls-d0779d360dc8">Supervised Learning to detect Phishing URLs</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Project Traffico builds traffic profiles using bus arrival times]]></title>
            <link>https://medium.com/@st_sg/project-traffico-builds-traffic-profiles-using-bus-arrival-times-d1c43c41db05?source=rss-7eef572e9eb1------2</link>
            <guid isPermaLink="false">https://medium.com/p/d1c43c41db05</guid>
            <category><![CDATA[transportation]]></category>
            <category><![CDATA[singapore]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[bus]]></category>
            <category><![CDATA[tutorial]]></category>
            <dc:creator><![CDATA[SingTat]]></dc:creator>
            <pubDate>Tue, 23 Oct 2018 07:07:39 GMT</pubDate>
            <atom:updated>2019-02-16T00:49:47.587Z</atom:updated>
            <content:encoded><![CDATA[<h4>A case study on the weekday and weekend profiles of Punggol Road (TPE)</h4><p>Residents in Punggol should have heard the good news that the new link between Kallang-Paya Lebar (KPE) / Tampines Expressway (TPE) interchange and Punggol Central would be <a href="https://www.straitstimes.com/singapore/transport/connection-between-punggol-and-kpe-tpe-to-open-by-end-november-one-year-ahead-of">available by end Nov 2018</a>, one year ahead of schedule. With another vital ingress and egress into and out of Punggol Town, the dynamics of Punggol road will surely change.</p><p>Here’s a simplified version of <a href="https://www.lta.gov.sg/content/ltaweb/en/roads-and-motoring/projects/also-in-progress/expansion-of-kpe---tpe-interchange-and-construction-of-link-road.html">how it looks</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bblX1a3g5poi8fInL7CGng.png" /><figcaption>New and existing ingress/egress of Punggol Town (©OpenStreetMap)</figcaption></figure><p>Typically, you enter or leave Punggol Town via the TPE, which is currently connected to Punggol Way or Punggol Road.</p><p>Before the dynamics of Punggol road are forever changed, I think it is a good time to do a recap. It all started with an <a href="https://medium.com/@st_sg/analysing-bus-arrival-times-part-1-c9989835aff6">analysis on bus arrival times</a> and a conclusion that they can be used to infer traffic conditions along bus routes. Using this property, I built <a href="https://medium.com/@st_sg/analysing-bus-arrival-times-part-2-bde5eca09443">Project Traffico</a> and set it up to monitor the top traffic hotspots in Singapore. Every month, congestion stats are tabulated and the most congested road wins the Snail Award. To date, quite unfortunately, roads along Punggol Road (TPE section) have <a href="https://www.maventechnologies.com.sg/traffico/stats">won the award every month</a> since inauguration.</p><p>On another front, Traffico is also able to build traffic profiles using bus journey times computed from arrival times. For a start, let us look at the traffic profiles of the two champions. I choose the data from April 2018 for illustrations below because there are no school holidays during that month so the data is less susceptible to “noise”.</p><p><strong>Punggol Road — TPE (Exit TPE towards Punggol)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4KKEWxL9-gFsO72SMkAIvA.jpeg" /><figcaption>Bus route from bus stop 65191 to 65071</figcaption></figure><p>These are the journey times from bus stop 65191 to 65071 for service number 3 across an entire day (630am to 11pm) for the month of April 2018.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*alQR6HOab28TJmhX_sBFHw.png" /><figcaption>Traffic Profile (April 2018)</figcaption></figure><p>Notice the significant difference between the weekday and weekend profiles:</p><ul><li>On weekdays, you can clearly see 2 peaks during the morning and evening peak hours.</li><li>On weekends, there are no morning peaks but journey times get longer around noon.</li></ul><p>Are weekday and weekend profiles always so different across roads in Singapore? Not quite. Consider the road along Lentor Ave (Exit SLE towards Yishun) and its traffic profile.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kXxNBzzUjC07Un2X9bt1Qg.jpeg" /><figcaption>Bus route from bus stop 55271 to 59011</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VE9tW05R2mPg5GFGB_QT9g.png" /><figcaption>Traffic Profile (April 2018)</figcaption></figure><p>As you can see, the profiles are quite similar, other than the typically longer journey times on weekdays.</p><p><strong>Punggol Road — TPE Junction (Towards Sengkang)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sQDc1SU_Lx56Wq7bOR1ykg.jpeg" /><figcaption>Bus route from bus stop 65079 to 65069</figcaption></figure><p>This is the profile from the journey times for service number 83 from bus stop 65079 to 65069.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ypa33D5NEo_2be4u8E43sA.png" /><figcaption>Traffic Profile (April 2018)</figcaption></figure><p>Again the weekday and weekend profiles are quite different. One look and you can see that you do not want to be on this stretch of road on weekend afternoons because it is usually congested. This is the main reason why this road has been the champion 8 times.</p><p>Are there any actual benefits in looking at these profiles? Well, for a start, you can try to avoid certain roads at certain times of the day. In fact, one of the first stories I covered under Traffico is a <a href="https://www.maventechnologies.com.sg/traffico/stories/punggol-road-traffic-profile">traffic profile of Punggol Road</a>. Understanding these profiles also helps us <a href="https://www.maventechnologies.com.sg/traffico/stories/punggol-road-tree-pruning">make better decisions</a>. Agencies involved in traffic engineering and maintenance (e.g. optimising traffic light sequences, detecting faulty traffic lights, etc.) could also tap on these profiles as references to the actual conditions on the ground.</p><p>Traffico demonstrates a creative use of bus arrival times to infer traffic conditions. On top of providing status on congestions, it is also able to build traffic profile of roads through those arrival times. I believe there is an unexplored potential in using these profiles to complement efforts in the areas of Smart Nation Initiatives, Urban Planning and Preemptive Maintenance.</p><p>With the opening of a new road directly into Punggol Central from the KPE/TPE interchange, the traffic profiles of Punggol road will likely change for the better. I look forward to a new Snail Award champion.</p><blockquote>Update</blockquote><blockquote>Check out the latest story on the traffic profile along Punggol Road after the opening of the new link to Punggol Central:<br><a href="https://www.maventechnologies.com.sg/traffico/stories/a-year-on-punggol-road">https://www.maventechnologies.com.sg/traffico/stories/a-year-on-punggol-road</a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d1c43c41db05" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Project KOPI & TEH (KAT) 2.0 — Shorthand notations for SG coffee and tea]]></title>
            <link>https://medium.com/@st_sg/project-kopi-and-teh-version-2-shorthand-notations-for-sg-coffee-and-tea-4c57d0952db8?source=rss-7eef572e9eb1------2</link>
            <guid isPermaLink="false">https://medium.com/p/4c57d0952db8</guid>
            <category><![CDATA[tea]]></category>
            <category><![CDATA[singapore]]></category>
            <category><![CDATA[culture]]></category>
            <category><![CDATA[design]]></category>
            <category><![CDATA[coffee]]></category>
            <dc:creator><![CDATA[SingTat]]></dc:creator>
            <pubDate>Thu, 11 Oct 2018 00:41:36 GMT</pubDate>
            <atom:updated>2019-03-21T02:13:17.752Z</atom:updated>
            <content:encoded><![CDATA[<h3>Project KOPI &amp; TEH (KAT) 2.0 — Shorthand notations for SG coffee and tea</h3><blockquote>This article is heavily influenced by the work done in <a href="https://medium.com/@st_sg/adding-a-binary-touch-to-singapore-coffee-kopi-and-tea-teh-culture-244534fdf4">Project KAT</a></blockquote><p>For some time, I have been intrigued by some local coffee chains with their shorthand notations of my favourite drink, TEH C SIEW-DAI (that’s tea with evaporated milk, less sweet):</p><ul><li>TC SD</li><li>TC-</li><li>TSi-</li></ul><p>That got me looking back at the work I have done for <a href="https://medium.com/@st_sg/adding-a-binary-touch-to-singapore-coffee-kopi-and-tea-teh-culture-244534fdf4">Project KAT</a>, which is now effectively version 1.0. After going through my notes, I found designs that I could work on to build my own version of shorthand notations. Since the general feedback for 1.0 is that it is too overwhelming (binary anyone?), I decide to use simplicity as the fundamental criterion for the design of these notations in 2.0.</p><p>This is my template of the shorthand notation in English.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qyjQr5u9nQkpNBc047Qpsw.jpeg" /><figcaption>Template in English</figcaption></figure><p>The fields in grey are optional. When they are not present, the default options apply. These defaults are represented by dots in the template for clarity only; there is no need to indicate these dots in the shorthand.</p><p>This is the template in local context.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vv7iQJXLATKFHEsT7l5ETQ.jpeg" /><figcaption>Template in Singapore context</figcaption></figure><p>The concept is simple to understand with these examples.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vqFV3mWsh0jC6WCAIXkhag.jpeg" /><figcaption>Shorthand notation for <strong>KOPI O GAO</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dSs9KpsKM6Mh82n4oUEr6g.jpeg" /><figcaption>Shorthand notation for <strong>TEH C PENG SIEW-DAI</strong></figcaption></figure><p>Some design considerations:</p><ul><li>Built on concepts in 1.0 for consistency, you should find the 5 categories of Base, Milk, Ice, Strength and Sweetness (BMISS) familiar and that the notations are just a different representation of the same 240 drinks covered in 1.0.</li><li>I use a forward slash / for cold drinks because it reminds me of the straw in 1.0 and it is very intuitive. I suppose nobody has time to draw ice cubes?</li><li>The sequence of categories in the local context is read in the order Base-Milk-Ice-Strength-Sweetness, although it is not evident from the template if you go with the typical left to right, top to bottom flow. I sacrificed a bit of clarity for simplicity, which should not pose a problem for intermediate readers. The sequence in English does not really matter (“Tea without milk, ice, strong, less sweet” or “Ice tea without milk, strong, less sweet” are both fine) but I tend to keep the same sequence for consistency.</li></ul><p>More examples?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vZk19Fq50qtmqhiNwE3CgQ.jpeg" /><figcaption>Shorthand notations of common drinks</figcaption></figure><p>Here’s the shorthand of all the drinks covered in 1.0. Why 240 drinks? There are 5 categories, each with 2, 4, 2, 4 and 4 options. That gives 256 (2 x 4 x 2 x 4 x 4) possibilities. Since 8 combinations of KOPI KOSONG and 8 combinations of TEH KOSONG are invalid, that leaves us with a grand total of 240 (256–16) drinks. Again, in reality, some of these drinks are not so common.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0T2-T9Exg6lRQzxb_eT3jw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t65oFKAEMxIdz9M-AM-6mA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SMGQEGhlbcNVVZHzuZpRZA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bwMZdM9X2QELJ0902CFXPg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yNFMWXznJwOd4Zv9HYCcow.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tm0DJnnxpyfIlbouITEWIg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rQ2dAaMpMpRxhe5aKHOWuA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C5MRTkDaMLcb9PQ6X4AEow.jpeg" /></figure><p>One final point. Unlike the decoding table in 1.0, this template can be intuitively extended to represent more drinks. Without making explicit changes to the template, you should have no problem understanding these notations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SjeQakD2qLKzr82hRuF21A.jpeg" /></figure><p>Essentially, this is my second attempt to systematically represent the local coffee and tea using the BMISS categories. The shorthand notations derived here are comprehensive, intuitive and simple to understand.</p><p>As with <a href="https://medium.com/@st_sg/adding-a-binary-touch-to-singapore-coffee-kopi-and-tea-teh-culture-244534fdf4">Project KAT</a>, I hope you are intrigued by the awesome scene of local coffee and tea culture in Singapore. Try them if you drop by. I can’t say you will like them, but I can say it will be an authentic experience.</p><blockquote>March 2019 Update: I have created some WhatsApp stickers for Android.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*HS21zgueQ5z6TBoynqaVBw.png" /><figcaption>KOPI &amp; TEH WhatsApp stickers</figcaption></figure><p>You can find them here:</p><p><a href="https://play.google.com/store/apps/details?id=sg.projectkopiandteh.wastickers">Project KOPI &amp; TEH - WhatsApp Stickers - Apps on Google Play</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4c57d0952db8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Project Traffico — A Creative Use of Bus Arrival Times]]></title>
            <link>https://medium.com/@st_sg/analysing-bus-arrival-times-part-2-bde5eca09443?source=rss-7eef572e9eb1------2</link>
            <guid isPermaLink="false">https://medium.com/p/bde5eca09443</guid>
            <category><![CDATA[singapore]]></category>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[open-data]]></category>
            <category><![CDATA[bus]]></category>
            <dc:creator><![CDATA[SingTat]]></dc:creator>
            <pubDate>Thu, 23 Nov 2017 06:54:06 GMT</pubDate>
            <atom:updated>2018-02-04T07:33:07.537Z</atom:updated>
            <content:encoded><![CDATA[<h3>Project Traffico — A Creative Use of Bus Arrival Times</h3><h3>Traffico (Traffic Observer) monitors bus arrival times to infer traffic conditions</h3><p>In <a href="https://medium.com/@st_sg/analysing-bus-arrival-times-part-1-c9989835aff6">Part 1</a>, I have shown that we can quite accurately compute the estimated journey time between bus stops through the use of backdated arrival times.</p><p>A straightforward application is to use them to infer traffic conditions. Indeed, I have always wanted to know how bad traffic congestions along Punggol Road are and now I am able to collect and visualise some real data.</p><blockquote>Developers who are keen to determine the speed of traffic along roads should check out LTA’s Traffic Speed Band API. This API allows you to query the current traffic speeds on roads, expressed in speed bands. I am aware of this API but I still proceeded to build Traffico as an exercise. Just a note from the current documentation (API User Guide dated 6 Oct 2017), I have no way to determine the unique ID of roads for this API and I have given my feedback to LTA.</blockquote><p>Automating the computation process, I was able to compile the estimated journey times for bus service 83 between bus stop code 65079 and 65069 during the off-peak hours for a few weeks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*6Mj7-O64I6251ynarxq6TA.jpeg" /></figure><p>As shown, on almost 160 occasions, the bus took 130+ seconds to travel between the 2 stops. We can use data in this chart to set a realistic threshold and if it has been exceeded, we know that the bus is taking longer than usual to travel between the stops, which means the road <strong>might</strong> be congested.</p><p>In order to mitigate the constraint shared in <a href="https://medium.com/@st_sg/analysing-bus-arrival-times-part-1-c9989835aff6">Part 1</a>, I have to use at least 2 services along a single road to strengthen the algorithm. The probability of bus bunching across multiple services is lower.</p><h3><strong>Screenshots</strong></h3><p>The images below are taken on 17 November 2017 with an iPad Mini. Through visual inspection (i.e. looking out of my window), I could see the congestion towards Sengkang.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/774/1*QcxAE8rG8ccQxuXrqROezA.png" /><figcaption>Home Page</figcaption></figure><p><strong>Punggol Road — TPE Junction</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/774/1*yk0c-Y-C5Q97Yy3F0qKNzQ.png" /><figcaption>Map and Details</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/774/1*ElPXO1rgLxYp8nJWP-O2YQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I5Kkdr6MQdRJRRxvsjuayQ.png" /><figcaption>Estimated Journey Times (Left: Small Screen; Right: Wide Screen)</figcaption></figure><p><strong>Punggol Road — TPE</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/774/1*_2jFfT28bbBT92JDLU7mmQ.png" /><figcaption>Map and Details</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/774/1*xyuw1dBHM2gT7HOJpi2mWQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LpU5ixFJu-Z1_OwahpJIaA.png" /><figcaption>Estimated Journey Times (Left: Small Screen; Right: Wide Screen)</figcaption></figure><h3><strong>Features</strong></h3><p>I have also added two features for registered users.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*baiLHMrKYWrkUbp6GrBOZw.png" /><figcaption>Email Alert</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8SqJx_M8xiQShTBWTGaW1A.png" /><figcaption>Chart with Historical Data</figcaption></figure><p>Well, the system is not perfect but generally it works. It takes a while for the system to react to actual changes on the ground because we are monitoring buses and updates are only possible when buses arrive at stops (usually every 3 to 15 mins).</p><p>You can find some interesting observations I have gathered from the historical data under <a href="https://www.maventechnologies.com.sg/traffico/stories">Stories</a> or you can try out <a href="https://www.maventechnologies.com.sg/traffico">Traffico</a> at <a href="https://www.maventechnologies.com.sg/traffico/">https://www.maventechnologies.com.sg/traffico</a>.</p><p>If you find these articles useful, please share them. Let me know if you have any queries or feedback.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bde5eca09443" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analysing Bus Arrival Times — Findings]]></title>
            <link>https://medium.com/@st_sg/analysing-bus-arrival-times-part-1-c9989835aff6?source=rss-7eef572e9eb1------2</link>
            <guid isPermaLink="false">https://medium.com/p/c9989835aff6</guid>
            <category><![CDATA[singapore]]></category>
            <category><![CDATA[open-data]]></category>
            <category><![CDATA[bus]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[tutorial]]></category>
            <dc:creator><![CDATA[SingTat]]></dc:creator>
            <pubDate>Mon, 20 Nov 2017 07:38:01 GMT</pubDate>
            <atom:updated>2018-02-04T07:25:04.771Z</atom:updated>
            <content:encoded><![CDATA[<h3>Analysing Bus Arrival Times — Findings</h3><p>I use bus apps to check bus arrival times so that I can walk quickly to the bus stop to catch the arriving bus. Most of the time, I will make it but occasionally, 2 things happen:</p><ul><li>I rush to the bus stop and see the bus leaving — the ‘just missed the bus’ phenomenon</li><li>I wait at the bus stop for the next couple of minutes seeing the ‘bus arriving’ message — the ‘bus arriving’ phenomenon</li></ul><p>Curious, I analysed the bus arrival times available from the Application Programming Interface (API) at the <a href="https://www.mytransport.sg/content/mytransport/home/dataMall.html">LTA DataMall</a> to look for clues. I found possible reasons behind the phenomenons and something more.</p><h3><strong>Setup</strong></h3><p>Let’s start with an illustration using 2 bus stops and a simple formula.</p><pre>[1] Journey time = Arrival at #2 — Arrival at #1</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p2SBXLqaHuM0YDM8MnPLWg.jpeg" /></figure><p>Now, let’s look at the actual data captured on 12 Nov 2017 for bus service 83 between 2 stops. I am able to ascertain the traffic conditions (i.e. smooth or heavy) through visual confirmation.</p><h3><strong>Smooth Traffic</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wkv2VjvyWfbbXgMbAvQrWA.jpeg" /><figcaption>Data captured from Bus Arrival API during smooth traffic</figcaption></figure><p>Let’s analyse timings at stop #1:</p><ul><li>At 8:48:04, the bus is predicted to reach at 8:48:37.</li><li>At 8:49:03, the bus is predicted to reach at 8:48:14. Technically this timing is backdated and should be classified as reporting rather than predicting.</li><li>At 8:50:03, the next bus is predicted to reach at 9:00:54.</li></ul><p>Interestingly, through my research, I discovered that a backdated arrival time quite accurately* reports the time a bus is actually at a stop (i.e. actual arrival time). Furthermore, the latest backdated arrival time is often the most accurate.</p><blockquote>*90% of the time, backdated arrival times are within 60s of the actual arrival times</blockquote><p>In other words, backdated arrival times are good estimates but are not timely; it does not help to know at 8:49:03 that the bus was at the stop at 8:48:14. Ideally, a bus app should disregard such ‘backdated’ timings. If not, you will see the ‘just missed the bus’ phenomenon.</p><p>Extending the same principles to bus stop #2, you can see a similar pattern.</p><p>With this knowledge, we can break down [1] further,</p><pre>[1a] Predicted journey time <br>= predicted arrival at #2 — predicted arrival at #1</pre><pre>[1b] Estimated journey time <br>= backdated arrival at #2 — backdated arrival at #1</pre><p>Remember, backdated times are more accurate so [1b] is more accurate than [1a].</p><p>From [1a], we can calculate that the predicted journey time is a flat 127s. I suppose this is probably determined by LTA using its arsenal of tools available under the <a href="https://www.lta.gov.sg/content/ltaweb/en/roads-and-motoring/managing-traffic-and-congestion/intelligent-transport-systems.html">Intelligent Transport System</a>.</p><p>From [1b], we can determine the estimated journey time as only 89s (08:50:06 minus 08:48:37).</p><p>Pictorially,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qf80n16J-L56jUyZliIFGQ.jpeg" /></figure><h3>Heavy Traffic</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W40_fDuH8ogxoZNdLvhbsw.jpeg" /><figcaption>Data captured from Bus Arrival API during heavy traffic</figcaption></figure><p>Let’s highlight backdated timings at stop #1:</p><ul><li>At 15:01:05, bus is reported to have arrived at 15:00:16.</li><li>At 15:02:04, bus is reported to have arrived at 15:00:41 (i.e. best backdated arrival time).</li></ul><p>Now look at the timings at stop #2. Firstly, the predicted journey time of 146s between the two stops have increased slightly from 127s. LTA probably obtained some feedback from its sources about the heavy traffic and added some buffers to it. Secondly, the arrival times at #2 are being pushed back repeatedly, which conceptually, gives us the dreaded ‘bus arriving’ phenomenon.</p><p>Again, note these backdated timings at #2:</p><ul><li>At 15:03:04, the bus is reported to have arrived at 15:02:53.</li><li>At 15:04:04, the bus is reported to have arrived at 15:03:40.</li><li>At 15:05:05, the bus is reported to have arrived at 15:04:18.</li><li>At 15:06:05, the bus is reported to have arrived at 15:05:19 (i.e. best backdated arrival time)</li></ul><p>Hence from [1b], the bus took 278s (15:05:19 minus 15:00:41) to travel between the same stops during heavy traffic. Once again, the predicted journey time [1a] of 146s is pretty off.</p><p>Yes, pictorially,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cvhXbodC-mO4O0YwAtcOTA.jpeg" /></figure><h3><strong>Constraint</strong></h3><p>When buses of the same service arrive in close proximity (i.e. bus bunching), arrival times are affected and interpreting [1b] becomes a challenge.</p><h3>Conclusion</h3><p>I have discovered that while backdated arrival times might not be suitable for bus apps, I found them good estimates for recording the arrival times of buses at stops and in turn, preferred parameters for calculating estimated journey times.</p><p>In fact, we can actually use them to infer the traffic conditions along specific stretch of roads where buses ply. This is the premise of <a href="https://www.maventechnologies.com.sg/traffico">Traffico</a> (<a href="https://www.maventechnologies.com.sg/traffico/">https://www.maventechnologies.com.sg/traffico</a>). In the <a href="https://medium.com/@st_sg/analysing-bus-arrival-times-part-2-bde5eca09443">next article</a>, I will discuss the development of this tool.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c9989835aff6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Adding a binary touch to Singapore coffee (kopi) and tea (teh) culture]]></title>
            <link>https://medium.com/@st_sg/adding-a-binary-touch-to-singapore-coffee-kopi-and-tea-teh-culture-244534fdf4?source=rss-7eef572e9eb1------2</link>
            <guid isPermaLink="false">https://medium.com/p/244534fdf4</guid>
            <category><![CDATA[singapore]]></category>
            <category><![CDATA[culture]]></category>
            <category><![CDATA[creative]]></category>
            <category><![CDATA[binary]]></category>
            <category><![CDATA[kopi]]></category>
            <dc:creator><![CDATA[SingTat]]></dc:creator>
            <pubDate>Fri, 10 Mar 2017 04:30:48 GMT</pubDate>
            <atom:updated>2019-03-21T02:19:56.922Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-NrjulWWsglbZuASbrCfrw.png" /><figcaption>A Kickstarter project to promote the awesome scene of local kopi and teh culture</figcaption></figure><p>A while back, I came across an incident where a patron had to walk 10m away to confirm his friend’s kopi or teh order. As an engineer, I start to ask myself if there is a better way to remember such orders. Knowing these local kopi and teh orders can be confusing because they are made up from different languages (e.g. KOPI C PENG SIEW-DAI), I started designing a binary sequence to systematically ‘step’ through each order. It turns out that it would get super complex if I were to include all the possibilities (e.g. GU-YOU, SUA, etc.) so I decided to only include some of the more common terms to start things off, though I still ended up with 240 orders.</p><p>Through this process however, I also came to appreciate more of these local terms and I am still fascinated by how these languages can come together. To some extent, it really shows how the different communities have integrated seamlessly instead of forming their own separate enclaves. That led to the cultural diversity we have today.</p><p>Hence, in this Kickstarter project, I chose to produce T-shirts. Here’s why.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0p241wuU4w6v43ctIUl_Hg.png" /><figcaption>Front of T-shirt</figcaption></figure><ul><li>It can really solve a problem; if I am wearing this, you can see it from 10m away. The numbers also make it easier for you to remember the order. I have developed an <a href="https://maventechnologies.com.sg/project-kopi-and-teh">interactive page </a>to help you with the computation. Also, for tourists who are not familiar with these local orders, that “KOPI C GAO KOSONG” label in front could help get you started.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PKkVha42_Se-CTipwwx2Kw.png" /><figcaption>Back of T-shirt</figcaption></figure><ul><li>It helps to spread the word; The decoding table at the back is in English to maximise reach to an international audience. If you are familiar with the local terms, you will understand the one to one mapping and the dashes denoting default options (which need not be articulated).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*0U6IixO8IsbaJIJ1Dlu0vA.png" /><figcaption>Decoding table in local context</figcaption></figure><ul><li>It is something I am confident of delivering; I did some research into T-shirt screen printing and heat transfer and revised my design a few times to make sure it can be printed. At least for now, I am more confident of delivering a T-shirt compared to other products.</li></ul><p>I chose 2 colours for these T-shirts: black or white. Being more familiar with C++ and Java (the programming language, not coffee) rather than colours or designs, I chose something ‘safer’ that should go well with your casual wear.</p><p>To help connect people to these terms / orders / flavours, I have also created a few simple comic strips loosely based on these KOPI and TEH “characters”. It is like a backstage pass where you can sneak up on them to hear what they are talking about (hence titled “Overheard”). You will discover that these characters behave in a certain way due to their flavours. For example in “CLIQUES”, the flavours with milk group themselves together while ostracising the one without. In “ENERGY”, the flavour with the most sugar gets hyperactive.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*QCe_X2-GiQhvbJZH_lpC-A.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*h5Kv8QnQ41Nzpue-92t3ag.jpeg" /></figure><p>I hope you find this project interesting and that it offers fresh perspectives on applying engineering concepts to culture. These T-shirts are suitable for locals, tourists, for yourself or as gifts. The campaign is currently still on (until 22 March 2017) at <a href="http://kck.st/2muPD7N">Kickstarter</a> and the comic strips are available at <a href="https://www.facebook.com/explore.sg.kat/posts">Facebook</a>. Let me know your queries or feedback :)</p><blockquote>March 2019 Update: I have added a “version 2” to Project KAT, which includes shorthand notations and some WhatsApp stickers. Check it out <a href="https://medium.com/@st_sg/project-kopi-and-teh-version-2-shorthand-notations-for-sg-coffee-and-tea-4c57d0952db8">here</a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=244534fdf4" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>