<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Data_Dream_J.O. on Medium]]></title>
        <description><![CDATA[Stories by Data_Dream_J.O. on Medium]]></description>
        <link>https://medium.com/@DataDreamJO?source=rss-42566753e5c8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*cirpBlb0Z0XgVDohyPJDdw.jpeg</url>
            <title>Stories by Data_Dream_J.O. on Medium</title>
            <link>https://medium.com/@DataDreamJO?source=rss-42566753e5c8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 14:16:49 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@DataDreamJO/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building an Automated Business Email Discovery Pipeline with Python]]></title>
            <link>https://medium.com/@DataDreamJO/building-an-automated-business-email-discovery-pipeline-with-python-e0469f3876e9?source=rss-42566753e5c8------2</link>
            <guid isPermaLink="false">https://medium.com/p/e0469f3876e9</guid>
            <category><![CDATA[web-scrapping]]></category>
            <category><![CDATA[revenue]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[crawl4ai]]></category>
            <dc:creator><![CDATA[Data_Dream_J.O.]]></dc:creator>
            <pubDate>Tue, 10 Nov 2026 12:02:54 GMT</pubDate>
            <atom:updated>2026-11-12T02:06:34.316Z</atom:updated>
            <content:encoded><![CDATA[<p>Date of creation: Monday, 9 November 2026, Last updated: Thursday, 12 November 2026</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z2qAV5n6iWPiNz37h8HSuw.png" /><figcaption>Figure 1: This figure illustrates the underlying business problem, the solution’s (program) overall workflow, and its expected contribution to improving lead data acquisition within the Revenue Department.</figcaption></figure><h3>Part 1: Introduction — The problem</h3><p>In B2B sales, identifying potential leads (or merchants) is an important part of driving business revenue. However, one of the challenges lies in finding reliable contact information to reach these potential customers.</p><p>It is possible to develop a Google Maps scraper to retrieve useful business information, such as company websites, phone numbers and physical addresses. However, email addresses are generally not displayed on Google Maps. Even when a business has an official website, its email contact information may not be publicly available.</p><p>One alternative is to purchase business contact datasets from external providers. While this may save time, the reliability of such information can be difficult to assess. Without knowing how the data was collected, maintained or validated, there is no guarantee that the email addresses are accurate or up to date.</p><p>Another possibility, with the growing capabilities of AI assistants such as ChatGPT and Claude, is to ask these tools to search websites and extract business contact information directly. However, the accuracy and consistency of the extracted information may vary. AI-powered browsing tools may also encounter difficulties navigating different website structures or accessing websites with <a href="https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots">anti-bot restrictions</a>, such as CAPTCHA challenges and Cloudflare protection. Furthermore, processing large numbers of companies individually may be constrained by usage limits and browsing capabilities, making it difficult to ensure consistent results at scale.</p><p>These challenges motivated me to explore building a dedicated small scale email discovery tool using Python. This pipeline functions to crawl for relevant websites, identify and extract publicly available potential email addresses, and determine the most probable business contact emails. By controlling the extraction and processing logic, the pipeline aims to provide a more structured and repeatable approach to gathering business emails.</p><p>Additionally, the program is designed to require minimal manual intervention, with a CSV file containing business names serving as its primary input. Ultimately, the goal is to make business email discovery more accessible and reliable through automation. I believe the program could also serve as a foundation for further development, with the potential to incorporate additional features, such as extracting other merchant information, allowing users to specify the information they wish to retrieve, and customising the structure and format of the generated output tables.</p><h3>Part 2: System Architecture — How it works?</h3><h4><strong>2.1 Input and initialisation</strong></h4><pre># Find and read files<br>df = pd.read_csv(path/&quot;Test.csv&quot;)[&quot;business_name&quot;]<br><br># Set up<br>today = datetime.today().strftime(&quot;%Y%m%d&quot;)<br>reader = easyocr.Reader([&quot;en&quot;])<br>headers = {&quot;User-Agent&quot;: (&quot;Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36&quot;)}<br>query = &quot; Singapore corporate office email&quot;<br>companies = df.to_list()<br>total = len(companies)<br>rows = [] <br><br># Loop through each company to extract a list of potential emails<br>for index, company_name in enumerate(companies):<br>    print(f&quot;\n=== Starting with {company_name} | {index + 1}/{total} ===&quot;)</pre><p>The whole process begins by reading a CSV file, and extracting the business names which will be individually fed into the program, to derive the websites that might contain the company’s email contact.</p><h4>2.2 Website Discovery</h4><pre># First, use google as the search engine for acquiring website links that might contain a company&#39;s email address<br>results = search_sites(company_name, query)<br># If anything goes wrong with google search, move on to DDGS<br>if not results:<br>    print(&quot;Captcha unsolved — using DDGS fallback&quot;)<br>    results = search_sites_2(company_name)</pre><p>The first stage of the pipeline involves identifying potential websites that may contain publicly available contact information for each business.</p><p>Google serves as the primary search engine due to its established search capabilities and broad coverage of relevant websites. This choice is supported by<a href="https://journals.sagepub.com/doi/abs/10.1177/09610006221138579"> Hoseinabadi and CheshmehSohrabi (2022)</a>, who compared Google, Yahoo, DuckDuckGo and Bing using a combined performance indicator, and discovered Google achieved the highest overall ranking among the four search engines.</p><p>Similarly, <a href="https://asistdl.onlinelibrary.wiley.com/doi/10.1002/pra2.758">Yagci et al. (2022)</a> found that Google returned more unique domains within its top search results compared to the other search engines examined. However, Google has anti-bot measures that can interrupt automated searches. To address this, I implemented a customised CAPTCHA-handling mechanism with multiple fallback strategies to help the pipeline overcome these interruptions.</p><p>When Google Search is unsuccessful, primarily due to difficulties in resolving Google’s reCAPTCHA challenges, the pipeline switches to Dux<strong> </strong>Distributed Global Search (DDGS) to retrieve alternative website links. DDGS offers the advantage of being an open-source Python metasearch library that provides programmatic access to <a href="https://haystack.deepset.ai/integrations/ddgs">multiple search services without requiring a paid search API</a>. Furthermore, its ability to aggregate results from different search providers reduces reliance on any single provider. Although individual search services may impose their own access restrictions, using multiple search backends provides alternative avenues for retrieving relevant website links, thereby improving the pipeline’s flexibility and resilience against search failures.</p><p>Once the search results have been collected, the identified website links are passed to the next stage for email extraction.</p><h4><strong>2.3 Two-Level Email Extraction</strong></h4><p>After identifying potential company websites, the pipeline proceeds to a two-level email extraction process designed to improve the chances of retrieving publicly available email addresses.</p><pre># First level scraping code: curl_cffi<br>p_emails, failed = first_level(results, headers, [], [])<br><br># Second level scraping code: crawl4ai + OCR chuncking<br>if failed:<br>    print(f&quot;\nList of failed links: {failed}&quot;)<br>    p_emails, failed = asyncio.run(second_level(path, headers, reader, failed, p_emails))<br><br># p_emails contains all potential emails, failed contains all unsuccessful website links</pre><p>Level 1: Initial Extraction</p><p>The first level uses curl_cffi to retrieve website content, followed by BeautifulSoup to parse the HTML. In the process, the raw HTML will be converted to a single Python string containing all visible text content before email extraction using regular expressions. In addition, a css selector is used to find all mailto: hyperlinks, which may contain email addresses embedded within the website&#39;s contact links. The extracted addresses are collated into a list called p_emails, which contains all potential email candidates.</p><p>A website is considered unsuccessful if the request does not return an HTTP 200 status code, or if no email addresses are found despite successfully accessing the page. These websites will be placed into a failed list (failed).</p><p>curl_cffi was selected because it is a proactive measure to combat anti-bot blocks caused by TLS fingerprinting by <a href="https://brightdata.com/blog/web-data/web-scraping-with-curl-cffi">impersonating browser TLS /JA3/HTTP/2 fingerprints</a>. Moreover, it is a <a href="https://dev.to/amendamax2025/bypassing-cloudflare-waf-and-akamai-in-python-using-tls-fingerprinting-the-curlcffi-guide-53l7">light weight</a> HTTP client library which allows webpage information to be retrieved without launching a full browser. However, curl_cffi does not execute JavaScript, as it <a href="http://curl_cffi documentation – Impersonation Guide (FAQ)">does not include a browser or JavaScript runtime</a>. As a result, websites that rely on JavaScript to retrieve and populate content dynamically may return a <a href="https://link.sc/blog/curl-cffi-web-scraping-guide">minimal HTML document</a> that does not contain the required information or data. Therefore, Crawl4AI is used as a browser-based alternative when <a href="https://docs.crawl4ai.com/core/quickstart/">JavaScript rendering is required</a>.</p><p>Level 2: Fallback Extraction</p><p>Websites that fail the initial extraction process are passed to the second level, which uses Crawl4AI to retrieve and process their content. Each website is given a maximum processing time of 120 seconds.</p><p>If crawling is successful, the pipeline uses the extracted Markdown to identify potential email addresses. If no email addresses are found, it captures a full-page screenshot and divides it into smaller sections for Optical Character Recognition processing (OCR), which is also known as <a href="https://unstructured.io/blog/easy-web-scraping-and-chunking-by-document-elements-for-llms">chunking</a>. Each section is scanned individually to identify potential email addresses, allowing the pipeline to examine the entire webpage, including content further down the page such as footers and contact sections.</p><p>If Crawl4AI fails to retrieve or process the website content, the pipeline skips Markdown extraction and proceeds directly to the screenshot-based OCR approach. This provides an alternative way to identify email addresses from the webpage’s rendered visual content, even when conventional content extraction is unsuccessful.</p><h4>2.4 Intermediate Data Collection and CSV Export</h4><p>After both extraction stages are completed, the results are consolidated into a dictionary containing the business name, websites searched and potential email addresses.</p><p>These records are appended to a list called rows.</p><p>To reduce the risk of losing previously collected results if the program encounters an error or is interrupted, the program periodically exports its progress into a CSV file after processing every seven businesses.</p><pre>row = {<br>&quot;business_name&quot;: company_name,<br>&quot;websites_searched&quot;: &quot;\n&quot;.join(item for item in results if isinstance(item, str) and item.strip()) if results else &quot; &quot;,<br>&quot;potential emails&quot;: &quot;\n&quot;.join(item for item in p_emails if isinstance(item, str) and item.strip()) if p_emails else &quot; &quot;}<br><br>rows.append(row)<br><br>print([f&quot;{name}: {answer} |&quot; for name, answer in row.items()])<br><br>output_path = output_folder / f&quot;{today}.csv&quot;<br><br>if len(rows)%7 == 0:<br>    pd.DataFrame(rows).to_csv(output_path, index = False)<br><br>... after loop ends ... <br><br>output_path_final = output_folder / f&quot;{today}_final.csv&quot;<br>pd.DataFrame(rows).to_csv(output_path, index=False)</pre><p>Once all businesses have been processed, the collected results are exported for further processing.</p><h4>2.5 Email Cleaning and Confidence Ranking</h4><p>After the extraction process, the collected email candidates undergo a cleaning and validation process to identify the most probable contact email for each business.</p><p>First, the email_validator library is used to validate each email address, with deliverability checks enabled. This checks whether the email follows the correct formatting rules and whether its domain is configured to receive emails. The latter is determined through DNS lookups, which examine the <a href="https://github.com/JoshData/python-email-validator/blob/main/email_validator/deliverability.py">domain’s mail exchange (MX) records, A or AAAA records</a>. However, this does not guarantee that the individual mailbox exists.</p><p>Next, the pipeline uses Python’s SequenceMatcher from the difflib library to calculate the similarity between the business name and both the email username and domain. The higher of the two similarity scores is retained as the email’s confidence score.</p><p>Finally, the email candidates are ranked according to their confidence scores. The highest-scoring candidate is selected, provided that it meets a predefined similarity threshold. This helps prioritise email addresses that are more closely associated with the business rather than selecting an arbitrary address from the extraction results.</p><pre># This part cleans the email and generates a new column that indicates the highest confidence email<br>df_phase1 = pd.read_csv(output_path)<br>process_companies_yield_emails(df_phase1, output_folder, today)</pre><h3>Part 3: Demonstration</h3><p>To demonstrate the pipeline in practice, I have included a short video showing its execution, from providing the input CSV file containing business names to generating the final output.</p><p>The video illustrates the automated workflow, including website discovery, the two-level email extraction process, and the subsequent cleaning, validation and ranking of potential email addresses.</p><p>Please note that the CAPTCHA bypass mechanisms are implemented to facilitate uninterrupted data scraping and they fall outside the primary scope of this demonstration.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FzxEv_ewdQrQ%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DzxEv_ewdQrQ&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FzxEv_ewdQrQ%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/8407ea0c67a1aef77ee9763b42eaf8e5/href">https://medium.com/media/8407ea0c67a1aef77ee9763b42eaf8e5/href</a></iframe><h3>Part 4: Results</h3><p>From the initial test involving 13 businesses, the pipeline identified 7 high-confidence email candidates. While this may appear to be a relatively modest result, it is important to consider the scope of the search. The pipeline currently retrieves results primarily from the first page of Google Search, which limits the number of potential websites available for email extraction.</p><p>Based on my experience experimenting with DDGS, retrieving a larger pool of approximately 30 website links per business could potentially improve email coverage, yielding at least one potential email candidate for a reasonable number of businesses. However, this also comes with varying levels of accuracy, as a larger pool of search results may introduce irrelevant websites and unrelated email addresses.</p><p>In addition, <a href="https://arxiv.org/abs/2207.07330">Yagci et al. (2022)</a> found that search engines do not return identical results, suggesting that combining multiple search engines could provide access to a wider variety of sources. In the current implementation, Google is prioritised, while DDGS serves as a fallback when Google searches are unsuccessful. However, the Google search process has generally been able to proceed without requiring the fallback. Thus, the actions of DDGS are not displayed in the demonstration.</p><p>One potential improvement would therefore be to expand the search beyond Google’s first page through pagination and incorporate additional results from DDGS. This would increase the number of unique websites available for extraction and potentially improve the pipeline’s overall email coverage. Moreover, retrieving a larger pool of websites would provide alternative sources when certain websites cannot be accessed due to anti-bot protections, reducing the detrimental impact of access restrictions on the overall extraction process while increasing the chances of identifying the correct email addresses for each business.</p><pre>Column1 | business_name         | potential emails                 | highest_confidence_email       | websites_searched<br>--------+-----------------------+----------------------------------+--------------------------------+------------------<br>0       | Challenger Singapore  | tb@challenger.sg                 | amk@challenger.sg              | https://links.sgx.com/1.0.0/corporate-announcements/DXFPCZRUOZ0YVW0B/422962_CHALLENGER%20_press%20release_New%20flagship.pdf<br>        |                       | dpo@challenger.sg                |                                | https://links.sgx.com/1.0.0/corporate-information/1809<br>        |                       | nex@challenger.sg                |                                | https://online.challenger.sg/pages/trade-in?srsltid=AYCX9vLY5voUxL3AbIoPpU9hXdbg84xzCy0rqdjVuL06AutPL473lSxm<br>        |                       | ws@challenger.sg                 |                                | https://www.zoominfo.com/p/Jordan-Chan/13093616712<br>        |                       | kt@challenger.sg                 |                                | https://www.capitaland.com/sg/malls/bugisjunction/en/stores/challenger-flagshipstore.html<br>        |                       | bp@challenger.sg                 |                                | https://www.mycareersfuture.gov.sg/companies/challenger-technologies-198400182K<br>        |                       | ep@challenger.sg                 |                                | https://www.companies.sg/business/200207221E/CHALLENGER-SINGAPORE-PTE-LTD<br>        |                       | vc@challenger.sg                 |                                | https://support.challenger.sg/hc/en-us/articles/4419948559503-Warranty-Technical-Support-Guide<br>        |                       | np@challenger.sg                 |                                | https://clementi320.wordpress.com/2012/01/13/challenger_locations/<br>        |                       | yt@challenger.sg                 |                                |<br>        |                       | sls@pixels.sg                    |                                |<br>        |                       | ts@challenger.sg                 |                                |<br>        |                       | cp@challenger.sg                 |                                |<br>        |                       | jp@challenger.sg                 |                                |<br>        |                       | fc@challenger.sg                 |                                |<br>        |                       | imm@challenger.sg                |                                |<br>        |                       | ps@challenger.sg                 |                                |<br>        |                       | fcmini@challenger.sg             |                                |<br>        |                       | hm@challenger.sg                 |                                |<br>        |                       | fim@mig.sg                       |                                |<br>        |                       | ssc@challenger.sg                |                                |<br>        |                       | us@challenger.sg                 |                                |<br>        |                       | cm@challenger.sg                 |                                |<br>        |                       | pp@challenger.sg                 |                                |<br>        |                       | fcm27@challenger.sg              |                                |<br>        |                       | fcm19@challenger.sg              |                                |<br>        |                       | tp@challenger.sg                 |                                |<br>        |                       | amk@challenger.sg                |                                |<br>--------+-----------------------+----------------------------------+--------------------------------+------------------<br>1       | Gain City             | FirstMiddleLast@gaincity.com     | FirstLast@gaincity.com         | https://www.gaincity.com/feedback/?SID=2agplpj0uk824b7jp5jeohdra0<br>        |                       | JohnMichaelDoe@gaincity.com      |                                | https://www.gaincity.com/service-connections/extended-warranty<br>        |                       | gciwish@gaincity.com             |                                | https://www.gaincity.com/service-connections/honest-practice<br>        |                       | sales@leadiq.com                 |                                | https://leadiq.com/c/gain-city/5a1d98c82300005b0087668d<br>        |                       | FirstLas@gaincity.com            |                                | https://www.gaincity.com/store_location_map_mobile<br>        |                       | LastFirst@gaincity.com           |                                | https://www.gaincity.com/feedback/?SID=0vb05jqmqi95qg3s4bgdfceb84<br>        |                       | DoeJohn@gaincity.com             |                                | https://www.gaincity.com/gain-city-mobile-apps<br>        |                       | JohnDoe@gaincity.com             |                                | https://www.gaincity.com/feedback?doc_rec_type=PKS&amp;pks_no=01373991&amp;salesman_id=11569&amp;technician_id=M-9600<br>        |                       | support@leadiq.com               |                                | https://www.gaincity.com/promotion-events/newsletter<br>        |                       | FirstLast@gaincity.com           |                                |<br>--------+-----------------------+----------------------------------+--------------------------------+------------------<br>2       | Isetan Singapore      | contactus@isetan.com.sg          | isetansin@isetan.com.sg        | https://my.linkedin.com/company/isetan-singapore<br>        |                       | isetansin@isetan.com.sg          |                                | https://links.sgx.com/1.0.0/corporate-announcements/H9PDOIQ6E8K2H2QQ/667744_Isetan_-_28_April_2021_AGM_Minutes__Final__signed_.pdf<br>        |                       | bizinfo@sg.marshallcavendish.com |                                | https://trvstage.isetan.com.sg/store-directory/<br>        |                       | sg22014@contact.page             |                                | https://sg22014-isetan-s-ltd.contact.page/<br>        |                       | hello@contact.page               |                                | https://www.isetan.com.sg/new-isetan-membership/<br>        |                       |                                  |                                | https://mu.linkedin.com/company/isetan-singapore?trk=similar-pages_result-card_full-click<br>        |                       |                                  |                                | https://jp.linkedin.com/company/isetan-singapore<br>        |                       |                                  |                                | https://rocketreach.co/lim-tay-email_22861242<br>        |                       |                                  |                                | https://www.timesbusinessdirectory.com/company-details/isetan-s-ltd/<br>--------+-----------------------+----------------------------------+--------------------------------+------------------<br>3       | TANGS                 |                                  |                                | https://tangs.com/member/signup?fbclid=IwAR1o35vHr3RctCctDP-d6-rKCdzi6hJ0wnPGK5S5pWX6ZRRIg6u_V_6nVio<br>        |                       |                                  |                                | https://tangs.com/careers/<br>        |                       |                                  |                                | https://tangs.com/content/heritage<br>        |                       |                                  |                                | https://tangs.com/member/signup<br>        |                       |                                  |                                | https://tangs.com/content/old-soles/shop-in-shop-content-page/shop-in-shop-content-page/f7da54b2-68da-4286-b6c9-2c4a186b3260<br>        |                       |                                  |                                | https://tangs.com/store/tang-plaza/floor/l4<br>        |                       |                                  |                                | https://tangs.com/careers/career-job-opening-jobs/c47ecfb4-4833-4542-b8dc-2b6440eade53<br>        |                       |                                  |                                | https://tangs.com/content/intercontinental-singapore/shop-in-shop-content-page/add1e6a8-b840-473b-854f-9a4ff0a9dde8<br>        |                       |                                  |                                | https://tangs.com/store/tang-plaza/store-in-good-company<br>--------+-----------------------+----------------------------------+--------------------------------+------------------<br>4       | Metro Singapore       | sg37148@contact.page             | XXXXXXXXX@metro.com.sg         | https://www.nlb.gov.sg/main/article-detail?cmsuuid=968a2277-f8df-45d6-b3d0-0ce546321fb9<br>        |                       | sg149298@contact.page            |                                | https://sg149298-metro-woodlands.contact.page/<br>        |                       | hello@contact.page               |                                | https://www.metroholdings.com.sg/retail.html<br>        |                       | XXXXXXXXX@metro.com.sg           |                                | https://sg37148-metro-holdings.contact.page/<br>        |                       |                                  |                                | https://sgnamecard.com.sg/metro_1.html<br>        |                       |                                  |                                | https://www.facebook.com/p/METRO-Singapore-100089307297995/<br>        |                       |                                  |                                | https://www.metroholdings.com.sg/update2020/Metro%20-%20CSR%20Announcement%20-1%20Aug%202020.pdf<br>        |                       |                                  |                                | https://www.metroholdings.com.sg/update2022/Metro%20-%20CSR%20Press%20Release%20-1%20Dec%202022%20-%20FINAL.pdf<br>        |                       |                                  |                                | https://www.facebook.com/p/METRO-Singapore-100054313819363/<br>--------+-----------------------+----------------------------------+--------------------------------+------------------<br>5       | Takashimaya Singapore | bizinfo@sg.marshallcavendish.com | contact-csc@takashimaya.com.sg | https://www.takashimayasc.com.sg/shopping-centre-vouchers/<br>        |                       | contact-csc@takashimaya.com.sg   |                                | https://www.facebook.com/takashimayasg/?locale=lv_LV<br>        |                       | name@domain.com                  |                                | https://www.emis.com/php/company-profile/SG/Takashimaya_Singapore_Ltd_en_2439948.html<br>        |                       |                                  |                                | https://www.takashimayasc.com.sg/tourists/<br>        |                       |                                  |                                | https://www.facebook.com/takashimaya.sg/<br>        |                       |                                  |                                | https://www.timesbusinessdirectory.com/company-details/takashimaya-singapore-ltd/<br>        |                       |                                  |                                | https://www.wesleyconnect.com/listingpage/sg/Takashimaya-Singapore-Ltd<br>        |                       |                                  |                                | https://contactout.com/company/Takashimaya-Singapore-Ltd-2038<br>        |                       |                                  |                                | https://www.takashimaya.com.sg/about-us/takashimaya-stores/?srsltid=AYCX9vKJgxuspccqE3i9OMqI8LzDXyGkzaF-qBsglsKPpaFiwVZcS-du<br>--------+-----------------------+----------------------------------+--------------------------------+------------------<br>6       | UNIQLO Singapore      |                                  |                                | https://sg.linkedin.com/company/uniqlo<br>        |                       |                                  |                                | https://map.uniqlo.com/sg/en/detail/11302034<br>        |                       |                                  |                                | https://www.fastretailing.com/employment/en/uniqlo/sg/career/business.html<br>        |                       |                                  |                                | https://map.uniqlo.com/sg/en/detail/11300028<br>        |                       |                                  |                                | https://rocketreach.co/uniqlo-singapore-profile_b6abcc37c862212e<br>        |                       |                                  |                                | https://builtinsingapore.com/company/uniqlo<br>        |                       |                                  |                                | https://contactout.com/company/uniqlo-singapore-careers-8958953<br>        |                       |                                  |                                | https://www.uniqlo.com/sg/en/special-feature/cp/sustainability<br>        |                       |                                  |                                | https://faq-sg.uniqlo.com/pkb_Home_UQ_SG?id=kA0Ie000000TP7K&amp;l=en_US&amp;fs=RelatedArticle<br>--------+-----------------------+----------------------------------+--------------------------------+------------------<br>7       | H&amp;M Singapore         |                                  |                                | https://hmgroup.com/about-us/corporate-governance/company-management/<br>        |                       |                                  |                                | https://www.yelp.com/biz/h-and-m-singapore-11<br>        |                       |                                  |                                | https://www.facebook.com/p/HM-100064737154922/<br>        |                       |                                  |                                | https://www.facebook.com/p/HM-100089700198777/<br>        |                       |                                  |                                | https://hmgroup.com/about-us/contact-us/raise-a-concern/<br>        |                       |                                  |                                | https://www2.hm.com/en_my/customer-service.html<br>        |                       |                                  |                                | https://www.sgpbusiness.com/company/Hm-Group-Pte-Ltd<br>        |                       |                                  |                                | https://www.streetdirectory.com/businessfinder/email.php?idCompany=172681&amp;branchId=5890<br>        |                       |                                  |                                | https://recordowl.com/company/h-m-hennes-mauritz-pte-ltd<br>--------+-----------------------+----------------------------------+--------------------------------+------------------<br>8       | The Editor&#39;s Market   | business@theeditorsmarket.com    | business@theeditorsmarket.com  | https://www.facebook.com/theeditorsmarket/?locale=en_GB<br>        |                       | support@atome.sg                 |                                | https://sg160413-the-editor-s-market.contact.page/<br>        |                       | banner-mobile@2x.png             |                                | https://rocketreach.co/zi-wong-email_511522414<br>        |                       | sg160413@contact.page            |                                | https://www.facebook.com/p/The-Editors-Market-100089461072172/<br>        |                       | hello@contact.page               |                                | https://th.theeditorsmarket.com/business-enquiries<br>        |                       | ic-findstore@2x.png              |                                | https://me.theeditorsmarket.com/contact?srsltid=AYCX9vJ_wrHa2AgSzRPWmDnVsbYQPMB_1YjdvrmSBWh0YsrglT0v3muw<br>        |                       | merchant@atome.sg                |                                | https://www.theeditorsmarket.com/customer-care?srsltid=AYCX9vI4UYf0REIHnFImHnttrHo6W72dvEYJ-uXRuT-4wHF4Oq-bVVfG<br>        |                       |                                  |                                | https://www.atome.sg/en-sg/paylater-merchants/the-editor-s-market<br>        |                       |                                  |                                | https://mm.theeditorsmarket.com/business-enquiries<br>--------+-----------------------+----------------------------------+--------------------------------+------------------<br>9       | Skechers Singapore    | copyright@openrice.com           |                                | https://sg.openrice.com/en/singapore/retail/r-skechers-r328460<br>        |                       |                                  |                                | https://ph.openrice.com/en/singapore/retail/r-skechers-sengkang-r270776<br>        |                       |                                  |                                | https://sg.openrice.com/en/singapore/retail/r-skechers-arab-street-r297883<br>        |                       |                                  |                                | https://dealer-locator.hunterdouglas.asia/sg<br>        |                       |                                  |                                | https://www.waze.com/lv/live-map/directions/sg/singapore/skechers-singapore-pte-ltd?to=place.ChIJJTr4t_QX2jER60ZJt-CMemk<br>        |                       |                                  |                                | https://sg.openrice.com/en/singapore/retail/r-skechers-punggol-r271281<br>        |                       |                                  |                                | https://www.facebook.com/fb-answers/skechers-corporate-office-contact/?locale=ur_PK<br>        |                       |                                  |                                | https://www.companies.sg/places/category/shoes?page=6<br>        |                       |                                  |                                | https://eresources.nlb.gov.sg/webarchives/details/www.skechers.com.sg.html<br>--------+-----------------------+----------------------------------+--------------------------------+------------------<br>10      | Nike Singapore        | contact@elucidate.co             |                                | https://www.companies.sg/business/S94FC4703J/NIKE-INC<br>        |                       |                                  |                                | https://www.nike.com/be/en/retail/directory/singapore/other<br>        |                       |                                  |                                | https://www.volza.com/company-profile/nike-singapore-office-92239179/<br>        |                       |                                  |                                | https://www.instagram.com/explore/locations/244160805/nike-singapore-hq-singapore/?hl=en<br>        |                       |                                  |                                | https://www.facebook.com/pages/Nike-Singapore-HQ-Singapore/288616357871913<br>        |                       |                                  |                                | https://maps.apple.com/place?auid=10038896664469009044&amp;lsp=9902<br>        |                       |                                  |                                | https://snatchjobs.sg/Companies/details?slug=1739<br>        |                       |                                  |                                | https://elucidate.co/directory/nike-global-trading-pte-ltd-ngtpsgs1<br>        |                       |                                  |                                | https://www.nike.com/ca/retail/s/nike-factory-store-changi-city-point<br>--------+-----------------------+----------------------------------+--------------------------------+------------------<br>11      | adidas Singapore      |                                  |                                | https://rocketreach.co/adidas-singapore-pte-ltd-profile_b5434747f68ecef9<br>        |                       |                                  |                                | https://www.adidas.com.my/en/help/sea-contact/contact-us<br>        |                       |                                  |                                | https://www.companies.sg/company-identifiers/199400638K<br>        |                       |                                  |                                | https://careers.adidas-group.com/jobs?location=%5B%7B%22country%22%3A%22Singapore%22%2C%22city%22%3A%22Singapore%22%7D%5D&amp;locale=en<br>        |                       |                                  |                                | https://news.adidas.com/contactus<br>        |                       |                                  |                                | https://rocketreach.co/jester-ang-email_92666899<br>        |                       |                                  |                                | https://sg.ltddir.com/companies/adidas-singapore-pte-ltd/<br>        |                       |                                  |                                | https://www.adidas.com.sg/lookbook/SC-98768398<br>        |                       |                                  |                                | https://www.adidas-group.com/<br>--------+-----------------------+----------------------------------+--------------------------------+------------------<br>12      | Sephora Singapore     | contact@sephora.sg               | contact@sephora.sg             | https://www.bloomberg.com/profile/company/0553744D:SP<br>        |                       |                                  |                                | https://sgmark.org/winners/sephora-head-office-ngee-ann-city/<br>        |                       |                                  |                                | https://www.zoominfo.com/p/Farah-Naser/7180745024<br>        |                       |                                  |                                | https://sephorasg.zendesk.com/hc/en-us/articles/18804538674585-I-have-been-contacted-from-a-Sephora-email-is-it-legitimate<br>        |                       |                                  |                                | https://maps.apple.com/place?place-id=I9F07DB4806D2ACA3<br>        |                       |                                  |                                | https://www.emis.com/php/company-profile/SG/Sephora_Asia_Pte_Ltd_en_12860254.html<br>        |                       |                                  |                                | https://www.facebook.com/sephorasingapore/posts/sephora-singapore-now-features-our-stockists-contact-us-details-conveniently-lis/119732541403506/<br>        |                       |                                  |                                | https://www.facebook.com/fb-answers/sephora-singapore-customer-service/?locale=ur_PK<br>        |                       |                                  |                                | https://www.facebook.com/fb-answers/sephora-office-singapore-address/?locale=lt_LT</pre><h3>Part 5: Challenges and further improvements</h3><p>There are several improvements that could be made to the current version of the pipeline:</p><ul><li><strong>Expanding Search Engine Coverage</strong>: The first, as mentioned earlier, was the incorporation of various search engines to gather as many search results as possible, thus increasing the pool of websites that might raise the chance of a successful email search.</li><li><strong>Identifying Multiple High-Confidence Email Candidates</strong>: The second improvement that could be made would be to allow more than one potential email to be identified as high confidence. Using my result as an example, Challenger Singapore has various outlets, but yet the Ang Mo Kio branch’s email (amk@challenger.sg) was the only one identified, skewing the accuracy of email discovery.</li><li><strong>Reducing Reliance on Google’s CAPTCHA-Handling Mechanism</strong>: Thirdly, instead of using Google browser for website discovery, it is best to find alternatives of DDGS that pull search results without paying for an API key, such as <a href="https://pypi.org/project/googlesearch-python/?utm_source=chatgpt.com">googlesearch-python</a> or <a href="https://pypi.org/project/wikipedia/?utm_source=chatgpt.com">wikipedia</a>, to reduce the pipeline’s reliance on Google’s CAPTCHA-handling mechanism, allowing for more development effort to be directed towards improving website discovery and email extraction rather than continually maintaining and adapting the mechanism to Google’s changing anti-bot measures.</li></ul><p>On the other hand, the current implementation of the pipeline and its potential improvements also present several challenges:</p><ul><li><strong>Increased processing time and filtering complexity:</strong> Expanding the search pool would naturally increase the number of websites that need to be processed. This would require additional processing time and more robust filtering mechanisms to maintain the relevance and quality of the extracted email candidates.</li><li><strong>The Ongoing Arms Race Against Anti-Bot Mechanisms:</strong> Websites are continuously improving their anti-bot protection through CAPTCHA, Cloudflare and other detection mechanisms. These measures can interrupt automated searches and prevent the pipeline from accessing otherwise publicly available contact information. As these protections evolve, the pipeline would also require continuous maintenance and adaptation to maintain reliable access.</li><li><strong>False Positives and Irrelevant Email Candidates:</strong> Extracting an email address does not necessarily mean that it belongs to the target business. Websites may contain third-party contact information, unrelated email addresses or even text that is mistakenly identified as an email address. Additional filtering and checks are therefore necessary to improve the relevance of the extracted candidates.</li></ul><h3>Part 6: Concluding statement</h3><p>Building this Python-based email discovery pipeline allowed me to explore how web scraping, automation and data processing can be combined to address a practical data collection problem. Throughout the project, I gained hands-on experience integrating different Python libraries while overcoming challenges involving anti-bot restrictions, inconsistent website structures and data quality.</p><p>Although the pipeline still has room for improvement, it provides a foundation for a more scalable and reliable email discovery solution. More importantly, the project reinforced my understanding that effective automation involves not just collecting data, but also ensuring its relevance, accuracy and reliability.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0469f3876e9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Retail Transaction Analysis in Microsoft Excel]]></title>
            <link>https://medium.com/@DataDreamJO/retail-transaction-analysis-in-microsoft-excel-13df6fd828e4?source=rss-42566753e5c8------2</link>
            <guid isPermaLink="false">https://medium.com/p/13df6fd828e4</guid>
            <category><![CDATA[marketing]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[interactive-content]]></category>
            <category><![CDATA[power-query]]></category>
            <category><![CDATA[excel]]></category>
            <dc:creator><![CDATA[Data_Dream_J.O.]]></dc:creator>
            <pubDate>Sun, 02 Aug 2026 17:13:03 GMT</pubDate>
            <atom:updated>2026-08-03T01:50:05.993Z</atom:updated>
            <content:encoded><![CDATA[<p>Date of creation: Sunday, 29 July 2026, Last updated: Sunday, 3 August 2026</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o45CZvxwM2TMI0Qc46aGCw.png" /></figure><h4><strong>Introduction</strong></h4><p>The <strong>Online Retail</strong> dataset was obtained from the UC Irvine Machine Learning Repository, available at: <a href="https://archive.ics.uci.edu/dataset/352/online+retail.">https://archive.ics.uci.edu/dataset/352/online+retail.</a> According to the repository, the dataset contains transactional records donated by a UK-based online retail company that operates exclusively through e-commerce, without a traditional brick-and-mortar store. The transactions span the period from <strong>1 December 2010 to 9 December 2011</strong>, with the majority of records originating from 2011.</p><blockquote><a href="https://drive.google.com/file/d/1G-c6sNfKWxsdNZzP3mnIvhhp3ay8TiRQ/view?usp=drive_link">Click here to view the Excel Project</a> (It has an interactive dashboard in section 6)</blockquote><p>The primary objective of this analysis is to evaluate the company’s overall business performance and gain deeper insights into its product portfolio, customer base, customer geographic distribution, and sales performance over the one-year period. In addition, an interactive Excel dashboard was developed to enable management to monitor key sales metrics, identify major revenue drivers, analyse customer purchasing behaviour, and support data-driven business decision-making.</p><h4><strong>Data Cleaning and Processing</strong></h4><p><strong>Originally, the data contains 8 columns:</strong></p><ul><li>InvoiceNo (categorical): a 6-digit integral number uniquely assigned to each transaction. If this code starts with letter ‘c’, it indicates a cancellation</li><li>StockCode (categorical): a 5-digit integral number uniquely assigned to each distinct product</li><li>Description (categorical): product name</li><li>Quantity (Integer): the quantities of each product (item) per transaction</li><li>InvoiceDate (Date): the day and time when each transaction was generated</li><li>UnitPrice (Continuous): product price per unit</li><li>CustomerID (Categorical): a 5-digit integral number uniquely assigned to each customer</li><li>Country (Categorical): the name of the country where each customer resides</li></ul><p><strong>Power Query is used to do the following data cleaning and transformation steps:</strong></p><p><strong>Column Renaming:</strong></p><ul><li>InvoiceNo was renamed to TransactionID</li><li>StockCode was renamed to ProductID</li><li>Description was renamed to ProductName</li><li>Country was renamed to CustomerResidenceCountry</li></ul><p><strong>New Columns Created:</strong></p><ul><li>Revenue: Calculated as UnitPrice x Quantity</li><li>Cancelled: Created to identify cancelled transactions using InvoiceNo as the reference.</li><li>Year, Quarter, and Month: Extracted from the InvoiceDate column.</li></ul><p><strong>Column Transformations:</strong></p><ul><li>Unique identifier fields (TransactionID, ProductID, and CustomerID) were converted to Text data type.</li><li>The value “EIRE” in the CustomerResidenceCountry column was standardised to “Ireland”.</li></ul><p><strong>Data Cleaning:</strong></p><ul><li>Inconsistent values in the ProductName column were cleaned by grouping {ProductID, ProductName}, count the number of occurences for each combination, remove duplicated sets, resulting in 3,958 unique ProductID- ProductName pairs, which were used as a reference table for XLOOKUP to populate ProductName Column.</li><li>Duplicate records were identified and removed</li><li>Approximately 1% of transactions contained missing ProductName values while approximately 25% contained missing CustomerID values. Missing CustomerID values were retained as they do not affect the overall sales, product, or geographic analyses conducted in this study.</li></ul><p>Following the completion of data cleaning, the dataset contained 41,981 transaction records for 2010 and 494,660 transaction records for 2011. The final dataset consists of the following fields: TransactionID, ProductID, ProductName, CustomerID, InvoiceDate, Year, Quarter, Month, CustomerResidenceCoiuntry, UnitPrice, Quantity, Revenue, Cancelled.</p><p><strong>Several Helper Columns </strong>were created to aid pivot table construction:</p><ul><li>UniqueFlag CustomerID per Month: This column identifies the first occurence of each customer within a given month, ensuring that each customer is counted only once in monthly customer metrics. This was achieved by group records by {CustomerID, Year, Month}, sorting each group by InvoiceDate, assigning an index starting from 1, and then expanding the grouped tables.</li><li>TransactionCount: Assigned a value of 1 for every transaction. This is used to calculate the total number of transactions for the Average Revenue per Transaction, and Cancellation Rate.</li><li>CancellationBoolean: Converts the Cancelled field into a numeric (0/1) indicator to enable the calculation of the Cancellation Rate.</li></ul><h4>A glimpse into the cleaned Online Retail Dataset…</h4><pre><br>//===============[]===========[]=====================================[]============[]==========================[]=============[]======[]=======[]=========[]===========[]==========[]=========[]===========[]=================================[]==================[]=====================\\<br>|| TransactionID || ProductID ||             ProductName             || CustomerID || CustomerResidenceCountry || InvoiceDate || Year || Month || Quarter || UnitPrice || Quantity || Revenue || Cancelled || UniqueFlag CustomerID per month || TransactionCount || CancellationBoolean ||<br>|]===============[]===========[]=====================================[]============[]==========================[]=============[]======[]=======[]=========[]===========[]==========[]=========[]===========[]=================================[]==================[]=====================[|<br>||        576580 || 22034     || ROBIN CHRISTMAS CARD                ||            || United Kingdom           || 15/11/2011  || 2011 || Nov   ||       4 ||         0 ||       -5 ||       0 || FALSE     ||                               0 ||                1 ||                   0 ||<br>||        574904 || 85017B    || ENVELOPE 50 BLOSSOM IMAGES          ||            || United Kingdom           || 07/11/2011  || 2011 || Nov   ||       4 ||         0 ||       11 ||       0 || FALSE     ||                               0 ||                1 ||                   0 ||<br>||        574905 || 85017B    || ENVELOPE 50 BLOSSOM IMAGES          ||            || United Kingdom           || 07/11/2011  || 2011 || Nov   ||       4 ||         0 ||      -22 ||       0 || FALSE     ||                               0 ||                1 ||                   0 ||<br>||        577972 || 84663A    || GRASS HOPPER WOODEN WALL CLOCK      ||            || United Kingdom           || 22/11/2011  || 2011 || Nov   ||       4 ||         0 ||        1 ||       0 || FALSE     ||                               0 ||                1 ||                   0 ||<br>||        577971 || 22606     || check                               ||            || United Kingdom           || 22/11/2011  || 2011 || Nov   ||       4 ||         0 ||        1 ||       0 || FALSE     ||                               0 ||                1 ||                   0 ||<br>||        577970 || 85135A    || YELLOW DRAGONFLY HELICOPTER         ||            || United Kingdom           || 22/11/2011  || 2011 || Nov   ||       4 ||         0 ||        4 ||       0 || FALSE     ||                               0 ||                1 ||                   0 ||<br>||        577968 || 85135B    || BLUE DRAGONFLY HELICOPTER           ||            || United Kingdom           || 22/11/2011  || 2011 || Nov   ||       4 ||         0 ||        2 ||       0 || FALSE     ||                               0 ||                1 ||                   0 ||<br>||        577966 || 85045     || GREEN CHRISTMAS TREE STRING 20LIGHT ||            || United Kingdom           || 22/11/2011  || 2011 || Nov   ||       4 ||         0 ||        2 ||       0 || FALSE     ||                               0 ||                1 ||                   0 ||<br>||        577964 || 37489A    || YELLOW/PINK FLOWER DESIGN BIG MUG   ||            || United Kingdom           || 22/11/2011  || 2011 || Nov   ||       4 ||         0 ||        2 ||       0 || FALSE     ||                               0 ||                1 ||                   0 ||<br>||        577961 || 23268     || SET OF 2 CERAMIC CHRISTMAS REINDEER ||            || United Kingdom           || 22/11/2011  || 2011 || Nov   ||       4 ||         0 ||        2 ||       0 || FALSE     ||                               0 ||                1 ||                   0 ||<br>||        577956 || 21145     || ANTIQUE GLASS PLACE SETTING         ||            || United Kingdom           || 22/11/2011  || 2011 || Nov   ||       4 ||         0 ||        3 ||       0 || FALSE     ||                               0 ||                1 ||                   0 ||<br>||        576594 || 21394     || RED POLKADOT BEAKER                 ||            || United Kingdom           || 15/11/2011  || 2011 || Nov   ||       4 ||         0 ||        1 ||       0 || FALSE     ||                               0 ||                1 ||                   0 ||<br>||        576569 || 47504K    || ENGLISH ROSE GARDEN SECATEURS       ||            || United Kingdom           || 15/11/2011  || 2011 || Nov   ||       4 ||         0 ||      -21 ||       0 || FALSE     ||                               0 ||                1 ||                   0 ||<br>||        576566 || 21830     || ASSORTED CREEPY CRAWLIES            ||            || United Kingdom           || 15/11/2011  || 2011 || Nov   ||       4 ||         0 ||     -186 ||       0 || FALSE     ||                               0 ||                1 ||                   0 ||<br>||        576578 || 22412     || METAL SIGN NEIGHBOURHOOD WITCH      ||            || United Kingdom           || 15/11/2011  || 2011 || Nov   ||       4 ||         0 ||      -37 ||       0 || FALSE     ||                               0 ||                1 ||                   0 ||<br>||        576585 || 84247E    || PACK OF 12 CHRISTMAS FUN CARDS      ||            || United Kingdom           || 15/11/2011  || 2011 || Nov   ||       4 ||         0 ||       22 ||       0 || FALSE     ||                               0 ||                1 ||                   0 ||<br>\\===============[]===========[]=====================================[]============[]==========================[]=============[]======[]=======[]=========[]===========[]==========[]=========[]===========[]=================================[]==================[]=====================//</pre><h4><strong>Section 1: The Overall Business Performance</strong></h4><p><strong>Purpose:</strong> The objective of this Section is to evaluate the overall business performance of the online retail company by summarising key monthly performance indicators. These metrics provide insights into the overall sales activity, customer engagement, and operational performance prior to further detailed analysis of sales trends, products, customers, and geographical markets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SF5O-quOSpVQGhs8sDzhNw.png" /><figcaption>Figure 1. Monthly Outlook for the business performance over the span of at least 1 year between December 2010 and December 2011, Number of distinct customers do not account for null CustomerID values. All figures are presented in GBP (£), Average Revenue per Customer is not calculated as 25% of transactions have missing CustomerID values.</figcaption></figure><p><strong>Observations:</strong></p><ul><li>The monthly business performance varied throughout the analysis period with noticeable fluctuations.</li><li>The <strong>highest revenue </strong>recorded is found in the <strong>fourth quarter 2011</strong>.</li><li>The <strong>best performing month</strong> is in November 2011, with the highest revenue (~1.4 million), quantity sold (~700k), transactions (~83k), and unique customers (~1.7k). The months, from September 2011 to November 2011, are some of the <strong>best consecutive performing months</strong> in terms of total revenue, quantity sold, number of customers and transactions.</li><li>The <strong>worst performing month</strong> is in December 2011 as it has the lowest revenue (~400k), quantity sold (~200k), transactions (~25k), and unique customers (686). However, this should be interpreted with caution as the data for December 2011 data only spans till the 9th and therefore does not represent a complete month. If this partial month was excluded, the weakest full-month performance was observed in either April or February 2011.</li><li>The <strong>average revenue per transaction</strong> reached its <strong>highest level</strong> in September 2011 (20.41 GBP/ transaction).</li><li>The <strong>average revenue per transaction is the lowest</strong> in January 2011 (16 GBP/ transaction).</li><li>The <strong>cancellation rates</strong> are the lowest between October 2011 and November 2011.</li></ul><p><strong>Overall Findings: </strong>The <strong>strongest period of performance occurred during the forth quarter of 2011</strong>, particularly from <strong>September to November 2011</strong>, where the business consistently recorded <strong>high revenue, sales volume, transaction counts, and customer activity</strong>. It also experienced <strong>the lowest cancellation rates </strong>when compared to the other full quarters (meaning not including December 2010). Among these months, <strong>November 2011 was the best-performing month</strong>, generating approximately £1.4 million in revenue, with around 700,000 units sold, 83,000 transactions, and 1,700 unique customers. <strong>The average revenue per transaction peaked in September </strong>at around £20.41 per transaction, <strong>suggesting customers placed higher-value orders during that month</strong>. The <strong>weakest period of performance occurred during the first quarter of 2011, particularly from January to February 2011</strong>. <strong>December 2011 recorded the lowest revenue, quantity sold, transaction counts, and customer activity.</strong></p><h4><strong>Section 2: Time Analysis</strong></h4><p>Below are some graphs that further illustrate the summarised findings from Section 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cgCXGkL4-oOgp60xYffu8w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ey2dx0oNhccaSU0Ln0QGIw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f_Z8y99rqFHquWmeqrfcTA.png" /></figure><h4><strong>Section 3: Product Analysis</strong></h4><p><strong>Observations on the top 10 products: </strong>The analysis identified the top 10 products based on total revenue and sales quantity. Product DOT generated the highest revenue at around £206 K, followed by 22423 and 85123A. In terms of sales volume, product 22197 recorded the highest quantity sold (sales volume) with around 56 K units, followed by 84077 and 85099B. The results show that the highest-revenue products are not always the highest-selling products by quantity.</p><blockquote>*ProductIDs are referred to here as they are a more accurate representation of the individual products compared to the actual product names.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zBtcoUloM-YyN6Cq4Jt9AA.png" /></figure><p><strong>Observations on the bottom 10 products: </strong>The bottom 10 Products with the greatest negative total revenue and sales volume might suggest various circumstances. <strong>Negative quantity might indicate that more units were returned than sold </strong>during the reporting period and <strong>negative revenue might imply that more money was refunded or credited back to the customers</strong>. These products might <strong>require further investigation </strong>to determine the underlying causes of the negative sales attributed to the negative quantity sold.</p><blockquote>* Negative quantity values might represent returned or cancelled items. These transactions reverse previous sales by recording the movement of products back into inventory, resulting in negative quantity, and hence generating the corresponding negative revenue.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qv6tSH0NInVNFuPvfZ_QqA.png" /></figure><p><strong>Pareto Analysis on Overall Top 10 Products by Revenue:</strong> The 80/20 Pareto analysis was performed on the top 10 revenue-generating products to examine their relative contribution within this group. Among the top 10 products with product IDs <strong>DOT, 22423, 85123A, 47566, 85099B, 23084, AND POST</strong> account for approximately 80% of the revenue generated by the top 10 products. However, it is paramount to note that these products only <strong>represent approximately 8.16% of the company’s total revenue. </strong>The top 10 products by revenue accounts for <strong>around 10% of the company’s total revenue</strong>.</p><h4><strong>Section 4: Customer Analysis</strong></h4><p><strong>Observations on the Top 10 Customers: </strong>There is a considerable overlap between the two rankings, where 9 out of the top 10 Customers appear in both the Top 10 by Revenue and Top 10 by Quantity lists. This suggests that the company&#39;s highest-value customers generally create revenue by purchasing large quantities of products rather than purchasing a few high-priced items.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M-Hlf1eGxXUUgnd_4Zva0A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WAUw47pmoNJYDlVS8ZoFDg.png" /></figure><p><strong>Pareto Analysis on the Top 10 Customers by Revenue: </strong>The Pareto Analysis shows the top 10 Customers by Revenue. The results show that the <strong>Top 10 customers by revenue</strong> contributed approximately <strong>14.07%</strong> of the company’s total revenue over the analysis period (December 2010 to December 2011).</p><p>Among these Top 10 customers, <strong>CustomerIDs 14646, 18102, 17450, 14911, 12415, 14156, and 17511</strong> accounted for approximately <strong>86.27%</strong> of the Top 10 customers’ combined revenue, representing around <strong>12.14%</strong> of the company’s total revenue. This indicates that while revenue among the Top 10 customers is concentrated within a relatively small number of key accounts, the business as a whole is not overly dependent on a handful of customers, as the Top 10 contribute only a modest proportion of total revenue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r5WPT3uoAT2uTs14KIpZbw.png" /><figcaption>Bullet Graph of Active Customers Over Time</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5lZLDYGnjaXgFPmvAJU7PA.png" /><figcaption>Time Series of Monthly New and Returning Customers</figcaption></figure><p><strong>Further Observations on Monthly New and Returning Customers:</strong> December 2010 had 948 new customers, the highest in the period between December 2010 and December 2011. However, after the initial month, new customers dropped from 421 in January 2011 to 167 in August 2011. There were small recoveries in March, September, and October 2011 but the overall trend was still downward. This suggests customer acquisition slowed over time. On the other hand, repeated customers grew from 362 in January 2011 390 in November 2011. This indicates that an increasing proportion of monthly sales came from existing customers rather than first-time buyers. But, in December 2011, only 41 new and 646 repeated customers were recorded. This sharp decline is likely because the dataset contains only up to 9th December 2011 data rather than the full month, so it should be interpreted with caution.</p><p><strong>Section 5: Geographical Analysis</strong></p><p><strong>Observations on Top 10 Countries by Revenue:</strong> The United Kingdom (UK) is the dominant market, generating around £8.17 million in revenue and 4.25 milkion units in sales volume. This is far ahead of every other country, indicating that the retail business is highly dependent on UK market. Moreover, revenue and sales volume rankings are generally consistent, where countries with higher sales volumes tend to generate higher revenues, suggesting a potential positive relationship between sales volume and revenue. Thirdly, Switzerland generates more revenue (~£56.4k) than Sweden (£36.6k) despite having lower sales volume (30.3k vs 35.6k). Lastly, Japan appears in the top 10 for sales volume (~25.2k) but not for revenue, implying lower revenue per unit compared with countries such as Belgium.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zKsN07HyizcaAWAGXgFWww.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BJ0UH-9p9jEuQQpATha6rg.png" /><figcaption>World Map: Distribution of Revenue by Customer Residence Countries</figcaption></figure><h4><strong>Conclusion</strong></h4><p>Overall, the analysis showed that the business performed strongest during the fourth quarter of 2011, with November as the best performing month based on various metrics. The study also identified the top-performing products, key customers, and the United Kingdom as the company’s primary market. These findings provide a clear overview of the company’s sales performance and customer purchasing patterns over the analysis period.</p><h4><strong>Citation</strong></h4><ul><li>Chen, D. (2015). Online Retail [Dataset]. UCI Machine Learning Repository. <a href="https://doi.org/10.24432/C5BW33">https://doi.org/10.24432/C5BW33</a>.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=13df6fd828e4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I built a simple Python function to automatically clean categorical data]]></title>
            <link>https://medium.com/@DataDreamJO/i-built-a-simple-python-function-to-automatically-clean-categorical-data-6f7f8ad174aa?source=rss-42566753e5c8------2</link>
            <guid isPermaLink="false">https://medium.com/p/6f7f8ad174aa</guid>
            <category><![CDATA[data-cleaning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[categorical-data]]></category>
            <category><![CDATA[kneelocator]]></category>
            <category><![CDATA[matplotlib]]></category>
            <dc:creator><![CDATA[Data_Dream_J.O.]]></dc:creator>
            <pubDate>Fri, 17 Jul 2026 16:15:20 GMT</pubDate>
            <atom:updated>2026-07-18T12:01:05.027Z</atom:updated>
            <content:encoded><![CDATA[<p>Date of creation: Friday, 17 July 2026; Last updated: Saturday, 18 July 2026</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gbVK87uACHM67XA8AUFrBA.png" /><figcaption>Figure 1. Cumulative frequency curve illustrating the threshold selected for grouping infrequently occurring categories into <strong>“Other”</strong>.</figcaption></figure><p><strong>Introduction</strong></p><p>A few semesters ago, I learned about this method during a collaborative class project. Although I didn’t come up with the idea and cannot claim credit for the underlying method, I reimplemented it from scratch to better understand its technique for cleaning categorical data by grouping infrequent values into an “Other” category. This post shares my implementation and explains how the method works.</p><p><strong>Problem statement</strong></p><p>Data cleaning is the absolute foundation of successful analytics. However, certain data types, especially those that are categorical in nature, might be difficult to clean when there are tons of records to investigate. Personally, I had experienced difficulty in cleaning categorical data when the words are actually supposed to be the same, but they come in different forms, like they can either be capitalised, wrongly spelt, lowercased, etc. So, rectifying such issues tends to be extremely time-consuming. This is problematic when my goal is to provide quick insights into the data rather than spending hours on cleaning and perfecting it.</p><p><strong>How the algorithm works?</strong></p><p>The algorithm works by leveraging the percentage cumulative frequency and KneeLocator function (from the Python library <a href="https://kneed.readthedocs.io/en/stable/">kneed</a>). This is the process:</p><ol><li>The function accepts a DataFrame and a categorical column as inputs. It first calculates the frequency of each category, generates the cumulative frequency distribution, and converts them into cumulative percentages.</li><li>The kneeLocator() takes in the following arguments: category indices as the x-axis, the percentage cumulative frequencies as the y-axis, curve = “concave”, and direction = “increasing”.</li><li>kneeLocator(…).knee returns the index of the detected knee point on the cumulative frequency curve, which is then used to generate the corresponding frequency threshold. Moreover, the generated curve visualises the detected threshold to make the preprocessing step more transparent.</li><li>Categories with frequencies below the detected threshold are renamed as “Other”, reducing the impact of infrequent categories while preserving important information.</li></ol><p><strong>Example of how the algorithm works</strong></p><p>To illustrate how the algorithm works, I used a housing dataset and applied it to clean the flat_type column.</p><p><strong>Before cleaning</strong>, the flat_type column contains several categories with low frequencies. After the algorithm is applied, categories with frequencies below a specified threshold are grouped into a single category named <strong>“Other”</strong>. In this example, the threshold is set to 599. This means any category in flat_type with fewer than 599 occurences is renamed to “<strong>Other</strong>”. The figures below show the flat_type column <strong>before</strong> and <strong>after</strong> the cleaning process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/888/1*D5XrMf5qhrv115Gevcq4Bg.png" /><figcaption><strong>Figure 2.</strong> Distribution of categories in the flat_type column before applying the cleaning algorithm.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*UKkf31Jf6K_GH3GUb2nQdg.png" /><figcaption><strong>Figure 3.</strong> Distribution of categories in the flat_type column after applying the cleaning algorithm (categories with fewer than 599 occurrences are grouped into <strong>&quot;Other&quot;</strong>).</figcaption></figure><p><strong>The algorithm</strong></p><pre>import pandas as pd <br>import matplotlib.pyplot as plt<br>from kneed import KneeLocator<br><br># Import data<br>df = pd.read_csv(&quot;...&quot;)<br><br># Function to clean column data<br><br>def clean_column(df, col):<br>    print(f&quot;Cleaning {col} column&quot;)<br>    ## Data<br>    new_df = df.copy()<br>    print(&quot;\n&quot; + &quot;=&quot; * 60)<br>    print(&quot;\033[1m&quot;+&quot;Original Data Shape&quot;+&quot;\033[0m&quot;)<br>    print(&quot;=&quot; * 60 + &quot;\n&quot;)<br>    print(df.shape)<br><br>    # Create cumulative frequency in percentage to create curve<br>    t = new_df[col].value_counts().reset_index() <br>    t[&quot;cumfreq&quot;] = t[&quot;count&quot;].cumsum() <br>    t[&quot;cumfreq&quot;] = t[&quot;cumfreq&quot;]*100/t[&quot;cumfreq&quot;].iloc[-1]<br>    print(t)<br><br>    ## Knee locator<br>    x = t.index<br>    y = t[&quot;cumfreq&quot;]<br><br>    knee = KneeLocator(<br>        x, y, curve = &quot;concave&quot;, direction = &quot;increasing&quot;<br>    )<br>    # This edge case overcomes the issue where knee.knee returns None<br><br>    if knee.knee is None:<br>        print(&quot;No clear threshold detected, data seems to be fine&quot;)<br>        return <br><br>    threshold = t.loc[knee.knee, &quot;count&quot;] # Create threshold<br><br>    ## Show threshold line <br>    print(&quot;\n&quot; + &quot;=&quot; * 60)<br>    print(&quot;\033[1m&quot;+f&quot;This is the cumulative frequency curve with the detected threshold, {threshold}&quot;+&quot;\033[0m&quot;)<br>    print(&quot;=&quot; * 60 + &quot;\n&quot;)<br><br>    fig, ax = plt.subplots()<br>    ax.plot(x, y, marker = &quot;o&quot;)<br>    ax.axvline(knee.knee, ls = &quot;--&quot;, color = &quot;red&quot;)<br>    plt.show()<br><br>    ## Remove rows that are insignificant<br><br>    rare = t.loc[t[&quot;count&quot;] &lt; threshold, col].to_list()<br>    df.loc[df[col].isin(rare), col] = &quot;Other&quot;<br><br>    ## Current data<br>    print(&quot;\n&quot; + &quot;=&quot; * 60)<br>    print(&quot;\033[1m&quot;+&quot;Data Shape and Category Distribution After Cleaning&quot;+&quot;\033[0m&quot;)<br>    print(&quot;=&quot; * 60 + &quot;\n&quot;)<br>    <br>    print(df.shape)<br>    print(df[col].value_counts())<br><br>    # sample table <br>    print(&quot;\n&quot; + &quot;=&quot; * 60)<br>    print(&quot;\033[1m&quot;+&quot;Sample data After Cleaning&quot;+&quot;\033[0m&quot;)<br>    print(&quot;=&quot; * 60 + &quot;\n&quot;)<br>    <br>    print(df.head())<br><br># An example of its implementation    <br>clean_column(df, &quot;flat_type&quot;)</pre><p><strong>Limitations and conclusion</strong></p><p>This approach relies solely on category frequency and does not account for slight writing variations in category names. Hence, inconsistent labels (e.g., Jake, jake, and Jakk) are treated as separate categories, even though they refer to the same entity. That means that if the category frequencies are relatively uniform, KneeLocator may not identify a meaningful threshold, and such inconsistencies would require additional preprocessing techniques to rectify.</p><p>Overall, I believe this simple algorithm can significantly reduce the time spent cleaning categorical data. Although this is a relatively simple implementation, I hope others will find it useful, build upon it, and find new approaches to data preprocessing.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6f7f8ad174aa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Manual to Macro: How I automated Excel Summing with VBA]]></title>
            <link>https://medium.com/@DataDreamJO/from-manual-to-macro-how-i-automated-excel-summing-with-vba-ca62311f4715?source=rss-42566753e5c8------2</link>
            <guid isPermaLink="false">https://medium.com/p/ca62311f4715</guid>
            <category><![CDATA[vba]]></category>
            <category><![CDATA[excel]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[vba-macros]]></category>
            <dc:creator><![CDATA[Data_Dream_J.O.]]></dc:creator>
            <pubDate>Wed, 30 Jul 2025 10:07:55 GMT</pubDate>
            <atom:updated>2025-08-01T06:11:23.666Z</atom:updated>
            <content:encoded><![CDATA[<p>Date of creation: Wednesday, 30 July 2025; Last updated: Friday, 1 August 2025</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*4_hvHMFZmqHJU5Gb.jpg" /></figure><p><strong>Introduction</strong></p><p>Many of us use the function =SUM() to add columns of data together. However, this task while might seem simple, can become repetitive and prone to errors, especially when dealing with larger datasets. To simplify this process, I have created a VBA macro that enables users to add multiple columns of data. Users can specify the starting row and the column where the result will be shown. Moreover, the program validates user input, skips rows with missing values, and halts execution if too many empty rows are detected.</p><p><strong>How does the Macro work?</strong></p><p>The macro automates the process of summing values across rows in Excel</p><ul><li><strong>Select columns to sum:</strong> It prompts the user to enter the column letter that should be included in the summation (e.g., A, B)</li></ul><pre>&#39;Indicate which boxes are to be added together and which are not supposed to be added<br>    Do While True<br>        Do<br>            answer = InputBox(&quot;Input the column&#39;s alphabetical value&quot;)<br>            answer = UCase(answer)<br>            If answer = &quot;&quot; Then Exit Sub<br>        Loop Until answer Like &quot;[A-Za-z]&quot;<br>        <br>        ReDim Preserve arr(array_size) &#39;Create array with specified size<br>        arr(array_size) = answer &#39;Insert value into array<br>        array_size = array_size + 1 &#39;Add 1 to recreate a new array size<br>        <br>        options = MsgBox(&quot;Anymore rows?&quot;, vbYesNoCancel) &#39;Check if the user needs to add more columns together<br>        If options = vbNo Then<br>            Exit Do<br>        ElseIf options = vbCancel Then<br>            Exit Sub<br>        End If<br>    Loop</pre><ul><li><strong>Select the result column:</strong> The user is then asked to specify the column where the total sum should be displayed (e.g., C)</li></ul><pre>&#39;Indicate which column is used to add in the results<br>    Do<br>        resultbox = InputBox(&quot;Specify the column&#39;s alphabetical value where the results will be stored&quot;)<br>        If resultbox = &quot;&quot; Then Exit Sub<br>    Loop Until resultbox Like &quot;[A-Za-z]&quot;</pre><ul><li><strong>Select the starting row: </strong>It allows the user to enter the starting row number where the summation should begin (e.g., row 2)</li></ul><pre>    Do<br>        rowbox = InputBox(&quot;Specify the starting row&quot;)<br>        If rowbox = &quot;&quot; Then Exit Sub<br>        i = rowbox<br>    Loop Until i Like &quot;[1-9]*&quot;</pre><ul><li><strong>Looping and Exit:</strong> The macro loops through each row, adds the specified column values for each row, and placed the total in the result column.</li></ul><pre>  Do While True<br>    <br>        Dim cancontinue As Boolean<br>        cancontinue = True<br>        <br>        For j = 0 To UBound(arr)<br>            If IsEmpty(Range(arr(j) &amp; i).Value) Then<br>                stopd = stopd + 1<br>                cancontinue = False<br>                Exit For<br>            End If<br>        Next j<br>        <br>        If cancontinue Then<br>            Dim total As Integer<br>            total = 0<br>            For j = 0 To UBound(arr)<br>                total = Range(arr(j) &amp; i).Value + total &#39; Adding values together<br>            Next j<br>            Range(resultbox &amp; i).Value = total<br>            stopd = 0<br>        End If<br>        <br>        i = i + 1 &#39; Loop counter for rows<br>        <br>        If stopd = 10 Then Exit Do &#39; After 10 empty cells are detected, the program will stop running&quot;<br>    Loop</pre><ul><li>It automatically stops after 10 consecutive empty rows are detected.</li></ul><pre>If stopd = 10 Then Exit Do &#39; After 10 empty cells are detected, the program will stop running&quot;</pre><p><strong>The Full VBA Code — Version 2</strong></p><pre>Sub add_values_ver2()<br>    &#39;Data_Dream_J.O.<br>    &#39;30 July 2025<br>    MsgBox &quot;This program aims to add a series of column values together&quot;<br>    <br>    &#39; Declarations<br>    Dim i As Variant, j As Integer, stopd As Integer, answer As Variant, array_size As Integer<br>    Dim arr() As Variant<br>    Dim options As VbMsgBoxResult<br>    Dim resultbox As String, rowbox As Variant<br>    <br>    &#39; Variables<br>    i = 2 &#39; Counter for rows<br>    stopd = 0 &#39; Counter for detecting empty cells<br>    array_size = 0<br>        <br>    &#39;Indicate which boxes are to be added together and which are not supposed to be added<br>    Do While True<br>        Do<br>            answer = InputBox(&quot;Input the column&#39;s alphabetical value&quot;)<br>            answer = UCase(answer)<br>            If answer = &quot;&quot; Then Exit Sub<br>        Loop Until answer Like &quot;[A-Za-z]&quot;<br>        <br>        ReDim Preserve arr(array_size) &#39;Create array with specified size<br>        arr(array_size) = answer &#39;Insert value into array<br>        array_size = array_size + 1 &#39;Add 1 to recreate a new array size<br>        <br>        options = MsgBox(&quot;Anymore rows?&quot;, vbYesNoCancel) &#39;Check if the user needs to add more columns together<br>        If options = vbNo Then<br>            Exit Do<br>        ElseIf options = vbCancel Then<br>            Exit Sub<br>        End If<br>    Loop<br>    <br>    &#39;Indicate which column is used to add in the results<br>    Do<br>        resultbox = InputBox(&quot;Specify the column&#39;s alphabetical value where results will be stored&quot;)<br>        If resultbox = &quot;&quot; Then Exit Sub<br>    Loop Until resultbox Like &quot;[A-Za-z]&quot;<br>    <br>    &#39;Indicate which is the starting row used<br>    Do<br>        rowbox = InputBox(&quot;Specify the starting row&quot;)<br>        If rowbox = &quot;&quot; Then Exit Sub<br>        i = rowbox<br>    Loop Until i Like &quot;[1-9]*&quot;<br>    <br>    &#39; The code basically adds two values together, and displays the value in the cells.<br>    Do While True<br>    <br>        Dim cancontinue As Boolean<br>        cancontinue = True<br>        <br>        For j = 0 To UBound(arr)<br>            If IsEmpty(Range(arr(j) &amp; i).Value) Then<br>                stopd = stopd + 1<br>                cancontinue = False<br>                Exit For<br>            End If<br>        Next j<br>        <br>        If cancontinue Then<br>            Dim total As Integer<br>            total = 0<br>            For j = 0 To UBound(arr)<br>                total = Range(arr(j) &amp; i).Value + total &#39; Adding values together<br>            Next j<br>            Range(resultbox &amp; i).Value = total<br>            stopd = 0<br>        End If<br>        <br>        i = i + 1 &#39; Loop counter for rows<br>        <br>        If stopd = 10 Then Exit Do &#39; After 10 empty cells are detected, the program will stop running&quot;<br>    Loop<br>    MsgBox &quot;Event completed&quot; &#39; Indicates the completion of the program<br>End Sub</pre><p><strong>An Overview of how the macro works</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*1vl1dSLRiAi95dbz_-sJug.gif" /></figure><p><strong>Limitations and possible improvements</strong></p><p>The macro works well for summing values across multiple fixed columns and rows. However, there are areas where it could be improved for more flexibility.</p><p>The most prominent limitation of this program is its <strong>inability to handle flexible row selection</strong>. The current macro requires the user to input <strong>a single desired starting row for summation</strong>.</p><p>For instance, if Column A has data from row 2, Column B starts from row 5, and Column C starts from row 3, the macro would assume that the summation all begins at row 2. As a consequence, <strong>rows with blank data will be ignored, and issues of incorrect calculations will occur.</strong></p><p>However, as most Excel data is structured in a tabular format, the <strong>need for dynamic row selection is often minimal.</strong> Additionally, introducing column-specific dynamic row selection could increase the <strong>risk of user error</strong>. For instance, dealing with multiple columns could cause the user to input the incorrect start and end row, leading to inaccurate calculations. Therefore, the <strong>trade-off between added flexibility and overall reliability would need to be carefully considered</strong> and possibly <strong>implemented with appropriate safeguards.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ca62311f4715" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Comparative Evaluation of KNN, Decision Tree, and Logistic Regression for Heart Disease…]]></title>
            <link>https://medium.com/@DataDreamJO/a-comparative-evaluation-of-knn-decision-tree-and-logistic-regression-for-heart-disease-375ae346c0c0?source=rss-42566753e5c8------2</link>
            <guid isPermaLink="false">https://medium.com/p/375ae346c0c0</guid>
            <category><![CDATA[r-programming-assignment]]></category>
            <category><![CDATA[healthcare]]></category>
            <category><![CDATA[logistic-regression-model]]></category>
            <category><![CDATA[knn]]></category>
            <category><![CDATA[decision-tree-classifier]]></category>
            <dc:creator><![CDATA[Data_Dream_J.O.]]></dc:creator>
            <pubDate>Thu, 10 Apr 2025 16:41:21 GMT</pubDate>
            <atom:updated>2025-04-11T03:43:51.105Z</atom:updated>
            <content:encoded><![CDATA[<h3>A Comparative Evaluation of KNN, Decision Tree, and Logistic Regression for Heart Disease Prediction</h3><p>Date of creation: Thu, 10 April 2025; Last updated: Thu, 10 April 2025</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/0*asRgEMFP93ObDh6F" /></figure><h3><strong>Introduction</strong></h3><p>According to the World Health Organisation (WHO), cardiovascular diseases (CVD) are the leading cause of death worldwide, claiming approximately 17.9 million lives each year. Cardiovascular disease encompasses conditions affecting the heart and blood vessels. Heart disease refers to any condition that impairs the heart’s function, including congenital defects, cardiac muscle diseases, blocked arteries, malfunctioning valves, and abnormal rhythms. Thus, the goal of this report is to analyse the dataset and identify the best predictive model for determining heart disease status using R programming.</p><h3><strong>Dataset</strong></h3><p>The file “heart-disease-dsa.csv” is a modified version of data from the UC Irvine Machine Learning Repository. It contains 300 rows of patient records, 12 features, and a categorical response variable. The modal category, “category 2” (fixed defect), was used to replace the two missing values in the “<strong>blood.disorder</strong>” column because there are a few missing values overall, and the number of patients in each category varies significantly.</p><pre>age sex chest.pain  bp chol fbs rest.ecg heart.rate angina st.depression vessels blood.disorder disease disease.numeric<br>1   63   1          3 145  233   1        0        150      0           2.3       0              1      no               0<br>2   37   1          2 130  250   0        1        187      0           3.5       0              2      no               0<br>3   41   0          1 130  204   0        0        172      0           1.4       0              2      no               0<br>4   56   1          1 120  236   0        1        178      0           0.8       0              2      no               0<br>5   57   0          0 120  354   0        1        163      1           0.6       0              2      no               0<br>6   57   1          0 140  192   0        1        148      0           0.4       0              1      no               0<br>7   56   0          1 140  294   0        0        153      0           1.3       0              2      no               0<br>8   44   1          1 120  263   0        1        173      0           0.0       0              3      no               0<br>9   52   1          2 172  199   1        1        162      0           0.5       0              3      no               0<br>10  57   1          2 150  168   0        1        174      0           1.6       0              2      no               0<br>11  54   1          0 140  239   0        1        160      0           1.2       0              2      no               0<br>12  48   0          2 130  275   0        1        139      0           0.2       0              2      no               0<br>13  49   1          1 130  266   0        1        171      0           0.6       0              2      no               0<br>14  64   1          3 110  211   0        0        144      1           1.8       0              2      no               0<br>15  58   0          3 150  283   1        0        162      0           1.0       0              2      no               0<br>16  50   0          2 120  219   0        1        158      0           1.6       0              2      no               0<br>17  66   0          3 150  226   0        1        114      0           2.6       0              2      no               0<br>18  43   1          0 150  247   0        1        171      0           1.5       0              2      no               0<br>19  69   0          3 140  239   0        1        151      0           1.8       2              2      no               0<br>20  59   1          0 135  234   0        1        161      0           0.5       0              3      no               0<br>21  44   1          2 130  233   0        1        179      1           0.4       0              2      no               0<br>22  42   1          0 140  226   0        1        178      0           0.0       0              2      no               0<br>23  61   1          2 150  243   1        1        137      1           1.0       0              2      no               0<br>24  40   1          3 140  199   0        1        178      1           1.4       0              3      no               0<br>25  71   0          1 160  302   0        1        162      0           0.4       2              2      no               0<br>26  59   1          2 150  212   1        1        157      0           1.6       0              2      no               0<br>27  51   1          2 110  175   0        1        123      0           0.6       0              2      no               0<br>28  65   0          2 140  417   1        0        157      0           0.8       1              2      no               0<br>29  53   1          2 130  197   1        0        152      0           1.2       0              2      no               0<br>30  41   0          1 105  198   0        1        168      0           0.0       1              2      no               0<br>31  65   1          0 120  177   0        1        140      0           0.4       0              3      no               0<br>32  44   1          1 130  219   0        0        188      0           0.0       0              2      no               0<br>33  54   1          2 125  273   0        0        152      0           0.5       1              2      no               0<br>34  51   1          3 125  213   0        0        125      1           1.4       1              2      no               0<br>35  46   0          2 142  177   0        0        160      1           1.4       0              2      no               0<br>36  54   0          2 135  304   1        1        170      0           0.0       0              2      no               0<br>37  54   1          2 150  232   0        0        165      0           1.6       0              3      no               0<br>38  65   0          2 155  269   0        1        148      0           0.8       0              2      no               0<br>39  65   0          2 160  360   0        0        151      0           0.8       0              2      no               0<br>40  51   0          2 140  308   0        0        142      0           1.5       1              2      no               0<br>41  48   1          1 130  245   0        0        180      0           0.2       0              2      no               0<br>42  45   1          0 104  208   0        0        148      1           3.0       0              2      no               0<br>43  53   0          0 130  264   0        0        143      0           0.4       0              2      no               0<br>44  39   1          2 140  321   0        0        182      0           0.0       0              2      no               0<br>45  52   1          1 120  325   0        1        172      0           0.2       0              2      no               0<br>46  44   1          2 140  235   0        0        180      0           0.0       0              2      no               0<br>47  47   1          2 138  257   0        0        156      0           0.0       0              2      no               0<br>48  53   0          2 128  216   0        0        115      0           0.0       0              2      no               0<br>49  53   0          0 138  234   0        0        160      0           0.0       0              2      no               0<br>50  51   0          2 130  256   0        0        149      0           0.5       0              2      no               0<br>51  66   1          0 120  302   0        0        151      0           0.4       0              2      no               0<br>52  62   1          2 130  231   0        1        146      0           1.8       3              3      no               0<br>53  44   0          2 108  141   0        1        175      0           0.6       0              2      no               0<br>54  63   0          2 135  252   0        0        172      0           0.0       0              2      no               0<br>55  52   1          1 134  201   0        1        158      0           0.8       1              2      no               0<br>56  48   1          0 122  222   0        0        186      0           0.0       0              2      no               0<br>57  45   1          0 115  260   0        0        185      0           0.0       0              2      no               0<br>58  71   0          2 110  265   1        0        130      0           0.0       1              2      no               0<br>59  54   1          1 108  309   0        1        156      0           0.0       0              3      no               0<br>60  52   1          3 118  186   0        0        190      0           0.0       0              1      no               0<br>61  41   1          1 135  203   0        1        132      0           0.0       0              1      no               0<br>62  58   1          2 140  211   1        0        165      0           0.0       0              2      no               0<br>63  35   0          0 138  183   0        1        182      0           1.4       0              2      no               0<br>64  51   1          2 100  222   0        1        143      1           1.2       0              2      no               0<br>65  45   0          1 130  234   0        0        175      0           0.6       0              2      no               0<br>66  44   1          1 120  220   0        1        170      0           0.0       0              2      no               0<br>67  62   0          0 124  209   0        1        163      0           0.0       0              2      no               0<br>68  54   1          2 120  258   0        0        147      0           0.4       0              3      no               0<br>69  51   1          2  94  227   0        1        154      1           0.0       1              3      no               0<br>70  29   1          1 130  204   0        0        202      0           0.0       0              2      no               0<br>71  51   1          0 140  261   0        0        186      1           0.0       0              2      no               0<br> [ reached &#39;max&#39; / getOption(&quot;max.print&quot;) -- omitted 229 rows ]</pre><h3><strong>Procedure</strong></h3><p>First, Exploratory Data Analysis will assess each feature’s meaning, contributions to health implications, and associations with the response variable. Next, K-Nearest Neighbors (KNN), Decision Tree (DT), and Logistic Regression (LR) will be individually evaluated using the True Positive Rate (TPR) to find their optimal configurations. For KNN, both even and odd values of K will be tested to address potential ties in classification, only if an even value of K gives the highest TPR. The best models will then be compared based on TPR, precision, ROC curves, and AUC values. Lastly, the strengths and limitations of each approach will be discussed and the most suitable model for the dataset will be identified.</p><h3><strong>Part 1 EDA: Exploring the variables and association</strong></h3><p><strong>0. Response variable: disease</strong></p><p>“<strong>disease” </strong>is a binary categorical response variable, which indicates the presence or absence of heart disease in the patients, where yes = presence of heart disease and no = absence of heart disease. There are 138 “yes” values and 162 “no” values in the “<strong>disease” </strong>variable. “<strong>disease.numeric</strong>” is a numerical representation of “<strong>disease</strong>” used for logistic regression, where 1 = presence, 0 = absence.</p><p><strong>1. Association between age and heart disease [weak]</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*jdGCKBd91yEECAW-00LnhQ.png" /></figure><p>“<strong>age</strong>” is a discrete input variable that indicates the age of the patient in years. According to the <a href="https://www.nia.nih.gov/health/heart-health/heart-health-and-aging">National Institute on Aging</a>, as the person ages, there might be changes to the heart and blood vessels, such as slower heartbeat rates and thickening of the heart. This could increase a person’s risk of heart disease and other related health problems.</p><p><strong>Observations: </strong>The median age of those without heart disease is around 52, which is lower than the median age of those with heart disease, which is around 58. Furthermore, for the group without heart disease, the first and third quartiles of the age distribution are approximately 44 and 59, respectively. For the group with heart disease, the first and third quartiles are around 52 and 62. Thus, there might be a weak or negligible association between age and heart disease, as the boxplots show considerable overlap despite differences in medians.</p><p><strong>2. Association between sex and heart disease [strong]</strong></p><p>“<strong>sex</strong>” is a binary categorical input variable represented numerically, and it represents the biological characteristics of each patient, where 1 = Male and 0 = Female. In terms of <a href="https://www.healthxchange.sg/news/heart-diseases-kill-more-men-at-earlier-age-than-women-straits-times-online">cardiovascular health</a>, Oestrogen in females aids in the removal of bad cholesterol and provides some protection against heart disease, whereas testosterone in males is linked to an increased risk of heart disease.</p><pre>Contingency table with conditional probabilities and odds ratio:<br>Pr(disease = yes |sex = M) = 0.5533981, Pr(disease = no |sex = M) = 0.4466019<br>Pr(disease = yes |sex = F)  = 0.2553191, Pr(disease = no |sex = F) = 0.7446809<br>Odds of having heart disease among males = 1.23913<br>Odds of having heart disease among females = 0.3428571<br>Odds ratio for odds of having heart disease among males over odds of having heart disease among females = 3.61413</pre><p><strong>Observations: </strong>The probability of having heart disease given that the patient is female is 0.2553, which is lower than the probability of having heart disease given the patient is male, which is 0.5534. Moreover, the odds of having heart disease among males is 3.614 times the odds of having heart disease among females. Thus, these observations suggest a strong association between sex and heart disease, with males being more susceptible to heart disease.</p><p><strong>3. Association between chest.pain and heart disease [strong]</strong></p><p>“<strong>chest.pain</strong>”<strong> </strong>is a categorical input variable consisting of four different types of chest pain represented numerically. <a href="https://www.ncbi.nlm.nih.gov/books/NBK559016/">0 = typical angina</a> is a symptom of myocardial ischemia, and it is characterised by persistent heavy-like pressure at the centre of the chest or anginal equivalent caused by exertion. <a href="https://www.narayanahealth.org/blog/atypical-chest-pain-symptoms-and-causes">1 = atypical angina</a> is characterised by sharp chest pain. <a href="https://www.medicalnewstoday.com/articles/nonanginal-chest-pain">2 = non-anginal pain</a> is characterised by chest pain in people without heart disease. <a href="https://www.ncbi.nlm.nih.gov/books/NBK536915/">3 = asymptomatic </a>is characterised by the lack of chest discomfort despite a reduced flow of oxygen-rich blood to the heart.</p><pre>Contingency table with conditional probabilities:<br>Pr(disease = yes | chest.pain = 0) = 0.7323944, Pr(disease = no |chest.pain = 0) =  0.2676056<br>Pr(disease = yes | chest.pain = 1) = 0.1800000, Pr(disease = no |chest.pain = 1) =  0.8200000<br>Pr(disease = yes | chest.pain = 2) = 0.2093023, Pr(disease = no |chest.pain = 2) =  0.7906977<br>Pr(disease = yes | chest.pain = 3) = 0.3181818, Pr(disease = no |chest.pain = 3) =  0.6818182</pre><p><strong>Observations:</strong> The probability of having heart disease varies across different types of chest pains, in the following order (from lowest to highest): atypical angina (0.1800), non-anginal pain (0.2093), asymptomatic (0.3182), typical angina (0.7324). This might suggest a strong association between chest pain (chest.pain) and heart disease, where patients with typical angina are more susceptible to heart disease.</p><p><strong>4. Association between bp and heart disease [weak]</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/0*1XlIuq4Zcvyn5C9R" /></figure><p><strong>“bp” </strong>is a discrete input variable that indicates the resting blood pressure of the patient measured in mmHg upon admission. <a href="https://www.heart.org/en/health-topics/high-blood-pressure/understanding-blood-pressure-readings">Hypertension (high blood pressure)</a> is a “silent killer” because it displays no symptoms. It is defined by a blood pressure reading of 130/80 mmHg or higher and it can damage blood vessels, increasing the risk of serious cardiovascular issues like heart failure and heart attacks.</p><p><strong>Observations: </strong>The median resting blood pressure of those without heart disease is around 130 mmHg, which is identical to the median resting blood pressure of those with heart disease. Furthermore, for the group without heart disease, the first and third quartiles of resting blood pressure are approximately 120 mm Hg and 140 mm Hg, respectively. For the group with heart disease, the first and third quartiles are around 120 mm Hg and 145 mm Hg. Therefore, these findings suggest that there may be a weak or negligible association between resting blood pressure (bp) and heart disease, as the boxplots show considerable overlap and the medians are identical.</p><p><strong>5. Association between chol and heart disease [weak]</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/0*WmqDjjnfx7hrdt81" /></figure><p><strong>“chol” </strong>is a discrete input variable that indicates the patient’s serum cholesterol level in mg/dL. <a href="https://www.myheart.org.sg/health/risk-factors/high-blood-cholesterol/">High blood cholesterol levels</a> are characterised by elevated low-density lipoprotein cholesterol levels, and they typically do not present any symptoms. However, high blood cholesterol is a significant risk factor for heart disease and the development of atherosclerotic disease.</p><p><strong>Observations: T</strong>he median serum cholesterol level of those without heart disease is around 234 mg/dL, which is lower than the median serum cholesterol level of those with heart disease, which is around 249 mg/dL. Furthermore, for the group without heart disease, the first and third quartiles of serum cholesterol levels are approximately 208 mg/dL and 266 mg/dL, respectively. For the group with heart disease, the first and third quartiles are around 217 mg/dL and 283 mg/dL. Therefore, these findings might suggest a weak or negligible association between the serum cholesterol levels (chol) and heart disease, as the boxplots show considerable overlap despite differences in medians.</p><p><strong>6. Association between fbs and heart disease [weak]</strong></p><p>“<strong>fbs</strong>” is a binary categorical input variable represented numerically, and it indicates whether the fasting blood sugar level is higher than 120 mg/dL, where 1 = true and 0 = false. Fasting blood sugar levels are measured after not eating anything the night before. Typically, the mean fasting blood sugar level falls between <a href="https://www.who.int/data/gho/indicator-metadata-registry/imr-details/2380#:~:text=When%20fasting%20blood%20glucose%20is,separate%20tests%2C%20diabetes%20is%20diagnosed.">70 mg/dL and 100 mg/dL</a>. If it is <a href="https://www.mayoclinic.org/diseases-conditions/diabetes/diagnosis-treatment/drc-20371451#:~:text=A%20fasting%20blood%20sugar%20level,Glucose%20tolerance%20test.">126 mg/dL or higher </a>on two separate tests, this indicates diabetes.<a href="https://www.niddk.nih.gov/health-information/diabetes/overview/preventing-problems/heart-disease-stroke#:~:text=High%20blood%20glucose%20from%20diabetes,can%20lead%20to%20heart%20disease.&amp;text=People%20with%20diabetes%20tend%20to,age%20than%20people%20without%20diabetes."> Elevated blood glucose levels</a> from diabetes can damage blood vessels and nerves that control the cardiovascular system, potentially leading to heart diseases.</p><pre>Contingency table with conditional probabilities and odds ratio:<br>Pr(disease = yes |fbs = true) = 0.4888889, Pr(disease = no |fbs = true) = 0.5111111<br>Pr(disease = yes |fbs = false)  = 0.4549020, Pr(disease = no |fbs = false) = 0.5450980<br>Odds of having heart disease among those where fasting blood sugar level &gt; 120 mg/dL is true = 0.9565217<br>Odds of having heart disease among those where fasting blood sugar level &gt; 120 mg/dL is false = 0.8345324<br>Odds ratio for odds of having heart disease among those where fasting blood sugar level &gt; 120 mg/dL is true over odds of having heart disease among those where fasting blood sugar level &gt; 120 mg/dL is false = 1.146177</pre><p><strong>Observations: </strong>The probability of having heart disease given that fbs = true is 0.4889, which is slightly higher than the probability of having heart disease given that fbs = false, which is 0.4549. Moreover, the odds of having heart disease among those with fbs = true is 1.146 times the odds of having heart disease among those with fbs = false. Thus, these observations might suggest a weak or negligible association between fasting blood sugar levels above 120 mg/dL (fbs) and heart disease, as the odds ratio is close to 1 and there are minimal differences between the conditional probabilities.</p><p><strong>7. Association between rest.ecg and heart disease [strong]</strong></p><p>“<strong>rest.ecg</strong>” is a categorical input variable consisting of three different results of the resting electrocardiogram (ECG) represented numerically. 0 = normal indicates normal ECG readings. <a href="https://jamanetwork.com/journals/jama/fullarticle/188761">1 = ST-T wave abnormalities</a> are characterised by an abnormality in the ST segment or T wave which could be associated with a risk of mortality caused by myocardial infarction, coronary heart disease, and cardiovascular disease.<a href="https://www.ncbi.nlm.nih.gov/books/NBK557534/#:~:text=The%20presence%20of%20LVH%20forecasts,with%20the%20regression%20of%20LVH."> 2 = possible or definite left ventricular hypertrophy</a> is characterised by thickening of the left ventricle which indicates an increased risk of cardiovascular morbidity and mortality.</p><pre>Contingency table with conditional probabilities:<br>Pr(disease = yes | rest.ecg = 0) = 0.5448276 Pr(disease = no |rest.ecg = 0) =  0.4551724<br>Pr(disease = yes | rest.ecg = 1) = 0.3708609, Pr(disease = no |rest.ecg = 1) =  0.6291391<br>Pr(disease = yes | rest.ecg = 2) = 0.7500000, Pr(disease = no |rest.ecg = 2) =  0.2500000</pre><p><strong>Observations:</strong> The probability of having heart disease varies across different types of resting ECG results, in the following order (from lowest to highest): ST-T wave abnormalities (0.3709), normal (0.5448), possible or definite left ventricular hypertrophy (0.7500). This might suggest a strong association between resting ECG readings (rest.ecg) and heart disease, where patients with possible or definite left ventricular hypertrophy are more susceptible to heart disease.</p><p><strong>8. Association between heart.rate and heart disease [strong]</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/0*pU8skEfbwlOh_Lh9" /></figure><p><strong>“heart.rate” </strong>is a discrete input variable that indicates the patient’s maximum heart rate achieved during exercise testing. <a href="https://pmc.ncbi.nlm.nih.gov/articles/PMC7523886/#:~:text=Maximal%20heart%20rate%20(HRmax)%20is%20an%20essential,during%20exercise%20(10)%2C%20and%20exercise%20prescription%20(2%2C5%2C16%2C33).&amp;text=Based%20on%20these%20results%2C%20clinicians%20should%20use,rehabilitation%20programs%20based%20on%20these%20HRmax%20values.">Maximal heart rate (HRmax)</a> is crucial for healthcare professionals in evaluating cardiovascular response during exercise testing and prescribing exercise. <a href="https://pmc.ncbi.nlm.nih.gov/articles/PMC3065291/#:~:text=Chronotropic%20incompetence%20(CI)%2C%20broadly,and%20quality%2Dof%2Dlife.">A low HRmax</a> may indicate chronotropic incompetence, suggesting potential heart issues. In contrast, <a href="https://www.healthline.com/health/running-heart-rate#safety">a very high heart rate</a> can increase the risk of cardiac events like arrhythmias and chest pain.</p><p><strong>Observations: </strong>The median HRmax of those without heart disease is around 161 bpm, which is higher than the median HRmax of those with heart disease, which is around 142 bpm. Furthermore, for the group without heart disease, the first and third quartiles of HRmax are approximately 148 bpm and 172 bpm, respectively. For the group with heart disease, the first and third quartiles are around 125 bpm and 156 bpm. This does not indicate a significant overlap. Therefore, these findings might suggest a strong association between HRmax (heart.rate) and heart disease, where individuals with lower HRmax are more susceptible to heart disease.</p><p><strong>9. Association between angina and heart disease [strong]</strong></p><p>“<strong>angina</strong>” is a binary categorical input variable represented numerically, and it indicates whether the patient has experienced exercise-induced angina, where 1 = yes and 0 = no. <a href="https://www.mayoclinic.org/diseases-conditions/angina/symptoms-causes/syc-20369373">Exercise-induced angina</a> is only noticeable during exercise due to increased oxygen demand and it signifies reduced blood flow to the heart which indicates risk of heart disease.</p><pre>Contingency table with conditional probabilities and odds ratio:<br>Pr(disease = yes |angina = yes) = 0.7676768, Pr(disease = no |angina = yes) = 0.2323232<br>Pr(disease = yes |angina = no)  = 0.3084577, Pr(disease = no |angina = no) = 0.6915423<br>Odds of having heart disease among those with exercise induced angina = 3.304348<br>Odds of having heart disease among those without exercise induced angina = 0.4460432<br>Odds ratio for odds of having heart disease among those  with exercise induced angina over odds of having heart disease among those without exercise induced angina = 7.408135</pre><p><strong>Observations: </strong>The probability of having heart disease given that angina = yes is 0.7677, which is higher than the probability of having heart disease given that angina = no, which is 0.3085. Moreover, the odds of having heart disease among those with exercise-induced angina is 7.408 times the odds of having heart disease among those without exercise-induced angina. Thus, these observations might suggest a strong association between exercise-induced angina (angina) and heart disease, where individuals with exercise-induced angina are more susceptible to heart diseases.</p><p><strong>10. Association between st.depression and heart disease [strong]</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/774/0*tTVPwTV_GACb1xR-" /></figure><p><strong>“st.depression” </strong>is a continuous input variable that indicates the extent of the dip in the ST segment on an ECG during exercise compared to resting. <a href="https://pmc.ncbi.nlm.nih.gov/articles/PMC1768611/#:~:text=Exercise%20induced%20ST%20segment%20depression,diagnosis%20of%20obstructive%20coronary%20atherosclerosis.&amp;text=It%20has%20also%20been%20associated,coronary%20artery%20disease%20(CAD).">Exercise-induced ST-segment depression relative to rest</a> is a strong indicator of myocardial ischemia and potential risk for coronary artery disease.</p><p><strong>Observations: </strong>The median Exercise-induced ST depression of those without heart disease is around 0.2 mm, which is lower than the median Exercise-induced ST depression of those with heart disease, which is around 1.4 mm. Furthermore, for the group without heart disease, the first and third quartiles of Exercise-induced ST depression are approximately 0.0 mm and 1.1 mm, respectively. For the group with heart disease, the first and third quartiles are around 0.6 mm and 2.5 mm. This does not indicate a significant overlap. Therefore, these findings might suggest a strong association between Exercise-induced ST depression (st.depression) and heart disease, where individuals with higher Exercise-induced ST depression are more susceptible to heart disease.</p><p><strong>11. Association between vessels and heart disease [strong]</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/838/0*r1MntaSiuvjTYqlj" /></figure><p><strong>“vessels” </strong>is a discrete input variable that indicates the number of major vessels identified using fluoroscopy. The presence of <a href="https://ijcrt.org/papers/IJCRT2105926.pdf">fluoroscopy-coloured major vessels</a> indicates poor blood flow through these vessels, where a greater number of coloured vessels is associated with a higher risk of heart disease.</p><p><strong>Observations:</strong> The median number of fluoroscopy-detected major vessels in individuals without heart disease is around 0, while those with heart disease have a median of about 1. The first and third quartiles for the group without heart disease are approximately 0, with a few outliers from 1 to 4, whereas for the group with heart disease, the first and third quartiles are around 0 and 2, respectively. This indicates minimal overlap between the two groups. Thus, this might suggest a strong association between the number of fluoroscopy-detected major vessels (vessels) and heart disease, where individuals with a higher count of major vessels are more susceptible to heart disease.</p><p><strong>12. Association between blood.disorder and heart disease [strong]</strong></p><p><strong>“blood.disorder” </strong>is a categorical input variable consisting of three different blood disorder classifications. 1 = normal refers to no observable perfusion (blood flow) abnormalities. <a href="https://link.springer.com/article/10.1007/s12350-020-02492-8">2 = fixed defect</a> are characterised by abnormal perfusion both at rest and stress, which might indicate permanent heart damage from myocardial infarction.<a href="https://link.springer.com/article/10.1007/s12350-020-02492-8"> 3 = reversible defect </a>are characterised by normal perfusion at rest but reduced perfusion during stress which signifies the presence of living ischemic tissues with poorly-functioning coronary blood flow reserve.</p><pre>Contingency table with conditional probabilities:<br>Pr(disease = yes | blood.disorder = 1) = 0.6666667, Pr(disease = no |blood.disorder = 1) =  0.3333333<br>Pr(disease = yes | blood.disorder = 2) = 0.2242424, Pr(disease = no |blood.disorder = 2) =  0.7757576<br>Pr(disease = yes | blood.disorder = 3) = 0.7606838, Pr(disease = no |blood.disorder = 3) =  0.2393162</pre><p><strong>Observations:</strong> The probability of having heart disease varies across different types of blood disorders, in the following order (from lowest to highest): fixed defect (0.2242), normal (0.6667), reversible defect (0.7607). This might suggest a strong association between the types of blood disorders (blood.disorder) and heart disease, where patients with reversible defects are more susceptible to heart disease.</p><h3><strong>Part 2 Methods: Building KNN, DT and LR Classifiers</strong></h3><p><strong>Overview: </strong>The new dataset, “<strong>heartdb_new</strong>”, is a modified version of the original CSV dataset (<strong>heartdb</strong>), by removing features with weak associations Three categorical variables were quantitatively and meaningfully relabelled in “<strong>heartdb_new</strong>” due to KNN implementation: (i) chest.pain by severity of chest pain and clinical relevance, (ii) rest.ecg by severity of cardiovascular health risk, and blood.disorder by severity of heart tissue damage and perfusion. Further modifications to “<strong>heartdb_new</strong>” will occur during the development of K-nearest neighbours (KNN), decision trees (DT), and logistic regression (LR) classifiers.</p><p><strong>1. Finding the best KNN model</strong></p><p><strong>Summary:</strong> The optimal KNN model is identified by the best k value that maximises the average True positive rate (TPR). The nominal variables, sex and angina, were excluded. The features: chest.pain*, rest.ecg*, heart.rate, st.depression, vessels, blood.disorder* were scaled to create the “<strong>scaled_heartdb_new</strong>” dataset and concatenated with the column of the disease variable (disease) to form “<strong>scaled.data</strong>” dataset. “<strong>scaled.data</strong>” consists of 300 rows and 7 columns. A 5-fold cross-validation was performed with values of k from 1 to the floor of sqrt(300) ~ 17.</p><p><strong>Observations:</strong> The highest average TPR achieved was 0.7680 at k = 6. However, the possibility of ties in predicted categorical responses is concerning. Among the odd k values, the best result was a TPR of 0.7614 at k = 7. Thus, the preferred KNN model is at k = 6, but further assessment with k = 7 is recommended.</p><p><strong>2. Finding the best DT model</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rE2zgu9yWQ9KMIFc" /><figcaption>Figure 7. This figure shows four tree diagrams of minsplit 1 (top left), 2 (top right), 3 (bottom left) and 4 (bottom right), where the full dataset, “<strong>heartdb_new_2</strong>”, is used. This illustrates that there are no apparent differences in the structure of the decision tree for all four different minsplits. The values of their TPR, precision, and AUC are also identical. From here, we can see blood.disorder is the most important feature, followed by chest.pain, st.depression, vessels, and heart.rate.</figcaption></figure><p><strong>Summary: </strong>The best DT model is determined by finding the best minsplit value that yields the highest TPR. This segment uses all 8 features, regardless of data type. The data set used is <strong>“heartdb_new_2”</strong>, consisting of 300 rows and 9 columns. It contains the categorical response (disease). All categorical features are converted to factors. Five-fold cross-validation is performed with minsplit values from 1 to 100.</p><p><strong>Observations: </strong>The average TPR decreases from 0.7306 to 0.6478 with increasing minsplit values. Four minsplit values, 1 to 4, yield the highest average TPR of 0.7306.</p><p><strong>3. Finding the best LR model</strong></p><p><strong>Summary: </strong>The best logistic regression model is chosen based on the highest TPR and balance in complexity. The dataset, “<strong>heartdb_logistic</strong>”, contains 300 rows and 8 features (with categorical data as factors) and a numerically labelled response variable (disease.numeric). A summary revealed that chest.pain, rest.ecg, and blood.disorder are the least significant features. The modified dataset, “<strong>heartdb_logistic_edit</strong>”, removes the least significant feature after each 5-fold cross-validation cycle and average TPR calculation.</p><p><strong>Observations: </strong>The removal of chest.pain and rest.ecg yields the highest average TPR (0.7998). Removing blood.disorder leads to a drop in average TPR (0.7294).</p><p><strong>4. A visual comparison of the performance of KNN, DT and LR on the full data set.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ecqSHKzGMtrdspYU" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/978/0*TlqudXxQvdT71T2b" /></figure><p><strong>5. Pros and cons of each classifier</strong></p><p><strong>KNN pros:</strong></p><ul><li>It is a <a href="https://www.ibm.com/think/topics/knn#:~:text=Advantages%20*%20Easy%20to%20implement:%20Given%20the,when%20compared%20to%20other%20machine%20learning%20algorithms.">simple algorithm</a> that is easy to comprehend and implement.</li><li>KNN uses fewer hyperparameters, like k value and distance metric, compared to other machine learning algorithms.</li><li>It possesses a <a href="https://www.analyticsvidhya.com/blog/2023/02/lazy-learning-vs-eager-learning-algorithms-in-machine-learning/">fast training phase</a> by storing the training dataset and performing computations only when a prediction is needed.</li></ul><p><strong>KNN cons:</strong></p><ul><li>Its prediction time is <a href="https://www.analyticsvidhya.com/blog/2023/02/lazy-learning-vs-eager-learning-algorithms-in-machine-learning/">slow and time-consuming</a> as the algorithm is <a href="https://app.myeducator.com/reader/web/1421a/11/q07a0/">computationally intensive</a>, and this issue worsens with larger training datasets.</li><li>KNN is affected by the <a href="https://towardsdatascience.com/k-nearest-neighbors-and-the-curse-of-dimensionality-e39d10a6105d/#:~:text=The%20k%2Dnearest%20neighbors%20algorithm%20hinges%20on%20data%20points%20being,algorithm%20to%20provide%20valuable%20predictions.">“curse of dimensionality”</a>, where the distance between data points becomes less meaningful as the dimensionality (number of features) increases, which can diminish prediction accuracy.</li><li>It is <a href="https://dl.acm.org/doi/10.1145/2801081.2801116">sensitive to noise</a>, meaning that factors such as noisy data, outliers, and mislabelled data can negatively impact prediction outcomes.</li></ul><p><strong>DT pros:</strong></p><ul><li>DT can handle both <a href="https://scikit-learn.org/stable/modules/tree.html">numerical and categorical data</a> seamlessly.</li><li>It requires minimal data preparation compared to other machine learning algorithms, which require preparations like data scaling and one-hot encoding.</li><li>It copes<a href="https://www.analyticsvidhya.com/blog/2021/05/25-questions-to-test-your-skills-on-decision-trees/#:~:text=Decision%20Trees%20are%20not%20sensitive,are%20prone%20to%20sampling%20errors."> well with outliers and noise</a> since they do not affect how data is split, with <a href="https://insidelearningmachines.com/decision_trees_robust_to_outliers/#:~:text=In%20general%2C%20Decision%20Trees%20are,will%20show%20sensitivity%20to%20outliers.">overfitted DT </a>as an exception.</li></ul><p><strong>DT cons:</strong></p><ul><li>DT is prone to <a href="https://www.gyansetu.in/blog/decision-trees-advantages-and-disadvantages/">overfitting</a> when it is too deep, complex, and unpruned, which makes its predictions more influenced by noise than by the actual patterns in the data.</li><li>It can be biased toward the dominant classes when imbalanced datasets are used, leading to poor predictive performance.</li><li>If the training data does not accurately represent real-world scenarios, the model might make inaccurate predictions with new data.</li></ul><p><strong>LR pros:</strong></p><ul><li>Logistic regression performs well with datasets that are <a href="https://careerfoundry.com/en/blog/data-analytics/what-is-logistic-regression/">linearly separable</a>, which means that the positive and negative classes can be separated by a line.</li><li>It is a <a href="https://www.linkedin.com/pulse/deep-dive-logistic-regression-scalebuildai/">straightforward</a> method to implement and interpret the outcomes.</li><li>It can <a href="https://www.tutorialspoint.com/pros-and-cons-of-different-classification-models">assist in feature selection</a> by evaluating the strength and significance of each feature.</li></ul><p><strong>LR cons:</strong></p><ul><li><a href="https://www.tutorialspoint.com/pros-and-cons-of-different-classification-models">Outliers can influence the model’s estimated coefficients</a> which makes it sensitive to extreme values and leads to less reliable predictions.</li><li>The <a href="https://www.grammarly.com/blog/ai/what-is-logistic-regression/#7">outcomes predicted are limited to discrete values</a>, which makes it unsuitable for predicting continuous values.</li><li>It may not provide accurate results if the <a href="https://careerfoundry.com/en/blog/data-analytics/what-is-logistic-regression/">sample size is too small</a> due to overfitting.</li></ul><h3><strong>Part III Conclusion: The Best Model/Classifier</strong></h3><p>In conclusion, it is recommended to use KNN (k = 7, excluding “sex” and “anginal”) for heart disease prediction. This model strikes a better balance as it yields the second-best AUC (0.9373), precision (0.8898), and TPR (0.8188). A high TPR is <a href="https://www.datacamp.com/tutorial/auc">crucial in a medical setting</a> as it ensures fewer heart disease patients are missed (i.e., false negatives). High precision ensures that healthy individuals are less likely to be mistakenly diagnosed with heart disease (i.e., false positives). A high AUC indicates better model performance in <a href="https://www.evidentlyai.com/classification-metrics/explain-roc-curve">distinguishing positive and negative cases</a>.</p><p>In comparison to the other three models, the KNN model (k = 6, excluding “sex” and “anginal”) offers the highest AUC (0.9430) and precision (0.8983), but it falls short in TPR (0.7681), which is critical in disease prediction. Moreover, LR generally performs the worst across all three metrics, with a TPR of 0.7971, precision of 0.8271, and AUC of 0.9021, making it the least competitive among the models tested. Lastly, DT (minsplit = 1, w. all features) has the highest TPR (0.8478), which is desirable as the consequences of misdiagnosing a disease patient as healthy (i.e. false negative) outweigh misclassifying a healthy individual with heart disease (i.e. false positive). Hence, it is acceptable for its precision (0.8540) to be lower than at least two classifiers. However, its AUC (0.9007) indicates that it performs slightly worse than the others. But, DT can be considered a strong runner-up especially when maximising TPR is the top priority rather than attaining balance.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=375ae346c0c0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Predicting Term Deposit Subscriptions: A Data-Driven Approach]]></title>
            <link>https://medium.com/@DataDreamJO/predicting-term-deposit-subscriptions-a-data-driven-approach-ca61c6a4b181?source=rss-42566753e5c8------2</link>
            <guid isPermaLink="false">https://medium.com/p/ca61c6a4b181</guid>
            <category><![CDATA[confusion-matrix]]></category>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[predictive-analytics]]></category>
            <category><![CDATA[decision-tree-classifier]]></category>
            <category><![CDATA[random-forest-algorithm]]></category>
            <dc:creator><![CDATA[Data_Dream_J.O.]]></dc:creator>
            <pubDate>Wed, 08 Jan 2025 15:31:48 GMT</pubDate>
            <atom:updated>2025-01-10T12:01:49.045Z</atom:updated>
            <content:encoded><![CDATA[<p>Date of creation: Friday, 16 August 2024 (Posted officially on Wednesday, 8 January 2025)</p><p>Last updated: Friday, 10 January 2025</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rsxf3j_iQ4uNnPLaq8BpjQ.png" /></figure><h3>Introduction</h3><p>This report is created with the aim to understand the factors that largely contributes to the question: what spurs a customer to purchase a fixed-term deposit?</p><p>According to <a href="https://www.investopedia.com/terms/t/termdeposit.asp">Investopedia’s definition</a>, a term deposit is a savings account where your money is held for a fixed period, ranging from one month to a few years. Term deposits usually offer higher interest rates than regular savings accounts, which let you withdraw money anytime.</p><p>The data is obtained from <a href="http://archive.ics.uci.edu/dataset/222/bank+marketing">UC Irvine Machine Learning Repository</a>. The data is related with direct marketing campaigns of a Portuguese banking institution. The marketing campaigns were based on phone calls. According to the website, there are four csv datasets:</p><ol><li><strong>bank-additional-full.csv</strong>: contains 20 features and 41,188 rows of data, ordered by date from May 2008 to November 2010.</li><li><strong>bank-additional.csv</strong>: contains 20 features and 4,119 rows of data. These rows were randomly selected from <strong>bank-additional-full.csv.</strong></li><li><strong>bank-full.csv</strong>: contains 17 features and 45211 rows of data, ordered by date.</li><li><strong>bank-.csv</strong>: contains 17 features and 4521 rows of data. These rows were randomly selected from <strong>bank-full.csv</strong>.</li></ol><p>Thus, this report aims to identify observational facts and trends from each features, determine a set of features that strongly influences the likelihood of subscription to the term deposits, and create a <a href="https://termdepositpredictivemodel.streamlit.app/">working predictive model</a>.</p><h3>Datasets: data cleaning and processing</h3><p>Initially, the four csv files were merged bank-additional-full.csv, bank-additional.csv, bank-full.csv, bank-.csv. But, it was found that every row will contain at least one null/ empty value. Hence, dropping rows with null values will lead to an empty dataset. The null values are found in these columns: ‘day_of_week’, ‘emp.var.rate’, ‘cons.price.idx’, ‘cons.conf.idx’, ‘euribor3m’, ‘nr.employed’, ‘balance’, ‘day’. Hence, csv files that were mean to be supplements were removed to focus on the main parameters. Since bank-.csv is a subset of bank-full.csv, we will use bank-full.csv to conduct our observations of the data.</p><p>bank client data:</p><ul><li>Age: Numeric</li><li>Job: Type of job (Categorical: “admin”, “unknown”, “unemployed”, “management”, “housemaid”, “entrepreneur”, “student”, “blue-collar”, “self-employed”, “retired”, “technician”, “services”)</li><li>Marital Status: Marital status (Categorical: “married”, “divorced” (includes widowed), “single”)</li><li>Education: Level of education (Categorical: “unknown”, “secondary”, “primary”, “tertiary”)</li><li>Credit Default: Has credit in default? (Binary: “yes”, “no”)</li><li>Balance: Average yearly balance in euros (Numeric)</li><li>Housing Loan: Has a housing loan? (Binary: “yes”, “no”)</li><li>Personal Loan: Has a personal loan? (Binary: “yes”, “no”)</li><li>Contact Type: Contact communication type (Categorical: “unknown”, “telephone”, “cellular”)</li><li>Day: Last contact day of the month (Numeric)</li><li>Month: Last contact month of the year (Categorical: “jan”, “feb”, “mar”, …, “nov”, “dec”)</li><li>Duration: Last contact duration in seconds (Numeric)</li><li>Campaign: Number of contacts performed during this campaign for this client (Numeric, includes last contact)</li><li>Pdays: Number of days since the client was last contacted from a previous campaign (Numeric, -1 means the client was not previously contacted)</li><li>Number of Contact Attempts: Number of contacts performed before this campaign for this client (Numeric)</li><li>Previous Outcome: Outcome of the previous marketing campaign (Categorical: “unknown”, “other”, “failure”, “success”)</li><li>Subscribed: Has the client subscribed to a term deposit?(Binary: “yes”, “no”)</li></ul><h4>final_bank.csv dataset (processed bank-full.csv):</h4><pre>Age           Job Marital Status  Education Credit Default  Balance  \<br>0       58    management        married   tertiary             no     2143   <br>1       44    technician         single  secondary             no       29   <br>2       33  entrepreneur        married  secondary             no        2   <br>3       47   blue-collar        married    unknown             no     1506   <br>4       33       unknown         single    unknown             no        1   <br>...    ...           ...            ...        ...            ...      ...   <br>45206   51    technician        married   tertiary             no      825   <br>45207   71       retired       divorced    primary             no     1729   <br>45208   72       retired        married  secondary             no     5715   <br>45209   57   blue-collar        married  secondary             no      668   <br>45210   37  entrepreneur        married  secondary             no     2971   <br><br>      Housing Loan Personal Loan Contact Type  Day Month  Duration  Campaign  \<br>0              yes            no      unknown    5   may       261         1   <br>1              yes            no      unknown    5   may       151         1   <br>2              yes           yes      unknown    5   may        76         1   <br>3              yes            no      unknown    5   may        92         1   <br>4               no            no      unknown    5   may       198         1   <br>...            ...           ...          ...  ...   ...       ...       ...   <br>45206           no            no     cellular   17   nov       977         3   <br>45207           no            no     cellular   17   nov       456         2   <br>45208           no            no     cellular   17   nov      1127         5   <br>45209           no            no    telephone   17   nov       508         4   <br>45210           no            no     cellular   17   nov       361         2   <br><br>       Pdays  Number of Contact Attempts Previous Outcome Subscribed  <br>0         -1                           0          unknown         no  <br>1         -1                           0          unknown         no  <br>2         -1                           0          unknown         no  <br>3         -1                           0          unknown         no  <br>4         -1                           0          unknown         no  <br>...      ...                         ...              ...        ...  <br>45206     -1                           0          unknown        yes  <br>45207     -1                           0          unknown        yes  <br>45208    184                           3          success        yes  <br>45209     -1                           0          unknown         no  <br>45210    188                          11            other         no  <br><br>[45211 rows x 17 columns]</pre><h3>Relationships between data and subscribers</h3><p>Before we create the predictive model, we need to observe the trends in regards to the dataset.</p><h3>Age and subscriber relationships: observations</h3><pre>Bin width  Subscribed  Not Subscribed  Total Subscribed(%)<br>0   18.00 to 21.85          55             121    176         31.2%<br>1   21.85 to 25.70         265             895   1160         22.8%<br>2   25.70 to 29.55         608            3329   3937         15.4%<br>3   29.55 to 33.40         854            6956   7810         10.9%<br>4   33.40 to 37.25         772            6554   7326         10.5%<br>5   37.25 to 41.10         523            5076   5599          9.3%<br>6   41.10 to 44.95         307            3232   3539          8.7%<br>7   44.95 to 48.80         419            4057   4476          9.4%<br>8   48.80 to 52.65         335            3445   3780          8.9%<br>9   52.65 to 56.50         313            2973   3286          9.5%<br>10  56.50 to 60.35         336            2598   2934         11.5%<br>11  60.35 to 64.20         161             217    378         42.6%<br>12  64.20 to 68.05          89             123    212         42.0%<br>13  68.05 to 71.90          59             106    165         35.8%<br>14  71.90 to 75.75          76              96    172         44.2%<br>15  75.75 to 79.60          62              69    131         47.3%<br>16  79.60 to 83.45          32              57     89         36.0%<br>17  83.45 to 87.30          16              11     27         59.3%<br>18  87.30 to 91.15           2               5      7         28.6%<br>19  91.15 to 95.00           4               1      5         80.0%</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2H71YSvKMtL-kReyAgLH3A.png" /><figcaption><strong>Observations</strong>:</figcaption></figure><ul><li>The graph indicates that <strong>most of the callees are aged between approximately 30 to below 40 years</strong>. This age group constitutes the majority of those called about term deposits.</li><li>When considering percentages, individuals <strong>under 29 and over 60 show a higher inclination to subscribe</strong>, with the highest participation rate observed at <strong>80.0%</strong>.</li><li>Overall, the data suggests a possible relationship between age and the likelihood of subscribing to term deposits.</li></ul><h3>Profession and subscriber relationship: observations</h3><pre>Profession  Subscribed  Not Subscribed  Total Subscribed(%)<br>0          admin.         631            4540   5171         12.2%<br>1     blue-collar         708            9024   9732          7.3%<br>2    entrepreneur         123            1364   1487          8.3%<br>3       housemaid         109            1131   1240          8.8%<br>4      management        1301            8157   9458         13.8%<br>5         retired         516            1748   2264         22.8%<br>6   self-employed         187            1392   1579         11.8%<br>7        services         369            3785   4154          8.9%<br>8         student         269             669    938         28.7%<br>9      technician         840            6757   7597         11.1%<br>10     unemployed         202            1101   1303         15.5%<br>11        unknown          34             254    288         11.8%</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MVr75ZuPkG_BPb5ExsJr9Q.png" /><figcaption><strong>Observations</strong>:</figcaption></figure><ul><li>The graph indicates that <strong>most callees work in blue-collar, management, or technical professions</strong>.</li><li>Among all professions, <strong>students are the most inclined to subscribe to term deposits</strong>, with around <strong>28.7%</strong> agreeing to the subscription, followed by <strong>retired individuals</strong>.</li><li>Employed individuals appear less likely to subscribe, and if unemployed callees are assumed to be of working age, it can be deduced that <strong>people of working age are generally less inclined to subscribe to term deposits</strong>.</li></ul><h3>Marital status and subscriber relationships: observations</h3><pre>Marital Status  Subscribed  Not Subscribed  Total Subscribed(%)<br>0       divorced         622            4585   5207         11.9%<br>1        married        2755           24459  27214         10.1%<br>2         single        1912           10878  12790         14.9%</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/1*nTM2cVOLLOKeb_q8u5VlaA.png" /><figcaption><strong>Observations</strong>:</figcaption></figure><ul><li>The graph indicates that <strong>most of the respondents (callees) are married</strong>.</li><li>Among <strong>single respondents, a higher proportion appears willing to subscribe</strong> to term deposits compared to other marital statuses. However, this group represents only 14.9% of the total single respondents.</li><li>Overall, although the majority of the respondents are married, <strong>single callees are slightly more likely to subscribe to term deposits</strong>.</li></ul><h3>Education background and subscriber relationships: observations</h3><pre>Education  Subscribed  Not Subscribed  Total Subscribed(%)<br>0    primary         591            6260   6851          8.6%<br>1  secondary        2450           20752  23202         10.6%<br>2   tertiary        1996           11305  13301         15.0%<br>3    unknown         252            1605   1857         13.6%</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/1*nC9okSXE0bGIY3eH0sLLGg.png" /><figcaption><strong>Observations</strong>:</figcaption></figure><ul><li>The graph indicates that <strong>most callees have secondary education</strong> as their highest level of education.</li><li>Individuals with <strong>tertiary education appear slightly more willing to subscribe to term deposits</strong>. However, the <strong>subscription rate of around 15.0% does not differ significantly from those with other educational backgrounds</strong>.</li><li>Overall, although the majority of callees have secondary education, there is a slight tendency for individuals with secondary education to subscribe to term deposits, though the difference is not substantial.</li></ul><h3>Credit Default background and Subscriber Relationships: observations</h3><pre>Credit Default  Subscribed  Not Subscribed  Total Subscribed(%)<br>0             no        5237           39159  44396         11.8%<br>1            yes          52             763    815          6.4%</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/587/1*mSyIIrLD83Sqa5mTfcT9eA.png" /><figcaption><strong>Observations</strong>:</figcaption></figure><ul><li>The graph indicates that <strong>most callees have a clean credit record</strong>, meaning they do not have a history of failing to repay loans or meet other financial obligations.</li><li>Subscription rates differ between the two groups: 11.8% of those with clean credit records subscribed to term deposits, compared to 6.4% of those without clean credit records.</li><li>Although the majority of callees with clean records dominate the sample size, the <strong>difference in subscription rates suggests that having a clean credit record might positively influence the likelihood of subscribing to term deposits</strong>.</li></ul><h3>Bank balance and subscriber relationships: observations</h3><pre>Bin width  Subscribed  Not Subscribed  Total Subscribed(%)<br>0   -1500 to -1000           3              94     97          3.1%<br>1    -1000 to -500          33             516    549          6.0%<br>2        -500 to 0         172            2901   3073          5.6%<br>3         0 to 500        1972           17899  19871          9.9%<br>4      500 to 1000         854            6078   6932         12.3%<br>5     1000 to 1500         475            3266   3741         12.7%<br>6     1500 to 2000         369            2026   2395         15.4%<br>7     2000 to 2500         255            1374   1629         15.7%<br>8     2500 to 3000         237            1025   1262         18.8%<br>9     3000 to 3500         166             798    964         17.2%<br>10    3500 to 4000         133             619    752         17.7%<br>11    4000 to 4500          93             454    547         17.0%<br>12    4500 to 5000          84             421    505         16.6%<br>13    5000 to 5500          75             334    409         18.3%<br>14    5500 to 6000          47             281    328         14.3%<br>15    6000 to 6500          28             232    260         10.8%<br>16    6500 to 7000          35             189    224         15.6%<br>17    7000 to 7500          29             146    175         16.6%<br>18    7500 to 8000          30             138    168         17.9%<br>19    8000 to 8500          21             150    171         12.3%<br>20    8500 to 9000          17             102    119         14.3%<br>21    9000 to 9500          18              74     92         19.6%<br>22   9500 to 10000           6              66     72          8.3%<br>23  10000 to 10500          14              71     85         16.5%<br>24  10500 to 11000          19              47     66         28.8%<br>25  11000 to 11500          11              46     57         19.3%<br>26  11500 to 12000          11              40     51         21.6%<br>27  12000 to 12500          11              37     48         22.9%<br>28  12500 to 13000          11              37     48         22.9%<br>29  13000 to 13500           5              40     45         11.1%<br>30  13500 to 14000           2              34     36          5.6%<br>31  14000 to 14500           4              22     26         15.4%<br>32  14500 to 15000           3              13     16         18.8%<br>33  15000 to 15500           4              25     29         13.8%<br>34  15500 to 16000           0              13     13          0.0%<br>35  16000 to 16500           0              21     21          0.0%<br>36  16500 to 17000           0              21     21          0.0%<br>37  17000 to 17500           1              19     20          5.0%<br>38  17500 to 18000           3              17     20         15.0%<br>39  18000 to 18500           3               3      6         50.0%<br>40  18500 to 19000           2               6      8         25.0%<br>41  19000 to 19500           2              10     12         16.7%<br>42  19500 to 20000           0               8      8          0.0%</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DFlRWhrSNh7nZizRXW6ofQ.png" /><figcaption><strong>Observations</strong>:</figcaption></figure><ul><li>The <strong>majority of individuals</strong> contacted regarding term deposits have bank balances between <strong>0 and below 500 euros</strong>.</li><li>The <strong>highest percentage of subscribers (50.0%) are in the bank balance range of 18,000 to below 18,500 euros</strong>. However, this percentage is based on only 6 individuals, so it may <strong>not be statistically significant</strong>.</li><li>The <strong>majority of subscriber percentages fall within the 10% to 20% range</strong>.</li><li>Overall, the data suggests a <strong>slight potential relationship between age and the likelihood of subscribing to term deposits</strong>.</li></ul><h3>Housing Loans and Subscriber Relationships: observations</h3><pre>Housing Loan  Subscribed  Not Subscribed  Total Subscribed(%)<br>0           no        3354           16727  20081         16.7%<br>1          yes        1935           23195  25130          7.7%</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*88VKVXkLLGN91aDc8oseCA.png" /><figcaption><strong>Observations</strong></figcaption></figure><ul><li>The graph indicates that <strong>majority of the callees have housing loans</strong>.</li><li>People without housing loans have a higher likelihood of subscribing to term deposits (16.7%) compared to those with housing loans (7.7%). This suggests a <strong>potential correlation between the absence of a housing loan and the preference for term deposits</strong>.</li></ul><h3>Personal Loan and Subscriber Relationships: observations</h3><pre>Personal Loan  Subscribed  Not Subscribed  Total Subscribed(%)<br>0            no        4805           33162  37967         12.7%<br>1           yes         484            6760   7244          6.7%</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*tFV4xlxc2MvjFxahZGjIGA.png" /><figcaption><strong>Observations</strong>:</figcaption></figure><ul><li>the graph indicates that <strong>most of the callees do not have a personal loan</strong>.</li><li>While most callees do not have a personal loan, the presence of a <strong>personal loan appears to have an impact on the likelihood of subscribing to term deposits</strong>. Specifically, 12.7% of those without personal loans subscribed to term deposits, compared to just 6.7% of those with personal loans.</li></ul><h3>Contact type and Subscriber Relationship: observations</h3><pre>Contact Type  Subscribed  Not Subscribed  Total Subscribed(%)<br>0     cellular        4369           24916  29285         14.9%<br>1    telephone         390            2516   2906         13.4%<br>2      unknown         530           12490  13020          4.1%</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/663/1*jZQqmHm2wyufLtynb2rW5Q.png" /><figcaption><strong>Observations</strong>:</figcaption></figure><ul><li>The <strong>majority of callees were contacted via cellular communication</strong>. Among them, those who were contacted cellularly have a slightly higher likelihood of subscribing to term deposits (14.9%) compared to those contacted via telephone (13.4%) and other means (4.1%).</li><li>Overall, the data suggests that being <strong>contacted cellularly may have a slight positive influence on the likelihood of subscribing to term deposits</strong>.</li></ul><h3>Dates and Subscriber Relationships: observations</h3><pre>Date Subscribed(%)  Total<br>0   11 jan         58.3%     12<br>1   12 jan         59.1%     22<br>2   13 jan         50.0%     18<br>3   14 jan         56.5%     23<br>4   15 jan         50.0%     14<br>..     ...           ...    ...<br>83  22 dec         71.4%      7<br>84  28 dec         56.2%     16<br>85  29 dec         53.8%     13<br>86   3 dec         61.5%     13<br>87   9 dec         55.0%     20</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*knPC_TB88EJHeSuuMxeWSw.png" /></figure><pre>Date Subscribed(%)  Total<br>0   jan         10.1%   1403<br>1   feb         16.6%   2649<br>2   mar         52.0%    477<br>3   apr         19.7%   2932<br>4   may          6.7%  13766<br>5   jun         10.2%   5341<br>6   jul          9.1%   6895<br>7   aug         11.0%   6247<br>8   sep         46.5%    579<br>9   oct         43.8%    738<br>10  nov         10.2%   3970<br>11  dec         46.7%    214</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/565/1*CQfX-Ak5_-TAk-hfTHWn-w.png" /><figcaption><strong>Observations</strong>:</figcaption></figure><ul><li>The months of March (52.0%), September (46.5%), October (43.8%), and December (46.7%) show significantly higher subscription rates compared to other months, where subscriptions typically hover around 10%.</li></ul><h3>Duration and Subscriber relationships: observations</h3><pre>Bin width  Subscribed  Not Subscribed  Total Subscribed(%)<br>0       0 to 100         113           10611  10724          1.1%<br>1     100 to 200         763           13265  14028          5.4%<br>2     200 to 300         946            7184   8130         11.6%<br>3     300 to 400         666            3763   4429         15.0%<br>4     400 to 500         519            1987   2506         20.7%<br>5     500 to 600         445            1147   1592         28.0%<br>6     600 to 700         415             707   1122         37.0%<br>7     700 to 800         317             425    742         42.7%<br>8     800 to 900         264             254    518         51.0%<br>9    900 to 1000         205             152    357         57.4%<br>10  1000 to 1100         168             112    280         60.0%<br>11  1100 to 1200         121              99    220         55.0%<br>12  1200 to 1300          87              58    145         60.0%<br>13  1300 to 1400          70              32    102         68.6%<br>14  1400 to 1500          51              37     88         58.0%<br>15  1500 to 1600          42              21     63         66.7%<br>16  1600 to 1700          25               7     32         78.1%<br>17  1700 to 1800          17               9     26         65.4%<br>18  1800 to 1900          14              14     28         50.0%<br>19  1900 to 2000          11               9     20         55.0%</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l4LMjbMmyXKqhMWpATOO2g.png" /><figcaption><strong>Observations</strong>:</figcaption></figure><ul><li>The distribution is right-skewed (positively skewed), indicating that <strong>most calls have shorter durations, with fewer calls having longer durations</strong>.</li><li>There appears to be a trend where <strong>longer call durations are associated with a higher likelihood of subscription</strong>, although there is a dip at around 1700 seconds.</li><li>Overall, the data strongly suggests that <strong>longer call durations correlate with an increased likelihood of the callee subscribing to term deposits</strong>.</li></ul><h3>Campaign (number of calls to the same person for the term deposits) and subscriber relationships: observations</h3><pre>Bin width  Subscribed  Not Subscribed  Total Subscribed(%)<br>0          0           0               0      0          0.0%<br>1          1        2561           14983  17544         14.6%<br>2          2        1401           11104  12505         11.2%<br>3          3         618            4903   5521         11.2%<br>4          4         317            3205   3522          9.0%<br>..       ...         ...             ...    ...           ...<br>59        59           0               0      0          0.0%<br>60        60           0               0      0          0.0%<br>61        61           0               0      0          0.0%<br>62        62           0               0      0          0.0%<br>63        63           0               1      1          0.0%</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a-iD0jV0orW5P8He3HK6lQ.png" /><figcaption><strong>Observations</strong>:</figcaption></figure><ul><li>The distribution is right-skewed (positively skewed), indicating that <strong>most callees were contacted fewer than approximately 12 times</strong>.</li><li>There appears to be a <strong>downward trend</strong>, where an <strong>increasing number of contacts with the same callee is associated with a lower likelihood of subscription</strong>, although there are spikes at 11, 17, 21, 24, 29, and 32 contacts.</li><li>Overall, the data strongly suggests that a <strong>higher number of contacts correlates with a decreased likelihood of the callee subscribing to term deposits</strong>.</li></ul><h3>Days Since Last Contact and subsciber relationships: options</h3><pre>Bin width  Subscribed  Not Subscribed  Total Subscribed(%)<br>0      0 to 20          22             136    158         13.9%<br>1     20 to 40          16              45     61         26.2%<br>2     40 to 60          28              40     68         41.2%<br>3     60 to 80          40              89    129         31.0%<br>4    80 to 100         506             485    991         51.1%<br>5   100 to 120         102             370    472         21.6%<br>6   120 to 140          50             223    273         18.3%<br>7   140 to 160          50             278    328         15.2%<br>8   160 to 180         100             573    673         14.9%<br>9   180 to 200         448             685   1133         39.5%<br>10  200 to 220          39             180    219         17.8%<br>11  220 to 240          21             118    139         15.1%<br>12  240 to 260          24             325    349          6.9%<br>13  260 to 280          60             421    481         12.5%<br>14  280 to 300          53             233    286         18.5%<br>15  300 to 320          37             238    275         13.5%<br>16  320 to 340          34             437    471          7.2%<br>17  340 to 360          70             755    825          8.5%<br>18  360 to 380          56             574    630          8.9%<br>19  380 to 400          34              28     62         54.8%<br>20  400 to 420          14               9     23         60.9%<br>21  420 to 440          20               9     29         69.0%<br>22  440 to 460          14              12     26         53.8%<br>23  460 to 480          19              16     35         54.3%<br>24  480 to 500           4              10     14         28.6%<br>25  500 to 520           5               7     12         41.7%<br>26  520 to 540           6              10     16         37.5%<br>27  540 to 560           8               7     15         53.3%<br>28  560 to 580           3               2      5         60.0%</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OnUhRk3PSx7TWasqEVjisg.png" /><figcaption><strong>Observations</strong></figcaption></figure><ul><li><strong>Most callees</strong> were contacted between <strong>80 and below 200 days after their first call</strong>.</li><li>The <strong>highest subscription percentage, 69.0%, was observed 420 to below 440 days</strong> after the first contact. However, this figure may be <strong>statistically insignificant due to the small sample size of only 29 individuals</strong>.</li><li>For statistically significant data (30 or more callees), subscription rates were highest when callees were approached <strong>80 to below 100 days after the first contact</strong>, with a percentage of approximately 51.1%.</li><li>Considering <strong>data with more than 30 callees</strong>, there appears to be an <strong>increasing trend in subscription rates from 0 to less than 100 days</strong>, followed by a <strong>decreasing trend from 100 to less than 380 days</strong>, with a noticeable <strong>spike from 380 to below 400 days</strong>, since last contact.</li></ul><h3>Number of contact attempts per person for other campaigns (before subscription of term deposits) and subscriber relationships: observations</h3><pre>Bin width  Subscribed  Not Subscribed  Total Subscribed(%)<br>0     0 to 5        4888           38800  43688         11.2%<br>1    5 to 10         321             841   1162         27.6%<br>2   10 to 15          65             168    233         27.9%<br>3   15 to 20           6              59     65          9.2%<br>4   20 to 25           4              27     31         12.9%<br>5   25 to 30           2              15     17         11.8%<br>6   30 to 35           1               3      4         25.0%<br>7   35 to 40           0               5      5          0.0%<br>8   40 to 45           0               2      2          0.0%<br>9   45 to 50           0               0      0          0.0%<br>10  50 to 55           0               1      1          0.0%</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WH6DQhbX7wqOvkemvNHrSg.png" /><figcaption><strong>Observations</strong>:</figcaption></figure><ul><li><strong>Most callees were contacted between 0 and below 5 times</strong> for other campaigns.</li><li>For statistically significant data (30 or more callees), the likelihood of subscription was <strong>lower for 0 to below 5 contact attempts (11.2%)</strong> and <strong>higher for 5 to below 15 contact attempts (approximately 27.0%)</strong>.</li><li>These findings suggest a <strong>potential relationship between the number of contact attempts and the callee’s willingness to subscribe to the other plans</strong>.</li></ul><h3>Previous outcome and subscriber relationships: observations</h3><pre>Previous calling outcome  Subscribed  Not subscribed  Total Subscribed (%)<br>0                  failure         618            4283   4901          12.6%<br>1                    other         307            1533   1840          16.7%<br>2                  success         978             533   1511          64.7%<br>3                  unknown        3386           33573  36959           9.2%</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/649/1*3unzl15TP4Cn0j9-mRM9Dw.png" /><figcaption><strong>Observations</strong>:</figcaption></figure><ul><li>The graph shows that the data <strong>does not provide information on whether most of the callees had been previously contacted for other marketing schemes</strong>. This lack of data could limit the ability to fully understand how prior contact might influence the likelihood of subscribing to the term deposits.</li><li>However, current data shows that <strong>people who were previously contacted and subscribed to prior marketing campaigns have a much higher likelihood of subscribing to term deposits (64.7%) compared to those who did not subscribe to previous campaigns (12.6%)</strong>.</li></ul><p><strong>Summary: Data and its properties</strong></p><p>At the start of the project, the data obtained from the <a href="http://archive.ics.uci.edu/dataset/222/bank+marketing">UC Irvine Machine Learning Repository</a> were processed and cleaned using the pandas library.</p><p>For each potential feature, observations were made:</p><ul><li>Age: Younger individuals (under 29) and older individuals (above 60) were more likely to subscribe, with most of the population falling between 30–40 years old.</li><li>Professions: Blue-collar, management, and technical professionals showed strong participation in the survey, with students being the most inclined to subscribe.</li><li>Marital status: Married individuals were predominant, but singles showed a higher inclination to subscribe.</li><li>Education background: A majority had secondary education, while those with tertiary education showed slightly higher subscription rates.</li><li>Credit Records: Clean credit records were associated with a higher likelihood of subscribing to term deposits.</li><li>Bank balance in euros: Those with higher balances, especially between 18,000–18,500 euros, were more likely to subscribe.</li><li>Housing loans: People without housing loans had a greater likelihood of subscribing to term deposits.</li><li>Personal Loans: Individuals without personal loans also showed higher subscription rates.</li><li>Type of contact: Contact through cellular communication was the most common, and this method had a slight positive impact on subscription likelihood.</li><li>Time-related features: The months of March, September, October, and December had significantly higher subscription rates, and call duration also played a role, with longer calls correlating with higher subscription chances.</li><li>Campaign-related features: A lower number of contacts was associated with a higher likelihood of subscription, as well as fewer contacts for other campaigns.</li></ul><p>Despite having access to a myriad amount of data, linear relationships seem rare, and the low subscriber percentage and imbalanced sampling makes it difficult to determine if the data has any significant impact on the predictive ability of the model.</p><h3>Implementation of the Multiple Linear Regression Model</h3><p>Initially, a Multiple Linear Regression model was developed after one-hot encoding the categorical features. Using the scikit-learn library, the R-squared values for both training and testing datasets were notably low: 0.304 (training) and 0.294 (testing). Despite attempts to enhance performance using Lasso and Ridge regression, feature removal, and backtracking-based feature selection, the results remained suboptimal. Upon reviewing the graphs, it was observed that the majority of the features are categorical in nature. The low performance of the model suggests that multiple regression may not be the optimal choice for mixed dataset containing a high proportion of categorical data.</p><h3>Implementation of the Decision Tree</h3><p>Based on sources such as <a href="https://dataheroes.ai/blog/decision-trees-an-overview-and-practical-guide/#:~:text=In%20conclusion%2C%20decision%20trees%20offer,and%20applicable%20to%20various%20domains.">link 1</a>, <a href="https://datascience.stackexchange.com/questions/418/best-way-to-classify-datasets-with-mixed-types-of-attributes">link 2</a>, <a href="https://towardsdatascience.com/decision-tree-classifier-explained-in-real-life-picking-a-vacation-destination-6226b2b60575">link 3</a>, it is evident that tree-based algorithms are often better suited for handling mixed datasets, particularly those with a high proportion of categorical data. Additionally, Random Forests have been shown to outperform individual Decision Tree classifiers by aggregating predictions from multiple decision trees, as highlighted by <a href="https://www.kdnuggets.com/2022/08/decision-trees-random-forests-explained.html#:~:text=Random%20forests%20typically%20perform%20better,up%20with%20a%20final%20prediction.">link 4</a> and <a href="https://corp.yonyx.com/automate-support/advantage-of-decision-tree/">link 5</a>.</p><p>Using the dataset prior to feature selection, an evaluation of a single decision tree using the scikit-learn library’s metrics module revealed suboptimal performance for the predictive model. While the model demonstrated high accuracy and specificity, it struggled with low precision, sensitivity, and F-score (the harmonic mean of precision and sensitivity). This indicates the decision tree was ineffective in detecting true positive cases and had limited ability to accurately predict whether an individual was likely to subscribe to the term deposits. The data below shows the predictive ability of a single decision tree prior to feature selection:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/564/1*eLj3C9erulzB7DB_v9qajg.png" /><figcaption>Accuracy:0.8715992037159921<br>Precision:0.46419098143236076<br>Sensitivity:0.4861111111111111<br>Specificity:0.9238884702336096<br>F-score: 0.47489823609226595</figcaption></figure><p>Using a backtracking algorithm for feature selection, the following 9 features were identified as yielding the most optimal results: ‘Marital Status,’ ‘Credit Default,’ ‘Contact Type,’ ‘Day,’ ‘Month,’ ‘Duration,’ ‘Pdays,’ ‘Number of Contact Attempts,’ and ‘Previous Outcome.’ The model’s performance metrics are as follows:</p><ul><li>Accuracy:0.8794514487945145</li><li>Precision:0.5049594229035167</li><li>Sensitivity:0.508628519527702</li><li>Specificity:0.9308651303362296</li><li>F-score: 0.5067873303167421</li></ul><p>While these results show some improvement in the model’s predictive ability, the low precision, sensitivity, and F-score indicate that the model still struggles to effectively predict true positive cases, making it less than ideal as a predictive tool. However, the confusion matrix revealed that both the training and testing datasets have a significantly higher proportion of true negatives (‘no’ cases) compared to true positives (‘yes’ cases) in the target column (Subscribed column). This indicates that the data is highly imbalanced, which might potentially reduce the model’s ability to accurately predict positive outcomes.</p><h3>Implementation of the Random Forest model</h3><h3>Random Forest Model Implementation Post-Feature Selection: Performance Evaluation Using Train-Test Split and Confusion Matrix Results</h3><p>With the use of Random Forests and the features selected using backtracking feature selection algorithm for single trees, the model shows an improved precision but lowered sensitivity and F-score compared to the single decision tree post feature selection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*14jMu6dPsnbWKZpMuh9WTg.png" /><figcaption>Accuracy:0.9020128290201282<br>Precision:0.5997150997150997<br>Sensitivity:0.41033138401559455<br>Specificity:0.9649451097804391<br>F-score: 0.48726851851851855</figcaption></figure><h3>Evaluation of Random Forest Performance Post-Feature Selection: Average Metrics from Stratified K-Fold Cross-Validation on Five Data Samples</h3><p>However, after applying StratifiedKFold, a technique commonly used for imbalanced datasets to generate five different training and testing data sets for the mean calculation of the model’s performance, the model’s performance decreased substantially.</p><p>StratifiedKFold 1:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*NDuuIQNdNk1mCQBoExKr9Q.png" /><figcaption>Accuracy:0.8802388587858012<br>Precision:0.16216216216216217<br>Sensitivity:0.005671077504725898<br>Specificity:0.996117720726362<br>F-score: 0.010958904109589041</figcaption></figure><p>StratifiedKFold 2:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*3UwPL5gBgxaN8eiTAhjyVg.png" /><figcaption>Accuracy:0.7336872373368724<br>Precision:0.026629292221443588<br>Sensitivity:0.03595080416272469<br>Specificity:0.8260488415779587<br>F-score: 0.030595813204508857</figcaption></figure><p>StratifiedKFold 3:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*ZtL3lwAme1eRgh0Na912sw.png" /><figcaption>Accuracy:0.767750497677505<br>Precision:0.07154605263157894<br>Sensitivity:0.08223062381852551<br>Specificity:0.8585921843687375<br>F-score: 0.07651715039577836</figcaption></figure><p>StratifiedKFold 4:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*JLKRDaNCUaQjD5aUpQO73g.png" /><figcaption>Accuracy:0.5760893607608936<br>Precision:0.06842923794712286<br>Sensitivity:0.20793950850661624<br>Specificity:0.624874749498998<br>F-score: 0.10297215071378422</figcaption></figure><p>StratifiedKFold 5:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*LO25VtQGKNuf8JkRz2ZBWQ.png" /><figcaption>Accuracy:0.4362972793629728<br>Precision:0.14749519986035958<br>Sensitivity:0.7986767485822306<br>Specificity:0.38827655310621245<br>F-score: 0.24900545159864446</figcaption></figure><p>Average from all 5 StratifiedKFold datasets:</p><ul><li>Accuracy:0.678812646784809</li><li>Precision:0.09525238896453343</li><li>Sensitivity:0.22609375251496458</li><li>Specificity:0.7387820098556537</li><li>F-score: 0.094009894004461</li></ul><h3>Random Forest Model Implementation with K-Fold Stratification, Undersampling, and Manual Hyperparameter Tuning</h3><p>As a result, undersampling was applied to balance the dataset while retaining the original data. While oversampling using SMOTE was considered, generating over 6,000 synthetic data points for balance did not seem ideal. After performing undersampling and utilizing random forests with StratifiedKFold for cross-validation, the model showed significant improvement in its predictive ability. The evaluation metrics after these adjustments were as follows:</p><p>Before hyperparameter tuning, the average metrics from five readings were as follows:</p><ul><li>Accuracy: 70.8%</li><li>Precision: 75.8%</li><li>Sensitivity: 58.1%</li><li>Specificity: 83.6%</li><li>F-score: 64.2%</li></ul><p>After implementing hyperparameter tuning, the results improved to:</p><ul><li>Accuracy: 76.6%</li><li>Precision: 78.5%</li><li>Sensitivity: 72.6%</li><li>Specificity: 80.6%</li><li>F-score: 75.1%</li></ul><h3>Reevaluating Feature Selection with Original Data: A Random Forest Approach Using Undersampling, StratifiedKFold, and Hyperparameter Tuning</h3><p>Results from the Backtracking feature selection program:</p><pre>Processing 7: 100%|█████████████████████| 11440/11440 [3:06:33&lt;00:00,  1.02it/s]<br>features 7: [&#39;Age&#39;, &#39;Marital Status&#39;, &#39;Credit Default&#39;, &#39;Housing Loan&#39;, &#39;Duration&#39;, &#39;Campaign&#39;, &#39;Previous Outcome&#39;]<br>Accuracy:0.8073380793414577<br>Precision:0.7997272426945468<br>Sensitivity:0.818310730694461<br>Specificity:0.796372549194943<br>F-score: 0.8078840791345551<br><br>Processing 8: 100%|█████████████████████| 12870/12870 [3:15:59&lt;00:00,  1.09it/s]<br>features 8: [&#39;Age&#39;, &#39;Job&#39;, &#39;Education&#39;, &#39;Balance&#39;, &#39;Housing Loan&#39;, &#39;Duration&#39;, &#39;Campaign&#39;, &#39;Previous Outcome&#39;]<br>Accuracy:0.8073382580988261<br>Precision:0.8016714627779542<br>Sensitivity:0.8154785899387109<br>Specificity:0.7992100552085029<br>F-score: 0.8072875609220425<br><br>Processing 9: 100%|█████████████████████| 11440/11440 [3:45:02&lt;00:00,  1.18s/it]<br>features 9: [&#39;Age&#39;, &#39;Job&#39;, &#39;Marital Status&#39;, &#39;Education&#39;, &#39;Credit Default&#39;, &#39;Balance&#39;, &#39;Housing Loan&#39;, &#39;Duration&#39;, &#39;Previous Outcome&#39;]<br>Accuracy:0.8090403857584004<br>Precision:0.8071368345406317<br>Sensitivity:0.810751082440763<br>Specificity:0.8073352016353306<br>F-score: 0.807439504835284<br><br>Processing 10: 100%|██████████████████████| 8008/8008 [2:37:29&lt;00:00,  1.18s/it]<br>features 10: [&#39;Age&#39;, &#39;Job&#39;, &#39;Marital Status&#39;, &#39;Education&#39;, &#39;Credit Default&#39;, &#39;Balance&#39;, &#39;Housing Loan&#39;, &#39;Personal Loan&#39;, &#39;Duration&#39;, &#39;Previous Outcome&#39;]<br>Accuracy:0.8073369621079068<br>Precision:0.8031271514586058<br>Sensitivity:0.8145319796191741<br>Specificity:0.8001525521637192<br>F-score: 0.806881314387099<br><br>Processing 11: 100%|██████████████████████| 4368/4368 [1:26:21&lt;00:00,  1.19s/it]<br>features 11: [&#39;Age&#39;, &#39;Job&#39;, &#39;Marital Status&#39;, &#39;Education&#39;, &#39;Balance&#39;, &#39;Housing Loan&#39;, &#39;Personal Loan&#39;, &#39;Duration&#39;, &#39;Campaign&#39;, &#39;Pdays&#39;, &#39;Previous Outcome&#39;]<br>Accuracy:0.8072435613830458<br>Precision:0.8087087553440183<br>Sensitivity:0.8031894669258683<br>Specificity:0.8113054924144196<br>F-score: 0.8042769606276645<br><br>Processing 12: 100%|████████████████████████| 1820/1820 [36:11&lt;00:00,  1.19s/it]<br>features 12: [&#39;Age&#39;, &#39;Job&#39;, &#39;Marital Status&#39;, &#39;Credit Default&#39;, &#39;Balance&#39;, &#39;Housing Loan&#39;, &#39;Personal Loan&#39;, &#39;Duration&#39;, &#39;Campaign&#39;, &#39;Pdays&#39;, &#39;Number of Contact Attempts&#39;, &#39;Previous Outcome&#39;]<br>Accuracy:0.8074336251547368<br>Precision:0.8161772165296476<br>Sensitivity:0.7925986268516846<br>Specificity:0.8222794118962075<br>F-score: 0.8024311103152222<br><br>Processing 13: 100%|██████████████████████████| 560/560 [11:10&lt;00:00,  1.20s/it]<br>features 13: [&#39;Age&#39;, &#39;Job&#39;, &#39;Education&#39;, &#39;Credit Default&#39;, &#39;Balance&#39;, &#39;Housing Loan&#39;, &#39;Personal Loan&#39;, &#39;Day&#39;, &#39;Duration&#39;, &#39;Campaign&#39;, &#39;Pdays&#39;, &#39;Number of Contact Attempts&#39;, &#39;Previous Outcome&#39;]<br>Accuracy:0.804598265159742<br>Precision:0.8077687020495409<br>Sensitivity:0.7977051003929156<br>Specificity:0.8114956013828059<br>F-score: 0.8022015772006018<br><br>Processing 14: 100%|██████████████████████████| 120/120 [02:23&lt;00:00,  1.20s/it]<br>features 14: [&#39;Age&#39;, &#39;Job&#39;, &#39;Marital Status&#39;, &#39;Education&#39;, &#39;Credit Default&#39;, &#39;Balance&#39;, &#39;Housing Loan&#39;, &#39;Personal Loan&#39;, &#39;Day&#39;, &#39;Month&#39;, &#39;Duration&#39;, &#39;Campaign&#39;, &#39;Pdays&#39;, &#39;Number of Contact Attempts&#39;]<br>Accuracy:0.7881506209584077<br>Precision:0.8012600449563744<br>Sensitivity:0.765190207331446<br>Specificity:0.8111205698619163<br>F-score: 0.7813862651628745<br><br>Processing 15: 100%|████████████████████████████| 16/16 [00:19&lt;00:00,  1.21s/it]<br>features 15: [&#39;Age&#39;, &#39;Job&#39;, &#39;Marital Status&#39;, &#39;Education&#39;, &#39;Credit Default&#39;, &#39;Balance&#39;, &#39;Housing Loan&#39;, &#39;Personal Loan&#39;, &#39;Day&#39;, &#39;Month&#39;, &#39;Duration&#39;, &#39;Campaign&#39;, &#39;Pdays&#39;, &#39;Number of Contact Attempts&#39;, &#39;Previous Outcome&#39;]<br>Accuracy:0.7856927071462728<br>Precision:0.8112015941905586<br>Sensitivity:0.7442122281379158<br>Specificity:0.8271891593177537<br>F-score: 0.7731878230517365<br><br>Processing 16: 100%|██████████████████████████████| 1/1 [00:01&lt;00:00,  1.46s/it]<br>features 16: [&#39;Age&#39;, &#39;Job&#39;, &#39;Marital Status&#39;, &#39;Education&#39;, &#39;Credit Default&#39;, &#39;Balance&#39;, &#39;Housing Loan&#39;, &#39;Personal Loan&#39;, &#39;Contact Type&#39;, &#39;Day&#39;, &#39;Month&#39;, &#39;Duration&#39;, &#39;Campaign&#39;, &#39;Pdays&#39;, &#39;Number of Contact Attempts&#39;, &#39;Previous Outcome&#39;]<br>Accuracy:0.764333346740136<br>Precision:0.7837328804315062<br>Sensitivity:0.7224752438062569<br>Specificity:0.8062031322375092<br>F-score: 0.7485329405287535</pre><p>The model’s performance has improved substantially. However, in hopes of boosting the performance even further, backtracking feature selection was applied again to identify the best-performing features. Nine features were selected, which differed from the previous selections used with the single decision tree model. These features are: ‘Age’, ‘Job’, ‘Marital Status’, ‘Education’, ‘Credit Default’, ‘Balance’, ‘Housing Loan’, ‘Duration’, ‘Previous Outcome’, and ‘Subscribed’. The results after this feature selection are as follows:</p><ul><li>Accuracy:0.8090403857584004</li><li>Precision:0.8071368345406317</li><li>Sensitivity:0.810751082440763</li><li>Specificity:0.8073352016353306</li><li>F-score: 0.807439504835284</li></ul><h3>Reevaluating Hyperparameter tuning and Sample table finding with post feature selection Data: A Random Forest Approach Using Undersampling, StratifiedKFold</h3><p>Results from hyperparameter tuning and searching the best sample data for Random Forest model:</p><pre>Table selection: 100%|████████████████████████| 100/100 [06:27&lt;00:00,  3.88s/it<br>{&#39;Random State for no_sample&#39;: 71, &#39;Accuracy&#39;: 0.8112125112281972, &#39;Precision&#39;: 0.8126358193215546, &#39;Sensitivity&#39;: 0.8077274019812108, &#39;Specificity&#39;: 0.8147083177591823, &#39;F-score&#39;: 0.8085226493719521}</pre><p>By reevaluating hyperparameter tuning and by finding the best sample dataset to be used for fitting the random forest model, we tested 100 different possible types of tabular datasets. These datasets were generated through different sets of true negative (“no”) data, and the resulting table showed improvement in the model’s performance:</p><ul><li>Accuracy: 0.8112125112281972</li><li>Precision: 0.8126358193215546</li><li>Sensitivity: 0.8077274019812108</li><li>Specificity: 0.8147083177591823</li><li>F-score’: 0.8085226493719521</li></ul><p>Although StratifiedKFold cross-validation could alter the metrics, the results have remained quite stable. These improved metrics indicate that the model has a high chance of correctly determining whether a person would or would not subscribe to the term deposits offered by the bank. Afterwards, with some assistance from ChatGPT, an interactive predictive model was created using the Streamlit library: <a href="https://termdepositpredictivemodel.streamlit.app/">Term Deposit predictive model</a>. Although the project concludes here, I believe that by using other predictive models, the metrics of this model could be further enhanced.</p><p><strong>Citation</strong>:</p><p>In this study, I used the dataset described in Moro et al. (2014) to predict the success of bank telemarketing. The dataset is publicly available for research purposes. Please refer to the original paper for full details on the dataset:</p><p>Moro, S., Cortez, P., &amp; Rita, P. (2014). A Data-Driven Approach to Predict the Success of Bank Telemarketing. Decision Support Systems, In press. <a href="http://dx.doi.org/10.1016/j.dss.2014.03.001">http://dx.doi.org/10.1016/j.dss.2014.03.001</a>.&quot;</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ca61c6a4b181" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unveiling Social Sentiments: Exploring Sentiment Analysis in Social Media for Insightful Reason…]]></title>
            <link>https://medium.com/@DataDreamJO/unveiling-social-sentiments-exploring-sentiment-analysis-in-social-media-for-insightful-reason-e9e857e1b375?source=rss-42566753e5c8------2</link>
            <guid isPermaLink="false">https://medium.com/p/e9e857e1b375</guid>
            <category><![CDATA[israel-palestine-conflict]]></category>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[confusion-matrix]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Data_Dream_J.O.]]></dc:creator>
            <pubDate>Mon, 25 Mar 2024 14:31:31 GMT</pubDate>
            <atom:updated>2024-03-26T09:52:51.677Z</atom:updated>
            <content:encoded><![CDATA[<h3>Unveiling Social Sentiments: Exploring Sentiment Analysis in Social Media for Insightful Reason Finding</h3><p><a href="https://trinket.io/python3/9750190e12?outputOnly=true&amp;runOption=run">Click here for the author’s name</a></p><p>Date of creation: Sun, 4 March 2024; Last updated: Sun, 24 March 2024</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/783/1*frg9wkf6szUW36sz3gE3gQ.jpeg" /></figure><h3><strong>Disclaimer! — Please read before delving into this report</strong></h3><p>This report aims to neutrally examine people’s perceptions surrounding the Israel-Palestine conflict. It is important to note that the representation of perspectives in this report is approached with the utmost respect for all parties involved. While differing viewpoints are presented, the aim is not to endorse or discredit any particular stance but rather to illuminate the diversity of opinions that exist within the discourse surrounding the Israel-Palestine conflict. <strong>Please refrain from reading this further if it causes any discomfort</strong>.</p><h3>Introduction: brief summary and objectives</h3><p>The Israel-Gaza conflict<a href="https://www.aljazeera.com/news/2024/1/21/hamas-says-october-7-attack-was-a-necessary-step-admits-to-some-faults">[source link]</a> is a deeply emotive and multifaceted issue that evokes a wide range of perspectives and opinions. In the international community, sentiments vary widely depending on geopolitical considerations, historical perspectives, and values <a href="https://en.wikipedia.org/wiki/International_reactions_to_the_Israel%E2%80%93Hamas_war">[source link]</a>. Social media sentiments also show the complexity and emotional intensity of the issue, with various subreddits on Reddit expressing different perspectives. Some subreddits express support for Gaza or a Palestinian perspective, such as r/Palestine, r/Gaza, and r/FreePalestine. Other subreddits express support for Israel or an Israeli perspective, such as r/Israel, r/Zionism, and r/IDF.</p><p>It is noted that opinions on this matter vary widely and can change over time, and there is no single, unified perspective held by everyone on either side. The results can vary depending on the methodology and demographics of the sample population. Ultimately, this report aims to investigate people’s sentiment on the conflict and identify key reasons for their views.</p><h3>Data cleaning and processing</h3><h4>Summary of data collection from Reddit</h4><p>To start off, I use <strong>ParseHub</strong>, a free web scrapper, to gather comments from <a href="https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vTuuZODci4TQXJnAm5f_ToPrGtSvqh-EcJ905gW4wFSQBQLk3mNxFfpw0qArN2GnI4Jzu5nIdP2WKM6/pubhtml?gid=1651583220&amp;single=true">four different subreddits</a>. The data collected are cleaned and processed using <strong>Excel</strong>. The four different subreddits include <strong>r/Israel_Palestine, r/centrist, r/Palestine, and r/Israel</strong>. These are categorised into three different groups: <strong>Neutral, Palestine, and Israel</strong>, based on their <strong>assumed perspective of the Israel-Palestine conflict</strong>.</p><pre>Ideological Spectrum  Total number of comments  Total number of commenters  \<br>0                 All                       1070                         692   <br>1              Neutral                       312                         133   <br>2            Palestine                       425                         304   <br>3               Israel                       333                         255   <br><br>   Largest contribution  Comments to commenter ratio  <br>0                    33                     1.548480  <br>1                    33                     2.345865  <br>2                     6                     1.398026  <br>3                     7                     1.305882</pre><p>The following is a brief description of four subreddits and what they are dedicated to:</p><ul><li>r/Palestine: This subreddit is focused on discussions, news, and content related to Palestine. It covers a wide range of topics including politics, culture, history, and current events from a Palestinian perspective.</li><li>r/Israel: This subreddit is focused on discussions and content related to Israel. It covers various aspects of Israeli society, politics, culture, news, and events from an Israeli perspective.</li><li>r/Centrist: This subreddit is dedicated to discussions from a centrist perspective. It aims to provide a platform for individuals with moderate or centrist political views to discuss a wide range of topics including politics, policy, and current events.</li><li>r/Israel_Palestine: This subreddit is specifically dedicated to discussions about the Israeli-Palestinian conflict. It provides a platform for users to share news, analysis, and opinions, and engage in discussions about the conflict from various perspectives.</li></ul><p>A total of 1070 comments have been collected from four different subreddits, with 692 individual commenters. The ratio of comments to commenters indicates that, on average, each commenter from Palestine and Israel provides one comment, while in neutral subreddits, each commenter provides an average of 2 comments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/819/1*CSzk2cejGo2uepSQQfwMeA.png" /></figure><h3>Sentiment analysis of the Israel Gaza conflict</h3><h4>Methodology</h4><p><strong>Objective</strong></p><ul><li>This section is used to identify the overall sentiments towards the Israel-Palestine conflict and compare human sentiment analysis with VADER sentiment analysis.</li></ul><p><strong>Tools</strong></p><p>The tools used for sentiment analysis include:</p><ul><li>Natural language Toolkit(NLTK): Used for tokenisation, stop words removal, lemmatisation and stemming, and lexicon-based analysis (VADER Sentimental Analysis).</li><li>Pandas: Used for viewing tabular contents</li><li>Matplotlib: Provides a graphical view of the sentimental distribution</li><li>sklearn: Used for creating a confusion matrix display to verify the accuracy of VADER sentiment analysis</li></ul><p>930 comments have been randomly sampled from the data. 310 comments were gathered from each Ideological Spectrum (Neutral, Palestine, Israel). The collected data were further processed to remove null values and duplicates using Pandas and Excel cleaning to ensure a fair representation of different ideological perspectives. These comments will be processed using natural language processing and analysed using VADER Sentiment analysis. A pie chart will then be plotted to showcase the division of sentiments.</p><h3>Sentimental analysis using NLTK Vader</h3><h4>Text preprocessing and algorithm development</h4><p>Data analysis of <a href="https://docs.google.com/spreadsheets/d/e/2PACX-1vTuuZODci4TQXJnAm5f_ToPrGtSvqh-EcJ905gW4wFSQBQLk3mNxFfpw0qArN2GnI4Jzu5nIdP2WKM6/pubhtml?gid=1611295965&amp;single=true">930 comments</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/522/1*vlHD9vCV_St6GjLouxj6PQ.png" /><figcaption>According to the sample data, 34.301% of the individuals exhibit a Negative sentiment regarding the <br>Israel-Palestine conflict.<br>According to the sample data, 15.269% of the individuals exhibit a Positive sentiment regarding the <br>Israel-Palestine conflict.<br>According to the sample data, 50.430% of the individuals exhibit a Neutral sentiment regarding the <br>Israel-Palestine conflict.</figcaption></figure><h4>A look into the comparison between Human and VADER Sentiment Analysis</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/552/1*_HFZTlErZVv0ogR6anCnTg.png" /><figcaption>Similarity between Human and Vader predicted values: 50.667%</figcaption></figure><p><strong>Conclusion</strong></p><p>An analysis of sentiment within the data indicates that the majority adopts a neutral position regarding the Israel-Palestine conflict. To assess the similarity between human and VADER sentiment analysis, a set of 75 comments was sampled. Each of these samples underwent manual inspection to ascertain whether it conveyed negative, positive, or neutral sentiments. The manually assigned values were then contrasted with the predicted values derived from the Vader analysis. Results revealed an accuracy rate of 50.7%. The primary source of inaccuracies predominantly arises from the subtle distinction between sentimentally neutral and negative comments.</p><p>Examining the comments reveals a blurred distinction between negativity and neutrality. The subreddit’s query revolved around whether the commentators perceive the conflict in Gaza as genocide. Consequently, individuals in the comment section often articulate their viewpoints with limited clarity regarding their stance on the conflict’s sides. As a result, the tool may struggle to fully comprehend the emotional and motivational context. Conversely, sentiment analysis inherently involves subjectivity, with diverse interpretations of text sentiment among individuals. NLTK’s VADER endeavors to accommodate this subjectivity through its lexicon and rule-based methodology, but its outcomes may still fluctuate depending on how the text is interpreted.</p><h3>Analysing key rationales influencing people’s stances on the Israel-Palestine conflict</h3><h4>Methodology</h4><p><strong>Objective</strong></p><p>This section aims to delineate the primary factors influencing perspectives on the Israel-Palestine conflict.</p><p><strong>Tools</strong></p><p>The analysis employs several libraries including:</p><ul><li>Pandas: Viewing and creating tabular data</li><li>nltk: Text preprocessing using word tokenisation, stop words removal and stemming.</li><li>Matplotlib: Graphically view the frequency of words that appear in the comments</li><li>string: To access modules involving ascii characters, numbers and punctuations.</li><li>emoji: To convert emojis to written text.</li><li>sumy and ChatGPT 3.5 (website): Summarise comments and provide in depth view into the comments</li><li>OS: This module functions to remove this warning: huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks…</li></ul><p><strong>Procedure</strong></p><p>Step 1: Text Preprocessing</p><p>Firstly, The dataset consists of 1070 comments gathered from four subreddits. Each comment will undergo tokenisation, stop word removal, and stemming to simplify words to their root forms. This normalization process reduces text complexity, treating various word forms identically, thus enhancing text analysis accuracy.</p><p>Step 2: Graphical Representation of Common Words</p><p>An interactive program visualises word distribution. Words appearing more than 37 times are depicted in a line and dot plot.</p><p>Step 3: Visual Analysis of Top 30 Words</p><p>The 30 most frequently occurring words are selected for further investigation. This involves selecting comments that contains these words, and summarising them to obtain meaningful insights.</p><h3>Text preprocessing and analysis</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xlooi9Drj74-7PiyYhml6Q.png" /></figure><p>The analysis identified a total of 4188 distinct words from the examined texts. The graph provided above facilitates the examination of words occurring more than 37 times within the analysed comments. A total of 104 words, each with a frequency of 38 or higher, were identified and sorted in ascending order of occurrence.</p><p>Reviewing the top 30 words may yield insights into prevailing perceptions of the conflict: “go”, “right”, “rape”, “get”, “make”, “attack”, “civilian”, “one”, “say”, “war”, “hostag”, “know”, “genocid”, “kill”, “think”, “would”, “even”, “gaza”, “isra”, “want”, “ `` “, “ ‘’ “, “like”, “palestinian”, “n’t”, “‘s”, “peopl”, “hama”, “israel”, and “ ’ “</p><p>From these 30 words, punctuations like “ `` “, “ ‘’ “, “ ’ “, as well as words like “n’t”, and “‘s”, were manually removed as they convey no meaning.</p><p>Upon examining the most frequently viewed 25 words from the comment list, stemmed words were manually converted to notable English terms. This is the result: “go”, “right”, “rape”, “get”, “make”, “attack”, “civilian”, “one”, “say”, “war”, “hostage”, “know”, “genocide”, “kill”, “think”, “would”, “even”, “Gaza”, “Israelis”, “want”, “like”, “Palestinian”, “people”, “Hamas”, “Israel”</p><p>Examining the remaining 25 words:</p><ul><li>Words such as “attack”, “civilian”, “genocide”, “hostage”, “kill”, “people”, “rape”, and “war” could indicate situations commonly associated with crimes and violations of international humanitarian law.</li><li>Nouns indicating a group of people, organisation, or country include: “Gaza”, “Hamas”, “Israel”, “Israelis/Israeli”, and “Palestinian”.</li><li>Words that do not have much meaning but still could provide some contextual perspective include: “even”, “get”, “go”, “know”, “like”, “make”, “one”, “right”, “say”, “think”, “want”, and “would”.</li></ul><p>For simplicity, words closely related to war crimes and international humanitarian violations (WCIHL) are examined as they are assumed to provide insights into the main reasons behind people’s perspectives on the conflict. These words include “attack”, “civilian”, “genocide”, “hostage”, “kill”, “people”, “rape”, and “war”. Other words in the top 25 list are not examined.</p><p>When these words are used in comments related to the Israel-Gaza conflict, they could imply various things depending on the context and the way they are used. Here are some potential implications:</p><p>“Attack”: Refers to military actions taken by one side against the other, which could range from airstrikes to ground incursions.</p><p>“Civilian”: Refers to non-combatant individuals, often implying harm or risk to innocent bystanders caught in the conflict.</p><p>“Genocide”: Implies systematic and deliberate destruction of a particular group, often associated with accusations of severe human rights violations.</p><p>“Hostage”: Suggests individuals being held against their will, possibly as bargaining chips or to exert control over a population.</p><p>“Kill”: Indicates loss of life, implying casualties resulting from the conflict, whether civilian or military.</p><p>“People”: Generally refers to the population affected by the conflict, often highlighting the human impact.</p><p>“Rape”: Implies sexual violence, which unfortunately has been documented in conflicts as a tool of intimidation and control.</p><p>“War”: Describes the overall state of conflict between the involved parties, indicating a broader context of violence and hostility.</p><p>In the context of the Israel-Gaza conflict, these words could describe aspects such as accusations of disproportionate force and violations of international law. It’s important to consider the source and context of such comments, as they may reflect differing perspectives and biases.</p><blockquote><strong><em>Note</em></strong><em>: The word “isra” is assumed to be “Israeli/Israelis” in the English format.</em></blockquote><pre>import nltk<br>from nltk.tokenize import word_tokenize<br>from nltk.corpus import stopwords<br>from nltk.stem import PorterStemmer<br><br>text = &quot;Israelis, Israel, Israeli&quot;<br>tokens = word_tokenize(text.lower())<br><br>for x in tokens:<br>    if x in string.punctuation:<br>        tokens.remove(x)<br><br>filtered_tokens = [token for token in tokens if token not in stopwords.words(&quot;english&quot;)]<br><br>Lemmatiser = PorterStemmer()<br>processed_tokens = [Lemmatiser.stem(token) for token in filtered_tokens]<br><br>text_list = [&quot;Israelis&quot;, &quot;Israel&quot;, &quot;Israeli&quot;]<br><br>result_list = []<br><br>for x in range(len(text_list)):<br>    result_list.append(f&quot;{text_list[x]} =&gt; {processed_tokens[x]}&quot;)<br><br>print(&quot;\033[1m&quot;+&quot;Word before text proprocessing&quot; + &quot;\033[0m&quot;, &quot;=&gt;&quot;, &quot;\033[1m&quot;+&quot;Word after text proprocessing&quot; + &quot;\033[0m&quot;)<br><br>for x in result_list:<br>    print(x)</pre><pre>Word before text proprocessing =&gt; Word after text proprocessing<br>Israelis =&gt; isra<br>Israel =&gt; israel<br>Israeli =&gt; isra</pre><h3>Summarisation of comments using Sumy and ChatGPT 3.5</h3><p><a href="https://docs.google.com/spreadsheets/d/e/2PACX-1vTuuZODci4TQXJnAm5f_ToPrGtSvqh-EcJ905gW4wFSQBQLk3mNxFfpw0qArN2GnI4Jzu5nIdP2WKM6/pubhtml?gid=286925799&amp;single=true">Comments containing indicators of WCIHL — before Sumy and ChatGPT 3.5</a></p><p><a href="https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vTuuZODci4TQXJnAm5f_ToPrGtSvqh-EcJ905gW4wFSQBQLk3mNxFfpw0qArN2GnI4Jzu5nIdP2WKM6/pubhtml?gid=2013321951&amp;single=true">Comments containing indicators of WCIHL — after Sumy</a></p><p>Note: Summarisation of comments by ChatGPT 3.5 is not included in this report. WCIHL is an abbreviation for war crimes and international humanitarian law.</p><h3><strong>Observations</strong></h3><p>The following shows some of the topics that were discussed in the comments:</p><p><strong>Sentiments revolving around Israel:</strong></p><p>Looking into the opinions made about Israel, the military’s actions against Palestine following the attacks by Hamas on October 7th, 2023, garnered significant criticism. The bulk of these critiques revolve around concerns for Palestinian casualties and welfare, accusations of land seizure, propagandistic narratives, and a perceived lack of effort in hostage rescue efforts.</p><p>Regarding the negative comments directed at Israel:</p><ul><li>The majority of criticisms stem from accusations that the Israeli military carrying out indiscriminate killing and genocide against Palestinian citizens e.g. (attack: comment 5), (people: comment 5), (people: comment 14), (genocide: comment 33), (rape: comment 30), (kill: comment 30), (civilian: comment 36), (civilian: comment 37), (kill: comment 39), (people: comment 93), (people: comment 107), (kill: comment 44). /</li><li>There have also been some comments that accuse Israel of land appropriation e.g. (civilian: comment 1), (war: comment 28), (hostage: comment 48), (people: comment 59), (people: comment 85), (people: comment 120). /</li><li>There have been comments that indicate people believe that Israel controls the political narratives to suit their interest, ranging from allowing October 7th attack to happen or using rape as a tool for rallying public sentiments e.g. (attack: comment 24), (attack: comment 27), (attack: comment 60), (genocide: comment 49), (kill: comment 53), (people: comment 131), (war: 25), (war: comment 29). /</li><li>Additionally, there are accusations of Israel neglecting the well-being of its hostages in favour of military pursuits e.g. (hostage: comment 4), (hostage: comment 7), (hostage: comment 8), (hostage: comment 14), (hostage: comment 24), (hostage: comment 25), (hostage: comment 26), (hostage: comment 34), (people: comment 128), (people: comment 133). /</li></ul><p>However, some comments express support for Israel’s military response:</p><ul><li>Some believe that Israel has the right to defend their borders against Hamas and Palestinian aggression e.g. (attack: comment 22), (attack: comment 43), (civilian: comment 12), (civilian: comment 22), (genocide: 14), (genocide: 25), (kill: comment 29)./</li><li>Some comments assert that Israel has not committed genocide and has conducted military operations in accordance with international laws e.g. (civilian: comment 10), (civilian: comment 11), (civilian: comment 12), (genocide: 4), (genocide: 8), (kill: comment 12), (kill: comment 21), (people: comment 16), (people: comment 56), (people: comment 84)./</li></ul><p><strong>Sentiments revolving around Hamas and Palestinians</strong></p><p>Regarding the perceptions of Hamas and the Palestinian populace:</p><ul><li>A majority of comments criticize them for exacerbating the conflict between Israel and Palestine e.g. (attack: comment 1), (attack: comment 4), (attack: comment 23), (attack: comment 25), (attack: comment 50), (attack: comment 78), (kill: comment 9), (people: comment 13), (people: comment 55). /</li><li>Some comments indicate frustrations with Hamas’s use of civilians as shields e.g. (people: comment 21), (kill: comment 67), (kill: comment 81), (people: comment 135)./</li><li>Some individuals find Hamas’s treatment of its citizens morally unacceptable, pointing to issues such as inadequate living conditions e.g. (attack: comment 4), aid misappropriation e.g. (civilian: comment 11), and blending in with civilian populations e.g. (civilian: comment 28), (people: comment 62), (people: comment 135)./</li><li>Comments also show strong hatred against Hamas and Palestinians for their involvement in war crimes against Israelians, or supporting terrorism e.g. (attack: comment 70), (attack: comment 71), (attack: comment 77), (attack: comment 75), (attack: comment 78), (civilian: comment 12), (people: comment 70), (people: comment 171), (rape: comment 48). /</li><li>There are however some comments that support Hamas and Palestinians as they view it as resistance against Israel’s oppression e.g. (attack: comment 48), (attack: comment 64), (civilian: comment 1), (civilian: comment 8), (civilian: comment 13)./</li><li>Some comments show concern that the Israel offensive against Hamas would radicalise Palestinians e.g. (attack: comment 23), (attack: comment 29), (attack: comment 30), (kill: comment 34), (kill: comment 35), (people: comment 51), (people: comment 68)./</li></ul><p><strong>Media propaganda and scepticism</strong></p><p>Several comments have shown scepticism about the accuracy of information passed on by the media.</p><ul><li>Most comments point out that the media provide information that lacks substantiation, and undermines the accuracy of the news with some claiming that its purpose is to provoke public sentiment or fuel propaganda narratives e.g. (rape: comment 6), (rape: comment 10), (rape: comment 20), (rape: comment 39), (genocide: comment 53), (people: comment 110), (people: comment 113), (people: comment 117), (people: comment 121), (people: comment 122)./</li><li>This is especially so when the news touches on the topic of rape e.g. (rape: comment 6), (rape: comment 10), (rape: comment 15), (rape: comment 16), (rape: comment 17)./</li></ul><p><strong>Political views</strong></p><p>Many people believe that the violent conflicts between Israel and Palestine are influenced by external parties. Some claim that Iran sees Israel as a threat, especially after the Israel-Saudi peace talks, and uses its relationship with Hamas to launch attacks on Israel e.g. (attack: comment 18), (attack: comment 39)./</p><p>Furthermore, some comments suggest that the international response has been ineffective in de-escalating the conflict e.g. (war: comment 2), (war: comment 84), (people: comment 98), and (people: comment 99)./</p><p><strong>Humanitarian concerns and empathy</strong></p><p>From the comments, it’s evident that most people are frustrated with plight of the civilians caught in the war between Israel and Palestine.</p><p>The comments typically discuss war crimes committed against civilians, including:</p><ul><li>The destruction of civilian livelihoods e.g. (people: comment 82), (people: comment 114), (people: comment 124), (attack: comment 4), (war: comment 9), (war: comment 10), (war: comment 11), (war: comment 26), (war: comment 56)./</li><li>The history of sexual violence perpetrated by Israel, Hamas, Palestinian citizens, and PLO (Palestine Liberation Organisation) e.g. (rape: comment 12), (rape: comment 27), (rape: comment 36), (rape: comment 47), (rape: comment 48), (rape: comment 49)./</li></ul><h3>Conclusion</h3><p>The discourse surrounding the Israel-Palestine conflict involves a wide range of concerns and viewpoints. People’s comments are often influenced by their personal beliefs, sources of information, biases, and empathy. After analysing the comments, several distinct themes emerge regarding how people perceive the Israel-Palestine conflict. These themes include individual perceptions of Israel, Palestine, the media, the international community, and humanitarian concerns.</p><p>It’s evident from the comments that the Israel-Palestine conflict has led some people to question the accuracy of media reports and has created an environment where taking sides and division are prevalent. People often choose sides based on historical references, past conflicts, and the impact of the ongoing conflict on civilians caught in the middle.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e9e857e1b375" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding the impacts of money on an individual’s mode of transportation in Singapore]]></title>
            <link>https://medium.com/@DataDreamJO/understanding-the-impacts-of-money-on-an-individuals-mode-of-transportation-in-singapore-14c88c52cd2d?source=rss-42566753e5c8------2</link>
            <guid isPermaLink="false">https://medium.com/p/14c88c52cd2d</guid>
            <category><![CDATA[projects]]></category>
            <category><![CDATA[data-visualisation]]></category>
            <category><![CDATA[transportation]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Data_Dream_J.O.]]></dc:creator>
            <pubDate>Fri, 08 Dec 2023 10:39:14 GMT</pubDate>
            <atom:updated>2023-12-08T10:39:14.358Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://trinket.io/python3/9750190e12?outputOnly=true&amp;runOption=run">Click here for the author’s name</a></h4><p>Date of creation: Sun, 17 September 2023; Last updated: Mon, 8 December 2023</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*20e7s69f5fFM_4gqYG-LDg.jpeg" /></figure><h3>Overview…</h3><p>Money is a lifeline for humans. It motivates our thoughts when purchasing goods and services for comfort or daily essentials. The type of transportation is a perfect embodiment of a product that satisfies the needs and wants of an individual.</p><p>Since the 1990s, the Singapore government has implemented measures to discourage vehicle ownership to combat vehicle overpopulation<a href="https://www.drivelah.sg/blog/demystifying-coe-everything-you-need-to-know-about-owning-a-car-in-singapore-clgw79tcq827003tpbuo1mz8gb/#:~:text=The%20Certificate%20of%20Entitlement%20(COE)%20is%20a%20permit%20that%20grants,road%20and%20control%20traffic%20congestion.">[source link]</a>. Some of these prominent measures include the ERP system(Electronic Road Pricing system), Vehicle Quota System(VQS, also unofficially known as COE Quotas), and COE(Certificate of Entitlement)<a href="https://www.channelnewsasia.com/singapore/coe-quota-policy-cars-vehicles-history-record-prices-2843996">[source link]</a>. Even after purchasing a vehicle, vehicle owners have to keep aside savings for expenses such as road tax, petrol, parking, car servicing, car loan instalments, and car insurance among others<a href="https://www.singtel.com/personal/products-services/lifestyle-services/stepup/stepupdiscover/cost-of-owning-a-car">[source link]</a>. This means that owning a vehicle is more than just scoring the COE bid, it&#39;s about having enough money to purchase and maintain it for the next ten years.</p><blockquote><em>The COE is a license that permits vehicle ownership to an individual for a decade in Singapore</em><a href="https://onemotoring.lta.gov.sg/content/onemotoring/home/buying/upfront-vehicle-costs/certificate-of-entitlement--coe-.html#:~:text=Understanding%20COE,-All%20vehicles%20in&amp;text=To%20register%20a%20vehicle%2C%20you%20must%20first%20place%20a%20bid,exercises%20conducted%20twice%20a%20month."><em>[source link]</em></a><em>.</em></blockquote><p>Therefore, only a group of Singaporeans with greater buying power can own a vehicle in Singapore. With this idea in mind, this begs the question: To what extent does a person’s financial situation affect their choice of transport?</p><p>Hence, this report aims to investigate the correlation between personal finance and transportation choices. To better comprehend this matter, we will need to understand and identify the influence of monthly household income on a person’s mode of transport. We will be looking at data from the <strong>years 2000 to 2022</strong>.</p><h3>To what extent does the monthly household income of a person influence his/her mode of transport in Singapore?</h3><p>First, we will provide an overview of the population of private transportation in Singapore, specifically cars and motorcycles, and the trends in average monthly household incomes per person, which includes the average monthly household income per person (AMHI) and average monthly household employed income per person(AMHEI) from 2012 to 2022. After we have understood this, we aim to identify any connection between the number of cars and motorcycles in Singapore and the average monthly household income per person. We hypothesise that as Singapore&#39;s average monthly household income rises, there will be a corresponding increase in the number of cars and motorcycles on the roads.</p><p><strong>Methodology</strong></p><p>For this investigation, we will gather the relevant data:</p><ul><li>Monthly household income per person in Singapore from 2012 to 2022<a href="https://tablebuilder.singstat.gov.sg/table/CT/17812">[source link]</a></li><li>Motor vehicle population in Singapore from 2012 to 2022<a href="https://www.lta.gov.sg/content/dam/ltagov/who_we_are/statistics_and_publications/statistics/pdf/MVP01-1_MVP_by_type.pdf">[source link]</a></li></ul><p>Next, we will create a combination chart called “The number of privately-owned vehicles and the average monthly household income from 2012 to 2022”.</p><p>This graph will include information from 2012 to 2022 about:</p><ul><li>The number of privately owned cars</li><li>The number of privately owned motorcycles</li><li>The average monthly household income per person(AMHI)</li><li>The average monthly household employed income(AMHEI)</li></ul><p>Linear regression will analyse the causal relationship between monthly household income per person and the number of motor vehicles (cars and motorcycles) on the roads.</p><blockquote><em>Footnotes: The resident household refers to a household where the household reference person is a Singapore citizen or permanent resident. The resident-employed household refers to a resident household with at least one employed person. The average monthly household income excludes Employer CPF Contributions </em><a href="https://tablebuilder.singstat.gov.sg/table/CT/17812"><em>[Average and Median Household Income from Work]</em></a><em>. </em><strong><em>This average only refers to the monthly income of one person</em></strong><em>. For instance, if one person in a household of four is employed, his income will be divided by four to derive the income per household member.</em></blockquote><p>Note:</p><ul><li><strong>AMHEI</strong> — Average monthly household employed income per household member among resident households</li><li><strong>AMHI</strong> — Average monthly household income per household member among resident employed</li></ul><pre>Year  Number of Cars  Number of motorcycles  AMHI  AMHEI<br>0   2012          535233                 143286  2625   2891<br>1   2013          540063                 144307  2662   2939<br>2   2014          536882                 144404  2814   3141<br>3   2015          519645                 143279  2993   3314<br>4   2016          504160                 142439  2979   3340<br>5   2017          502187                 141304  3071   3481<br>6   2018          509302                 136842  3165   3602<br>7   2019          515036                 140398  3241   3735<br>8   2020          519132                 140782  3163   3647<br>9   2021          532204                 141594  3335   3780<br>10  2022          532348                 142453  3554   4063</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dcVqaonVF2omWeHf-FOcAg.png" /></figure><blockquote><strong><em>Observations for the graph of “The number of privately-owned vehicles and monthly household income from 2012 to 2022”</em></strong><em>:</em></blockquote><ul><li>Visually, there are no relationships between the number of vehicles and the collective average monthly household income per person(AMHI, AMHEI).</li><li>As the years increase from 2012 to 2022, there is a steady incline in average monthly household income per person(AMHI, AMHEI).</li><li>As the years increase from 2012 to 2022, there aren’t any major or minor changes in the car and motorcycle population.</li></ul><p>There does not seem to be a meaningful trend based on the information presented in the graph between the number of vehicles and the average monthly household income per person (AMHI, AMHEI) in Singapore. In addition, the car population is consistently five times larger than motorcycles from 2012 to 2022. This is interesting because according to <a href="https://coe.sgcharts.com/">SGCharts</a>, motorcycles have a much lower COE than cars which means that motorcycles provide a cheaper means of travel than cars. Therefore, other factors might be involved in shaping the number of vehicles in Singapore. Now, we will use linear regression to find the casual relationships between the number of vehicles and the average monthly household income per person(AMHI, AMHEI).</p><pre>Year  Number of Cars  Number of motorcycles  \<br>Year                   1.000000       -0.227709              -0.523450   <br>Number of Cars        -0.227709        1.000000               0.608844   <br>Number of motorcycles -0.523450        0.608844               1.000000   <br>AMHI                   0.970654       -0.264328              -0.504472   <br>AMHEI                  0.979958       -0.287563              -0.540074   <br><br>                           AMHI     AMHEI  <br>Year                   0.970654  0.979958  <br>Number of Cars        -0.264328 -0.287563  <br>Number of motorcycles -0.504472 -0.540074  <br>AMHI                   1.000000  0.995932  <br>AMHEI                  0.995932  1.000000</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/653/1*poOXkWe0xcby0eRBbYhtpA.png" /></figure><pre>List of correlation coefficients:<br><br>Number of Cars against AMHI correlation coefficient(r): -0.2643279733353691<br>Number of Motorcycles against AMHI correlation coefficient(r): -0.5044722372414379<br>Number of Cars against AMHEI correlation coefficient(r): -0.28756341105865957<br>Number of Motorcycles against AMHEI correlation coefficient(r): -0.5400740915353085</pre><blockquote><strong><em>Observations for the &quot;Graphs of the number of privately-owned vehicles against AMHI/AMHEI&quot;:</em></strong></blockquote><blockquote><em>There is a weak negative linear correlation between the number of motorcycles and the average monthly household income per person (AMHI/AMHEI), which is around -0.500. Similarly, the relationship between the number of cars and the average monthly household income per person (AMHI/AMHEI) shows a weak negative linear correlation, around -0.200.</em></blockquote><blockquote><em>The relationship between the number of motorcycles and the average monthly household income per person (AMHI/AMHEI) is stronger than the relationship between the number of cars and the average monthly household income per person (AMHI/AMHEI).</em></blockquote><blockquote><em>From the graphs and statistics, it can be inferred that there is a weak negative linear relationship between the number of vehicles and the average monthly household income per person (AMHI/AMHEI).</em></blockquote><p>We employed linear regression analysis to investigate the relationship between the number of vehicles and the average monthly household incomes(AMHI, AMHEI). Our findings indicate that there exists a weak negative linear correlation between these two variables. Specifically, as the monthly household income increases, the number of privately owned vehicles(cars and motorcycles) in Singapore tends to decrease. Based on these observations, we can make two assumptions.</p><p>For one, with the rise in average monthly household incomes, stricter measures to reduce vehicle population may have led to decreased usage. On the other hand, the relationships between the average monthly household income and vehicle population in Singapore might not exist, and no definitive conclusions can be drawn. This is mainly because the correlation coefficient, which is a benchmark for indicating the reliability of a relationship between two factors, is extremely low for all four linear regression graphs.</p><p>Based on the data used in the previous graphs, it is important to investigate the correlation between the number of privately owned vehicles that are less than a year old (i.e. &lt;1yo) and the average monthly household income per person (AMHI, AMHEI). This is because the vehicle population data used in the previous graphs consists of vehicles with different age distributions. Therefore, it is possible that the average monthly household income per person would only greatly influence the number of vehicles if they are categorized in the same year. For example, a person&#39;s decision to purchase a car during the year 2012 could be greatly influenced by the average monthly household income per person in 2012 but not by the average monthly household income per person in 2018.</p><p><strong>Methodology</strong></p><p>For this investigation, we will collect data on the number of privately owned cars and motorcycles from 2012 to 2022. The data sources for this information can be found at <a href="https://www.lta.gov.sg/content/dam/ltagov/who_we_are/statistics_and_publications/statistics/pdf/MVP01-3-1_Cars_by_age.pdf">link1</a> and <a href="https://www.lta.gov.sg/content/dam/ltagov/who_we_are/statistics_and_publications/statistics/pdf/MVP01-3-2_MTC_by_age.pdf">link2</a>, respectively.</p><p>We will create a combination chart called &quot;The number of privately-owned vehicles(&lt;1yo) and the average monthly household income from 2012 to 2022&quot;.</p><p>Using this data, we will create a combination chart titled &quot;The Number of Privately-owned Vehicles (&lt;1yo) and Monthly Household Income from 2012 to 2022.&quot; This chart will include information from 2012 to 2022, specifically:</p><ul><li>The number of privately-owned cars(&lt;1yo)</li><li>The number of privately-owned motorcycles(&lt;1yo)</li><li>The average monthly household income per person(AMHI)</li><li>The average monthly household employed income(AMHEI)</li></ul><p>We will use linear regression to analyse the data and determine if there are any significant causal relationships between the monthly household income per person and the number of new vehicle ownerships (cars and motorcycles that are less than 1 year old) from 2012 to 2022.</p><p>Please note that our data will exclude tax-exempted vehicles and vehicles for off-the-road use (RU plates).</p><pre>Year  Number of Cars  Number of motorcycles  AMHI  AMHEI<br>0   2012           27297                   9852  2625   2891<br>1   2013           21952                  11571  2662   2939<br>2   2014           28547                   8068  2814   3141<br>3   2015           57387                   7354  2993   3314<br>4   2016           87157                   8178  2979   3340<br>5   2017           91614                   9451  3071   3481<br>6   2018           79961                  11943  3165   3602<br>7   2019           72090                  19246  3241   3735<br>8   2020           44162                  10463  3163   3647<br>9   2021           44985                  11415  3335   3780<br>10  2022           30168                  12015  3554   4063</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y8nrE1xgorUZp0s1sQH4_A.png" /></figure><blockquote><em>Observations for the graph of “The number of privately-owned vehicles(&lt;1yo) and the average monthly household income from 2012 to 2022”:</em></blockquote><ul><li>Between 2012 and 2017, there was a noticeable increase in the average monthly household income per person (AMHI, AMHEI), which coincided with a sharp rise in new car ownerships. However, between 2017 and 2022, despite a continuous increase in the average monthly household income per person (AMHI, AMHEI), there was a significant drop in new car ownership.</li><li>There was a steady increase in motorcycle ownership from 2015 to 2019 as the average monthly household income per person (AMHI, AMHEI) increased. However, there was a slight decline in motorcycle ownership from 2013 to 2015 and from 2019 to 2022 despite the steady growth in the average monthly household income per person (AMHI, AMHEI).</li><li>The highest number of new car ownerships was in 2017, with a total of 91614 cars, while the greatest number of new motorcycle ownerships was in 2019, with 19246 motorcycles.</li></ul><pre>Year  Number of Cars  Number of motorcycles  \<br>Year                   1.000000        0.205183               0.419855   <br>Number of Cars         0.205183        1.000000               0.087316   <br>Number of motorcycles  0.419855        0.087316               1.000000   <br>AMHI                   0.970654        0.259106               0.395508   <br>AMHEI                  0.979958        0.274340               0.438719   <br><br>                           AMHI     AMHEI  <br>Year                   0.970654  0.979958  <br>Number of Cars         0.259106  0.274340  <br>Number of motorcycles  0.395508  0.438719  <br>AMHI                   1.000000  0.995932  <br>AMHEI                  0.995932  1.000000</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/693/1*-GO-NNQ2YCZgqoVTDcwAJA.png" /></figure><pre>List of correlation coefficients:<br><br>Number of Cars against AMHI correlation coefficient(r): 0.25910647653893676<br>Number of Motorcycles against AMHI correlation coefficient(r): 0.3955079933287862<br>Number of Cars against AMHEI correlation coefficient(r): 0.27433991495140697<br>Number of Motorcycles against AMHEI correlation coefficient(r): 0.4387186116869273</pre><blockquote><strong><em>Observations for the &quot;Graphs of the number of privately-owned vehicles(&lt;1yo) against AMHI/AMHEI&quot;:</em></strong></blockquote><ul><li>There is a weak positive linear correlation (around 0.400) between the number of motorcycles and the average monthly household income per person (AMHI/AMHEI). Similarly, there is a weak positive linear correlation (around 0.250) between the number of cars and the average monthly household income per person (AMHI/AMHEI).</li><li>The relationship between the number of motorcycles and the average monthly household income per person (AMHI/AMHEI) is stronger than the relationship between the number of cars and the average monthly household income per person (AMHI/AMHEI).</li><li>From the acquired graphs and statistics, it can be inferred that an increase in average monthly household income per person leads to a corresponding increase in the number of vehicle ownership, as all four graphs depict weak, positive linear relationships.</li></ul><p>After analysing the data for &quot;The number of privately-owned vehicles (&lt;1yo) and monthly household income per person from 2012 to 2022,&quot; the results indicate a weak connection between the average household income per person (AMHI/AMHEI) and the number of new vehicle ownership between 2012 to 2022. Therefore, we cannot draw any meaningful conclusions from the data we have analysed so far. It&#39;s worth noting that as the average monthly household income per person increases, the number of vehicle owners also increases. However, since the correlation coefficient is low, the observations derived from the linear graphs remain inconclusive.</p><p>Further research has revealed that the COE quota system is responsible for controlling the vehicle population in Singapore. It does so by imposing restrictions on the number of newly registered vehicles each year. This is also why the number of cars on the roads in Singapore has always been five times higher than the number of motorcycles. This is supported by evidence provided through the following links: <a href="https://www.valuechampion.sg/average-cost-coe-car-ownership-sg">link1</a>, <a href="https://coe.sgcharts.com/">link2</a> and <a href="https://www.lta.gov.sg/content/dam/ltagov/who_we_are/statistics_and_publications/statistics/pdf/COE_Result_2015_2019.pdf">link3</a>. Therefore, the average monthly household income of a person does not have any impact on the vehicle population and the number of newly registered vehicles in Singapore every year.</p><p>Since we cannot assume that the average monthly household income per person has an impact on the number of vehicles on the road, we will have to investigate if there is a correlation between monthly household income and transportation choices, we will analyse data from the years 2000, 2005, 2010, 2015, and 2020. We will examine the difference in monthly household income reported per person and see if it affects their choice of transportation.</p><p><strong>Methodology</strong></p><p>We will be analysing the transportation methods of resident working persons aged 15 years and over, using data obtained through Singstats: Mode of Transport<a href="https://www.singstat.gov.sg/find-data/search-by-theme/population/mode-of-transport/publications-and-methodology">[source link]</a>.</p><p>To investigate how the monthly household income affects people&#39;s transportation methods, we will categorise the mode of transport into five categories: Privately-owned cars Privately-owned motorcycles (including scooters) Public transport (MRT, LRT, Bus) Taxis/private hire Walking.</p><p>We will create grouped bar charts based on data from 2000, 2005, 2010, 2015, and 2020. The bar chart will contain the following information:</p><ul><li>The distribution of monthly household income</li><li>The different modes of transport.</li><li>The proportion of users who use different modes of transportation.</li></ul><p>To calculate the proportion of vehicle users for each monthly household income, we will use this formula:</p><ul><li>The proportion of users for a type of vehicle(%) = Number of users for a type of vehicle / Total number of vehicle users(privately-owned cars, privately-owned motorcycles(including scooters), public transport(MRT, LRT, Bus), Taxis/private hire, Walking) x 100%</li></ul><p>Please note:</p><ul><li>The monthly household income from work includes employer CPF contributions.</li><li>The data from the years 2010, 2015, and 2020 do not include Singaporeans residing in Singapore who were working overseas for at least 6 months.</li><li>Data about private hires was only collected in the year 2020. The years 2000, 2005,2010, and 2015 do not contain any data about private hires.</li></ul><p><strong>Data from year 2000, 2005, 2010, 2015, 2020:</strong></p><pre>Data from year 2000<br><br>             Mode of transport Below $1,000 $1,000 - $1,999 $2,000 - $2,999  \<br>0         Privately-owned cars        4.07%           8.88%          14.62%   <br>1  Privately-owned motorcycles        4.06%           7.93%           8.79%   <br>2            Public transport        71.22%          65.80%          64.35%   <br>3                      Walking       19.74%          16.54%          11.27%   <br>4           Taxis/private hire        0.92%           0.84%           0.97%   <br><br>  $3,000 - $3,999 $4,000 - $4,999 $5,000 - $5,999 $6,000 - $6,999  \<br>0          18.50%          20.66%          24.19%          28.54%   <br>1           8.28%           7.43%           6.40%           5.85%   <br>2          63.17%          63.29%          62.05%          58.68%   <br>3           8.87%           7.44%           6.09%           5.25%   <br>4           1.18%           1.18%           1.26%           1.68%   <br><br>  $7,000 - $7,999 $8,000 &amp; Over  <br>0          30.71%        48.96%  <br>1           4.72%         2.42%  <br>2          58.32%        43.18%  <br>3           4.62%         3.30%  <br>4           1.63%         2.14%  <br><br><br>Data from year 2005<br><br>             Mode of transport Below $1,000 $1,000 - $1,999 $2,000 - $2,999  \<br>0         Privately-owned cars        5.23%           6.00%          11.28%   <br>1  Privately-owned motorcycles        4.05%           6.99%           8.14%   <br>2            Public transport        68.70%          67.79%          66.47%   <br>3                      Walking       21.73%          18.92%          13.62%   <br>4           Taxis/private hire        0.29%           0.30%           0.49%   <br><br>  $3,000 - $3,999 $4,000 - $4,999 $5,000 - $5,999 $6,000 - $6,999  \<br>0          14.95%          18.30%          21.34%          25.74%   <br>1           8.17%           8.04%           7.08%           5.63%   <br>2          65.12%          63.51%          61.92%          58.74%   <br>3          10.86%           9.11%           8.25%           8.20%   <br>4           0.90%           1.04%           1.41%           1.69%   <br><br>  $7,000 - $7,999 $8,000 &amp; Over  <br>0          28.72%        47.35%  <br>1           5.37%         2.53%  <br>2          57.69%        42.42%  <br>3           6.56%         4.90%  <br>4           1.66%         2.81%  <br><br><br>Data from year 2010<br><br>             Mode of transport Below $1,000 $1,000 - $1,999 $2,000 - $2,999  \<br>0         Privately-owned cars        5.25%           5.73%          10.08%   <br>1  Privately-owned motorcycles        2.96%           4.97%           6.25%   <br>2            Public transport        68.33%          67.17%          68.59%   <br>3                      Walking       22.68%          21.68%          14.38%   <br>4           Taxis/private hire        0.78%           0.45%           0.70%   <br><br>  $3,000 - $3,999 $4,000 - $4,999 $5,000 - $5,999 $6,000 - $6,999  \<br>0          13.25%          16.04%          20.19%          22.11%   <br>1           6.41%           6.68%           6.64%           5.96%   <br>2          66.98%          65.46%          63.05%          62.36%   <br>3          12.38%          10.74%           9.16%           8.13%   <br>4           0.97%           1.08%           0.95%           1.44%   <br><br>  $7,000 - $7,999 $8,000 &amp; Over  <br>0          24.83%        42.87%  <br>1           5.40%         2.85%  <br>2          60.96%        46.96%  <br>3           7.44%         5.18%  <br>4           1.37%         2.13%  <br><br><br>Data from year 2015<br><br>            Mode of transport  Below $1,000 $1,000 - $1,999 $2,000 - $2,999  \<br>0         Privately-owned cars        5.17%           5.11%           7.90%   <br>1  Privately-owned motorcycles        2.59%           4.12%           4.37%   <br>2            Public transport        68.10%          65.77%          66.95%   <br>3                      Walking       23.71%          24.01%          19.56%   <br>4           Taxis/private hire        0.43%           0.99%           1.22%   <br><br>  $3,000 - $3,999 $4,000 - $4,999 $5,000 - $5,999 $6,000 - $6,999  \<br>0           9.81%          12.16%          13.68%          15.27%   <br>1           4.90%           5.69%           5.55%           4.78%   <br>2          70.47%          67.57%          66.91%          67.87%   <br>3          13.57%          13.71%          12.20%          10.74%   <br>4           1.25%           0.87%           1.66%           1.34%   <br><br>  $7,000 - $7,999 $8,000 &amp; Over  <br>0          16.91%        32.37%  <br>1           4.92%         3.25%  <br>2          66.87%        55.97%  <br>3           9.92%         6.79%  <br>4           1.38%         1.62%  <br><br><br>Data from year 2020<br><br>             Mode of transport Below $1,000 $1,000 - $1,999 $2,000 - $2,999  \<br>0         Privately-owned cars        5.68%           5.80%           8.45%   <br>1  Privately-owned motorcycles        1.88%           2.37%           3.64%   <br>2            Public transport        57.93%          64.10%          64.67%   <br>3                      Walking       33.09%          26.38%          21.81%   <br>4           Taxis/private hire        1.42%           1.35%           1.43%   <br><br>  $3,000 - $3,999 $4,000 - $4,999 $5,000 - $5,999 $6,000 - $6,999  \<br>0           9.25%           9.56%          12.43%          12.80%   <br>1           4.52%           5.42%           5.18%           6.06%   <br>2          67.24%          67.01%          66.26%          66.40%   <br>3          16.94%          15.16%          13.62%          12.50%   <br>4           2.04%           2.86%           2.50%           2.24%   <br><br>  $7,000 - $7,999 $8,000 &amp; Over  <br>0          14.78%        29.53%  <br>1           5.86%         3.16%  <br>2          64.73%        55.84%  <br>3          11.78%         7.65%  <br>4           2.85%         3.82%</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EPlxIOwWiHVJHf0orGrqGw.png" /></figure><p><strong>Summary for Privately-owned cars:</strong></p><p>Singaporeans with a monthly income per person of $8,000 and over usually use privately owned cars as a means of daily transport(48.95880707%).</p><p><strong>Summary for Privately-owned motorcycles:</strong></p><p>Singaporeans with a monthly income per person of $2,000 — $2,999 usually use privately owned motorcycles as a means of daily transport(8.788109177%).</p><p><strong>Summary for Public Transport:</strong></p><p>Singaporeans with a monthly income per person of Below $1,000 usually use Public transport as a means of daily transport(71.21558385%).</p><p><strong>Summary for Walking:</strong></p><p>Singaporeans with a monthly income per person of Below $1,000 usually use Walking as a means of daily transport(19.74099467%).</p><p><strong>Summary for Taxis/private hire:</strong></p><p>Singaporeans with a monthly income per person of $8,000 and over usually use Taxis/private hire as a means of daily transport(2.144953901%).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u4g4E0Ak6k9XCQbrHh8NaQ.png" /></figure><p><strong>Summary for Privately-owned cars:</strong></p><p>Singaporeans with a monthly income per person of $8,000 and over usually use privately owned cars as a means of daily transport(47.34631132%).</p><p><strong>Summary for Privately-owned motorcycles:</strong></p><p>Singaporeans with a monthly income per person of $3,000 — $3,999 usually use privately owned motorcycles as a means of daily transport(8.171797534%).</p><p><strong>Summary for Public Transport:</strong></p><p>Singaporeans with a monthly income per person of Below $1,000 usually use Public transport as a means of daily transport(68.70095479%).</p><p><strong>Summary for Walking:</strong></p><p>Singaporeans with a monthly income per person of Below $1,000 usually use Walking as a means of daily transport(21.729306870000002%).</p><p><strong>Summary for Taxis/private hire:</strong></p><p>Singaporeans with a monthly income per person of $8,000 and over usually use Taxis/private hire as a means of daily transport(2.80709524%).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mN2sd03ZksK1p6_2qENDAw.png" /></figure><p><strong>Summary for Privately-owned cars:</strong></p><p>Singaporeans with a monthly income per person of $8,000 and over usually use privately owned cars as a means of daily transport(42.86998683%).</p><p><strong>Summary for Privately-owned motorcycles:</strong></p><p>Singaporeans with a monthly income per person of $4,000 — $4,999 usually use privately owned motorcycles as a means of daily transport(6.678653794000001%).</p><p><strong>Summary for Public Transport:</strong></p><p>Singaporeans with a monthly income per person of $2,000 — $2,999 usually use Public transport as a means of daily transport(68.58584613000001%).</p><p><strong>Summary for Walking:</strong></p><p>Singaporeans with a monthly income per person of Below $1,000 usually use Walking as a means of daily transport(22.68482594%).</p><p><strong>Summary for Taxis/private hire:</strong></p><p>Singaporeans with a monthly income per person of $8,000 and over usually use Taxis/private hire as a means of daily transport(2.1338114519999998%).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F2D1C1QBTghgRHxLdW39hQ.png" /></figure><p><strong>Summary for Privately-owned cars:</strong></p><p>Singaporeans with a monthly income per person of $8,000 and over usually use privately owned cars as a means of daily transport(32.37422177%).</p><p><strong>Summary for Privately-owned motorcycles:</strong></p><p>Singaporeans with a monthly income per person of $4,000 — $4,999 usually use privately owned motorcycles as a means of daily transport(5.694980695%).</p><p><strong>Summary for Public Transport:</strong></p><p>Singaporeans with a monthly income per person of $3,000 — $3,999 usually use Public transport as a means of daily transport(70.46750285%).</p><p><strong>Summary for Walking:</strong></p><p>Singaporeans with a monthly income per person of $1,000 — $1,999 usually use Walking as a means of daily transport(24.00568182%).</p><p><strong>Summary for Taxis/private hire:</strong></p><p>Singaporeans with a monthly income per person of $5,000 — $5,999 usually use Taxis/private hire as a means of daily transport(1.663585952%).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MduB2NH-6kQl9Mha1gh1ng.png" /></figure><p><strong>Summary for Privately-owned cars:</strong></p><p>Singaporeans with a monthly income per person of $8,000 and over usually use privately owned cars as a means of daily transport(29.530216120000002%).</p><p><strong>Summary for Privately-owned motorcycles:</strong></p><p>Singaporeans with a monthly income per person of $6,000 — $6,999 usually use privately owned motorcycles as a means of daily transport(6.055035219%).</p><p><strong>Summary for Public Transport:</strong></p><p>Singaporeans with a monthly income per person of $3,000 — $3,999 usually use Public transport as a means of daily transport(67.24361264999999%).</p><p><strong>Summary for Walking:</strong></p><p>Singaporeans with a monthly income per person of Below $1,000 usually use Walking as a means of daily transport(33.09014907%).</p><p><strong>Summary for Taxis/private hire:</strong></p><p>Singaporeans with a monthly income per person of $8,000 and over usually use Taxis/private hire as a means of daily transport(3.819623772%).</p><p>We will now create a program that summarises the most frequent users of each type of transport from the collective years of 2000, 2005, 2010, 2015, and 2020.</p><pre>Most frequent user for each mode of transport:<br><br>Based on data collected in the years 2000, 2005, 2010, 2015, and 2020, it appears that people living in Singapore <br>with a monthly household income of [&#39;$8,000 &amp; Over&#39;] tend to use <br>Privately-owned cars as their preferred mode of transportation more frequently than other household income groups.<br><br><br>Based on data collected in the years 2000, 2005, 2010, 2015, and 2020, it appears that people living in Singapore <br>with a monthly household income of [&#39;$4,000 - $4,999&#39;] tend to use <br>Privately-owned motorcycles as their preferred mode of transportation more frequently than other household income groups.<br><br><br>Based on data collected in the years 2000, 2005, 2010, 2015, and 2020, it appears that people living in Singapore <br>with a monthly household income of [&#39;Below $1,000&#39;, &#39;$3,000 - $3,999&#39;] tend to use <br>Public transport  as their preferred mode of transportation more frequently than other household income groups.<br><br><br>Based on data collected in the years 2000, 2005, 2010, 2015, and 2020, it appears that people living in Singapore <br>with a monthly household income of [&#39;Below $1,000&#39;] tend to use <br>Walking as their preferred mode of transportation more frequently than other household income groups.<br><br><br>Based on data collected in the years 2000, 2005, 2010, 2015, and 2020, it appears that people living in Singapore <br>with a monthly household income of [&#39;$8,000 &amp; Over&#39;] tend to use <br>Taxis/private hire as their preferred mode of transportation more frequently than other household income groups.</pre><p><strong>Observations</strong>:</p><p>Based on the data from the <strong>years 2000, 2005,2010,2015,2020:</strong></p><p>It has been observed that individuals with a monthly household income of <strong>$8,000 or more</strong> tend to use privately owned<strong> cars, taxis, and private hires</strong> more frequently compared to those with other monthly household incomes.</p><p>Those earning a monthly household income of <strong>$4,000-4,999</strong> use <strong>privately-owned motorcycles</strong> the most.</p><p>It has also been found that individuals with a reported monthly household income of <strong>below $1,000 or $3,000 - 3,999</strong> are the most frequent users of <strong>public transport</strong> compared to other users of different monthly household incomes.</p><p>It has been found that people with a monthly household income <strong>below $1,000</strong> are more likely to use <strong>walking</strong> as their mode of transport compared to those with higher monthly household incomes.</p><p><strong>Using visual aids from grouped bar chats from the years from 2000 to 2020:</strong></p><p><strong>Privately-owned cars</strong>:</p><p>The bar chart, spanning the years 2000 to 2020, consistently shows a negative skew in the data distribution. This suggests that as the monthly household income of a person increases, the more likely they will own a car as their preferred mode of transport. In simpler terms, people from more affluent backgrounds are more likely to own a privately owned car.</p><p><strong>Privately-owned motorcycles</strong>:</p><p>In the year 2000, the data distribution was slightly positively skewed. As the years progressed from 2000 to 2010, the data distribution gradually shifted to form a symmetrical distribution. However, from 2010 to 2020, the data distribution gradually shifted again, this time becoming positively skewed.</p><p>To put it simply, in the year 2000, people with lower monthly household income predominantly used privately owned motorcycles. Between 2000 and 2010, motorcycle ownership became increasingly popular among people with moderate monthly household income. From 2010 to 2020, individuals with higher monthly household income were more likely to purchase and use motorcycles. However, this trend is not observed among those who earn $8,000 or more.</p><p><strong>Public transport</strong>:</p><p>Based on the graphs spanning the years 2000 to 2020, it is evident that public transport is the most commonly used mode of transportation, with over 50% of people using it, regardless of their monthly household income.</p><p>The bar chart data distribution from 2000 to 2010 was positively skewed, while it became symmetrical from 2010 to 2020. This implies that between 2000 and 2010, individuals with higher monthly household incomes were less likely to use public transport.</p><p>In 2015, people with monthly household incomes ranging from below $1,000 to $7,999 showed similar interest in public transport, while those earning $8,000 and above were less likely to use it. By 2020, the majority of public transport users were those earning between $2,000 and $7,999 per month.</p><p>To summarize, over the years from 2000 to 2020, public transport became increasingly popular among individuals with moderate monthly household incomes.</p><p><strong>Walking</strong>:</p><p>From 2000 to 2020, the data consistently shows a positive skew, indicating that people with higher monthly household incomes are less likely to use walking as their primary mode of transportation.</p><p><strong>Taxis/private hire</strong>:</p><p>Based on the graphs spanning from the year 2000 to 2020, it is evident that less than 10% of people across all monthly household income brackets prefer using taxis or private hires. This makes taxis or private hires the least popular choice for transportation.</p><p>Furthermore, the data from 2000 to 2020 is consistently negatively skewed. In 2020, a larger proportion of people regardless of monthly household income opted for taxis or private hires.</p><p>This implies that as the monthly household income of an individual increases, their preference for taxis/private hires also increases. Moreover, the slight rise in the percentage of people using taxis or private hires in 2020 may indicate that this mode of transport is becoming more popular. Alternatively, the inclusion of private hires data in 2020 could have boosted the proportion of people using either of these two services.</p><p><strong>...</strong></p><p>Based on the data collected, it can be inferred that a person&#39;s mode of transportation in Singapore is heavily influenced by their monthly household income. However, just by looking at the graphs, there remains an unanswered question: Why is there a change in the preference for motorcycles and public transportation among those who earn a moderate to high monthly household income?</p><p>To answer this question, we will need to determine if the proportion of privately owned motorcycles and public transport users in Singapore is closely related to the changes in the Consumer Price Index (CPI) from 2000 to 2022. This is because since CPI measures the change in consumers&#39; purchasing power, it is often a strong key factor that influences an individual&#39;s economic decisions <a href="https://www.investopedia.com/terms/c/consumerpriceindex.asp#:~:text=Annual%20CPI%20Formula&amp;text=The%20annual%20CPI%20is%20reported,current%20market%20prices%20are%20appreciating">[source link]</a>. Hence, using currently available data and data about the CPI from 2000 to 2022, we will see if the proportion of motorcycles and public transport users over the years from 2000 to 2020 has a close relation to the CPI changes in Singapore. If no visible relationships are found between the CPI and the proportion of vehicle users(privately-owned motorcycles, public transport), we will use text analysis methods to understand the general sentiment about these modes of transportation among the people of Singapore.</p><h3>Analysing the relationship between motorcycle and public transport users and the CPI changes in Singapore between the years 2000 to 2020</h3><p><strong>Methodology</strong></p><p>As part of this investigation, we will gather the following data:</p><ul><li>Changes in CPI in Singapore between the years 2000 and 2020 <a href="https://www.singstat.gov.sg/whats-new/latest-news/cpi-highlights">[source link]</a></li><li>Proportions of motorcycle and public transport users based on monthly household income of less than $1,000 to $8,000 and above between the years 2000 and 2020 <a href="https://www.singstat.gov.sg/find-data/search-by-theme/population/mode-of-transport/publications-and-methodology">[source link]</a></li></ul><p>We will create two types of graphs for each monthly household income:</p><ul><li>A graph of the proportion of motorcycle users against the CPI</li><li>A graph of the proportion of public transport users against the CPI</li></ul><p>These graphs will help us determine the strength of the connection between owning a motorcycle or using public transport and the CPI. By analyzing this information, we can determine if the increase in change in prices paid by consumers for goods and services is responsible for the changes in transportation types.</p><p>Please note:</p><ul><li>We are using 2019 as the base year for the Consumer Price Index (CPI).</li></ul><p><strong>Annual data of the consumer price index(CPI):</strong></p><pre>                                    Variables           2000           2001  \<br>0                                         CPI         74.435          75.19   <br>1     CPI changes from previous year(decimal)  0.01347947444  0.01014307785   <br>2  CPI changes from previous year(percentage)          1.35%          1.01%   <br><br>              2002            2003           2004            2005  \<br>0           74.896          75.261         76.518          76.877   <br>1  -0.003910094427  0.004873424482  0.01670187747  0.004691706527   <br>2           -0.39%           0.49%          1.67%           0.47%   <br><br>             2006          2007           2008  ...           2013  \<br>0          77.617        79.251         84.504  ...         98.474   <br>1  0.009625765834  0.0210520891  0.06628307529  ...  0.02358505275   <br>2           0.96%         2.11%          6.63%  ...          2.36%   <br><br>            2014             2015             2016            2017  \<br>0         99.483           98.963           98.436          99.004   <br>1  0.01024635945  -0.005227023713  -0.005325222558  0.005770246658   <br>2          1.02%           -0.52%           -0.53%           0.58%   <br><br>             2018            2019      2020           2021           2022  <br>0          99.438             100    99.818        102.119         108.37  <br>1  0.004383661266  0.005651762908  -0.00182  0.02305195456  0.06121289868  <br>2           0.44%           0.57%    -0.18%          2.31%          6.12%  <br><br>[3 rows x 24 columns]</pre><pre>years      CPI_changes      CPI<br>0   2000    0.01347947444   74.435<br>1   2001    0.01014307785    75.19<br>2   2002  -0.003910094427   74.896<br>3   2003   0.004873424482   75.261<br>4   2004    0.01670187747   76.518<br>5   2005   0.004691706527   76.877<br>6   2006   0.009625765834   77.617<br>7   2007     0.0210520891   79.251<br>8   2008    0.06628307529   84.504<br>9   2009   0.005964214712   85.008<br>10  2010    0.02823263693   87.408<br>11  2011    0.05247803405   91.995<br>12  2012     0.0457633567   96.205<br>13  2013    0.02358505275   98.474<br>14  2014    0.01024635945   99.483<br>15  2015  -0.005227023713   98.963<br>16  2016  -0.005325222558   98.436<br>17  2017   0.005770246658   99.004<br>18  2018   0.004383661266   99.438<br>19  2019   0.005651762908      100<br>20  2020         -0.00182   99.818<br>21  2021    0.02305195456  102.119<br>22  2022    0.06121289868   108.37</pre><pre>Data for privately-owned motorcycles:<br><br>   years  Below $1,000  $1,000 - $1,999  $2,000 - $2,999  $3,000 - $3,999  \<br>0   2000      0.040592         0.079343         0.087881         0.082789   <br>1   2005      0.040475         0.069862         0.081355         0.081718   <br>2   2010      0.029594         0.049728         0.062451         0.064136   <br>3   2015      0.025862         0.041193         0.043742         0.049031   <br>4   2020      0.018811         0.023724         0.036350         0.045169   <br><br>   $4,000 - $4,999  $5,000 - $5,999  $6,000 - $6,999  $7,000 - $7,999  \<br>0         0.074323         0.064017         0.058479         0.047187   <br>1         0.080415         0.070787         0.056344         0.053683   <br>2         0.066787         0.066422         0.059577         0.053956   <br>3         0.056950         0.055453         0.047819         0.049180   <br>4         0.054169         0.051838         0.060550         0.058640   <br><br>   $8,000 &amp; Over  <br>0       0.024203  <br>1       0.025256  <br>2       0.028531  <br>3       0.032475  <br>4       0.031613  <br><br>Data for public transport:<br><br>   years  Below $1,000  $1,000 - $1,999  $2,000 - $2,999  $3,000 - $3,999  \<br>0   2000      0.712156         0.658013         0.643511         0.631726   <br>1   2005      0.687010         0.677919         0.664685         0.651186   <br>2   2010      0.683274         0.671653         0.685858         0.669811   <br>3   2015      0.681034         0.657670         0.669502         0.704675   <br>4   2020      0.579265         0.640987         0.646743         0.672436   <br><br>   $4,000 - $4,999  $5,000 - $5,999  $6,000 - $6,999  $7,000 - $7,999  \<br>0         0.632878         0.620550         0.586834         0.583196   <br>1         0.635110         0.619195         0.587371         0.576875   <br>2         0.654599         0.630517         0.623573         0.609627   <br>3         0.675676         0.669131         0.678691         0.668680   <br>4         0.670065         0.662575         0.664022         0.647349   <br><br>   $8,000 &amp; Over  <br>0       0.431775  <br>1       0.424230  <br>2       0.469590  <br>3       0.559650  <br>4       0.558394</pre><p>After we have created the CSV files Motorcycle data.csv, PT data.csv, and CPI data.csv, we will now proceed to plot and visualise the graphs and their casual relationships.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EFNA6vGkY_lvFinjlwB2dQ.png" /></figure><p><strong>Observations:</strong> The changes in CPI in percentages vary a lot over the years. From the graph, it can be deduced that the maximum change in CPI is around 6% and the lowest is around -0.1. This change in CPI does not show any visible trends. The CPI on the other hand shows a slow but steady increase over the years.</p><p><strong>Correlation coefficients — Motorcycle users, CPI, and CPI changes</strong></p><pre>                  CPI_changes      CPI<br>years              -0.476965  0.967767<br>CPI_changes         1.000000 -0.486160<br>CPI                -0.486160  1.000000<br>Below $1,000        0.354683 -0.964563<br>$1,000 - $1,999     0.389234 -0.960690<br>$2,000 - $2,999     0.479001 -0.995423<br>$3,000 - $3,999     0.454222 -0.997180<br>$4,000 - $4,999     0.417054 -0.962542<br>$5,000 - $5,999     0.610870 -0.857028<br>$6,000 - $6,999     0.539797 -0.303553<br>$7,000 - $7,999    -0.082705  0.441609<br>$8,000 &amp; Over      -0.481272  0.993508</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*glQQK3jg4qyeMvFzxgqL4A.png" /></figure><pre>List of correlation coefficients:<br><br>For the graph titled Below $1,000, the correlation coefficient for the linear regression is -0.96456285151768<br>For the graph titled $1,000 - $1,999, the correlation coefficient for the linear regression is -0.9606901363616943<br>For the graph titled $2,000 - $2,999, the correlation coefficient for the linear regression is -0.9954227414838254<br>For the graph titled $3,000 - $3,999, the correlation coefficient for the linear regression is -0.9971800330611993<br>For the graph titled $4,000 - $4,999, the correlation coefficient for the linear regression is -0.9625415758469829<br>For the graph titled $5,000 - $5,999, the correlation coefficient for the linear regression is -0.857027574440089<br>For the graph titled $6,000 - $6,999, the correlation coefficient for the linear regression is -0.3035525410108271<br>For the graph titled $7,000 - $7,999, the correlation coefficient for the linear regression is 0.4416085972904689<br>For the graph titled $8,000 &amp; Over, the correlation coefficient for the linear regression is 0.9935077502477125</pre><blockquote><strong><em>Observations:</em></strong></blockquote><ul><li>The correlation coefficients for all subplots except for the graphs titled &quot;$6,000-6,999&quot; and &quot;$7,000-7,999&quot; indicate strong linear relationships between the proportion of privately-owned motorcycle users and the consumer price index (CPI) from 2000 to 2020. Therefore, the connection between the proportion of privately-owned motorcycle users and CPI over these years is highly reliable.</li><li>The two graphs labelled &quot;$6,000-6,999&quot; and &quot;$7,000-7,999&quot; display a weak negative linear relationship and a weak positive linear relationship, respectively. As a result, there is no reliable connection between the proportion of privately-owned motorcycle users and CPI from the years 2000 to 2020.</li><li>There is a decreasing trend in motorcycle ownership among households with a monthly household income of below $1,000 to $6,999 as the CPI increases from 2000 to 2020.</li><li>From 2000 to 2020, motorcycle ownership shows an increasing trend among households with a monthly household income of at least $7,000 as the CPI rises.</li></ul><p>For the graph labelled &quot;$6,000-6,999&quot;, there is an outlier present in the 2015 data. After the removal of the outlier, we have found that the graph shows a very strong positive linear relationship. This indicates that from 2000 to 2020, there was an increasing trend of motorcycle ownership among households with a monthly household income of at least $6,000 as the CPI rises.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/1*YSf24K7I6ML6c8wLnKl3Ug.png" /><figcaption>For the edited graph titled “$6,000 — $6,999”, the correlation coefficient for the linear regression is 0.8135006865975575 after the removal of the outlier from 2015.</figcaption></figure><p><strong>Correlation coefficients — Public Transport users, CPI, and CPI changes</strong></p><pre>                  CPI_changes      CPI<br>years              -0.476965  0.967767<br>CPI_changes         1.000000 -0.486160<br>CPI                -0.486160  1.000000<br>Below $1,000        0.429601 -0.703805<br>$1,000 - $1,999     0.499098 -0.625500<br>$2,000 - $2,999     0.453355  0.131643<br>$3,000 - $3,999    -0.436520  0.872588<br>$4,000 - $4,999    -0.462232  0.990956<br>$5,000 - $5,999    -0.655117  0.961715<br>$6,000 - $6,999    -0.527422  0.985283<br>$7,000 - $7,999    -0.550601  0.963545<br>$8,000 &amp; Over      -0.590645  0.980797</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/1*W5uT95uQ-6R6MOpV5dfKMw.png" /></figure><pre>List of correlation coefficients:<br><br>For the graph titled Below $1,000, the correlation coefficient for the linear regression is -0.7038045873289007<br>For the graph titled $1,000 - $1,999, the correlation coefficient for the linear regression is -0.6255002703765156<br>For the graph titled $2,000 - $2,999, the correlation coefficient for the linear regression is 0.1316431415383064<br>For the graph titled $3,000 - $3,999, the correlation coefficient for the linear regression is 0.8725879322067612<br>For the graph titled $4,000 - $4,999, the correlation coefficient for the linear regression is 0.9909560416076708<br>For the graph titled $5,000 - $5,999, the correlation coefficient for the linear regression is 0.9617145729597453<br>For the graph titled $6,000 - $6,999, the correlation coefficient for the linear regression is 0.9852831833213392<br>For the graph titled $7,000 - $7,999, the correlation coefficient for the linear regression is 0.9635448602316137<br>For the graph titled $8,000 &amp; Over, the correlation coefficient for the linear regression is 0.9807970552698716</pre><blockquote><strong><em>Observations:</em></strong></blockquote><ul><li>The correlation coefficients for all subplots, except for the graphs titled “Below $1,000”, “$1,000–1,999”, and “$2,000–2,999”, show a very strong linear relationship between the proportion of public transport users and the consumer price index (CPI) from the years 2000 to 2020. This indicates that the link between the proportion of public transport users and CPI during this period is highly reliable.</li><li>The two graphs titled &quot;Below $1,000&quot; and &quot;1,000−1,000−1,999&quot; show a strong positive linear relationship. Therefore, there is a reliable connection between the proportion of public transport users and CPI from the years 2000 to 2020.</li><li>The graph titled &quot;$2,000-2,999&quot; shows a weak negative linear relationship. Therefore, the connection between the proportion of public transport users and CPI from the years 2000 to 2020 is unreliable.</li><li>People with a monthly household income of below $1,000, and $1,000-1,999 show a declining trend in public transport usage as CPI increases from 2000 to 2020.</li><li>As the CPI increased from 2000 to 2020, the use of public transport increased among households earning at least $3,000 monthly.</li><li>It is not possible to draw accurate observations about individuals with a monthly household income of $2,000-2,999 as the correlation between the rising CPI and the percentage of public transport users from 2000 to 2020 is highly unreliable and does not provide any useful information to explain the cause-effect relationship.</li></ul><p>In conclusion, it is found that the increasing consumer price index over the years from 2000 to 2020 has a strong influence on motorcycle and public transport usage by people from different monthly household income backgrounds.</p><p>For motorcycle users, it is found that as the CPI increases over the years, there is a decrease in the proportion of motorcycle users with a monthly household income of less than $6,999. On the other hand, people with a monthly household income of more than or equal to $7,000 show an increase in motorcycle ownership over the years.</p><p>It is also worth noting that the increase in CPI has also encouraged more people from different financial backgrounds to utilise public transport. However, the exception can be found for those earning less than or equal to $1,999. From the graphs, we can deduce that the rising cost of living has discouraged those earning below $1,999 from using public transport. Therefore, it is a question if the rising cost of living has encouraged them to take a more economical mode of transportation like walking.</p><p>Hence, we will investigate the proportion of vehicle users(privately owned cars, walking, taxi/private hire) with monthly household incomes below $1,000 and $1,000-1,999 against the CPI from 2000 to 2020. This investigation aims to see if people with low monthly household incomes would take an economical approach to transportation when the CPI increases over the years.</p><h3>Analysing the relationship between the proportion of users (privately-owned cars, walking, taxi/private hire) with monthly household incomes below $1,000 and $1,000-1,999 against the CPI from 2000 to 2020</h3><p><strong>Methodology</strong></p><p>For this investigation, we will collect the relevant data:</p><ul><li>CPI changes in Singapore from the years 2000 to 2020<a href="https://www.singstat.gov.sg/whats-new/latest-news/cpi-highlights">[source link]</a></li><li>The proportion of people with a monthly household income of below $1,000 and $1,000-1,999 who use cars, walking, and taxi/private hires as a mode of transport from the years 2000 to 2020<a href="https://www.singstat.gov.sg/find-data/search-by-theme/population/mode-of-transport/publications-and-methodology">[source link]</a></li></ul><p>From the years 2000 to 2020, we will plot the graphs:</p><ul><li>The proportion of users walking against the CPI for people with a monthly household income of below $1,000</li><li>The proportion of users walking against the CPI for people with a monthly household income of $1,000-1,999</li><li>The proportion of users using cars against the CPI for people with a monthly household income of below $1,000</li><li>The proportion of users using cars against the CPI for people with a monthly household income of $1,000-1,999</li><li>The proportion of users using taxis/private hire against the CPI for people with a monthly household income of below $1,000</li><li>The proportion of users using taxis/private hire against the CPI for people with a monthly household income of $1,000-1,999</li></ul><p>Note:</p><ul><li>For the Consumer Price Index (CPI), 2019 is used as the base year</li></ul><pre>Data for walking:<br><br>   years  Below $1,000  $1,000 - $1,999<br>0   2000      0.197410         0.165403<br>1   2005      0.217293         0.189150<br>2   2010      0.226848         0.216828<br>3   2015      0.237069         0.240057<br>4   2020      0.330901         0.263769<br><br>Data for privately-owned cars:<br><br>   years  Below $1,000  $1,000 - $1,999<br>0   2000      0.040665         0.088839<br>1   2005      0.052331         0.060019<br>2   2010      0.052525         0.057281<br>3   2015      0.051724         0.051136<br>4   2020      0.056787         0.057996<br><br>Data for taxis/private hire:<br><br>   years  Below $1,000  $1,000 - $1,999<br>0   2000      0.009178         0.008402<br>1   2005      0.002891         0.003050<br>2   2010      0.007760         0.004510<br>3   2015      0.004310         0.009943<br>4   2020      0.014236         0.013524<br><br>CPI and CPI changes from 2000 to 2022:<br><br>    years  CPI_changes      CPI<br>0    2000     0.013479   74.435<br>1    2001     0.010143   75.190<br>2    2002    -0.003910   74.896<br>3    2003     0.004873   75.261<br>4    2004     0.016702   76.518<br>5    2005     0.004692   76.877<br>6    2006     0.009626   77.617<br>7    2007     0.021052   79.251<br>8    2008     0.066283   84.504<br>9    2009     0.005964   85.008<br>10   2010     0.028233   87.408<br>11   2011     0.052478   91.995<br>12   2012     0.045763   96.205<br>13   2013     0.023585   98.474<br>14   2014     0.010246   99.483<br>15   2015    -0.005227   98.963<br>16   2016    -0.005325   98.436<br>17   2017     0.005770   99.004<br>18   2018     0.004384   99.438<br>19   2019     0.005652  100.000<br>20   2020    -0.001820   99.818<br>21   2021     0.023052  102.119<br>22   2022     0.061213  108.370</pre><pre>                <br>#walking and CPI correlation<br>                 CPI_changes       CPI<br>years              -0.476965  0.967767<br>CPI_changes         1.000000 -0.486160<br>CPI                -0.486160  1.000000<br>Below $1,000       -0.458361  0.763506<br>$1,000 - $1,999    -0.455950  0.970728<br><br>#cars and CPI correlation<br>                 CPI_changes       CPI<br>years              -0.476965  0.967767<br>CPI_changes         1.000000 -0.486160<br>CPI                -0.486160  1.000000<br>Below $1,000       -0.299269  0.699504<br>$1,000 - $1,999     0.304340 -0.714349<br><br>#taxis/private hire and CPI correlation<br>                 CPI_changes       CPI<br>years              -0.476965  0.967767<br>CPI_changes         1.000000 -0.486160<br>CPI                -0.486160  1.000000<br>Below $1,000        0.024405  0.345914<br>$1,000 - $1,999    -0.583137  0.687508</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a6fvrEwsJfxc7KO54NEdzw.png" /></figure><pre>List of correlation coefficients:<br><br>For the graph titled Below $1,000(walking), the correlation coefficient for the linear regression is 0.7635057516777868<br>For the graph titled $1,000 - $1,999(walking), the correlation coefficient for the linear regression is 0.9707284508641514<br>For the graph titled Below $1,000(privately-owned cars), the correlation coefficient for the linear regression is 0.6995040316696415<br>For the graph titled $1,000 - $1,999(privately-owned cars), the correlation coefficient for the linear regression is -0.7143487554895666<br>For the graph titled Below $1,000(Taxis/private hire), the correlation coefficient for the linear regression is 0.34591429799608964<br>For the graph titled $1,000 - $1,999(Taxis/private hire), the correlation coefficient for the linear regression is 0.6875080469326131</pre><blockquote><strong><em>Observations</em></strong></blockquote><ul><li>About the walking data, the correlation coefficients for both subplots show strong positive linear relationships between the proportion of people using walking as a mode of transport and the consumer price index(CPI) from 2000 to 2020. Therefore, the connection between the proportion of users walking and CPI is very reliable.</li><li>For privately owned cars, individuals with a monthly household income of below $1,000 show strong positive linear relationships between the proportion of privately owned car users and the consumer price index(CPI) from 2000 to 2020. On the other hand, people with a monthly household income of $1,000-1,999 show strong negative linear relationships between the proportion of people using privately owned cars and the consumer price index(CPI) from 2000 to 2020.</li><li>No reliable observations can be derived between the proportion of taxis/private hire users with a monthly household income of below $1,000 and the CPI as the causative relationship is extremely unreliable. People with a monthly household income of $1,000-1,999 show strong positive linear relationships between the proportion of people using taxis/private hire as a mode of transport and the CPI from 2000 to 2020.</li></ul><p>When asked whether the rising cost of living has encouraged people with lower household incomes to use a more economical mode of transport such as walking, the answer is a resounding yes. There is a strong correlation between the use of walking as a mode of transport and the increase in the CPI from 2000 to 2020. This could be an indication that public transport will be less affordable for people on lower household incomes.</p><p>However, contrary to the previous assumption, a small minority with a monthly household income below 1,000𝑎𝑛𝑑1,000���1,000-1,999 show increasing use of private cars and taxis/private hire over the years as the CPI rises. This allows some assumptions to be made as to why this is happening:</p><ul><li>The monthly household income is not a representation of the total wealth of a person or of the source of their monthly income. This is because</li><li>Some people may have accumulated wealth by saving extensively</li><li>Prior to the census, some may have had a higher monthly household income.</li><li>They may not have declared their income from other sources.</li><li>Assuming that the information collected by Singstat is survey-based, they may have misreported their monthly household income or car ownership.</li><li>The relationship between the proportion of users and the CPI over the years may be strong. However, the relationship is not entirely reliable. The graph shows that over the years 2000 to 2020, the data do not always show an increasing trend in the use of taxis or cars as a means of transport. In order to make a definitive statement about the casual relationship, more annual data are needed from 2000 to 2020.</li><li>For those with an average monthly household income of $1,000-1,999, there may be an increased need for point-to-point services. Alternatively, taxi fares may be more affordable.</li><li>There is a rising demand for long-distance commuting and early attendance to work and events, fuelling the need for those on lower incomes to secure costly modes of transportation, such as acquiring used cars.</li></ul><p>Overall, given the strong correlation between the proportion of people with a monthly household income of $1,999 or less who walk and the CPI, which is stronger than the correlation for private cars and taxis/private hire, it remains arguable that pricing public transport may not be feasible for people on lower monthly household incomes.</p><p>Based on the given assumptions, we will examine whether travel distance affects the trends shown in the graphs. Specifically, we will examine the increase in car use among those earning less than $1,000 and the increasing preference for taxis or private hire among those earning between $1,000-$1,999 from 2000 to 2020. Our hypothesis proposes that the average travel distance has escalated over time, potentially necessitating cars or taxis/private hire for transportation.</p><h3>Does the increase in travelling distance over the years from 2010 to 2020 result in an increase usage of cars and taxis/private hires for transport?</h3><p><strong>Methodology</strong></p><p>For this research, we will look at data from 2010 to 2020. The relevant data will be obtained from Singstats: Mode of Transport<a href="https://www.singstat.gov.sg/find-data/search-by-theme/population/mode-of-transport/publications-and-methodology">[source link]</a>. We will need to collect the following data:</p><ul><li>The proportion of people with monthly household incomes below $1,000 who use privately-owned cars, taxis/private hires from 2010 to 2020.</li><li>The proportion of people with monthly household incomes of $1,000-1,999 who use privately-owned cars, taxis/private hires from 2010 to 2020.</li></ul><p>We will plot four types of graphs:</p><ul><li>The proportion of people taking up to 15, 16 - 30, 31 - 45, 46 - 60, and more than 60 minutes(mins) to travel.</li><li>The proportion of car users with a monthly house income below $1,000 and $1,000-1,999 in the years 2010, 2015, and 2020.</li><li>The proportion of users of taxis/private hire with monthly household incomes of below $1,000 and $1,000-1,999 in the years 2010, 2015, and 2020.</li><li>The proportion of people who use walking as a mode of transport with monthly household incomes of below $1,000 and $1,000-1,999 in the years 2010, 2015, and 2020.</li></ul><p>To calculate the proportion of vehicle users for each monthly household income, we will use this formula:</p><ul><li>The proportion of users of a type of vehicle(%) = Number of users of a type of vehicle / Total number of vehicle users(includes all types of vehicles listed in Singstat) x 100%</li></ul><pre>Travel time taken from the year 2010, 2015, and 2020:<br><br>   years Up to 15 mins 16 - 30 mins 31 - 45 mins 46 - 60 mins  \<br>0   2010        17.70%       37.85%       20.99%       17.28%   <br>1   2015        14.51%       34.52%       21.70%       20.67%   <br>2   2020        14.15%       32.85%       22.95%       20.71%   <br><br>  More than 60 mins  <br>0             6.19%  <br>1             8.60%  <br>2             9.34%  <br><br>Proportion of vehicles users with monthly household income of Below $1,000 and $1,000-1,999 taken from the year 2010, 2015, and 2020:<br><br>    Year Privately-owned car users, Below $1,000  \<br>0  2010                                   4.66%   <br>1  2015                                   4.76%   <br>2  2020                                   5.29%   <br><br>  Privately-owned car users, $1,000 - $1,999  \<br>0                                      4.89%   <br>1                                      4.55%   <br>2                                      5.30%   <br><br>  Taxis/private hire users, Below $1,000  \<br>0                                  0.69%   <br>1                                  0.40%   <br>2                                  1.33%   <br><br>  Taxis/private hire users, $1,000 - $1,999 Walking, Below $1,000  \<br>0                                     0.38%                20.11%   <br>1                                     0.88%                21.83%   <br>2                                     1.24%                30.85%   <br><br>  Walking, $1,000 - $1,999  <br>0                   18.51%  <br>1                   21.37%  <br>2                   24.11%</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hc-XW2zi19bHDwJNnKyRRw.png" /></figure><pre>Summary for the travelling time taken by people in Singapore in Up to 15 mins:<br><br>There is a drop in values by -3.193% from 17.699% in 2010 to 14.507% in 2015<br>There is a drop in values by -0.353% from 14.507% in 2015 to 14.154% in 2020<br><br>Summary for the travelling time taken by people in Singapore in 16 - 30 mins:<br><br>There is a drop in values by -3.325% from 37.846% in 2010 to 34.521% in 2015<br>There is a drop in values by -1.668% from 34.521% in 2015 to 32.854% in 2020<br><br>Summary for the travelling time taken by people in Singapore in 31 - 45 mins:<br><br>There is an increase in values by 0.708% from 0.2098791371% in 2010 to 0.2169572486% in 2015<br>There is an increase in values by 1.251% from 0.2169572486% in 2015 to 0.2294631873% in 2020<br><br>Summary for the travelling time taken by people in Singapore in 46 - 60 mins:<br><br>There is an increase in values by 3.394% from 0.172795551% in 2010 to 0.2067351326% in 2015<br>There is an increase in values by 0.038% from 0.2067351326% in 2015 to 0.2071125103% in 2020<br><br>Summary for the travelling time taken by people in Singapore in More than 60 mins:<br><br>There is an increase in values by 2.416% from 0.06186752729% in 2010 to 0.08602818247% in 2015<br>There is an increase in values by 0.732% from 0.08602818247% in 2015 to 0.09335294302% in 2020</pre><p>The graph shows that most people, regardless of their monthly household income, take 16-30 minutes to travel. However, over the years from 2010 to 2020, the proportion of people taking up to 15 minutes and 16-30 minutes to travel will decrease and the proportion of people taking 31 minutes or more to travel will increase. This may mean that the time it takes to travel may be longer in the future.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o4Riw8P5wjA8gUDFas0big.png" /></figure><p>From the graphs, we can see that from the years 2010 to 2020:</p><ul><li>There is an increase in the number of car users with monthly household incomes of below $1,000 and $1,000–1,999.</li><li>There is an increase in the number of taxis/private hire users with monthly household incomes of below $1,000 and $1,000–1,999.</li><li>There is an increase in the number of people who walk with monthly household incomes of below $1,000 and $1,000–1,999.</li></ul><p>With the increasing need to travel longer distances, the proportion of car owners and taxi/private hire users is higher in 2020 compared to 2010 and 2015. Although more data is needed to properly assess the trend, it can be inferred from the graphs that the increased need to travel longer distances has forced those with lower household incomes to find alternative modes of transport. However, this only applies to a minority. Despite the need to travel longer distances, walking has remained the most popular mode of transport over the years.</p><h3>Conclusion: To what extent does the monthly household income of a person influence his/her mode of transport in Singapore?</h3><p>In a nutshell, a person’s financial situation has a strong influence on his or her mode of transport in Singapore. This is apparent when using monthly household income data as a measure of individual wealth.</p><p>The COE quota system controls the vehicle population in Singapore, which makes it difficult to determine whether the rising average monthly household income per person (AMHI, AMHEI) has an impact on the growth of vehicle ownership.</p><p>Fortunately, a look at the difference in reported monthly household incomes can provide insights into how a person’s income can have an impact on their means of transport. Based on data from 2000 to 2020, we find that those with higher monthly household incomes are more likely to own a car or use taxis or private hire to get around. People with a lower monthly household income are more likely to use walking as a means of transport. Public transport and motorcycles have seen a unique shift in popularity over the years. People with medium to high monthly household incomes have become increasingly open to using public transport over the years. The same can be said for the users of motorcycles as well. Using the Consumer Price Index (CPI) in Singapore, we found that the popularity of owning a motorcycle decreases with increasing CPI, except for those with a monthly household income of $6,000 or more. The popularity of using public transport increases with increasing CPI, except for those with a monthly household income of less than or equal to $1,999.</p><p>Those with a monthly household income of $1,999 or less may find it difficult to afford public transport fares due to the rising cost of goods and services as measured by the CPI. As a result, the proportion of people with a monthly household income of $1,999 or less have increasingly used walking as their preferred mode of transport as the CPI has risen. However, there has been a small increase in car ownership and use of taxis or private hire for a small minority of those with monthly household incomes below $1,000 and $1,000–1,999 respectively. This could be due to increased distance travelled, hidden wealth, unreported second incomes, misinformation, unreliable casual relationships, affordability of taxi or private hire fares and many other factors. Over the years, there has been an increase in the time and possibly the distance required for travel, which may have led to an increase in the use of cars and taxis or private hire cars for transport by those with monthly household incomes of $1,999 or less. However, it is important to note that correlation does not necessarily imply causation. Establishing a causal relationship between vehicle ownership and the financial situation of individuals would require further research. Furthermore, as the data collected for the linear regression analysis are only from the years 2000, 2005, 2010 and 2020, more data from the years between 2000 and 2020 are needed to make a definitive statement about the causal effect of an individual’s financial situation on vehicle ownership.</p><p>Thank you for reading</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=14c88c52cd2d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>