<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Xian Jin Seow on Medium]]></title>
        <description><![CDATA[Stories by Xian Jin Seow on Medium]]></description>
        <link>https://medium.com/@xianjinseow92?source=rss-ca9d66f32b7c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Q-ka1LKyuAWJJmYaZ5gffA.png</url>
            <title>Stories by Xian Jin Seow on Medium</title>
            <link>https://medium.com/@xianjinseow92?source=rss-ca9d66f32b7c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 08:18:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@xianjinseow92/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Things I wish I knew before joining Metis Singapore Data Science Bootcamp]]></title>
            <link>https://medium.com/data-science/things-i-wish-i-knew-before-joining-metis-singapore-data-science-bootcamp-8768b9e9fda2?source=rss-ca9d66f32b7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/8768b9e9fda2</guid>
            <category><![CDATA[jobs]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-science-bootcamp]]></category>
            <category><![CDATA[experience]]></category>
            <category><![CDATA[time]]></category>
            <dc:creator><![CDATA[Xian Jin Seow]]></dc:creator>
            <pubDate>Thu, 23 Apr 2020 11:58:52 GMT</pubDate>
            <atom:updated>2020-05-10T04:03:05.006Z</atom:updated>
            <content:encoded><![CDATA[<h4>My personal experience of Singapore’s Metis Data Science bootcamp as a person without a coding or statistics background.</h4><p>This isn’t meant to be a short 5 minutes read (and it shouldn’t be anyway if you want an insight into what you’re getting yourself into). Spoiler alert: Every single day is an uphill battle.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*m1Z-8-KLg2xwg07spKGKag.jpeg" /><figcaption>Actually me in Metis’ Bootcamp, 7 Oct — 27 Dec 2019, colorized. Photo via <a href="https://me.me/i/this-past-month-or-two-or-three-me-kick-me-67b79115ddbb45a5a6a99d6c20819c28">me.me</a></figcaption></figure><h3>The Bootcamp Structure</h3><p>The Metis Singapore Data Science Bootcamp is a 12-week long intensive program focused on Python Programming where you learn about Exploratory Data Analysis (EDA), Statistical Modelling and Big Data tools through the completion of 5 projects.</p><p>These Bootcamps are known for being notoriously intense — and they are. Imagine cramming 1–2 years’ worth of information into just 3 months.</p><p>Every single day, I start my mornings with Pair Programming exercises, grab a 5 minutes break, then go straight to lessons immediately after. I’d break for lunch and come back for more lessons, which typically end around late afternoon at 3–4 pm.</p><p>While lessons have ended, I wasn’t done with my day yet. Whatever free time I had was spent struggling to keep up with concepts I had learned the day before (or even further back) whilst at the same time, working on self-initiated projects with 2 week deadlines. And this whole cycle repeats itself for the entire 3 months.</p><p>While it has been excruciatingly difficult, I can’t say I didn’t benefit from it at all. Prior to the Bootcamp, I was a self-studying junkie (Youtube, LeetCode, reading Data Science articles) and thinking I was progressing along fine. Until I got into the Bootcamp — which severely challenged that perspective. I’m glad I went for it because I would’ve been stuck in my little bubble thinking everything was fine, when in-fact, wasn’t. The bootcamp made me realize that there was <strong><em>a lot </em></strong>more to learn in the field of Data Science, and equipped me with enough knowledge to <strong><em>legitimately </em></strong>get started on my own.</p><h3>Do not underestimate the Bootcamp</h3><p>It is <strong><em>extremely</em></strong> Statistics and Math-intensive, and these are the things I wish I knew before joining as well as the recommendations I have for you that are not a <strong><em>GOOD TO HAVE</em></strong><em> </em>but a <strong><em>MUST HAVE</em></strong> before joining one.</p><h3>Things you MUST have</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/531/1*I-x7UieYqtKb3RagmMDlzg.jpeg" /><figcaption>Photo via <a href="https://www.amazon.com/Funny-Calculus-Algebra-Mathematics-Coffee/dp/B01HTWQZA0">amazon.com</a>. Pretty much sums up what you must have for the camp. Buy and bring this mug along with you to the bootcamp for extra points</figcaption></figure><h4>1. You MUST love Graphs</h4><p>Surprise surprise. This is a Data-Related field. Obviously most, if not all, of your findings are going to come out in the form of graphs. You have to love looking at graphs, working with them, tweaking them (bar charts, line plots, scatter plots, etc.) and you definitely have to love sharing your findings with them.</p><h4>2. You are skeptical of everything</h4><p>You question everything and want to verify the truth behind facts/figures presented to you. This is important because you are in a data-related field and you want to know if the data you’re getting is valid.</p><p>If this sounds like you, that’s perfect because Statistics serves as a tool that allows you to make <em>sense of, quantify and also test the validity</em> of a dataset, which gives you confidence that your hypothesis is true and your model works. Which leads me to my next point.</p><h4>3. You MUST have a good grasp on Statistics</h4><p>You also must know how (or at least appreciate) to apply Statistics in a <strong>pragmatic </strong>manner. With only 3 months in the bootcamp, the hyper-speed pace of each passing day isn’t gonna wait for you to fully understand a concept before you’re slapped in the face by another the very next day. Which also leads me to my next point.</p><h4>4. You <strong>MUST have good Math skills</strong></h4><p>I’m talking about calculus, like log, ln, differential equations, and etc. To understand statistical concepts and appreciate how to use them, you will need a sound grasp of mathematics. If you don’t, you’ll start realizing that you will be left in the dust as each day passes by.</p><p>If you don’t have these, I’d say you’ll be in for a rough ride. I didn’t even know what Linear Regression was prior to joining, and I recommend that you gain at least an <strong><em>intermediate level </em></strong>of Statistics and Math if you want to get the best bang for your buck and survive this bootcamp.</p><p>If you think you’re well-prepared and the 4 aforementioned points like fun, then perfect! We can move on to talking about how you can cope with the coming 3 months.</p><h3>Keeping up with the pace of the Bootcamp</h3><h4><strong>1. Ask when in doubt</strong></h4><p>If you don’t know and you don’t ask — you’re only hurting yourself because you’ll stay clueless if you don’t. It’s not absurd if you don’t know the content (I mean — that’s why you signed up for the bootcamp, right?). It’s only absurd when you’re not clarifying your doubts when you know you have one.</p><h4>2. You MUST be okay with <strong><em>not knowing everything</em></strong></h4><p>I don’t mean this in the sense that you don’t try your best and that it’s okay to slack off when you have lapses of knowledge — but realize that trying to get a grasp on <strong><em>everything </em></strong>before you start moving to the next step results in you <strong><em>being way too slow</em></strong><em> and </em><strong><em>can lead you to end up not doing anything.</em></strong></p><p>This has stunted me from even starting a project. I wanted to know every single bit of code, study Linear Regression first and arm myself with a full arsenal of knowledge before beginning my project. As a result, I was <strong><em>left with very little time to do any actual work</em></strong> nearing the end and <strong><em>produced less-than-desirable results</em></strong>.</p><p>You’re never going to know everything at every stage of the way, and you’re never going to be fully prepared. I never thought I’d be so cliche as to use a overused quote, but as Nike put it,</p><blockquote>“Just Do It”</blockquote><p>And really — just do it. There are going to be many times where you don’t know anything, and that’s fine. That’s why you’re in the bootcamp in the first place. If anything, you can always consult Google. You’re gonna be using new code you have completely zero clue about even as you type it into your Jupyter Notebooks, and that’s okay. Because you <strong><em>will</em> </strong>learn after you <em>use </em>it.</p><h4>3. You MUST have the <strong><em>GRIT</em></strong><em> </em>to stay disciplined and push through</h4><p>You always have to move forward every single day.</p><p>The best way to learn is through an <strong><em>ITERATIVE PROCESS</em></strong>. When you start a project, you will hit many roadblocks. And as painful as it is, you have to live with not knowing what to expect — sometimes even not knowing how to google for an answer or not even knowing the right questions. It’s fine. Just stay disciplined and push through. If you don’t know, don’t stay stuck.</p><p>Try asking questions. Try googling. Try talking to people, try consulting your classmates, your lecturer, or even students from the previous batch (<a href="https://medium.com/u/e105999c34d2">Black Raven (James Ng)</a> and <a href="https://medium.com/u/af46618e5594">Edward Bingei</a> helped me out a lot). You’re going to make a lot of friends and among them, your best will be Google (seriously, Google’s a lifesaver. Oh, and StackOverflow too).</p><p>After you solve a problem in your project, you’ll feel a great sense of achievement — and then immediately get stuck again on a new issue (which is pretty normal, by the way). Which then brings me to my next point...</p><h4>4. You MUST love Continuous Learning</h4><p>Things in this field are constantly moving, changing. To be honest, coding used to be a unicorn skill back then. Now everybody pretty much every Tom, Dick and Harry can code. You have to stay at the top, and to be at the top, you have to be learning continuously. You <strong><em>have </em></strong>to love the learning process.</p><h4>5. Understand that Project Design is an Iterative Process</h4><p>Apple wasn’t built in a day and neither are any of the projects you embark on during this camp. Understand that a project is an on-going, built-on-previous-attempts process.</p><p>You can have an impressive end-goal of what you want out of your project in mind, and draw out a huge Gantt chart of how you want to go about completing it, but the reality is, there are going to be a lot of obstacles that prevent you from doing so (things never going to go 100% according to plan anyway).</p><p>Don’t be so hung up if weren’t able to build something amazing on the first try. Focus on <strong><em>completing </em></strong>something simple first, then improve once you’re done!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G9fw-fXenZ-dzqlPN9CSGw.jpeg" /><figcaption>Image via my own bootcamp lecture notes</figcaption></figure><p>I think the above image presents the concept of Project Design the best. The idea of the creation of cars never stemmed from having the idea of a car itself. It spawned from the<strong> <em>overarching idea</em></strong> of an increasing demand and need for better transportation options.</p><p>Likewise, your projects will follow a process very much like this one. An overarching idea of what to achieve, and you improve on it (maybe even create a different product than you had initially envisioned) after CREATING YOUR FIRST FINISHED PRODUCT.</p><h3>Phew, the bootcamp sounds like a lot of work.</h3><p>Yeah, it is a lot. That’s why it’s called a bootcamp. And know that it’s going to be hard — and that’s okay. You’re not alone. If you must know, I suffered A LOT during this time. Think staying up till 4am to rehearse and create slides for your presentation, and just getting a measly 2 hours of sleep before waking up to shower, freshen up and head over to Metis for your presentation.</p><p><strong>But it does get better</strong>. Because with each passing project, you become more proficient in coding, more familiar with Python and its syntax, the appropriate Statistical Packages to use, and become better with presentation and communication (which, I must say, is great because I love speaking to people). The journey is painful but when you look back, you’ll be proud of what you have achieved — having survived that crazy and intense schedule and also <strong><em>having 5 Data Science projects</em></strong> under your belt.</p><p>Just like how I emphasized that Project Design is an iterative process, so is your learning. Here are some tips you can keep in mind when you do your projects while you keep up the fast-paced syllabus.</p><h4>Tip 1: Start Early and Study Ahead</h4><p>Keep ahead of the syllabus. Even if you haven’t covered the statistical concept(which model to use) or programming syntax to be used in your projects in class yet, read ahead and do your project alongside the lesson plans.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/347/1*x8WMJ-2DCOIGoKlXHCBikw.gif" /><figcaption>Red bar = curriculum pace. Grey bar = well.. ideally you. Photo via <a href="https://www.geeksaresexy.net/2013/07/31/playing-online-videos-vs-buffering-the-race-gif/">geeksaresexy.net</a></figcaption></figure><p>You’re Gromit laying down the tracks and the train’s the curriculum. It be like that. Get your project done up till the point where the instant you learn about your new statistical model in class, you’re ready to plop it into your project as that lesson ends.</p><h4>Tip 2: Multi-Task</h4><p>Time management will be a huge issue because of the tight deadlines in this camp. However, these deadlines don’t have to be your downfall.</p><p>Your projects will always end with a <strong><em>presentation </em></strong>and a <strong><em>blog post</em></strong> (Not like this one, silly. Like my <a href="https://medium.com/@xianjinseow92/creating-a-flask-app-to-classify-consumer-complaints-using-natural-language-processing-6c8778c9eb48">other blog posts about my projects</a>). Don’t wait till you’ve finished writing all your code and analysis then you begin to start on them.</p><p>Do your slides and write your blog post <strong><em>concurrently </em></strong>while your model trains, or when you’re running a really long web-scraper. Get creative with how you can slot bits and pieces of work in!</p><h4>Tip 3: Decide on Project Ideas even before the camp begins</h4><p>The requirement of graduating the Bootcamp is the completion of all 5 projects in the syllabus, 4 of which are self-initiated and self-led projects, each with an overarching statistical concept that you are expected to demonstrate after you’ve learned them (please follow <strong>Tip 1: Start early and Study Ahead</strong>).</p><p>Project 2: Linear Regression <br>Project 3: Supervised Learning<br>Project 4: Natural Language Processing<br>Project 5: Passion Project (whatever you want)</p><p>I suggest coming up with <strong><em>some </em></strong>idea of what you want to do before going into the camp, because most of your time will be spent learning the bootcamp material rather than brainstorming on what to do for the projects.</p><p>It doesn’t have to be very concrete, because you can refine these ideas as you progress with the bootcamp (and you have your instructor and your classmate to bounce ideas off with too). But you have to at least have an inkling of what you want to do if you don’t want to be fussing over what to do while also keeping up with very technically-heavy material of the bootcamp.</p><h4>Ok thanks but that still doesn’t really give me a concrete idea of what projects I should be doing</h4><p>Remember why you’re in this camp. Are you doing it in hopes of employment? Are you doing just to up-skill? Perhaps a quote from The <a href="https://www.youtube.com/channel/UC4xKdmAXFh4ACyhpiQ_3qBw">TechLead</a> can shed some light on this.</p><blockquote>“I’m not good at coding, but my at least code has business value.”</blockquote><p>Ok, it isn’t actually verbatim but it’s pretty much inferred from his video <a href="https://www.youtube.com/watch?v=xqgH9j3x2OE">here</a>. What value does your code/project bring to the world? A good coder is not a memory storage (one who remembers all the syntax in a coding language), but rather one who knows what value his PRODUCT can bring to the world. Likewise, your projects should be one where people can see value in. Ask yourself these questions:</p><ul><li>What do employers want to see?</li><li>What business value do I want to provide?</li><li>What burning question do I have about the world and what is the answer? Can anyone else benefit from this knowledge?</li></ul><p>You can refer to <a href="https://archive.ics.uci.edu/ml/index.php">this</a> for ideas. If not, you’re also more than welcome to <a href="https://towardsdatascience.com/catching-a-welcher-classifying-a-credit-card-defaulter-f4b21547a618?source=your_stories_page---------------------------">explore the blogs I’ve written about the projects I’ve done</a> during the camp!</p><p>If you don’t feel a shred of worry from everything I’ve said so far, GREAT! Go on a holiday before the camp because those 3 months are going to be one helluva ride. If not, forget the holiday and commit your time to preparing for it the very best you can because… those 3 months are going to be one helluva ride.</p><h3>Personal Takeaways</h3><p>I’d also like to add, in my <strong><em>personal</em></strong> <strong><em>opinion</em></strong>, Data Science Bootcamps are something you should go for ONLY when you’ve already <strong><em>decided </em></strong>you want to pursue a career in Data Science. Not when you’re still <strong><em>exploring </em></strong>if it’s a career for you. Otherwise, every single day in the Bootcamp will be torture. There have been students who have dropped out of the program (I’m not trying to scare you, just telling you what actually happened).</p><h4>Learn by doing, not by reading</h4><p>If there’s anything the time-intensive structure of the bootcamp taught me, it’s that you <strong><em>learn by doing </em></strong>and<strong><em> </em></strong>not by reading.</p><p>There’s really no point accruing all those Coursera or Udemy certifications if you can’t do anything of value with that knowledge. Showcase what you can <strong><em>DO </em></strong>to the world. As <a href="https://www.youtube.com/channel/UCV0qA-eDDICsRR9rPcnG7tw">JomaTech</a> once said,</p><blockquote>“If you made an interesting analysis, but no one is there to read your analysis… Did you really make an analysis?”<br>- JomaTech on <a href="https://www.youtube.com/watch?v=MfP-P8EHGBo&amp;t=12m12s">5 Tips For Getting A Data Science Job</a></blockquote><p>Very wise.</p><p>I personally intend to enrich my learning by working on projects that focuses on real-world applications. In the process, I’m sure I’ll come across many things I don’t know, and I’ll be learning things that solve the problem, rather than learning MANY things that aren’t useful.</p><h4>You actually have to chill out sometimes</h4><p>I know I said a lot of things about the bootcamp being intense, multi-tasking, being diligent and disciplined in studying, and staying ahead of the syllabus. But the fact is; nobody can work 18/hours a day, everyday for 3 months. <em>Burnout is very common in the bootcamp</em>. You gotta take a chill pill sometimes.</p><p>Give yourself a reward yourself after every project. Take a day off to get refreshed, grab some drinks with your classmates, whatever. Just disconnect yourself from the material then get back in once you’re done recharging.</p><h3><strong><em>Final Thank Yous</em></strong></h3><p>I really want to say I have been quite fortunate to have been surrounded by amazing classmates (of whom were leaps and bounds above my skill level) who were very helpful in guiding me along the way when I encountered any difficulties in the curriculum.</p><p>We also played a lot (we went for more than a few drinks) and that helped offset quite a lot of the pressure from the bootcamp.</p><p>Thanks for reading and if you wanna hit me up, you can do so over at my <a href="https://www.linkedin.com/in/seow-xian-jin/">Linkedin</a> or at xianjinseow92@gmail.com.</p><p>Cheers!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8768b9e9fda2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/things-i-wish-i-knew-before-joining-metis-singapore-data-science-bootcamp-8768b9e9fda2">Things I wish I knew before joining Metis Singapore Data Science Bootcamp</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating a Flask App to Classify Consumer Complaints using Natural Language Processing…]]></title>
            <link>https://medium.com/data-science/creating-a-flask-app-to-classify-consumer-complaints-using-natural-language-processing-6c8778c9eb48?source=rss-ca9d66f32b7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/6c8778c9eb48</guid>
            <category><![CDATA[consumer-complaints]]></category>
            <category><![CDATA[word-embeddings]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[flask]]></category>
            <dc:creator><![CDATA[Xian Jin Seow]]></dc:creator>
            <pubDate>Fri, 17 Apr 2020 09:43:46 GMT</pubDate>
            <atom:updated>2020-04-19T15:38:22.746Z</atom:updated>
            <content:encoded><![CDATA[<h3>Creating a Flask App to Classify Consumer Complaints using Natural Language Processing (Multi-class Classification with TFIDF/Word2Vec)</h3><p>In part 1 of this series, I will cover the workflow in obtaining the working model that churns out these classifications. This post will be continued in part 2, which involves building the Flask App of this model. So if you’re solely interested in the Flask portion, click <strong><em>here (still WIP)</em></strong> to head on over to that post!</p><p>Without further ado, let’s jump into part 1 of this series.</p><pre>I am dissatisfied with the current outcome of a dispute that was initiated with Discover Card regarding a single transaction that occurred on XXXX/XXXX/2015 in the amount of {$280.00}. I have corresponded with Discover Card at least four times since XXXX/XXXX/2015 ( which I have enclosed as an attachment to this complaint ). I believe that the credit card issuer has violated consumer protection laws by failing to implement the Special Rule for Credit Card Purchase protection despite overwhelming paperwork evidence submitted by me that shows the merchant has conducted business in bad faith less favorable to the consumer. I have sustained a monetary loss as a result of merchants bad faith and intent. I have patiently utilized the internal Discover Card dispute process over the past three months with the credit card issuer always favoring the merchant ; I have repeatedly submitted irrefutable paperwork evidence that has shown that the merchant has conducted business in bad faith. I have tried in good faith to address my complaint with the merchant and Discover Card but believe that I will not receive a favorable outcome.<br></pre><pre>I suffered a work-related injury in XXXX and am now permanently XXXX. My income has dropped substantially in the past 5 years and I am now on a low fixed income. On XXXX separate occasions from XXXX to XXXX I had zero income and had to use my Sams Club card to purchase food, thus running up debt out of necessity. I am currently in the process of attempting to pay down my Sams Club ( Synchrony Bank ) card, and stopped using the card some time ago. I have always made at least minimum payments and have never missed a payment. Despite this, my interest rate has been unilaterally raised three times in the past two years, and is now at 23.15 %. I called a Sams Club account rep today to file a complaint over the phone, because I am never going to be able to pay down this card when I am paying almost {$50.00} a month in interest and can only afford to pay the minimum payment + {$4.00} or {$5.00} dollars. They would not work with me, which I expected. In my opinion, Synchrony Bank is taking unfair advantage of recent interest rate hikes to gouge customers, especially those who are financially unable to make substantial monthly payments on their accounts. Therefore I am contacting the CFPB to file a complaint through which I might receive some relief.</pre><h3>HOLD UP!</h3><p>Did you read read the wall of text above? Chances are you didn’t (because I wouldn’t either). Who has the time to read a whole chunk of text with no headers, no summary and terrible formatting (bold, spacing, etc.) that potentially holds no relevance to you?</p><p>If you actually read the texts above, you’d realize they’re actually customer complaints. And as Mr Gates once mentioned:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/388/1*DGA2A0dq504pILm5EuJL3A.jpeg" /><figcaption>Well if he says it, it must be true. (source: <a href="https://me.me/i/your-most-unhappy-customers-are-your-greatest-source-of-learning-21856189">photo</a> via <a href="https://me.me/">me.me</a>)</figcaption></figure><p>There is so much we can learn from our customers’ complaints. The best brands always connect with their consumers and make decisions revolving around them (think Paramount, who actually pushed the release of the Sonic movie to a later date and changed the design of our favorite blue hedgehog because of the terrible reviews and the countless memes that ensued).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*idyH05VznE7NxVvVIWMhqA.jpeg" /><figcaption>Truly horrific design (left), but kudos to you, Paramount, for redesigning (right) and listening to your audience (source: <a href="https://www.yahoo.com/author/tom-butler">Tom Butler</a> via <a href="https://sg.news.yahoo.com/sonic-hedgehog-trailer-sega-redesign-133549564.html">sg.news</a>)</figcaption></figure><p>Understanding your customers is important and all, but it doesn’t change the fact that a wall of text from an angry customer is not only strenuous to read through, but also demotivating. And often times it’s not going to be just one complaint.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SOsf-0QOfkm4GCS5_a62Pw.png" /><figcaption>Ok yes if you look closely, they are all the same complaint. But you get my drift.</figcaption></figure><p>Typically, in banks (or any customer service department of large companies), they see thousands of complaints like these on a daily basis and they’re not exactly going to be as pleasantly phrased and formatted as the ones above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*DD_LWypzKBWiL594PRstFQ.jpeg" /><figcaption>Actual image of customer complaining /s (source: via <a href="https://yandex.fr/news/story/Penzenskij_chinovnik_nabrosilsya_s_kulakami_na_zhurnalista--174f8bff44d7cae0ccd8bb37b57a21f8?lr=143&amp;stid=Gn7E&amp;persistent_id=89413115&amp;lang=ruuk&amp;rubric=Penza&amp;from=story">yandex</a>)</figcaption></figure><p>Furthermore, the complaints can be filed into the wrong sections by the consumers themselves, and it is going to be a huge pain to go through every single complaint and direct them to the relevant departments to be dealt with. And needless to say, these slow processing times also lead to slow response times to the customers, leading to your already-angry customers becoming angrier.</p><p>Man, if only there were some way to quickly categorize each complaint in order to speed up your reading process.</p><h3>Project Objective</h3><p>Well — turns out there is, which is the very purpose of this project. In my project, I developed a model that can correctly classify a customer complaint according to it’s product type, with an <strong><em>accuracy of 83%.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vjw0aSQDfmoclwgkiU3dfA.png" /></figure><p>The app utilizes a machine learning algorithm in conjunction with Natural Language Processing (NLP) methodologies to process text to make a prediction on the type of product the complaint is referring to.</p><h3>Defining the Problem</h3><p>This is essentially a <strong><em>supervised (labelled) </em></strong>text multi-class classification problem, of which our goal is to make a prediction (assign to correct category) with a new input (complaint). In the steps below, I will take you through the following:</p><ul><li>Data Exploration (Understanding the dataset)</li><li>Data (text) Pre-processing and the general steps one might take in an NLP project</li><li>Model selection and why Micro-F1 is used as an evaluation metric</li><li>Selected model final test and results</li></ul><p>In Part 2 of the Series…</p><ul><li>Flask App demonstration &amp; how to build one</li></ul><p>All the code to this project can be found <a href="https://github.com/xianjinseow92/projects/tree/master/Project_4_Consumer_Complaint_Text_MultiClassifier">here</a> at my github. Feel free to drop me a message should you need any clarification.</p><h3>Section 1: The Data</h3><p>The data used to train this model is taken from <a href="http://catalog.data.gov/dataset/consumer-complaint-database"><strong><em>here</em></strong></a>. And an explanation of what each column means can be found <a href="https://cfpb.github.io/api/ccdb/fields.html"><strong><em>here</em></strong></a>. It is essentially a labelled dataset provided by the Consumer Financial Protection Bureau (CFPB) about complaints received about financial products and services.</p><p>We observe that the dataset contains 1,437,716 rows of data and many, many rows containing null values.</p><pre>&lt;class &#39;pandas.core.frame.DataFrame&#39;&gt;<br>RangeIndex: 1437716 entries, 0 to 1437715<br>Data columns (total 18 columns):<br>Date received                   1437716 non-null object<br>Product                         1437716 non-null object<br>Sub-product                     1202551 non-null object<br>Issue                           1437716 non-null object<br>Sub-issue                       887045 non-null object<br>Consumer complaint narrative    463991 non-null object<br>Company public response         530104 non-null object<br>Company                         1437716 non-null object<br>State                           1414039 non-null object<br>ZIP code                        1304830 non-null object<br>Tags                            196144 non-null object<br>Consumer consent provided?      831624 non-null object<br>Submitted via                   1437716 non-null object<br>Date sent to company            1437716 non-null object<br>Company response to consumer    1437715 non-null object<br>Timely response?                1437716 non-null object<br>Consumer disputed?              768482 non-null object<br>Complaint ID                    1437716 non-null int64<br>dtypes: int64(1), object(17)<br>memory usage: 197.4+ MB</pre><p>However, since we are solely analyzing the complaint itself, we need only the Product and Consumer Complaint Narrative columns.</p><p>After only keeping the columns we need, removing the null values and renaming them into something nicer to work with, we are left with just 463,991 rows of data:</p><pre>&lt;class &#39;pandas.core.frame.DataFrame&#39;&gt;<br>Int64Index: 463991 entries, 4 to 1437711<br>Data columns (total 2 columns):<br>PRODUCT               463991 non-null object<br>CONSUMER_COMPLAINT    463991 non-null object<br>dtypes: object(2)<br>memory usage: 10.6+ MB</pre><p>Doing a little bit of EDA (exploratory data analysis) on the PRODUCT column, we observe 18 classes with the following distribution:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/999/1*XCVJ93h1b-7GEqmgbmGXzA.png" /></figure><p>Some of the classes (products), however, overlap. For instance, Credit card or prepaid card clash with Credit card and Prepaid card. Additionally, there is also a problem with having 463,991 rows to perform analysis on (I doubt your laptop will be able to handle the processing and model training at the later part of this project. Mine couldn’t).</p><p>After removing a few product classes, combining some classes together and reducing the number of observations by 90% (the specifics of which can be found in my jupyter notebook on <a href="https://github.com/xianjinseow92/projects/tree/master/Project_4_Consumer_Complaint_Text_MultiClassifier/notebooks">my github</a>), we are left with around 30,000 rows of complaints (of which is manageable for a laptop with 8GB ram).</p><pre>&lt;class &#39;pandas.core.frame.DataFrame&#39;&gt;<br>Int64Index: 31942 entries, 72 to 1437623<br>Data columns (total 2 columns):<br>PRODUCT               31942 non-null object<br>CONSUMER_COMPLAINT    31942 non-null object<br>dtypes: object(2)<br>memory usage: 748.6+ KB</pre><p>With the following distribution:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/804/1*ZUHF5X3DJ7VuYWptej7eMQ.png" /></figure><p>Not too shabby. Now we are ready to do some text pre-processing on our complaint data!</p><h3>Section 2: Text Pre-Processing</h3><h4>Section 2.1: Natural Language Preprocessing in a Nutshell</h4><p>Now that we have our data, we need to go through the cleaning process. For text data sets in general, this often includes:</p><ul><li>Removal of punctuation</li><li>Removal of stop words (things like “the”, “this”, “what”)</li><li>Lowercase</li><li>Tokenization</li><li>Stemming/lemmatization (reducing words down to their base form by removing suffixes like “-ed”, “-ing”)</li></ul><p>If you are dealing with online text and comments, you will require further removal of items like hyperlinks, usernames, and automated messages.</p><p>What pre-processing you do varies from project to project, and for this I achieved the best accuracy score doing all of the above except stemming and lemmatization. I considered using bi-grams, but that caused me to consistently run into memory issues, and I had to therefore move forward with only considering 1-gram (standard single-word tokenization).</p><p>Below is a sample illustration of punctuation removal, stop-word removal, lower-casing and thereafter tokenization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b7HHAiA9uObxQfeqG73HLg.jpeg" /><figcaption>Punctuation removal</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1eXiIiYfY0VgUYtjOkQ1Ew.jpeg" /><figcaption>Stopword removal</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pcNlns_OXEicgTqrQQM8jw.jpeg" /><figcaption>Lowercasing</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NlcyYwBBF6TsV9xBntljUA.jpeg" /><figcaption>Tokenization</figcaption></figure><h4>Section 2.2: Vectorization</h4><p>To turn words into something a machine learning algorithm can understand and process, we need to do something called <strong><em>vectorization</em></strong>. Put simply, this is the process of turning words into multi-dimensional vectors in such a way that the meaning or context of the word is correlated with where that vector points. <strong>In a sense, vectorization allows computers to quantify the meaning of words by mapping similar word meanings to similar vector spaces.</strong></p><h4>Section 2.3: Term Frequency — Inverse Document Frequency Vectorization</h4><p>For this application, we want <strong>context-based</strong> vectorization. This is where Term frequency inverse document frequency vectorization comes in (TF-IDF). This vectorization method looks at the number of times a word appears in a comment relative to the number of times it appears in other comments. <strong>Two things result in a higher TF-IDF score:</strong></p><ol><li><strong>Higher frequency of the word within the specific complaint being scored.</strong></li><li><strong>Lower frequency of the word across all other complaints.</strong></li></ol><h3>Section 3: Training Methodology</h3><h4>Section 3.1: Model Evaluation Score</h4><p>In a multi-class scenario, it isn’t as simple as Precision, Recall and F1 anymore as per a binary-classification (as with my previous project on <a href="https://towardsdatascience.com/catching-a-welcher-classifying-a-credit-card-defaulter-f4b21547a618">Classifying Defaulters</a>). It’s on a whole new level. I recommend referring to <a href="https://towardsdatascience.com/multi-class-metrics-made-simple-part-i-precision-and-recall-9250280bddc2">Boaz Shmueli</a>’s explanation on evaluation metrics in multi-class classifiers if you need a better understanding on the evaluation metrics used for models in a multi-classification scenario. He does a really good job explaining regarding precision, recall and F1 scoring on <strong><em>both binary</em></strong><em> </em>and <strong><em>multi-class</em></strong> classification metrics.</p><p>As with this mutli-classifcation case, I have decided to use <strong><em>Micro-F1</em></strong> as an evaluation metric between models for it measures the model’s <strong><em>overall accuracy</em></strong>.</p><h4>Section 3.2: Models Used</h4><p>I have decided to use the following models for the multi-class text classification:</p><ol><li>Multinomial Naive Bayes</li><li>Gaussian Naive Bayes</li><li>Logistic Regression</li><li>Random Forest</li><li>LinearSVC</li></ol><h3>Section 4: Model Selection</h3><p>Each model is <strong><em>cross-validated</em></strong> on a stratified <strong><em>5-fold split train-validation (of 80% of the whole dataset)</em></strong>. The remaining 20% data was held out and untouched to simulate how our final chosen model would perform in real-life scenarios where it would have to deal with inputs it had never seen before.</p><p>As always, the code behind how this was done can be found on my jupyter notebook (Section 3 onwards in the notebook) on <a href="https://github.com/xianjinseow92/projects/tree/master/Project_4_Consumer_Complaint_Text_MultiClassifier/notebooks">github</a>.</p><p>From our preliminary training results, we see that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*weCR2eNHpZpvJmDXoX7FSg.png" /></figure><p>Logistic Regression gives us the highest accuracy of 83%.</p><p>83% is a pretty good accuracy. But can we do better? We can’t be sure if it WILL be better, but we can always try and strive for improvement.</p><h4>Section 4.1: Stanford GloVe and Google’s Word2Vec</h4><p>Instead of utilizing Vectorization (Tf-idfVectorizer that we used above), we can utilize WordEmbedding on the complaints before feeding them into our classifiers. To understand more about WordEmbeddings, you can find it <a href="https://towardsdatascience.com/the-three-main-branches-of-word-embeddings-7b90fa36dfb9">here</a>.</p><p>Some methods such as Word2Vec and <a href="https://spacy.io/models">spaCy</a> involve pre-trained, multi-gigabyte models that are trained on hundreds of thousands, if not millions of documents and then reduce the meanings of words to a set of a couple hundred numbers. These models are excellent at preserving context and meaning of words in general, but they are slow and large.</p><p>I have chosen to use Standford’s GloVe (in increasing complexity) and Google’s Word2Vec for this project.</p><h4>Section 4.2: TfidfVectorizer &amp; WordEmbedding Results</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ftCqjFU5HXew9A9Bq-FJlw.png" /></figure><p>Compiling all results from the tests, we find that the vanilla TFidfVectorizer does way better than the WordEmbeddings, with the highest accuracy of 83%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7NgWHg-01kdRbP-t6CKhEQ.png" /></figure><p>Thus, <strong><em>TfidfVectorizer using Logistic Regression</em></strong> is selected as the text pre-processing and model combination to use for this project.</p><h3>Section 5: Testing our final selected Logistic Regression Classifier on unseen data</h3><p>Remember the 20% data we heldout earlier that we didn’t use in the cross-validation? Here comes the part where we use it as an estimate as to how well our model will perform on unseen data (i.e. how it would perform in real-life scenarios).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PMMG49eokf3zYAu33B82cg.png" /></figure><p>The model returns us a whooping <strong><em>83% accuracy on unseen data! </em></strong>Signifying that it can correctly classify a complaint 83% of the time! Now that we have our model, we can <em>finally</em> see it in action!</p><h3>Section 6: Flask Demo</h3><p>To be able to utilize our model in a user-friendly way, I created a Flask app that allows a complaint to be typed in. Thereafter, the app will predict which product category the complaint belongs to. Refer to this <strong><em>post (WIP) </em></strong>to find out how the Flask app is created!</p><pre>I am dissatisfied with the current outcome of a dispute that was initiated with Discover Card regarding a single transaction that occurred on XXXX/XXXX/2015 in the amount of {$280.00}. I have corresponded with Discover Card at least four times since XXXX/XXXX/2015 ( which I have enclosed as an attachment to this complaint ). I believe that the credit card issuer has violated consumer protection laws by failing to implement the Special Rule for Credit Card Purchase protection despite overwhelming paperwork evidence submitted by me that shows the merchant has conducted business in bad faith less favorable to the consumer. I have sustained a monetary loss as a result of merchants bad faith and intent. I have patiently utilized the internal Discover Card dispute process over the past three months with the credit card issuer always favoring the merchant ; I have repeatedly submitted irrefutable paperwork evidence that has shown that the merchant has conducted business in bad faith. I have tried in good faith to address my complaint with the merchant and Discover Card but believe that I will not receive a favorable outcome.</pre><p>Remember this horrible wall of text we saw at the beginning of the post? Let’s put our model to the test. Take your time to read it and guess which of these categories it falls under.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/804/1*ZUHF5X3DJ7VuYWptej7eMQ.png" /></figure><p>Now let’s put our app to the test.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgfycat.com%2Fifr%2Ffalseringediguana&amp;display_name=Gfycat&amp;url=https%3A%2F%2Fgfycat.com%2Ffalseringediguana&amp;image=https%3A%2F%2Fthumbs.gfycat.com%2FFalseRingedIguana-size_restricted.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=gfycat" width="2700" height="1444" frameborder="0" scrolling="no"><a href="https://medium.com/media/e45313bda86b68e8f7448eadf786cec2/href">https://medium.com/media/e45313bda86b68e8f7448eadf786cec2/href</a></iframe><p>Did you come to the same conclusion as the app did? Now let’s try this with a more difficult complaint, full of typos, caps and millennial lingo.</p><pre>Omg where is my money?????? AOSINIONSAD WHY DID YOU GUYS TAKE MY MONEY AWAY. SAOIDNSIOADNOIAODNNIOASDNSADNOSDNOASDN</pre><pre>I TRANSFERRED IT LAST NIGHT AND I WOKE UP TO NOTHING IN!!!!!! MY BANK ACCOUNT. HELP PLEASE!!!!!!!!!!!!!!!!!!!!!!!!!!!!!</pre><pre>I NEED THE MONEY OR ELSE I WILL BE HUNTED BY LOAN SHARKS!!!!!!!!</pre><pre>YOU *(#&amp;$) PEOPLE HAVE NO RIGHT TO DO THIS TO ME!!!! I NEED MY MONEY!!!!</pre><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgfycat.com%2Fifr%2Fclosecoarsehochstettersfrog&amp;display_name=Gfycat&amp;url=https%3A%2F%2Fgfycat.com%2Fclosecoarsehochstettersfrog&amp;image=https%3A%2F%2Fthumbs.gfycat.com%2FCloseCoarseHochstettersfrog-size_restricted.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=gfycat" width="2700" height="1444" frameborder="0" scrolling="no"><a href="https://medium.com/media/ff879450451b122365d45e8af97a2bfb/href">https://medium.com/media/ff879450451b122365d45e8af97a2bfb/href</a></iframe><p>We see that the app can handle messy complaints with no errors and correctly classify what product category they fall into as well.</p><h3>Section 7: App Benefits</h3><p>The utilization of this app can bring about a plethora of benefits, from freeing up manpower to increasing efficiency of sifting through customer complaints.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YWk_-axY4xcPlJ8O_9bdvw.png" /></figure><h3>Section 8: Future Work</h3><p>As an augmentation to this app to improve it’s usefulness, we can perhaps look into the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O9wNnSjzaddYk54suLStBg.png" /></figure><p><strong><em>Topic Summarization<br></em></strong>Currently, the app only classifies the complaint into the correct product category. While it does file the complaint into it’s correct category, the customer complaint officer would still have to read through the complaint itself. It would be nice for the app to be able to also provide a short abstract of what the complaint is about. This would be an interesting topic to look into and devise how to create this functionality.</p><p><strong><em>Complaint Prioritization<br></em></strong>We can look into also implementing Sentiment Analysis to provide some sort of complaint prioritization. Angrier-sounding complaints (i.e., angry customers) can be given a higher priority, since angry customers tend to provide the most problems. Our app would therefore have an automated sorting for the customer complaint officer as to which complaints need to be urgently attended to.</p><ul><li>ask if anyone would like to work on it together, can drop me a text</li></ul><h3>Section 9: Parting Words</h3><p>This has been by no means a short post, and hopefully an informative one in your journey in Data Science. Thank you for taking the time to read this far.</p><p>If you learned anything, find this helpful or think it might help someone in this field, do share this post with others!</p><p>As always, feel free to hit me up over on my <a href="https://www.linkedin.com/in/seow-xian-jin/">Linkedin </a>or my gmail at xianjinseow92@gmail.com!</p><p>Cheers.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c8778c9eb48" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/creating-a-flask-app-to-classify-consumer-complaints-using-natural-language-processing-6c8778c9eb48">Creating a Flask App to Classify Consumer Complaints using Natural Language Processing…</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Catching a Welcher: Classifying a Credit Card Defaulter]]></title>
            <link>https://medium.com/data-science/catching-a-welcher-classifying-a-credit-card-defaulter-f4b21547a618?source=rss-ca9d66f32b7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/f4b21547a618</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[credit-card-default]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[predictions]]></category>
            <category><![CDATA[classifier-project]]></category>
            <dc:creator><![CDATA[Xian Jin Seow]]></dc:creator>
            <pubDate>Tue, 19 Nov 2019 10:00:34 GMT</pubDate>
            <atom:updated>2019-11-20T08:25:55.268Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*ncw88R85hT-4r2wHyaFZ-w.jpeg" /><figcaption><a href="https://www.financialexpress.com/money/defaulted-on-loan-credit-card-payments-you-can-still-take-a-loan-heres-how/1704996/">Photo Credits</a></figcaption></figure><p>Before we begin, as always, the code to my project can be found on my <a href="https://github.com/xianjinseow92/projects/tree/master/Project_3_Credit_card_default_classifier">github</a>, and should you have anything you wish to ask about this project, please feel free to contact me via <a href="https://www.linkedin.com/in/seow-xian-jin/">linkedin</a>.</p><h3>Introduction</h3><h4>What it means to default on a credit card (From a Credit Card User point of View)</h4><p>In case you’re unclear on what defaulting on a credit card means, here’s the gist: After you’ve failed to make a payment on your credit card for 180 days (or as decided by your credit card company), your issuer assumes you’re probably never going to. At this point, the issuer can (and usually does) close your card, write off what you owe as bad debt and sell your account to a collections agency.</p><h4>Why is it important?</h4><p>Now your credit card issuer is out of the picture. Your debt belongs to a collector, and you’ll start getting flooded with phone calls asking you to pay the bill. Although you can send written notice asking them to stop calling you and there are <a href="http://www.consumer.ftc.gov/blog/facing-debt-collection-know-your-rights">laws limiting what debt collectors</a> can say and do when they contact you, you’ll still owe the debt. And if you don’t deal with it one way or another, you could get sued.</p><p>Also, the default will be reported to the <strong>three major credit bureaus (or more, depending on your country)</strong>. Your credit score will take a nosedive, and the blemish will stay on your credit report for <strong>up to seven years</strong>.</p><p>In short, defaulting on a credit card bill has serious consequences. You should make it a priority to handle this matter as soon as you can.</p><h4>What does it mean from a Bank’s Point of View?</h4><p>Well, it’s analogus to if someone asked to borrow some money from you, and never pays you back (I am severely oversimplifying it but you get it). You’re going to be 100 bucks short and the other person 100 bucks happier (I really need to work on my analogies).</p><p>A bad loan for the bank can therefore lead to a huge financial loss in the event of a defaulter and thus requires heavy risk-mitigation regarding this issue. Furthermore, the bank loaning process can be quite a manual and tiresome task in identifying who might be a potential defaulter. The loaning process is also quite ependent on the loan processors themselves, leaving a lot of it to subjectivity which might possibly turn into a bad loan in the future.</p><p>But then again, on the flipside, banks can also earn a lot from people who actually pay back their loans. Additionally, we don’t want to be turning people away who could really use that money to get their lives back on track or achieve their dreams (i.e. setting up a business they’ve always wanted to).</p><blockquote>How then, do we properly assess if someone is going to be a credit risk or not?</blockquote><p>What are the telltale signs of a person’s behavior that makes him/her a consistent welcher? <em>(yes, I am aware that I am quoting urban dictionary)</em></p><h3>Project Objective</h3><p>Therefore, this project aims to bridge this gap of uncertainty by utilizing a data-driven approach by using past data of credit card customers in <strong><em>conjunction with machine learning</em></strong> to predict whether or not a consumer will default on their credit cards.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uZyt9Z189siaNsAlIDtjEg.jpeg" /></figure><p>The goal behind using this model is to achieve two things:</p><ul><li>Bring more consistency to the loaning process and;</li><li>Investigate what the key drivers are behind a potential defaulter</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jkq_MUGVeHRhqvywIJxjyA.jpeg" /></figure><p>In addition to answering these pressing questions, I personally wanted to focus on the learning process as well; to become better/more efficient with a data science project workflow, and also enter a domain I was unfamiliar with (and uncomfortable with, having come from a Mechanical Engineering background) to step out of my comfort zone. With that being said, let’s delve into how one may embark on a Classifier Data Science Project.</p><h3>Section 1: The Data</h3><h4>Section 1.1: Dataset Description</h4><p>The dataset considered in this analysis is the <em>“Default of Credit Card Clients”</em> dataset released under the public liscense of <em>Creative Commons</em> and available on the <a href="https://www.kaggle.com/uciml/default-of-credit-card-clients-dataset">Kaggle website</a>.</p><p>This dataset contains <strong>30000 observations of 25 variables </strong>from a bank (and also a cash and credit card issuer in Taiwan); where each observation corresponds to a particular credit card client. Among the total 30000 observations, 6636 observations (22.1%) are cardholders with default payment.</p><p>The 25 variables in this dataset comprises of demographic variables (gender, education level, marriage status, and age) and financial variables of 6-months worth of payment data from April 2005 to September 2005 (amount of given credit, monthly repayment statuses, monthly amount of bill statements, and monthly amount of previous payments). The breakdown of what each variable means is stated in the next section.</p><h4>Section 1.2: Understanding the Variables</h4><p><strong>ID</strong>: ID of each client</p><p><strong>LIMIT_BAL</strong>: Amount of given credit in NT dollars (includes individual and family/supplementary credit</p><p><strong>SEX</strong>: Gender (1=male, 2=female)</p><p><strong>EDUCATION</strong>: (1=graduate school, 2=university, 3=high school, 4=others, 5=unknown, 6=unknown)</p><p><strong>MARRIAGE</strong>: Marital status (1=married, 2=single, 3=others)</p><p><strong>AGE</strong>: Age in years</p><p>— —</p><p><strong>PAY_0</strong>: Repayment status in September, 2005 (-1=pay duly, 1=payment delay for one month, 2=payment delay for two months, … 8=payment delay for eight months, 9=payment delay for nine months and above)</p><p><strong>PAY_2</strong>: Repayment status in August, 2005 (scale same as above)</p><p>…;</p><p><strong>PAY_6</strong>: Repayment status in April, 2005 (scale same as above)</p><p>— —</p><p><strong>BILL_AMT1</strong>: Amount of bill statement in September, 2005 (NT dollar)</p><p><strong>BILL_AMT2</strong>: Amount of bill statement in August, 2005 (NT dollar)</p><p>…;</p><p><strong>BILL_AMT6</strong>: Amount of bill statement in April, 2005 (NT dollar)</p><p>— —</p><p><strong>PAY_AMT1</strong>: Amount of previous payment in September, 2005 (NT dollar)</p><p><strong>PAY_AMT2</strong>: Amount of previous payment in August, 2005 (NT dollar)</p><p>…;</p><p><strong>PAY_AMT6</strong>: Amount of previous payment in April, 2005 (NT dollar)</p><p><strong>default.payment.next.month</strong>: Default payment (1=yes, 0=no)</p><h4>Section 1.3: Data Cleaning</h4><p>Some of the columns didn’t make sense to me, so I decided to rename them into more understandable terms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/1*joH7LU_3ELDTPlHxC-J3Tw.png" /></figure><p>We can also see that the columns contain no null values — and a clean DataFrame is always more than welcome.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/297/1*iJqJBYo-3qoSCNXZR85ayg.png" /></figure><p>While all is fine and dandy in the no-null neighborhood, things aren’t going so smoothly in other areas (it never does — that’s just a fact in data cleaning).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*_QZ7_Xlu4rZV6jQki8r-4Q.png" /></figure><p>We see from a simple df.describe().T that two things stand out: Education and Marriage.</p><h4>Education</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vMuNdToMacLskyfIQHNqBQ.png" /></figure><h4>Marriage</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xJ2Irot06W0ixXuqNSEV3A.png" /></figure><p>These don’t align with the variable breakdown as per <a href="https://www.kaggle.com/uciml/default-of-credit-card-clients-dataset">the dataset</a>, thus I have decided to group the weird entries as ‘<em>others</em>’ instead.</p><h4>Summarising data preprocessing</h4><p>Here is a quick recap of pre-processing manipulations we did.</p><ol><li>deault.payment.next.monthto DEFAULT</li><li>SEXto GENDER</li><li>PAY_0 to PAY_1</li><li>Grouped unknown EDUCATIONcategories (0,5,6) and re-assigned them to 4 (others)</li><li>Grouped unknown MARRIAGEcategories (0) and re-assigned them to 3 (others)</li></ol><h3>Section 2: Data Exploration</h3><h4>Section 2.1: Mapping the Target</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*QyamB1WWY34AahVfRJh8ug.jpeg" /><figcaption>Pretty skewed data if you ask me</figcaption></figure><p>Upon closer inspection of our dataset, we can see that there is some class imbalance, something of which we have to keep in mind when evaluating the efficacy of our model (i.e., we cannot use accuracy to measure our model’s effectiveness). Additionally an imbalanced class will also affect the performance of the trained model. There are, however, a few ways to deal with class imbalance:</p><ol><li>Oversample the Dataset</li><li>SMOTE the Dataset</li><li>Undersample the Dataset</li><li>Do nothing (yes, you heard me right. do nothing. sometimes, it gives you better results than fiddling with the dataset)</li></ol><h3>Section 3: Methodology</h3><h4>Section 3.1: Vanilla (untouched) Dataset and Models Used</h4><p>Now that we have our dataset prepped and features finalized, we are ready to do some training/cross-validation to get a baseline feel of how our models perform on the data.</p><p>The models I have selected for use are:</p><ol><li>Gaussian Naive Bayes</li><li>Logistic Regression</li><li>K-Nearest Neighbors</li><li>Decision Tree</li><li>Random Forest</li><li>Linear SVC</li></ol><h4>Section 3.2: Model Evaluation Metric</h4><p>As we have mentioned above, <strong><em>accuracy</em></strong> would not be an accurate (pun intended) metric to use to evaluate how each model performs over one another. F1 score will be used instead.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/485/1*Izi7YDL3WNOHqqLU3wrRyw.png" /></figure><p>I have chosen to evaluate the models based on their F1 score because it considers both precision and recall. In essence, we can take the F1 score as a harmonic mean of both the precision and recall.</p><p>In layman terms, <strong><em>Recall</em></strong>, in the case of credit defaulters refers to:</p><blockquote>Out of ALL the defaulters (True Positive and False Negatives), how many did our model actually get correct?</blockquote><p><strong><em>Precision </em></strong>refers to:</p><blockquote>How correct is our model based on it’s own predictions (True Positives and False Positives)?</blockquote><p>Additionally, <strong><em>recall</em></strong> will hold more importance over <strong><em>precision </em></strong>in the case of classifying defaulters, because we want to be able to catch as many potential defaulters as possible so as to not incur losses to the bank. This will be made clearer in the following sections.</p><h4>Section 3.3: Preliminary Model Testing — Obtaining a Baseline Performance using Vanilla Dataset</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/923/1*xPMddk5dlcKibRKuKTu0Kw.png" /></figure><p>From the preliminary results we get from testing our models on the vanilla dataset (no transformation or scaling done on the features), we can see that the F1 score comes out pretty low.</p><p>Now that we have a baseline, we can proceed to work on improving the F1 score.</p><h4>Section 3.3: Model Selection using F1 Score</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T11CTHox5mQS8gumORWQUQ.jpeg" /><figcaption>Model train/5-fold cross-validation on various processing done on dataset</figcaption></figure><p>Each of the 5 models were trained on different variations of the dataset; scaled and unscaled oversampled, undersampled, SMOTE and vanilla datasets. As mentioned previously, there are a few ways to deal with imbalanced datasets. Oversampling, undersmapling and SMOTE were applied in this project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*lylqtugEMkRLdPF_T_rVpw.png" /><figcaption>Scaled Vanilla Dataset</figcaption></figure><p>After doing all the train/cv tests, the dataset that gave us the highest F1 score is the scaled vanilla dataset, with Gaussian Naive Bayes taking the lead at <strong>0.518. </strong>However, the models were all trained on their default settings. Therefore, I have chosen to further hyper-parameter tune the leading two highest F1 scoring models (<strong>k-NN</strong> at <strong>0.417</strong> and <strong>RandomForest</strong> at <strong>0.41</strong>) in a bid to obtain a higher F1 score than <strong>0.518.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/983/1*rObJPuJ8XbV6rkQNfir6AA.png" /></figure><p>Despite hyper-parameter tuning k-NN and RandomForest, they each still gave an F1 score lower than Gaussian Naive Bayes’ <strong>0.51. </strong>Therefore, Gaussian Naive Bayes was chosen as the Model to classify the defaulters.</p><h4>Section 3.4: Optimizing Gaussian Naive Bayes for Credit Card Default Classification</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*5lZYJXXghccTHL1I_Xn8kg.jpeg" /><figcaption>Train/CV Split: Gaussian Naive Bayes Confusion Matrix (<strong>Threshold = 0.5</strong>)<strong> </strong>on scaled dataset</figcaption></figure><p>Upon plotting the confusion matrix of the Gaussian Naive Bayes with it’s default threshold of 0.5 (with F1 score of 0.517), we observe that it gives a very poor recall score. The above matrix tells us that the model only catches 56% of all the defaulters (570 out of 1019). Which means that <strong><em>misses</em></strong> <strong><em>44% of all the defautlers</em></strong>. This definitely does not spell good news for our hypothetical bank (very similar to credit card frauds) because we are letting a lot of non-loan-returning credit card defaulters go undetected. Therefore, there is a need to further optimize the threshold to increase the model’s recall.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*Pb90LV1xOgDZ2p2PqT-cuQ.jpeg" /><figcaption>Train/CV Split: Gaussian Naive Bayes Confusion Matrix (<strong>Threshold = 0.25</strong>)<strong> </strong>on scaled dataset</figcaption></figure><p>After performing some optimzation, it is found that at a threshold of 0.25 gives a better <strong><em>recall </em></strong>score<strong><em> </em></strong>of <strong>0.76</strong> (778 out of 1019) at the expense of decreased precision (we see that the False Positives increased to 1847).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*lE4Mi-4LHLK9ifVqsw1Row.jpeg" /><figcaption>More false positives with <strong>threshold = 0.25</strong></figcaption></figure><p>However, we have to understand this is an unavoidable tradeoff. A higher recall on the same model will lead to a <em>lower precision (higher False Positives) </em>since we essentially just adjusting the threshold, and not changing the model itself.</p><h4>What does a higher False Positive mean?</h4><p>A higher <em>False Positive </em>means that the model classifies more people as defaulters despite them not being actual defaulters. From our hypothetical bank’s point of view, this means that we will be inconveniencing people by labelling them as potential non-loan-returners.</p><p>The key consideration here is — is this tradeoff justifiable? How do we justify it as a good tradeoff? This actually depends on the goals of the business that you are building this model for, and the level of accuracy desired.</p><p>Since we don’t have an actual business advocating a score, we will decide a hypothetical case for ourselves — that we want to be more risk-adverse than precise. Therefore, having a <em>higher recall</em> to catch more defaulters<em> </em>outweighs the cost of inconveniencing more people (<em>lower precision).</em></p><h3>Section 4: Final Model Testing and Result Interpretation</h3><p>Now with our finalized permutation, using a <em>Gaussian Naive Bayes </em>model with a <em>threshold of 0.25</em> on a <em>scaled dataset</em>, we are ready to do our final testing by training our model on 80% of the original data (that we did our train/cross-validation on) and testing on the other 20% unseen holdout data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*XO7Ggk0xIMDJkrJDDBu7rQ.jpeg" /></figure><p>Final tests show that our model actually performs even better on unseen data — giving us a higher <strong><em>recall</em></strong> score of<strong> 0.79</strong> (1058 out of 1341). This means that our model is able to accurately catch around <strong><em>80%</em></strong> of all the defaulters.</p><h3>Section 5: Future Work</h3><p>This project was completed over a span of 2 weeks whilst juggling daily classes from 8am to 5pm. So of course, there is more that we can do to improve our model and analysis of this credit card defaulter case.</p><p>Given more time and resources, I’d like to do work on the following:</p><ul><li>Feature Engineering; coming up with new features given the current dataset that could potentially be better predictors of credit card defaulters</li><li>Hyper-parameter tune other models</li></ul><h3>Section 6: Interesting Insights from Dataset</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r2KTF5y9hWJm-9RgxDsL_w.png" /><figcaption>Feature Importance as plotted by Random Forest</figcaption></figure><p>From this graph, we can derive some interesting insights behind the behaviour of a defaulter. The top 3 telling drivers of whether someone is a defaulter or not boils down to their:</p><ul><li><strong><em>Repayment status in just 1 month (</em></strong>which means that just by looking at how late someone is on their payments in 1 month, we can tell if they will default or not)</li><li><strong><em>Age</em></strong></li><li><strong><em>Limit Balance</em></strong></li></ul><h3>Final Section: Lessons Learned</h3><h4>Feature Manipulation</h4><p>In this case, categorical columns like PAY_0 , …, PAY_6,MARRIAGE,EDUCATION may not have been represented the dataset in the best way. A better way could have been to do one-hot encoding (creating dummy variables) instead. Classifiers like RandomForests are great at segregating columns like these and could have resulted in a model that leads to better prediction of credit card defaulters.</p><h3>Parting Words</h3><p>As always, you can find the code to this project <a href="https://github.com/xianjinseow92/projects/tree/master/Project_3_Credit_card_default_classifier">here</a>, and my linkedin <a href="https://www.linkedin.com/in/seow-xian-jin/">here</a>. Please feel free to drop me a message should you have any questions regarding this project. Till next time!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f4b21547a618" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/catching-a-welcher-classifying-a-credit-card-defaulter-f4b21547a618">Catching a Welcher: Classifying a Credit Card Defaulter</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Project 2: Worth it? Predicting the Price of Used Cars in Singapore]]></title>
            <link>https://medium.com/data-science/project-2-worth-it-predicting-the-price-of-used-cars-in-singapore-e9afe63c75d0?source=rss-ca9d66f32b7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/e9afe63c75d0</guid>
            <category><![CDATA[car-prices]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[linear-regression]]></category>
            <dc:creator><![CDATA[Xian Jin Seow]]></dc:creator>
            <pubDate>Sun, 03 Nov 2019 14:02:19 GMT</pubDate>
            <atom:updated>2019-11-24T02:38:07.593Z</atom:updated>
            <content:encoded><![CDATA[<h3>Worth it? Predicting the Price of Used Cars in Singapore</h3><h3>Introduction</h3><p>The purpose of this project is to conduct a mini-project that encompasses the bulk of the data science process — from data collection (web-scraping: BeautifulSoup, Python), data cleaning, exploratory data analysis, model training and testing stage. The source of data comes <a href="https://www.sgcarmart.com/main/index.php">SgCarMart</a>, an online car sales portal in Singapore.</p><p><strong><em>Overview:<br></em></strong>The project was focused around a few core skills:</p><ul><li><em>Webscraping</em></li><li><em>Data Cleaning</em></li><li><em>Exposure to sklearn, statsmodel</em></li><li><em>Utilizing OLS Regression</em></li><li><em>Eliminating features that had an insignificant effect on the response variable by evaluating the p-values and R² value of the mode</em></li></ul><p>Buying a car may just be one of the biggest purchases in your life (especially in Singapore, where we are notorious for the exorbitant prices of our cars).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ISSDzCiVLEk9Lj_oMnLO-A.png" /><figcaption>Vin Diesel also seems to agree (<a href="https://www.asiaone.com/static/multimedia/gallery/130523_ffcars/pic35.html">photo credits here</a>)</figcaption></figure><p>As such, one should invest considerable time in thinking about what one is really looking for if one wants to own a car in Singapore.</p><p>However, there is a lot of complexity that goes into what determines the price of a car in Singapore (think taxes, COE, salesman up-selling and etc). Therefore, it can be a frustrating and time-consuming process to quantify what a value-for-your-money purchase of a car would be like.</p><h3>Objectives of this Project</h3><p>To bridge this gap, this project aims to <strong><em>utilize data</em></strong> to implement a <strong><em>Linear Regression Model</em></strong><em> </em>to investigate the factors that affect the price of <strong><em>used </em></strong>cars, and provide a form of a benchmark for comparison and quantify what a reasonable price range of a car would be.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0wi3K7Uz59PmwOu-1x1sZA.png" /></figure><p>The hope is that my model will be able to predict the price of a car given a few typical, easily accessible characteristics of a used car to inform potential car-owners to make better purchase decisions.</p><p>In this project, only passenger vehicles are considered (this is for personal use, after all). Commercial vehicles such as lorries/trucks, buses and vans that the common population do not use are not included in this analysis.</p><h3>Section 1: Web-Scraping</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/341/1*WO-0nEPd0rTUnkCGvwYkqQ.jpeg" /><figcaption><a href="http://aitechcentral.com/blog/web-scraping-techniques-python-beautifulsoup/">Photo credits</a></figcaption></figure><p>Ah, yes. The most tedious and time-consuming part of the project. Ask anybody who has been through Metis’s Bootcamp and they’d shudder when you mention “web-scraping”. I have written the code for web-scraping sgcarmart (including try-except functions to deal with errors that can come up during the actual scraping itself) should you intend to use it. The code can be found <a href="https://github.com/xianjinseow92/projects/tree/master/Project_2_SgCarMart%20Price%20Prediction_2">here</a> on my github.</p><p>The web-scraping process was done using Python 3.7.3 in conjunction with a package called BeautifulSoup.</p><p>This is done through iterating through <a href="https://www.sgcarmart.com/used_cars/listing.php?BRSR=0&amp;RPG=20&amp;AVL=2&amp;VEH=1">search pages containing 100 search results of individual listings of used cars</a>. The <a href="https://www.sgcarmart.com/new_cars/newcars_overview.php?CarCode=12518">links of each individual listing</a> is then appended to a list, and each individual link is accessed to scrape the features of a used car. Upon scraping these features, they are then appended to a dataframe.</p><h3>Part 2: The Data</h3><p>After web-scraping, my dataframe contained 3987 rows (observations) and 17 columns (features) of data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/996/1*Qs5SNIUo2oVB2EgYmUiseA.png" /><figcaption>A random sample of 5 rows of my dataframe</figcaption></figure><p>Upon inspection, there are null values in my dataframe. This is attributed to the fact that sometimes, not all sellers put up complete information of a car in a listing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*oGsyoIybYAG_vvVOxmq1Vg.png" /><figcaption>As seen above.</figcaption></figure><p>Because these entries are very specific to the car’s listing, I decided to drop rows of data with missing values since it will severely affect the results of my analysis. After dropping the NaN values, my dataframe was left with 2584 rows (observations) and 17 colums (features) of data — which is still a pretty sizable amount for training a Linear Regression Model.</p><p>A <strong><em>new feature, </em></strong>Car Age, was also created to replace “Manufactured Year” because it seemed more intuitive to me to understand how old a car is rather than when it was manufactured.</p><p>The vehicle types and car brands were also appended to the dataframe via the creation of dummy variables.</p><p><strong>Challenge: Segmenting the Vehicles</strong></p><p>Vehicles differ in their levels of prestige from a basic Chery QQ to a fancy Bentley Continental GT. Further insights may also be derived from segmenting vehicles in this dimension. Looking through the data, there are a few potential ways through which we can segment the vehicles:</p><ol><li><strong>Vehicle Type</strong> (VEHICLE_TYPE): There are 7 classes for Vehicle Type, namely Mid-Sized Sedan, Luxury Sedan, Hatchback, MPV, SUV, Sports Car, Stationwagon. Each vehicle is assigned only one Vehicle Type. This classification can be vague because premium and mass-market brands produce vehicles in all of these segments. For instance, a Toyota Camry, a BMW 520i and a Rolls-Royce Ghost are all classified under Luxury Sedan as per sgcarmart’s definition. Vehicle Type is a poor way to segment the vehicles.</li><li><strong>Brand</strong> (BRAND): The brand of a car can tell us a lot about the level of prestige of a vehicle. Just the make (brand) alone tells us a lot about the <strong><em>perceived desirability </em></strong>of a vehicle. For example, when we hear of someone driving a BMW, the first thought the common layman would have is likely “that is a prestigious brand”. Therefore, it makes sense to categorize the Vehicles into their respective brands.</li></ol><p>Based on my understanding of car brands in Singapore (and also with the help of Google), the cars are split into 6 segments are <em>Exotic, Ultra Luxury, Luxury, Mid Level, Economy </em>and <em>Budget</em>.</p><p>The proposed segments and their respective makes are as follows:</p><ol><li><strong>Exotic</strong>: Aston Martin, Ferrari, Lamborghini, McLaren</li><li><strong>Ultra Luxury</strong>: Bentley, Land Rover, Maserati, Porsche, Rolls-Royce</li><li><strong>Luxury</strong>: Audi, BMW, Jaguar, Jeep, Lexus, Lotus, Mercedes-Benz, Mitsuoka, Volvo</li><li><strong>Mid Level</strong>: Alfa Romeo, Chrysler, Infiniti, MINI, Opel, Saab, Volkswagen</li><li><strong>Economy</strong>: Chevrolet, Citroen, Fiat, Ford, Honda, Hyundai, Kia, Mazda, Mitsubishi, Nissan, Peugeot, Renault, Skoda, Ssangyong, Subaru, Suzuki, Toyota</li><li><strong>Budget</strong>: Daihatsu, Proton</li></ol><p>After segmenting the vehicles into their various categories of prestige — there poses a problem of having too many columns. Some of the car makes in the dataset are just too uncommon in Singapore and thus may skew the pricing due to their rarity as opposed to their perceived desirability. Therefore, I renamed the following brands into “Others”.</p><p>7. <strong>Others:</strong> Opel, Ssangyong, Proton, Daihatsu, Fiat, Alfa Romeo, Skoda, Hummer, Aston Martin, Lotus, Ford and Jeep.</p><p>Now that we have our dataframe prepped and ready, it’s time for the next step in building a Linear Regression Model.</p><h3>Part 3: Feature Selection</h3><p>The feature selection is first done without including the <strong><em>make </em></strong>of the cars to get a base case of how much a car would cost based on just their features (mileage, engine capacity, car age, open market value, etc). A heatmap of a correlation matrix of the features with respect to price is plotted in order to easily visualize any predictor variables (features) with multi-colinearity or poor fit with the price.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/997/1*2-NLr-5-72_mjGRXxnz0Bg.png" /><figcaption>Correlation Matrix Heatmap</figcaption></figure><p>We can immediately see from the heatmap (as demoted by the bright red spots) that <strong>OMV </strong>and <strong>ARF </strong>are HIGHLY correlated (<strong>0.96 correlation</strong>). And rightfully so — ARF, Additional Registration Fee (Christ, there are just way too many fees in involved in owning a car in Singapore), is actually value calculated based on a percentage of the <strong>OMV </strong>(Open Market Value) of the vehicle. We also observe that <strong>Engine Capacity </strong>and <strong>Road Tax </strong>have a high correlation — because <strong>Road Tax </strong>is essentially calculated from <strong>Engine Capacity</strong>. We don’t need these predictor variables with a high correlation because they’d just be explaining the same thing. The highly correlated variables are then dropped and we are finally left with this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*C1Ya2isjsHdGHTuCoKvkeQ.png" /><figcaption>A much more managaeable dataset with fewer features</figcaption></figure><p><strong>Pairplot: Visualizing Trends of Features with Price</strong></p><p>Upon plotting the pairplot, we see that the features (top row of the pairplots) follow a somewhat linear relationship with price.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wnT4NkDHuRmcbaujs6Ibjw.png" /><figcaption>Price vs Un-Transformed Features relationships</figcaption></figure><p>However, the data points seem to be a bit spaced out along the y-axis. Therefore, we can still improve the fit the data points by applying a log transformation onto price and mileage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/1*qPz5Roxa3HnyhEirhMfRFQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/1*YeLQN1GkCDliVmeGe877Tg.png" /><figcaption>Price Distribution before (left) and after (right) Applying a Log Transformation</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gDL2i0zGAyieztkL_XDAmg.png" /><figcaption>Log Price and Feature relationships after applying Log Transformation</figcaption></figure><p>Upon applying a log into price, we can see that the features now have a clearer linear relationship with log price (see first row of pairplots).</p><p>I have tried including the Brands of the cars into the Model. However, they introduced too much colinearity and noise into the model, causing it to produce unrealistically good results (very good R² scores). Therefore, they will not be included for this analysis. The finalized features for Training the Model are:</p><p><strong><em>Target Variable (1)</em></strong>: Log Price</p><p><strong><em>Predictor Variables (9):</em></strong><em> </em>Log Mileage (km), Engine Capacity (CC), Car Age (# of years), OMV (Open Market Value, $), COE Price ($), Days of COE Left (# of Days), Curb Weight (kg), Number of Owners (Integer), Transmission (1 or 0)</p><h3>Part 4: Model Selection</h3><p>With the finalized features, I considered using Simple Linear Regression, Lasso, Ridge, ElasticNet and Poylnomial Regression in Modelling the data. The steps below highlight my methodology in the selection of the best Model for this case.</p><p>I first found out what the best Alphas were for Lasso, Ridge, ElasticNet and ElasticNet, and then split the data into 60–20–20 Train / Cross-Validation / Test split for model selection. The best alphas of Lasso, Ridge and ElasticNet was used in this train/cross-validation test.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y79NvkELrCb2f7tsucvPgw.png" /></figure><p>The model is trained on 60% of the whole data set and validated against 20% of the whole data set. This train/validation is done over 5-fold validation for a more comprehensive way in obtaining the R² values for each of the tested models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ShP-naN078-_xTdMNGC4NA.png" /><figcaption>Mean R² scores of each Linear Model over 5-fold validation test</figcaption></figure><p>As seen above, the models all scored a mean R² score of 0.913 over the 5-fold train/cross-validation test, with Polynomial Regression having an exception of 0.829. Since they are so similar, I have chosen to use the most vanilla form of Linear Regression: <strong><em>Simple Linear Regression</em></strong> to model the data for its ease of use (no need for pesky hyperparameters unlike the other models).</p><h3>Part 5: Training / Testing the Simple Linear Regression Model</h3><p>After completing the train/cross-validation, it’s finally time to train the model (phew) on the whole dataset. This time, it is done in an 80–20 split (80% data to train model and 20% unseen data to test the model’s efficacy) to provide the model with more training data to improve its performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8r_F44cOnA_VRa_Vg1MZyg.png" /><figcaption>80% of data used for training, and 20% for testing</figcaption></figure><p>After running the test, the Simple Linear Regression returns an R² score of 0.91. This means that the model explains 91% of the variation of the data, with the other 9% unaccountable for due to random noise.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K-ZOW1C9YHZIB9zzAV4FMw.png" /></figure><h3>Part 6: Using the Model</h3><p>So what does this mean for us? From the results summary, it tells us that the features with the <strong><em>highest coefficients</em></strong> are Car Age and Engine CC (meaning these are the biggest influencers on price!) After doing some backwards conversion (don’t forget — our Target Variable was logged) the model essentially tells us this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rufk62CqyGHJfpK-u8eThw.png" /></figure><p>Results of which we can intuitively understand. The older a car gets, the more it’s value depreciates. Additionally, an increase in engine capacity is usually attributed to a bigger car or a more powerful car. Thus, an increase in engine CC would also lead to an increase in price. While we already know these things intuitively, the model provides us with an estimate to QUANTIFY how much these factors influence an increase in price.</p><p>How can this model be used? Well, with just the typical, easily accessible information of a car (car age, mileage, engine cc, etc.), we can get a benchmark of how much a car would cost based on just its physical features (without considering the brand).</p><p>A user can then cross-check this benchmark with the used car listing price and then decide for himself if he wants to accept a higher price based on the brand or other, special features of a car.</p><h3>Part 7: Insights from Model</h3><p><strong><em>When is a good time to buy a car?</em></strong></p><ul><li>NEWLY launched car models are normally priced higher. Be patient and wait for a while for the price to drop before you make your purchase.</li><li>The only time when prices go down is when <strong><em>COE </em></strong>price <strong><em>decreases</em></strong>. A genuine good buy is when the COE is low.</li></ul><p>And of course, apart from what the model tells us, more advice for anybody making a financial investment such as a car:</p><p><strong><em>Here are a few things you can ask yourself prior to buying a car in conjunction to using the Model:</em></strong></p><ol><li><em>What is my budget?<br></em>Set a budget, and DO NOT exceed it.</li><li><em>What type of car do I need?<br></em>Do you need a large MPV to ferry your whole family? Or is your driving so atrocious that you need a small car in order not to be too much of a road hazard? (I’m talking about myself, obviously.)</li><li><em>What </em><strong><em>brand</em></strong><em> of car do I want?<br></em>Narrow down your selection to a few brands. Choosing from too many brands can be very tiring and time-consuming. Generally, <em>Japanese </em>makes represent the best value-for-money (and interestingly, this is also reflected in the dataset that I have pulled from sgcarmart, with Toyota having the highest frequency of resale amongst all car brands). <em>Korean</em>, <em>Malaysian</em>, and <em>China</em> makes are for buyers with <strong><em>tight budgets</em></strong>.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*tMNEPygBRVWhOVbxJUMp-A.png" /></figure><h3>Part 8: For the Future</h3><p>There is a lot more that can be done with the current model I have created. In the future, I would like to also <strong><em>incorporate brands</em></strong> as part of the model’s predictive capabilities on top of using the physical features of the car for more granularity and customization for a user.</p><h3>Part 9: Interesting Finds</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/190/1*z-vbI_4vjAk_YLFln9zNBg.png" /><figcaption>That’s bollocks</figcaption></figure><p>Mercedes-Benz, a luxury brand, actually came in as the second highest resale car at a frequency of 374 out of the entire 2584 observations. Who would’ve thought? Were people unsatisfied with their Mercedes? Is it too expensive to upkeep? Or perhaps it’s because it’s easy to sell a Mercedes? That’s definitely some food for thought for further exploration.</p><h3>Part 10: Parting Words — Personal Learning Points</h3><ol><li>As much as possible, try not transform your Predicted Variable. It makes it difficult to explain to others how a change in your Predictors lead to an increase in your Prediction. In my case, I applied a log onto the target variable (price). Therefore, the results had to be interpreted in terms of percentage instead of actual, layman values (for eg. a unit increase in x will contribute to a 100 dollars increase in y).</li><li>Scale. The scale of your data is really important. For instance, my price value had such a different scale to mileage. Thus, the coefficients that came up in the model were <strong><em>extremely </em></strong>small. It is difficult for people (including myself) to get a sense of having an increase in 1km of mileage translating to 0.0000001 increase in price.</li><li>Therefore, before beginning any project, always have an audience in mind too. What insights do you want to bring to people? How do you value add to people? Frame your project in a way that you want to sell this model to people, and find the easiest possible way to explain and convince people of its efficacy.</li></ol><p>As always, if you think I can improve my code, if you have trouble reproducing the steps, or if you think this helped you, let me know in the comments!</p><p>You can also contact me on <a href="https://www.linkedin.com/in/seow-xian-jin/">linkedin</a>. And you can find my github <a href="https://github.com/xianjinseow92/projects">repo </a>here.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e9afe63c75d0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/project-2-worth-it-predicting-the-price-of-used-cars-in-singapore-e9afe63c75d0">Project 2: Worth it? Predicting the Price of Used Cars in Singapore</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Week 1 @ Metis Singapore Data Science Bootcamp — Project 1: Exploratory Data Analysis (EDA) on the…]]></title>
            <link>https://medium.com/data-science/week-1-metis-singapore-data-science-bootcamp-project-1-exploratory-data-analysis-eda-on-the-b35b8c5d4417?source=rss-ca9d66f32b7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/b35b8c5d4417</guid>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[seaborn]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <category><![CDATA[time]]></category>
            <dc:creator><![CDATA[Xian Jin Seow]]></dc:creator>
            <pubDate>Sun, 13 Oct 2019 10:48:48 GMT</pubDate>
            <atom:updated>2019-11-26T05:18:46.206Z</atom:updated>
            <content:encoded><![CDATA[<h3>Week 1 @ Metis Singapore Data Science Bootcamp — Project 1: Exploratory Data Analysis (EDA) on the NYC MTA, Turnstiles and Drinking out of the Proverbial Fire Hose</h3><p>If you are interested in the code for this project, you can find it at this <a href="https://github.com/xianjinseow92/projects/tree/master/NYC%20MTA%20Turnstile%20Project">repo</a>.</p><p>My first week at metis was nothing short of difficult and trying. And I’d say coupled with the intense back-to-back lessons and tight deadlines, I can safely say I’d just slept, on average, just 4 hours per night within the past week. And the most challenging to do by far was to set up my github.io (I’m kidding. The most challenging part was the project), of which I have given up on, so you see me here on medium.</p><p>This week’s focus was to get up to speed on the essential, basic tools for data analysis; python, matplotlib, seaborn and our workflow (Minimum Viable Product, Design Flow) centered around a project on MTA Subway Traffic Data. The goal of the project was to help a hypothetical non-profit organization which aimed to promote the awareness and participation of women to come up with an optimized strategy for canvasing.</p><p>We were given a lot of freedom to do what we want (and we all know, with great power comes great responsibility) to interpret the results and provide recommendations in a way we deemed was useful for this hypothetical organization’s purposes.</p><h4>Web Scraping:</h4><p>I’d say coming from a background with no coding whatsoever, the first biggest hurdle for me was to automate a way to collect the data. Of which I am deeply grateful for <a href="https://medium.com/u/75b5f5a46f52">Julia Kho</a>’s post on webscraping MTA Turnstile data, which can be found <a href="https://towardsdatascience.com/how-to-web-scrape-with-python-in-4-minutes-bc49186a8460">here</a>.</p><h4>Exploratory Data Analysis:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*zHnOSJZgb96H4AhfLzGWbw.jpeg" /><figcaption>Average Entries for the Top 7 NYC Stations with the Highest Average Entries</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nAiI-J_1d0UR-pElGaI7Sw.jpeg" /><figcaption>Split-Histograph Data Of an Average Entry by Day of the Week for the Top 7 NYC Stations with the Highest Average Entries</figcaption></figure><p>“Exploratory Data Analysis” (EDA) is a catch-all description of this project’s process, meaning pretty much anything from studying to visualization of a dataset in order to extract meaning from it up until the the point where your current data is useful enough to be doing extensive statistical modeling and/or hypothesis testing.</p><p>From the graphs as we see above, we observe that the stations with the highest volume of people are <strong><em>34 Street — Penn Station</em></strong>, <strong><em>42 Street — Grand Central</em></strong>, and <strong><em>34 Street — Herald Square</em></strong> (of which is true if you do a quick google search).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FJ9oI6NyM0fjZAVkXrc8OQ.jpeg" /></figure><p>Another interesting point to note is that the volume of people on the <strong><em>weekdays are higher</em></strong> as opposed to the weekends — vastly different from what I had expected (prior to scraping the data). I expected that the weekends would yield a higher volume of people as compared to weekdays (which is typical of Singapore).</p><p>Coming up with visualizations that tell a clear story about the data is one of the most enjoyable parts of EDA — these visualizations represent both the finished package of the information you’ve scraped together and your creative spin on how to frame that information. It’s also really interesting because it serves to defy one’s own assumptions and allows one to learn more about the world (being in Singapore, and not in NYC. I would’ve never known the traffic patterns in a foreign country without deep-diving into the data itself).</p><p>But before getting there, you usually have to spend some time battling the data to get it into a usable format with meaningful information. How my teammates and I felt after finishing up data cleaning can be best summed up in the picture below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/338/1*6zkDn9jvBnvXDc1DIOh0HA.gif" /><figcaption>True story</figcaption></figure><h4>Further Analysis:</h4><p>I’d like to delve deeper into this and instead <strong><em>only </em></strong>understanding the traffic by day, I’d also like to know how it fluctuates by the time of the day. Is it more packed in the Morning? Afternoon? Evening? Perhaps even at Night? There is so much more we can riff off and interpret from the data, and should time permit, I’d definitely be coming back to revisit this dataset.</p><h4>Shortcomings in Metrics Used:</h4><p><strong><em>Entry vs Exit</em></strong></p><p>A great data scientist is aware of their own biases, and must take heed and question the credibility of his/her hypothesis and test for accuracy. The metric we used here is ENTRY (the number of people ENTERING the station). But is using ENTRY as a metric effective? Why not consider exits? Would people be more receptive to canvasing (receiving flyers, more open to talking to surveyors, giving out their emails, etc.) EXITING the station instead? There are so many things to consider and should time permit, I would like to revisit this idea.</p><h4>Final Reflection:</h4><p>This first week has truly been akin to drinking out of a fire hose; from being bombarded with pair programming exercises first thing in the morning to theory-dense topics like time-complexity of code, a lot of post-class study has to be done in order to keep up with the material at hand. But oddly, it is vastly satisfying to be this busy, and to realize that there are yet so many interesting things out there to learn.</p><p>As a final reflection on what I’ve learned this week, I want to turn to a central problem in working life that’s often overlooked in the context of a classroom. When my instructor introduced the MTA project at the beginning of the week, he gave us a pointed warning that our deadlines at bootcamp will be unfair and that we need to <strong><em>abandon perfectionism</em></strong>. There’s always another step to take to try to make the data more polished, to expand the scope of your project and analysis, or to adjust your model for even slightly better accuracy. Yet in the real world, <strong><em>time is the most important resource</em></strong> and knowing when to constrain yourself is often <strong><em>more valuable than the quality of your ideas</em></strong>.</p><p><strong>Things I’d like to improve on and work on next:</strong></p><ol><li>Project management with git and gitHub</li><li>Learning to set proper deadlines for each stage of the project, i.e. webscraping, data cleaning, data processing, data exploration, interpretation and results, and then the final presentation. I will keep these in mind for the next project I do to dutifully practice the proper skillsets/habits in becoming not only a intelligent data scientist, but a practical one that understands real-world deadlines in delivering a useful product!</li></ol><p>Look forward to my next post!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b35b8c5d4417" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/week-1-metis-singapore-data-science-bootcamp-project-1-exploratory-data-analysis-eda-on-the-b35b8c5d4417">Week 1 @ Metis Singapore Data Science Bootcamp — Project 1: Exploratory Data Analysis (EDA) on the…</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introductory Post to.. Well, me.]]></title>
            <link>https://medium.com/@xianjinseow92/introductory-post-to-well-me-4c22ad113cb4?source=rss-ca9d66f32b7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/4c22ad113cb4</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Xian Jin Seow]]></dc:creator>
            <pubDate>Sun, 13 Oct 2019 09:08:33 GMT</pubDate>
            <atom:updated>2019-10-30T04:57:06.917Z</atom:updated>
            <content:encoded><![CDATA[<p>S’up everybody. This will be my first post ever in my Medium Blog. Over next 3 months, I will continuously post updates here on my experience in Singapore’s Metis Data Science Bootcamp (this is more of a personal thing — a way for me to track my own progress and also to inculcate discipline and consistency), as well as the projects I embark in over the span of the entire camp.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4c22ad113cb4" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>