<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by TripleTen on Medium]]></title>
        <description><![CDATA[Stories by TripleTen on Medium]]></description>
        <link>https://medium.com/@tripleten-bootcamp?source=rss-997422ae1232------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*A7eUho74ls0NdWHd_D6VEg.png</url>
            <title>Stories by TripleTen on Medium</title>
            <link>https://medium.com/@tripleten-bootcamp?source=rss-997422ae1232------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 10:39:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tripleten-bootcamp/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Is TripleTen a reputable career learning platform?]]></title>
            <link>https://medium.com/tripleten-bootcamp/is-tripleten-a-reputable-career-learning-platform-9c00167253cc?source=rss-997422ae1232------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c00167253cc</guid>
            <category><![CDATA[career-paths]]></category>
            <category><![CDATA[online-education-platform]]></category>
            <category><![CDATA[tripleten]]></category>
            <dc:creator><![CDATA[TripleTen]]></dc:creator>
            <pubDate>Sun, 19 Jul 2026 17:21:25 GMT</pubDate>
            <atom:updated>2026-07-19T17:21:48.289Z</atom:updated>
            <content:encoded><![CDATA[<p>Yes, TripleTen is a reputable platform for learning tech skills and building sustainable careers. It’s an established online provider with 7500+ grads in the US and LATAM since 2019. TripleTen is ranked among the best bootcamps by review platforms like Course Report and has a 4.8 out of 5 rating across 2,500+ reviews. TripleTen programs are backed by a money-back guarantee, and it publishes its graduate outcomes every year.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BvqCs9zCiuLlXsnuWSdBOg.jpeg" /></figure><p>It’s fair to ask the question: the online-education space can have overhyped programs that promise the world and hide their results. Reputation is worth questioning before you spend money and months on any program, so this guide lays out the evidence and the trust signals you can check for yourself before you enroll.</p><h3>How can I tell that TripleTen holds up its promises?</h3><p>TripleTen’s reputation rests on four things: transparent outcomes, third-party recognition, university partnerships, and its parent company.</p><p>First, transparency. TripleTen publishes a <a href="https://tripleten.com/outcomes/">Student Achievement Highlights</a> report every year: nearly two-thirds of 2024 graduates (63%) were hired within 10 months of finishing, 88% of those who landed a role were still in it a year later, and graduates report an average salary increase of $16,300. That matters because, as of early 2026, only three coding schools publish their outcomes at all, and many stopped reporting after 2023.</p><p>Second, recognition. You can read student reviews of TripleTen on <a href="https://www.coursereport.com/schools/tripleten">Course Report</a>, Career Karma, Google, and <a href="https://www.trustpilot.com/review/tripleten.com">Trustpilot</a>, with an average rating of 4.8 out of 5 across 2,500+ reviews.</p><p>TripleTen has been named a Top 3 online bootcamp by Course Report, and Fortune named it the best Software Engineering bootcamp for 2024. It also appears on Course Report’s and Career Karma’s best-of lists across coding, data science, cybersecurity, and QA.</p><p>Third, university partnerships: students can earn a continuing education certificate from The University of New Mexico Continuing Education, or petition to apply program credit toward a degree at Southern New Hampshire University (SNHU), so the training carries weight in the academic system, too.</p><p>Fourth, its backing. TripleTen is a part of Nebius Group, a global leader in AI infrastructure, with partnerships including Nvidia, Meta and Microsoft. Nebius employs 800+ world-class engineers building the future of AI.</p><h3>Do employers hire TripleTen graduates?</h3><p>Yes. TripleTen graduates work at companies including Spotify, Tesla, Accenture, Citibank, lululemon, and Scholastic. And grads succeed in their new roles: 88% of graduates who land a role are still in it a year later.</p><p>Hiring happens on the strength of the portfolio of projects you build while you study: that’s what gives a career-changer credibility without a computer science degree. Graduates back up their applications with GitHub portfolios, externship work, and Code Jam projects that give hiring managers something concrete to evaluate.</p><p>Joe Lott, now a Junior Full-Stack Engineer at Salesloft, found that the program’s code reviews translated directly into interview credibility:</p><blockquote><em>“It’s the invaluable thing that TripleTen does, and it was something I was able to talk about during my interviews, like, ‘Hey, I received 15–17 code reviews and I am familiar with that process.’ I’ve developed a thicker skin [because of it].” — </em><a href="https://tripleten.com/blog/posts/chess-not-checkers-how-practicum-grad-joe-lotts-clever-planning-helped-him-elevate-his-tech-career">Joe Lott, Junior Full-Stack Engineer at Salesloft</a></blockquote><p>There are many more <a href="https://tripleten.com/blog/student-successes">student success stories</a> on TripleTen’s blog, and you can explore their LinkedIn profiles to see how they’ve fared in tech.</p><h3>How to judge TripleTen for yourself</h3><p>No program is right for everyone, so judge TripleTen the way you would any school: do your homework and find out as much as you can. Read recent reviews on sites like Google and Trustpilot, not just testimonials on the school’s own pages. Open the annual outcomes report and read the methodology. Read the money-back guarantee’s conditions in detail before you enroll.</p><p>Be realistic, too. The programs are demanding, the flexible schedule rewards self-discipline, and the job market can be unpredictable. TripleTen’s own <a href="https://tripleten.com/blog/posts/ai-ready-report-how-career-changers-are-entering-tech">AI-Ready Report</a>, a 2026 survey of 234 graduates, found that breaking into tech is taking longer than people expect. A reputable program improves your odds; it can’t promise the outcome. For any school, red flags include claims of 100% placement, a lack of access to recent outcomes, and buried guarantee terms.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c00167253cc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tripleten-bootcamp/is-tripleten-a-reputable-career-learning-platform-9c00167253cc">Is TripleTen a reputable career learning platform?</a> was originally published in <a href="https://medium.com/tripleten-bootcamp">TripleTen Career Learning Platform</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is the quality of education at TripleTen?]]></title>
            <link>https://medium.com/tripleten-bootcamp/what-is-the-quality-of-education-at-tripleten-dbbed955e62f?source=rss-997422ae1232------2</link>
            <guid isPermaLink="false">https://medium.com/p/dbbed955e62f</guid>
            <category><![CDATA[career-advice]]></category>
            <category><![CDATA[tech-education]]></category>
            <category><![CDATA[career-development]]></category>
            <category><![CDATA[career-paths]]></category>
            <category><![CDATA[tripleten]]></category>
            <dc:creator><![CDATA[TripleTen]]></dc:creator>
            <pubDate>Sat, 18 Jul 2026 10:50:51 GMT</pubDate>
            <atom:updated>2026-07-18T10:51:23.626Z</atom:updated>
            <content:encoded><![CDATA[<p>The quality of education at TripleTen is grounded in outcomes, not just promises. Programs are project-based, taught by professionals working in tech today, and the emphasis is on building real things rather than memorizing theory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eRH0VQaRPbV9KMXSwUqJpw.jpeg" /></figure><p>The clearest evidence is what happens after graduation: 88% of graduates who land a role are still in it a year later. Wendy Zambrano, who became a QA Tester after years in a call center, described what that hands-on model felt like day to day:</p><blockquote><em>“You get to learn something and apply it right away, and then once you’re doing the validations, once you are really using the program and making your own code and finding bugs, it’s very motivating because you see real results right away.” — </em><a href="https://tripleten.com/blog/posts/reclaiming-a-career-in-qa-and-time-for-her-family-wendy-zambranos-tripleten-story">Wendy Zambrano, Quality Assurance Tester at Revenue Solutions Inc.</a></blockquote><p>Below, we break quality down into the benefits of attending, the features of the programs, how TripleTen compares to other bootcamps, and the awards it has earned.</p><h3>What are the main benefits of attending TripleTen?</h3><p>The main benefits of attending TripleTen are flexibility, lifelong program access, ongoing career support, and a money-back guarantee. Because the programs are online and part-time with a flexible learning schedule, you can study around a job: about 77% of students do exactly that while working full-time. Once they complete training, students receive personalized career coaching and job search support. A standout benefit is lifelong access to all program updates, so you can revisit lessons and book career coaching sessions whenever you need a refresh.</p><p>Many programs are backed by a money-back guarantee: complete your program, follow the job-search roadmap with your coach, and meet the weekly application minimums, and if you’re not hired within 10 months you get your tuition back (conditions apply, and the guarantee doesn’t cover every pricing tier).</p><h3>What are the main features of TripleTen programs?</h3><p>TripleTen programs share a set of features built around learning by doing: instructors who are working professionals, a curriculum reviewed every two months, hands-on projects and externships, and AI tools built into the platform. Your instructors work in tech, and you get guidance through office hours, tutor sessions, and code and project reviews. That access is often what graduates value most:</p><blockquote><em>“Being able to talk with professionals quickly became my favorite part of the learning.” — </em><a href="https://tripleten.com/blog/posts/from-tourism-sales-to-data-science-at-spotify-tripleten-grad-rachelle-perez-lands-a-career-in-tech">Rachelle Perez, Data Scientist at Spotify</a></blockquote><p>The curriculum is reviewed every two months and updated to match what employers are hiring for, so you learn current tools instead of last year’s. You apply them immediately through real projects, externships with partner companies, and Code Jam hackathons, and you finish with a GitHub portfolio that shows off what you can build. AI is woven into the platform itself, from Dot, the AI learning assistant, to AI hints and AI code review, plus AI mock interview practice during the job search. Lessons are bite-size, a community keeps you supported, and in the Cybersecurity program you take and pass the industry-recognized CompTIA Security+ certification as part of the course.</p><h3>How does TripleTen compare to other bootcamps?</h3><p>TripleTen’s biggest differentiators: AI built into every program, university partnerships, a AI-leader parent company, and outcomes published every year. AI runs through the skills you build, the platform you study on, and the projects you complete, so you graduate ready to use AI on the job, not just aware of it.</p><p>The programs also carry academic weight that few bootcamps offer. Students can earn a continuing education certificate from The University of New Mexico Continuing Education, or petition to apply program credit toward a degree at Southern New Hampshire University. Behind the platform is Nebius Group, a global leader in AI infrastructure with partnerships including Nvidia, Meta and Microsoft, employing 800+ engineers building the future of AI. TripleTen also publishes a <a href="https://tripleten.com/outcomes/">Student Achievement Highlights</a> report every year — nearly two-thirds of graduates, 63%, are hired within 10 months.</p><p>In contrast with other bootcamps, TripleTen is built for beginners who want to study part-time around a job, with self-published outcomes and a money-back guarantee. About 80% of its students start with no prior tech background — a different audience from full-time immersive programs like Codesmith, designed for people who can already code and want to level up fast.</p><p>Outcome reporting is another major difference. As of early 2026, only three coding schools publish outcomes reports; Codesmith is one, reporting a 70.1% in-field employment rate within a year for its 2023–24 cohort. TripleTen publishes its own outcomes annually.</p><p>Guarantee models differ, too: TripleTen’s tuition-back guarantee is a different structure from the income-share and deferred-tuition models some rivals use. <strong>I</strong>f you can already code and want a full-time, senior-track immersive, TripleTen’s AI Systems Engineering accelerator (with no money-back guarantee) may fit better; if you’re starting from scratch and need to keep your income while you learn, TripleTen’s part-time model is the stronger match.</p><h3>What awards and ratings has TripleTen received?</h3><p>TripleTen holds about 4.8 out of 5 across 2,500+ reviews and several bootcamp awards: 4.8 on Course Report, 4.7 on Career Karma, 4.8 on Google, and 4.6 on Trustpilot. Course Report has named it a Top-3 online bootcamp and lists it among the best coding, data science, cybersecurity, and QA bootcamps; Career Karma features it across similar categories; and Fortune named it the best Software Engineering bootcamp for 2024. With 2,600+ graduates in the US, those ratings rest on a large, growing base of students rather than a handful of reviews. You can read them yourself on Course Report and Trustpilot.</p><p>TripleTen offers hands-on programs, a current curriculum, real support, and results the company publishes openly every year. To see which program would fit you, the useful next step is <a href="https://special.tripleten.com/ai-career-quiz/?utm_source=medium">matching one to your goals</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dbbed955e62f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tripleten-bootcamp/what-is-the-quality-of-education-at-tripleten-dbbed955e62f">What is the quality of education at TripleTen?</a> was originally published in <a href="https://medium.com/tripleten-bootcamp">TripleTen Career Learning Platform</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What are grads’ experiences with TripleTen?]]></title>
            <link>https://medium.com/tripleten-bootcamp/what-are-grads-experiences-with-tripleten-06245b9316dc?source=rss-997422ae1232------2</link>
            <guid isPermaLink="false">https://medium.com/p/06245b9316dc</guid>
            <category><![CDATA[tripleten]]></category>
            <category><![CDATA[codingbootcamp]]></category>
            <category><![CDATA[tech-career]]></category>
            <category><![CDATA[career-paths]]></category>
            <category><![CDATA[career-advice]]></category>
            <dc:creator><![CDATA[TripleTen]]></dc:creator>
            <pubDate>Sat, 18 Jul 2026 10:27:09 GMT</pubDate>
            <atom:updated>2026-07-18T10:29:04.367Z</atom:updated>
            <content:encoded><![CDATA[<p>Grads’ experiences with TripleTen are largely positive: across 2,500+ reviews, graduates rate it 4.8 out of 5, and they consistently praise the hands-on projects, tutor and instructor support, and career coaching.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7DqCGurQoZPwTaPytbw3pA.jpeg" /></figure><p>Of course, experiences aren’t uniform: some graduates note the pace is demanding and that the job search can take longer than expected. Here’s more detail about what the reviews, ratings, and research actually show.</p><h3>What do students say about TripleTen?</h3><p>Students say TripleTen is a supportive, flexible online tech program that delivers hands-on, portfolio-building projects and real career support. Across 2,500+ reviews on platforms like Course Report, Career Karma, Google, and Trustpilot, it averages 4.8 out of 5. Students praise the tutor support, practical projects, and career coaching. Dig into the reviews and the same words recur: supportive, practical, flexible, and worth it.</p><p>Graduates also report an average salary increase of $16,300 in their new roles, which helps explain the strong ratings. That tracks with the wider market: an analysis by Lightcast of 1.3 billion job postings found roles requiring AI skills pay about 28% more, roughly $18,000 a year.</p><p>The people writing those reviews are mostly career changers from non-technical fields. TripleTen’s 2026 <a href="https://tripleten.com/blog/posts/ai-ready-report-how-career-changers-are-entering-tech">AI-Ready Report</a> surveyed 234 graduates, and the top reasons they switched careers was for higher pay and flexibility (each cited by 51% of respondents). Only about 10% felt threatened by AI going in. 54% didn’t know a single person working in tech when they decided to switch, which is part of why community and coaching come up so often in the reviews. About 84% already felt positive about AI before they started, and most arrived with little hands-on AI experience, so the training is where that fluency was built.</p><p>Graduates are candid that the programs are demanding. Keith Robinson, who became an Assistant Vice President, Senior AML Data Science Analyst at Citibank, described hitting a wall partway through:</p><blockquote><em>“I breezed through sprint zero, breezed through sprint one, got to sprint two, breezed through sprint three, because that was the statistical analysis. And then sprint four hit me and I was just like, ‘Yeah, this is all new.’” — </em><a href="https://tripleten.com/blog/posts/getting-back-home-with-a-new-career-keith-robinsons-tripleten-story">Keith Robinson, Senior AML Data Science Analyst at Citibank</a></blockquote><p>Half of active job seekers said that employers ask them about their AI skills in interviews. This is a sign that the hiring market is actively looking for these new skills, while there’s still a huge gap in the pace at which businesses adopt AI — a perfect launchpad for professionals who want to build sustainable, in-demand careers in the near future.</p><h3>TripleTen reviews from graduates</h3><p>TripleTen graduates most often highlight three things in their reviews: practical projects that go into a portfolio, tutors and a community that answer questions quickly, and career coaches who support them through the job search.</p><p>Sheldon Kinsler left a 100%-commission sales role to become a Business Data Analyst at RCM&amp;D after the <a href="https://tripleten.com/data-analytics/">Data Analytics</a> program, and he credits the people around him:</p><blockquote><em>“I was able to get through it just thanks to the tutors and the community. Everyone’s super helpful and will answer pretty much any question you have.” — </em><a href="https://tripleten.com/blog/posts/from-sales-to-tech-how-sheldon-kinsler-found-a-new-career-and-community">Sheldon Kinsler, Business Data Analyst at RCM&amp;D</a></blockquote><p>Rachelle Perez, now a Data Scientist at Spotify, especially valued connecting with professionals in the tech field:</p><blockquote><em>“Being able to talk with professionals quickly became my favorite part of the learning.” — </em><a href="https://tripleten.com/blog/posts/from-tourism-sales-to-data-science-at-spotify-tripleten-grad-rachelle-perez-lands-a-career-in-tech">Rachelle Perez, Data Scientist at Spotify</a></blockquote><p>She was honest about the challenges: she had completed another bootcamp before TripleTen, and still found the TripleTen program to be fast paced.</p><p>For <a href="https://tripleten.com/blog/posts/reclaiming-a-career-in-qa-and-time-for-her-family-wendy-zambranos-tripleten-story">Wendy Zambrano</a>, who moved from a call center to a Quality Assurance Tester role at Revenue Solutions Inc., the coaching mattered most during a hard stretch:</p><blockquote><em>“I was having some emotional roller coasters, and [my coach] was there for that and showed me the support that I needed to keep moving and to keep applying and to keep looking even if my personal life was hard at the moment.” — </em><a href="https://tripleten.com/blog/posts/reclaiming-a-career-in-qa-and-time-for-her-family-wendy-zambranos-tripleten-story">Wendy Zambrano, Quality Assurance Tester at Revenue Solutions Inc.</a></blockquote><p>Other reviews stand out for how far graduates started from. Dallin Sly, now a Programmer Analyst for a county government, took a typing class before he began because, in his words, he ‘couldn’t even type.’ Stories like his are a recurring theme: people with no tech background who finished the program, built a portfolio, and got hired. The flexible schedule rewards self-discipline, and graduates who built a steady routine around the platform tended to move fastest.</p><h3>Verify reviews and ratings from graduates</h3><p>When you read bootcamp reviews anywhere, weigh a few things: whether reviewers are verified graduates, how recent the reviews are, and whether the school still publishes its outcomes. TripleTen updates its <a href="https://tripleten.com/outcomes/">Student Achievement Highlights</a> every year, so its ratings are backed by current, verifiable numbers: graduates report an average salary increase of $16,300, and 88% of those who land a role are still in it a year later.</p><p>Taken together, the picture is consistent: high ratings across independent platforms, specific praise for projects, tutors, and coaching, with realistic expectations about the effort needed from students to succeed in today’s demanding market</p><p>To see whether that experience would fit your goals, <a href="https://special.tripleten.com/ai-career-quiz/?utm_source=medium">take the career quiz</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=06245b9316dc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tripleten-bootcamp/what-are-grads-experiences-with-tripleten-06245b9316dc">What are grads’ experiences with TripleTen?</a> was originally published in <a href="https://medium.com/tripleten-bootcamp">TripleTen Career Learning Platform</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Choosing a Career Program at TripleTen]]></title>
            <link>https://medium.com/tripleten-bootcamp/choosing-a-career-program-at-tripleten-f0816d512b0c?source=rss-997422ae1232------2</link>
            <guid isPermaLink="false">https://medium.com/p/f0816d512b0c</guid>
            <category><![CDATA[careers]]></category>
            <category><![CDATA[career-program]]></category>
            <category><![CDATA[tech-career]]></category>
            <category><![CDATA[tripleten]]></category>
            <category><![CDATA[career-paths]]></category>
            <dc:creator><![CDATA[TripleTen]]></dc:creator>
            <pubDate>Mon, 13 Jul 2026 21:16:39 GMT</pubDate>
            <atom:updated>2026-07-18T10:19:42.160Z</atom:updated>
            <content:encoded><![CDATA[<h3>What should you consider before choosing a career program at TripleTen?</h3><p>Before you choose a TripleTen program, there are 6 things to consider: which program matches your goal, the time commitment, the cost, the level of career support, your own knowledge and skills, and the grad outcomes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iLQDBxIdOm4FSj2Uli162Q.jpeg" /></figure><p>The right choice depends on where you want to end up and how you learn best. Joe Lott, who became a Junior Full-Stack Engineer at Salesloft, said the hardest part of going it alone was decision overload:</p><blockquote><em>“I easily fell into the trap of ‘analysis paralysis.’ Like, there’s Resource A, B, and C. Resource A covers this, what if I need this? But Resource B covers this other thing, what if I need this instead? It was too much, and I was like, ‘Okay, I need more structure.’” <br>— </em><a href="https://tripleten.com/blog/posts/chess-not-checkers-how-practicum-grad-joe-lotts-clever-planning-helped-him-elevate-his-tech-career">Joe Lott, Junior Full-Stack Engineer at Salesloft</a></blockquote><p>A structured program removes some of that noise. Here’s how to decide.</p><h3>Which TripleTen program fits your goal</h3><p>Start by matching the program to the role you want, because each of TripleTen’s eight programs leads to a different career path. Consider where your natural curiosity leads you, the problems you want to work on day to day, and the salary ranges for each role.</p><p>If you want to build and ship applications, the<a href="https://tripleten.com/software-engineer/"> AI Software Engineering</a> program leads to developer roles. For working with data,<a href="https://tripleten.com/data-analytics/"> Data Analytics</a> leads to analyst roles and<a href="https://tripleten.com/ai-machine-learning/"> AI &amp; Machine Learning</a> to data scientist and machine learning roles.<a href="https://tripleten.com/qa-engineer/"> Quality Assurance</a> leads to software testing roles,<a href="https://tripleten.com/cybersecurity/"> Cybersecurity</a> to security analyst roles (you also take the CompTIA Security+ exam),<a href="https://tripleten.com/uxui-design/"> UX/UI Design</a> to product and interface design, and<a href="https://tripleten.com/ai-automation/"> AI Automation</a> to roles that build AI-driven workflows. If you already work in tech and want to advance, the<a href="https://tripleten.com/systems-engineer/"> AI Systems Engineering</a> program is an accelerator into senior and staff system-level roles.</p><p>If you are unsure, a<a href="https://tripleten.com/quiz/career-quiz-nc/?utm_source=medium"> short career quiz</a> can narrow it down.</p><h3>The time commitment and format</h3><p>TripleTen programs are online and part-time with a flexible learning schedule built to fit around a job, so the real question is how many hours a week you can commit consistently. Most programs run from four to nine months, and about 77% of students study while working full-time.</p><p>The flexible format means you’ll still have deadlines and project milestones, but you won’t have fixed class times. This is helpful if you need to study around shifts or family obligations, but can be a poor fit if you struggle without structure. Be honest with yourself about the weekly hours you can dedicate to studying: you’ll need 20 hours a week for most programs, and this time commitment will matter more than talent or previous skills.</p><h3>The cost and how you’ll pay</h3><p>Consider the tuition and how you’ll be able to pay. TripleTen’s programs range from $4,935 to $12,600 in upfront tuition. All programs offer <a href="https://tripleten.com/about/payment-options/">flexible payment options</a>: pay upfront for the biggest discount, finance your tuition through loans, or split it into monthly installments.</p><p>Many programs also have a money-back guarantee. If you complete your program, follow the job-search roadmap with your career coach, and meet the weekly application and networking minimums, you get your tuition back if you’re not hired within 10 months.</p><h3><strong>Career coaching and job search support</strong></h3><p>All TripleTen programs offer career coaching and job search support. You complete your technical training first, then move into a dedicated Career Sprint of career projects and job-search prep before you start applying for jobs. This includes things like optimizing your LinkedIn profile and making sure you have a strong resume and portfolio.</p><p>During the job search, day-to-day guidance and task management happens in Percruit, TripleTen’s job-search platform. You can practice interviewing for HR and technical interviews with AI coaches, but as soon as you land an interview, you’ll do live mock interviews with a career coach. You will also be expected to take part in at least two career activities a month, such as workshops and industry-experience projects.</p><h3>Your starting point and expectations</h3><p>You don’t need a technical background (about 80% of TripleTen students start from another career entirely), but you should go in expecting a demanding program that rewards consistency. Set expectations early: you won’t understand everything at once, and it’s hard to predict how long the job search will take. Robert Pickett, who switched into data analytics later in his career, is proof that you can be a successful beginner at any age:</p><blockquote><em>“I’m nearly 60 years old, and I switched careers to something more data oriented. I’ve been using computers since Reagan was president. So if this old dog can learn something new, so can you” <br>— </em><a href="https://tripleten.com/blog/posts/7-strategies-to-launch-a-bia-journey">Robert Pickett, Data Analytics graduate</a></blockquote><p>His practical advice applies to anyone weighing the leap: start with the basics and build from there, and treat the program like a job with steady hours. Graduates who go in with that mindset tend to get the most out of it.</p><h3><strong>How to check the outcomes before you commit</strong></h3><p>Before enrolling in any online program, take a look at the outcome data. TripleTen publishes a <a href="https://tripleten.com/outcomes/">Student Achievement Highlights</a> report every year (the latest came out in April 2026) and holds about 4.8 out of 5 across 2,500+ reviews on <a href="https://www.coursereport.com/schools/tripleten">Course Report</a>, <a href="https://careerkarma.com/schools/tripleten/">Career Karma</a>, <a href="https://share.google/dZiV1yTylcnEyW6cX">Google</a>, and <a href="https://www.trustpilot.com/review/tripleten.com">Trustpilot</a>.</p><p>Some claims should be a red flag: claims of 100% placement, refusal to share detailed or recent outcomes, and guarantees whose conditions are buried. A program willing to show its numbers, its reviews, and its fine print is one you can evaluate honestly.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f0816d512b0c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tripleten-bootcamp/choosing-a-career-program-at-tripleten-f0816d512b0c">Choosing a Career Program at TripleTen</a> was originally published in <a href="https://medium.com/tripleten-bootcamp">TripleTen Career Learning Platform</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why I Chose TripleTen and Why It’s Been Worth It]]></title>
            <link>https://medium.com/tripleten-bootcamp/why-i-chose-tripleten-and-why-its-been-worth-it-42740b32e80a?source=rss-997422ae1232------2</link>
            <guid isPermaLink="false">https://medium.com/p/42740b32e80a</guid>
            <category><![CDATA[education]]></category>
            <category><![CDATA[business-intelligence]]></category>
            <category><![CDATA[career-change]]></category>
            <category><![CDATA[self-improvement]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[TripleTen]]></dc:creator>
            <pubDate>Thu, 12 Feb 2026 15:02:29 GMT</pubDate>
            <atom:updated>2026-07-13T17:17:49.155Z</atom:updated>
            <content:encoded><![CDATA[<h4>By <a href="https://www.linkedin.com/in/amit-sharma-ais/">Amit Sharma</a>, TripleTen student</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MUvgEDRRtO3AgSQcs7Fk9w.png" /></figure><p>Bootcamps: are they worth it? That is the biggest question people ask themselves before taking the leap. Trust me, I understand. It took me months to finally decide to join <a href="https://tripleten.com/">TripleTen</a>. My main questions were: Was it <a href="https://tripleten.com/blog/posts/the-return-on-investment-for-bootcamps">worth it</a>? Would it truly help me? Would I even be able to change my career after doing something I have done for over 15 years? Truth be told, I looked at myself in the mirror, and I was scared.</p><h4><strong>Realizing what matters most</strong></h4><p>I missed so many opportunities with my kids such as birthdays. My youngest one has autism, and I missed his graduation from his ABA therapy. When that happened, that hurt the most because these things in life only happen once. That is something I will never get back. I didn’t even realize what happened because working in retail management took all my time, and all I was worried about was paying the bills, making sure they had been taken care of by working consistently.</p><p>I looked up TripleTen because I heard about it on YouTube. I saw that most of the grads get new careers without any knowledge of tech in their background at all. I then searched on YouTube for <a href="https://www.youtube.com/watch?v=QaooXTSsJ68&amp;list=PLes02YtPKABmnqdONljB8q9Gpc6M5Nl3p">actual people who did TripleTen</a>, and in 9/10 videos I watched, their lives had changed. Some people came from backgrounds like mine, and others from backgrounds like teaching or sales. They found jobs in a whole new field with tech skills in less than a year, and I thought to myself “Can I really do this? I am not a tech person in any way — no social media accounts, none of that.”</p><h4><strong>Discovering business intelligence and a new path</strong></h4><p>So, I did the <a href="https://tripleten.com/quiz/career-quiz-nc/?form_position=header">TripleTen career quiz</a> twice because the first time I was nervous and decided, “No I am not doing this. It is too hard.” A month later, I did it again and came up with the same result: <a href="https://tripleten.com/data-analytics/">Business Intelligence Analytics</a>. I did some research on the program with TripleTen and outside sources. I saw that it is a growing field, and with the information I started gaining, it was as if I opened up a new world of opportunities for careers.</p><p>Then I took the call. I spoke with Miguel twice. I told him my fears: What if I failed, what do I do then? We talked about how this is a leap, a method of making a positive change. I told him it’s time to change my life and my career, and TripleTen was the start toward building a new me, a better me. I enrolled in TripleTen because it was flexible. I could still work and get my courses done on my time. TripleTen understands that there is life outside of their programs.</p><h4><strong>My experience at TripleTen so far</strong></h4><p>I knew I had made the right decision. I started in September 2025, and it was like learning a new language. However, I was excited because I saw people from all over the world taking that leap. I met an amazing learning coach, Nino, who has been there to remind me that I am not alone. I have learned so many new skills such as spreadsheets, Tableau, Power BI, and new tech.</p><p>My one quibble is that the program is all online, and I would like some chance to get together with my fellow students. I think that’s an opportunity for TripleTen to expand themselves. However, that doesn’t change how grateful I am for TripleTen. It opened my eyes to a whole new world of opportunities.</p><p>There’s more than retail. There is a chance to have not only a job, but a career: to grow and develop yourself personally and professionally. TripleTen opened that part of the world to me that I didn’t even know existed in the first place. I can’t wait to get into the career after finishing my program.</p><h4><strong>Looking ahead and advice for others</strong></h4><p>I hope one day to not work on Black Friday and to be able to stay up late with my kids after Thanksgiving dinner. I think TripleTen can help me do that. The best part is they help guide you through learning new soft skills, so it isn’t all focused on technology, but also on becoming a better person, understanding people. My TripleTen experience has been amazing so far, and I can’t wait to finish this.</p><p>One piece of advice I can give and will always give, no matter what you do, is to believe in yourself and never give up. TripleTen has shown me that by staying dedicated and putting in the work, you’re already halfway there. TripleTen says, “Learn the job. Get the job.” That is the goal I continue to strive for, and I hope you all take this leap, too.</p><p><em>To spark a new professional future like Amit has, </em><a href="https://tripleten.com/special/free-career-consultation/?utm_source=medium"><em>book a free career consultation with one of TripleTen’s expert advisors today</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42740b32e80a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tripleten-bootcamp/why-i-chose-tripleten-and-why-its-been-worth-it-42740b32e80a">Why I Chose TripleTen and Why It’s Been Worth It</a> was originally published in <a href="https://medium.com/tripleten-bootcamp">TripleTen Career Learning Platform</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[First Things First: A Brief Guide to Data Loading, Cleaning, and Exploration]]></title>
            <link>https://medium.com/tripleten-bootcamp/first-things-first-a-brief-guide-to-data-loading-cleaning-and-exploration-562cbc666c3d?source=rss-997422ae1232------2</link>
            <guid isPermaLink="false">https://medium.com/p/562cbc666c3d</guid>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[codingbootcamp]]></category>
            <category><![CDATA[coding-tutorial]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[TripleTen]]></dc:creator>
            <pubDate>Fri, 20 Aug 2021 14:51:00 GMT</pubDate>
            <atom:updated>2026-01-12T19:45:11.983Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>Part 3</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sKaJm6gKtFmAUnqyPGyMtg.png" /><figcaption>by Alex Kim, Consultant at TripleTen <a href="https://twitter.com/alex000kim">Twitter</a> | <a href="https://www.linkedin.com/in/alex000kim/">LinkedIn</a></figcaption></figure><p>Note: This is Part 3 of a three-part article. If you haven’t read <a href="https://medium.com/practicum-by-yandex/first-things-first-a-brief-guide-to-data-loading-cleaning-and-exploration-56f7222c0e0b">Part 1</a> and <a href="https://y-practicum.medium.com/first-things-first-a-brief-guide-to-data-loading-cleaning-and-exploration-847a20a1432f">Part 2</a>, we recommend that you do so first.</p><h3>Quick data exploration</h3><p>In <a href="https://medium.com/practicum-by-yandex/first-things-first-a-brief-guide-to-data-loading-cleaning-and-exploration-56f7222c0e0b">Part 1</a>, we loaded our LendingClub data; in <a href="https://y-practicum.medium.com/first-things-first-a-brief-guide-to-data-loading-cleaning-and-exploration-847a20a1432f">Part 2</a>, we performed preprocessing. Now our dataset is clean and ready for exploration.</p><p>What exactly happens at this stage depends on the business problem you’re dealing with. It typically involves things like data visualization, statistical analysis, and machine learning modeling.</p><p>Let’s see how we’d further explore one of the columns in our DataFrame, &#39;home_ownership&#39;. First, we&#39;ll see how many missing values are in this column, if any (again, by chaining together the pandas.Series.isna() and pandas.Series.sum() methods).</p><p>Then, we’ll get only the list of unique values in this column (pandas.Series.unique()) and their counts (pandas.Series.value_counts()).</p><p>Histogram plots are an excellent choice to visualize value counts. Here we are using the seaborn data visualization library.</p><p>We pass an optional order= parameter to the seaborn.countplot() method to order our histogram from the largest to smallest category.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all%25255B%2527home_ownership%2527%25255D.isna%2528%2529.sum%2528%2529%25250A%25253E%2525200&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all%2525255B%252527home_ownership%252527%2525255D.isna%252528%252529.sum%252528%252529%2525250A%2525253E%252525200&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/dc6102844acc4a526210ef80025a13e3/href">https://medium.com/media/dc6102844acc4a526210ef80025a13e3/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all%25255B%2527home_ownership%2527%25255D.unique%2528%2529%25250A%25253E%252520%25255B%2527RENT%2527%25252C%252520%2527OWN%2527%25252C%252520%2527MORTGAGE%2527%25252C%252520%2527ANY%2527%25255D%25250ACategories%252520%25284%25252C%252520object%2529%25253A%252520%25255B%2527ANY%2527%25252C%252520%2527MORTGAGE%2527%25252C%252520%2527OWN%2527%25252C%252520%2527RENT%2527%25255D&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all%2525255B%252527home_ownership%252527%2525255D.unique%252528%252529%2525250A%2525253E%25252520%2525255B%252527RENT%252527%2525252C%25252520%252527OWN%252527%2525252C%25252520%252527MORTGAGE%252527%2525252C%25252520%252527ANY%252527%2525255D%2525250ACategories%25252520%2525284%2525252C%25252520object%252529%2525253A%25252520%2525255B%252527ANY%252527%2525252C%25252520%252527MORTGAGE%252527%2525252C%25252520%252527OWN%252527%2525252C%25252520%252527RENT%252527%2525255D&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ed54b0921d45126435b1f693acbd9833/href">https://medium.com/media/ed54b0921d45126435b1f693acbd9833/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all%25255B%2527home_ownership%2527%25255D.value_counts%2528%2529%25250A%25253E%252520MORTGAGE%252520%252520%252520%252520239220%25250ARENT%252520%252520%252520%252520%252520%252520%252520%252520195406%25250AOWN%252520%252520%252520%252520%252520%252520%252520%252520%252520%25252060128%25250AANY%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520488%25250AName%25253A%252520home_ownership%25252C%252520dtype%25253A%252520int64&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all%2525255B%252527home_ownership%252527%2525255D.value_counts%252528%252529%2525250A%2525253E%25252520MORTGAGE%25252520%25252520%25252520%25252520239220%2525250ARENT%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520195406%2525250AOWN%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%2525252060128%2525250AANY%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520488%2525250AName%2525253A%25252520home_ownership%2525252C%25252520dtype%2525253A%25252520int64&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/d46fa406e93acdb6fb2f06f448587814/href">https://medium.com/media/d46fa406e93acdb6fb2f06f448587814/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dimport%252520seaborn%252520as%252520sns%25250A%25250Asns.countplot%2528x%25253D%2527home_ownership%2527%25252C%252520%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520data%25253Ddf_all%25252C%252520%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520order%25253Ddf_all%25255B%2527home_ownership%2527%25255D.value_counts%2528%2529.index%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dimport%25252520seaborn%25252520as%25252520sns%2525250A%2525250Asns.countplot%252528x%2525253D%252527home_ownership%252527%2525252C%25252520%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520data%2525253Ddf_all%2525252C%25252520%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520order%2525253Ddf_all%2525255B%252527home_ownership%252527%2525255D.value_counts%252528%252529.index%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/322c143ccc288216c8cf492ff2a85dfa/href">https://medium.com/media/322c143ccc288216c8cf492ff2a85dfa/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QW_pL5ijxQAYmh8OsdrOHQ.png" /></figure><p>The issue is that when we have many columns, as we do in this case, performing these steps will become very tedious very quickly.</p><p>Fortunately, a lot of data exploration can be automated. This reduces the amount of boilerplate code that you’d need to write to compute common statistics and relationships between variables. Below, we’ll show you how to use two different libraries for automating exploration.</p><h3>The pandas-profiling library</h3><p>Using this library involves three steps:</p><ol><li>Import the ProfileReport class from the library.</li><li>Instantiate a ProfileReport object with optional parameters. Here, we&#39;ll be running the analysis in minimal mode, which disables resource-intensive computations.</li><li>Display the resulting report as HTML embedded in a Jupyter notebook, <a href="https://towardsdatascience.com/bring-your-jupyter-notebook-to-life-with-interactive-widgets-bc12e03f0916">ipywidgets</a>, or in a separate HTML file.</li></ol><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dfrom%252520pandas_profiling%252520import%252520ProfileReport%25250A%25250Aprofile%252520%25253D%252520ProfileReport%2528df_all%25252C%252520title%25253D%252522Pandas%252520Profiling%252520Report%252522%25252C%252520minimal%25253DTrue%2529%25250Aprofile.to_widgets%2528%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dfrom%25252520pandas_profiling%25252520import%25252520ProfileReport%2525250A%2525250Aprofile%25252520%2525253D%25252520ProfileReport%252528df_all%2525252C%25252520title%2525253D%25252522Pandas%25252520Profiling%25252520Report%25252522%2525252C%25252520minimal%2525253DTrue%252529%2525250Aprofile.to_widgets%252528%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/073744025eef9435c6be940ada663956/href">https://medium.com/media/073744025eef9435c6be940ada663956/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j8njwi-Lyv-yQ1E8opNK-A.png" /></figure><h3>The sweetviz library</h3><p>Another way to get an interactive and automated data visualization report is to use<a href="https://github.com/fbdesignpro/sweetviz">sweetviz</a>.</p><ol><li>Create a report by passing the DataFrame to the sweetviz.analyze() method. We set the pairwise_analysis= parameter to &quot;off&quot; to disable expensive computations.</li><li>Call the show_notebook() method to display the interactive report inside the notebook.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TPTlSt6zX_r4MLlf8Ubadw.png" /></figure><p>There are other libraries that perform automated exploratory data analysis. Which one you choose is largely a matter of taste.</p><h3>Summary</h3><p>Everything we’ve described in this 3-part article can be viewed in <a href="https://github.com/alex000kim/data_cleaning_guide/blob/main/DataCleaningGuide.ipynb">this Jupyter notebook</a>.</p><p>While data cleaning may not be the most exciting process, it’s arguably one of the most crucial steps for extracting valuable insights from your data. What you do at this stage will affect what business decisions are made down the line.</p><p>Approach this with care: document your choices and assumptions, making them explicit to others and yourself. Ask questions like:</p><ul><li>Are these dates really in the time zone I think they are?</li><li>Are these observations really outliers? If so, is there an issue with how the data is being recorded in the database?</li><li>Do zeros replace missing values in my data?</li></ul><p>For an exhaustive guide to common data issues, we encourage you to check out <a href="https://github.com/Quartz/bad-data-guide">the Quartz guide to bad data</a>.</p><p>It might be a good idea to keep notes of all the unexpected and interesting things you observe. In a work setting, you can review these notes with someone who has in-depth knowledge of the data to see which direction you should go next.</p><h3>What’s next?</h3><p>If you’d like to continue working with this dataset by creating some data visualizations, we can recommend the following libraries for you to try out:</p><ul><li>For static plots: <a href="https://seaborn.pydata.org/">seaborn</a> or <a href="http://yhat.github.io/ggpy/">ggplot</a></li><li>For interactive plots: <a href="https://plotly.com/python/plotly-express/">plotly-express</a> or <a href="https://docs.bokeh.org/en/latest/index.html">bokeh</a></li></ul><p>Learn more about the data science bootcamp and other courses by checking out <a href="https://tripleten.com/">TripleTen</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=562cbc666c3d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tripleten-bootcamp/first-things-first-a-brief-guide-to-data-loading-cleaning-and-exploration-562cbc666c3d">First Things First: A Brief Guide to Data Loading, Cleaning, and Exploration</a> was originally published in <a href="https://medium.com/tripleten-bootcamp">TripleTen Career Learning Platform</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[First Things First: A Brief Guide to Data Loading, Cleaning, and Exploration]]></title>
            <link>https://medium.com/tripleten-bootcamp/first-things-first-a-brief-guide-to-data-loading-cleaning-and-exploration-847a20a1432f?source=rss-997422ae1232------2</link>
            <guid isPermaLink="false">https://medium.com/p/847a20a1432f</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[guides-and-tutorials]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[codingbootcamp]]></category>
            <dc:creator><![CDATA[TripleTen]]></dc:creator>
            <pubDate>Fri, 20 Aug 2021 13:20:15 GMT</pubDate>
            <atom:updated>2026-01-12T19:43:50.661Z</atom:updated>
            <content:encoded><![CDATA[<h4>Part 2</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FoNQfl0_DUtGnlIeYg_b9A.png" /><figcaption>by Alex Kim, Consultant at TripleTen <a href="https://twitter.com/alex000kim">Twitter</a> | <a href="https://www.linkedin.com/in/alex000kim/">LinkedIn</a></figcaption></figure><p>Note: This is Part 2 of a three-part article.</p><p>In <a href="https://medium.com/practicum-by-yandex/first-things-first-a-brief-guide-to-data-loading-cleaning-and-exploration-56f7222c0e0b">Part 1</a> of this article, we downloaded multiple CSV files containing LendingClub’s historical loan data. Here we’ll determine if this data has any issues and, if so, how to deal with them.</p><h3>Looking at our data</h3><p>Our fpath_list variable contains the paths of all the CSVs we downloaded. Let&#39;s take a quick look at one of them by calling the <a href="https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html">pandas.read_csv()</a><br>method and displaying the first and last few lines of the file:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf%252520%25253D%252520pd.read_csv%2528fpath_list%25255B0%25255D%2529%25250Adf.head%2528%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf%25252520%2525253D%25252520pd.read_csv%252528fpath_list%2525255B0%2525255D%252529%2525250Adf.head%252528%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/3bc7d7fc20d25b9146c04f136b6951d0/href">https://medium.com/media/3bc7d7fc20d25b9146c04f136b6951d0/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iqz5kdoNpxpv8do5Z02Tgg.png" /></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf.tail%2528%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf.tail%252528%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/25a525e4d88aca020ec8cd1a7b8db56b/href">https://medium.com/media/25a525e4d88aca020ec8cd1a7b8db56b/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zGzmXjbvzGfCa5zViZ45og.png" /></figure><p>Hopefully, you noticed that something doesn’t look right about the way this DataFrame is displayed. Let’s look at the list of columns:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf.columns%25250A%25253E%252520Index%2528%25255B%2527Notes%252520offered%252520by%252520Prospectus%252520%2528https%25253A%25252F%25252Fwww.lendingclub.com%25252Finfo%25252Fprospectus.action%2529%2527%25255D%25252C%252520dtype%25253D%2527object%2527%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf.columns%2525250A%2525253E%25252520Index%252528%2525255B%252527Notes%25252520offered%25252520by%25252520Prospectus%25252520%252528https%2525253A%2525252F%2525252Fwww.lendingclub.com%2525252Finfo%2525252Fprospectus.action%252529%252527%2525255D%2525252C%25252520dtype%2525253D%252527object%252527%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ddaa7067e8bcaa0b08dc1ecd9696f18a/href">https://medium.com/media/ddaa7067e8bcaa0b08dc1ecd9696f18a/href</a></iframe><p>Not what you expected, right?</p><p>We used read_csv() with default values for all of its parameters. While these default values work well for CSV files that are properly structured, the one we have is not.</p><h3>Problems with our files</h3><p>Our CSV files have two issues:</p><ol><li>They don’t have column names on the first line. The first line is just this string: &#39;Notes offered by Prospectus (&lt;https://www.lendingclub.com/info/prospectus.action&gt;)&#39;</li><li>They contain summary totals in the last two lines.</li></ol><h3>The solution: skiprows= and skipfooters=</h3><p>Below you see that we dealt with these issues by using the skiprows= and skipfooters= parameters. We encourage you to refer to the <a href="https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html">documentation</a> to see how to use these and other parameters of the read_csv() method.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf%252520%25253D%252520pd.read_csv%2528fpath_list%25255B0%25255D%25252C%252520skiprows%25253D1%25252C%252520skipfooter%25253D2%2529%252520%252523%252520ignore%252520the%252520first%252520line%252520and%252520the%252520last%252520two%252520lines%25250Adf.head%2528%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf%25252520%2525253D%25252520pd.read_csv%252528fpath_list%2525255B0%2525255D%2525252C%25252520skiprows%2525253D1%2525252C%25252520skipfooter%2525253D2%252529%25252520%25252523%25252520ignore%25252520the%25252520first%25252520line%25252520and%25252520the%25252520last%25252520two%25252520lines%2525250Adf.head%252528%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/be8f4fd66cfcae6c6224efa3df3b5a93/href">https://medium.com/media/be8f4fd66cfcae6c6224efa3df3b5a93/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bOdGGvdSz20amts15TgH_w.png" /></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf.tail%2528%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf.tail%252528%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/25a525e4d88aca020ec8cd1a7b8db56b/href">https://medium.com/media/25a525e4d88aca020ec8cd1a7b8db56b/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LSf9tkOVwopI6yyNwfJ2vw.png" /></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dall_cols%252520%25253D%252520df.columns%25250Aall_cols%25250A%25253E%252520Index%2528%25255B%2527id%2527%25252C%252520%2527member_id%2527%25252C%252520%2527loan_amnt%2527%25252C%252520%2527funded_amnt%2527%25252C%252520%2527funded_amnt_inv%2527%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%2527term%2527%25252C%252520%2527int_rate%2527%25252C%252520%2527installment%2527%25252C%252520%2527grade%2527%25252C%252520%2527sub_grade%2527%25252C%25250A%252520%252520%252520%252520%252520%252520%252520...%25250A%252520%252520%252520%252520%252520%252520%252520%2527orig_projected_additional_accrued_interest%2527%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%2527hardship_payoff_balance_amount%2527%25252C%252520%2527hardship_last_payment_amount%2527%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%2527debt_settlement_flag%2527%25252C%252520%2527debt_settlement_flag_date%2527%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%2527settlement_status%2527%25252C%252520%2527settlement_date%2527%25252C%252520%2527settlement_amount%2527%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%2527settlement_percentage%2527%25252C%252520%2527settlement_term%2527%25255D%25252C%25250A%252520%252520%252520%252520%252520%252520dtype%25253D%2527object%2527%25252C%252520length%25253D144%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dall_cols%25252520%2525253D%25252520df.columns%2525250Aall_cols%2525250A%2525253E%25252520Index%252528%2525255B%252527id%252527%2525252C%25252520%252527member_id%252527%2525252C%25252520%252527loan_amnt%252527%2525252C%25252520%252527funded_amnt%252527%2525252C%25252520%252527funded_amnt_inv%252527%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252527term%252527%2525252C%25252520%252527int_rate%252527%2525252C%25252520%252527installment%252527%2525252C%25252520%252527grade%252527%2525252C%25252520%252527sub_grade%252527%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520...%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252527orig_projected_additional_accrued_interest%252527%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252527hardship_payoff_balance_amount%252527%2525252C%25252520%252527hardship_last_payment_amount%252527%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252527debt_settlement_flag%252527%2525252C%25252520%252527debt_settlement_flag_date%252527%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252527settlement_status%252527%2525252C%25252520%252527settlement_date%252527%2525252C%25252520%252527settlement_amount%252527%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252527settlement_percentage%252527%2525252C%25252520%252527settlement_term%252527%2525255D%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520dtype%2525253D%252527object%252527%2525252C%25252520length%2525253D144%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/7292e8e313b7c140dc7adef5df70b8e6/href">https://medium.com/media/7292e8e313b7c140dc7adef5df70b8e6/href</a></iframe><p>We have a list of file paths for our CSV files, which, hopefully, all have the structure. (It would be time-consuming to tweak the data-loading code for each one of them.) If this is the case, the next step is to load all of them into one big DataFrame.</p><p>A quick and easy way to achieve this is to first create an empty list (df_list) that will store individual DataFrames (one per CSV file). To populate this empty list, we&#39;ll iterate through file paths. On every iteration, we’ll:</p><ol><li>Load a CSV file with read_csv() and the parameters we identified earlier</li><li>Make an assertion that <em>all</em> CSV files have the same columns as the very first file we looked at (for a brief intro to assertions, see <a href="https://stackoverflow.com/questions/5142418/what-is-the-use-of-assert-in-python">here</a>)</li><li>Append the newly created DataFrame to df_list</li></ol><p>After the loop is done, we’ll combine all DataFrames in the df_list into one (df_all) using pandas’ <a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.concat.html">concat</a> (concatenate) method.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_list%252520%25253D%252520%25255B%25255D%25250Afor%252520fpath%252520in%252520tqdm%2528fpath_list%2529%25253A%252520%252520%252520%252520%25250A%252509df%252520%25253D%252520pd.read_csv%2528fpath%25252C%252520skiprows%25253D1%25252C%252520skipfooter%25253D2%25252C%252520engine%25253D%2527python%2527%2529%25250A%252509assert%252520list%2528df.columns%2529%252520%25253D%25253D%252520list%2528all_cols%2529%25250A%252509df_list.append%2528df%2529%25250Adf_all%252520%25253D%252520pd.concat%2528df_list%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_list%25252520%2525253D%25252520%2525255B%2525255D%2525250Afor%25252520fpath%25252520in%25252520tqdm%252528fpath_list%252529%2525253A%25252520%25252520%25252520%25252520%2525250A%25252509df%25252520%2525253D%25252520pd.read_csv%252528fpath%2525252C%25252520skiprows%2525253D1%2525252C%25252520skipfooter%2525253D2%2525252C%25252520engine%2525253D%252527python%252527%252529%2525250A%25252509assert%25252520list%252528df.columns%252529%25252520%2525253D%2525253D%25252520list%252528all_cols%252529%2525250A%25252509df_list.append%252528df%252529%2525250Adf_all%25252520%2525253D%25252520pd.concat%252528df_list%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/903e54f90b000b3f07638671b2136ca1/href">https://medium.com/media/903e54f90b000b3f07638671b2136ca1/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all.shape%25250A%25253E%252520%2528495242%25252C%252520144%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all.shape%2525250A%2525253E%25252520%252528495242%2525252C%25252520144%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/7655d6ad42ed6afa0dfee61d28302fd7/href">https://medium.com/media/7655d6ad42ed6afa0dfee61d28302fd7/href</a></iframe><h3>Missing values and outliers</h3><p>Our dataset contains some missing values. Ideally, for every column that’s missing values, you’d like to know why values are missing and the preferred way of dealing with them (filling with some constant value, removing from the dataset, etc).</p><p>If you are lucky, there will either be an accompanying metadata file describing the reasons why some data is missing, or one of your colleagues might possess this information. In reality, this is rare, and most of the time, you’ll have to make some educated guesses and decide how to deal with these issues. Here, for simplicity, we’ll do the following:</p><ol><li>Calculate the percentage of values missing in each column</li><li>Remove columns that are missing more than a predefined (by us) PERCENT_MISSING_THRES threshold value</li></ol><p>Feel free to tweak this threshold in either direction and see how this changes the number of columns.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dpercent_missing_by_col%252520%25253D%252520df_all.isna%2528%2529.mean%2528%2529%25250Apercent_missing_by_col%25250A%25253E%252520id%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%2525201.000000%25250Amember_id%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%2525201.000000%25250Aloan_amnt%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%2525200.000000%25250Afunded_amnt%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%2525200.000000%25250Afunded_amnt_inv%252520%252520%252520%252520%252520%252520%252520%252520%252520%2525200.000000%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520...%252520%252520%252520%25250Asettlement_status%252520%252520%252520%252520%252520%252520%252520%2525200.988862%25250Asettlement_date%252520%252520%252520%252520%252520%252520%252520%252520%252520%2525200.988862%25250Asettlement_amount%252520%252520%252520%252520%252520%252520%252520%2525200.988862%25250Asettlement_percentage%252520%252520%252520%2525200.988862%25250Asettlement_term%252520%252520%252520%252520%252520%252520%252520%252520%252520%2525200.988862%25250ALength%25253A%252520144%25252C%252520dtype%25253A%252520float64&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dpercent_missing_by_col%25252520%2525253D%25252520df_all.isna%252528%252529.mean%252528%252529%2525250Apercent_missing_by_col%2525250A%2525253E%25252520id%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252525201.000000%2525250Amember_id%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252525201.000000%2525250Aloan_amnt%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252525200.000000%2525250Afunded_amnt%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252525200.000000%2525250Afunded_amnt_inv%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252525200.000000%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520...%25252520%25252520%25252520%2525250Asettlement_status%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252525200.988862%2525250Asettlement_date%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252525200.988862%2525250Asettlement_amount%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252525200.988862%2525250Asettlement_percentage%25252520%25252520%25252520%252525200.988862%2525250Asettlement_term%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252525200.988862%2525250ALength%2525253A%25252520144%2525252C%25252520dtype%2525253A%25252520float64&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/6510760b5f7272e5568620e0c7d5c3b2/href">https://medium.com/media/6510760b5f7272e5568620e0c7d5c3b2/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3DPERCENT_MISSING_THRES%252520%25253D%2525200.5%25250Acols_remaining%252520%25253D%252520df_all.columns%25255Bpercent_missing_by_col%252520%25253C%252520PERCENT_MISSING_THRES%25255D%25250Acols_remaining%25250A%25253E%252520Index%2528%25255B%2527loan_amnt%2527%25252C%252520%2527funded_amnt%2527%25252C%252520%2527funded_amnt_inv%2527%25252C%252520%2527term%2527%25252C%252520%2527int_rate%2527%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%2527installment%2527%25252C%252520%2527grade%2527%25252C%252520%2527sub_grade%2527%25252C%252520%2527emp_title%2527%25252C%252520%2527emp_length%2527%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%2527home_ownership%2527%25252C%252520%2527annual_inc%2527%25252C%252520%2527verification_status%2527%25252C%252520%2527issue_d%2527%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%2527loan_status%2527%25252C%252520%2527pymnt_plan%2527%25252C%252520%2527purpose%2527%25252C%252520%2527title%2527%25252C%252520%2527zip_code%2527%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%2527addr_state%2527%25252C%252520...%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%2527total_il_high_credit_limit%2527%25252C%252520%2527hardship_flag%2527%25252C%252520%2527debt_settlement_flag%2527%25255D%25252C%25250A%252520%252520%252520%252520%252520%252520dtype%25253D%2527object%2527%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3DPERCENT_MISSING_THRES%25252520%2525253D%252525200.5%2525250Acols_remaining%25252520%2525253D%25252520df_all.columns%2525255Bpercent_missing_by_col%25252520%2525253C%25252520PERCENT_MISSING_THRES%2525255D%2525250Acols_remaining%2525250A%2525253E%25252520Index%252528%2525255B%252527loan_amnt%252527%2525252C%25252520%252527funded_amnt%252527%2525252C%25252520%252527funded_amnt_inv%252527%2525252C%25252520%252527term%252527%2525252C%25252520%252527int_rate%252527%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252527installment%252527%2525252C%25252520%252527grade%252527%2525252C%25252520%252527sub_grade%252527%2525252C%25252520%252527emp_title%252527%2525252C%25252520%252527emp_length%252527%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252527home_ownership%252527%2525252C%25252520%252527annual_inc%252527%2525252C%25252520%252527verification_status%252527%2525252C%25252520%252527issue_d%252527%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252527loan_status%252527%2525252C%25252520%252527pymnt_plan%252527%2525252C%25252520%252527purpose%252527%2525252C%25252520%252527title%252527%2525252C%25252520%252527zip_code%252527%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252527addr_state%252527%2525252C%25252520...%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252527total_il_high_credit_limit%252527%2525252C%25252520%252527hardship_flag%252527%2525252C%25252520%252527debt_settlement_flag%252527%2525255D%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520dtype%2525253D%252527object%252527%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/41b8f5dae012ea7d04bc6e566a76d04f/href">https://medium.com/media/41b8f5dae012ea7d04bc6e566a76d04f/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all%252520%25253D%252520df_all%25255Bcols_remaining%25255D%25250Adf_all.shape%25250A%25253E%252520%2528495242%25252C%252520100%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all%25252520%2525253D%25252520df_all%2525255Bcols_remaining%2525255D%2525250Adf_all.shape%2525250A%2525253E%25252520%252528495242%2525252C%25252520100%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/fdbfb525ed611f7a6518e07608c015d2/href">https://medium.com/media/fdbfb525ed611f7a6518e07608c015d2/href</a></iframe><h3>Feature engineering</h3><p>Now we are moving on to feature engineering, the process of modifying or creating new features from existing ones. Feature engineering can be domain-specific and can get fairly complex.</p><p>Here, we’ll leave all numerical features (integers and floats) as is and focus our attention on object-type (which in pandas world often means string-type) columns. Let&#39;s see how many of them we have:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all.dtypes.value_counts%2528%2529%25250A%25253E%252520int64%252520%252520%252520%252520%252520%25252051%25250Afloat64%252520%252520%252520%25252026%25250Aobject%252520%252520%252520%252520%25252023%25250Adtype%25253A%252520int64&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all.dtypes.value_counts%252528%252529%2525250A%2525253E%25252520int64%25252520%25252520%25252520%25252520%25252520%2525252051%2525250Afloat64%25252520%25252520%25252520%2525252026%2525250Aobject%25252520%25252520%25252520%25252520%2525252023%2525250Adtype%2525253A%25252520int64&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/fef3a9574792e82a169a568969cd548d/href">https://medium.com/media/fef3a9574792e82a169a568969cd548d/href</a></iframe><p>Which columns are object type?</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all.select_dtypes%2528include%25253D%25255B%2527object%2527%25255D%2529.columns%25250A%25253E%252520Index%2528%25255B%2527term%2527%25252C%252520%2527int_rate%2527%25252C%252520%2527grade%2527%25252C%252520%2527sub_grade%2527%25252C%252520%2527emp_title%2527%25252C%252520%2527emp_length%2527%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%2527home_ownership%2527%25252C%252520%2527verification_status%2527%25252C%252520%2527issue_d%2527%25252C%252520%2527loan_status%2527%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%2527pymnt_plan%2527%25252C%252520%2527purpose%2527%25252C%252520%2527title%2527%25252C%252520%2527zip_code%2527%25252C%252520%2527addr_state%2527%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%2527earliest_cr_line%2527%25252C%252520%2527revol_util%2527%25252C%252520%2527initial_list_status%2527%25252C%252520%2527last_pymnt_d%2527%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%2527last_credit_pull_d%2527%25252C%252520%2527application_type%2527%25252C%252520%2527hardship_flag%2527%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%2527debt_settlement_flag%2527%25255D%25252C%25250A%252520%252520%252520%252520%252520%252520dtype%25253D%2527object%2527%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all.select_dtypes%252528include%2525253D%2525255B%252527object%252527%2525255D%252529.columns%2525250A%2525253E%25252520Index%252528%2525255B%252527term%252527%2525252C%25252520%252527int_rate%252527%2525252C%25252520%252527grade%252527%2525252C%25252520%252527sub_grade%252527%2525252C%25252520%252527emp_title%252527%2525252C%25252520%252527emp_length%252527%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252527home_ownership%252527%2525252C%25252520%252527verification_status%252527%2525252C%25252520%252527issue_d%252527%2525252C%25252520%252527loan_status%252527%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252527pymnt_plan%252527%2525252C%25252520%252527purpose%252527%2525252C%25252520%252527title%252527%2525252C%25252520%252527zip_code%252527%2525252C%25252520%252527addr_state%252527%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252527earliest_cr_line%252527%2525252C%25252520%252527revol_util%252527%2525252C%25252520%252527initial_list_status%252527%2525252C%25252520%252527last_pymnt_d%252527%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252527last_credit_pull_d%252527%2525252C%25252520%252527application_type%252527%2525252C%25252520%252527hardship_flag%252527%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252527debt_settlement_flag%252527%2525255D%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520dtype%2525253D%252527object%252527%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/3a9db716cce1fc7fc21fd9e15fe27558/href">https://medium.com/media/3a9db716cce1fc7fc21fd9e15fe27558/href</a></iframe><p>And what data do they contain?</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all.select_dtypes%2528include%25253D%25255B%2527object%2527%25255D%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all.select_dtypes%252528include%2525253D%2525255B%252527object%252527%2525255D%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/b4bc2e30c91d450318a277f760a8e704/href">https://medium.com/media/b4bc2e30c91d450318a277f760a8e704/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G0j5N1XVnQ5WgcFxF9Cxbw.png" /></figure><p>First, we notice a few datetime columns that were loaded as strings. Let’s convert them to datetime objects with pandas’ <a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.to_datetime.html">to_datetime()</a> method.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dfor%252520col%252520in%252520%25255B%2527issue_d%2527%25252C%252520%2527earliest_cr_line%2527%25252C%252520%2527last_pymnt_d%2527%25252C%252520%2527last_credit_pull_d%2527%25255D%25253A%252520%252520%252520%252520%25250A%252520%252520%252520%252520df_all%25255Bcol%25255D%252520%25253D%252520pd.to_datetime%2528df_all%25255Bcol%25255D%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dfor%25252520col%25252520in%25252520%2525255B%252527issue_d%252527%2525252C%25252520%252527earliest_cr_line%252527%2525252C%25252520%252527last_pymnt_d%252527%2525252C%25252520%252527last_credit_pull_d%252527%2525255D%2525253A%25252520%25252520%25252520%25252520%2525250A%25252520%25252520%25252520%25252520df_all%2525255Bcol%2525255D%25252520%2525253D%25252520pd.to_datetime%252528df_all%2525255Bcol%2525255D%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/f1ec007a522f2378aba26205ea741516/href">https://medium.com/media/f1ec007a522f2378aba26205ea741516/href</a></iframe><p>Next, we see a few columns (&#39;term&#39;, &#39;int_rate&#39;, &#39;revol_util&#39;, and &#39;emp_length&#39;) that contain numerical information but are stored as strings because of the unit names stored along with numbers (%, months, years, etc). For each of these columns, we&#39;ll need to come up with a function that takes a raw string value and outputs a cleaned-up numerical value (either integer or float, depending on the original data).</p><p>Below you see how we deal with the &#39;term&#39; column by defining the term_to_int function to map a value like &#39; 36 months&#39; to the number 36. Note that we use the strip() method inside this function. It turns out that all the values in the &#39;term&#39; column start with a leading space! Can you guess how we found that out? Hint: check out <a href="https://pandas.pydata.org/docs/reference/api/pandas.Series.values.html">pandas.Series.values</a>.</p><p>Once we have the function that transforms our values, we use the <a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.apply.html">pandas.Series.apply()</a> method to apply this function to our column.</p><p>We deal with the columns &#39;int_rate&#39;, &#39;revol_util&#39; and &#39;emp_length&#39; the same way, adjusting our transformation function as needed by the raw values contained in each column.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddef%252520term_to_int%2528x%2529%25253A%252520%252520%252520%252520%25250A%252520%252520%252520%252520x%252520%25253D%252520str%2528x%2529.strip%2528%2529%252520%252520%252520%252520%25250A%252520%252520%252520%252520x%252520%25253D%252520x.split%2528%2527%252520%2527%2529%252520%252520%252520%252520%25250A%252520%252520%252520%252520x%252520%25253D%252520x%25255B0%25255D%252520%252520%252520%252520%25250A%252520%252520%252520%252520return%252520int%2528x%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddef%25252520term_to_int%252528x%252529%2525253A%25252520%25252520%25252520%25252520%2525250A%25252520%25252520%25252520%25252520x%25252520%2525253D%25252520str%252528x%252529.strip%252528%252529%25252520%25252520%25252520%25252520%2525250A%25252520%25252520%25252520%25252520x%25252520%2525253D%25252520x.split%252528%252527%25252520%252527%252529%25252520%25252520%25252520%25252520%2525250A%25252520%25252520%25252520%25252520x%25252520%2525253D%25252520x%2525255B0%2525255D%25252520%25252520%25252520%25252520%2525250A%25252520%25252520%25252520%25252520return%25252520int%252528x%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/5cf3e240fb992cf6c5e2c6a85206cc80/href">https://medium.com/media/5cf3e240fb992cf6c5e2c6a85206cc80/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3D%252523%252520this%252520is%252520a%252520convenience%252520function%252520to%252520check%252520how%252520our%252520function%252520works%252520on%252520different%252520values%252520in%252520a%252520given%252520column%25250Adef%252520print_apply_results%2528col_name%25252C%252520func%25252C%252520df%2529%25253A%252520%252520%252520%252520%25250A%252520%252520%252520%252520for%252520val%252520in%252520df%25255Bcol_name%25255D.unique%2528%2529%25253A%252520%252520%252520%252520%252520%252520%252520%252520%25250A%252509%252509%252520%252520%252520%252520print%2528val%25252C%252520%252522--%25253E%252522%25252C%252520func%2528val%2529%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3D%25252523%25252520this%25252520is%25252520a%25252520convenience%25252520function%25252520to%25252520check%25252520how%25252520our%25252520function%25252520works%25252520on%25252520different%25252520values%25252520in%25252520a%25252520given%25252520column%2525250Adef%25252520print_apply_results%252528col_name%2525252C%25252520func%2525252C%25252520df%252529%2525253A%25252520%25252520%25252520%25252520%2525250A%25252520%25252520%25252520%25252520for%25252520val%25252520in%25252520df%2525255Bcol_name%2525255D.unique%252528%252529%2525253A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%2525250A%25252509%25252509%25252520%25252520%25252520%25252520print%252528val%2525252C%25252520%25252522--%2525253E%25252522%2525252C%25252520func%252528val%252529%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/7ae296a4681f2599a2bd69dc74461cf5/href">https://medium.com/media/7ae296a4681f2599a2bd69dc74461cf5/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dprint_apply_results%2528col_name%25253D%2527term%2527%25252C%252520func%25253Dterm_to_int%25252C%252520df%25253Ddf_all%2529%25250A%25253E%25252036%252520months%252520--%25253E%25252036%25250A%25252060%252520months%252520--%25253E%25252060&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dprint_apply_results%252528col_name%2525253D%252527term%252527%2525252C%25252520func%2525253Dterm_to_int%2525252C%25252520df%2525253Ddf_all%252529%2525250A%2525253E%2525252036%25252520months%25252520--%2525253E%2525252036%2525250A%2525252060%25252520months%25252520--%2525253E%2525252060&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/16b148eb241542b0a8f61a96446d1b88/href">https://medium.com/media/16b148eb241542b0a8f61a96446d1b88/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all%25255B%2527term%2527%25255D%252520%25253D%252520df_all%25255B%2527term%2527%25255D.apply%2528term_to_int%2529%25250Adf_all.head%2528%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all%2525255B%252527term%252527%2525255D%25252520%2525253D%25252520df_all%2525255B%252527term%252527%2525255D.apply%252528term_to_int%252529%2525250Adf_all.head%252528%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/9fae0640246ede9e2a9bf89152ad4626/href">https://medium.com/media/9fae0640246ede9e2a9bf89152ad4626/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rsKbEOOHBXfZyhLHVa_lRA.png" /></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddef%252520int_rate_to_float%2528x%2529%25253A%252520%252520%252520%252520%25250A%252520%252520%252520%252520x%252520%25253D%252520str%2528x%2529.strip%2528%2529%252520%252520%252520%252520%25250A%252520%252520%252520%252520x%252520%25253D%252520x.replace%2528%2527%252525%2527%25252C%252520%2527%2527%2529%252520%252520%252520%25250A%252520%252520%252520%252520return%252520float%2528x%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddef%25252520int_rate_to_float%252528x%252529%2525253A%25252520%25252520%25252520%25252520%2525250A%25252520%25252520%25252520%25252520x%25252520%2525253D%25252520str%252528x%252529.strip%252528%252529%25252520%25252520%25252520%25252520%2525250A%25252520%25252520%25252520%25252520x%25252520%2525253D%25252520x.replace%252528%252527%25252525%252527%2525252C%25252520%252527%252527%252529%25252520%25252520%25252520%2525250A%25252520%25252520%25252520%25252520return%25252520float%252528x%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/c468d65aeb67a44cca37b38b8c47500d/href">https://medium.com/media/c468d65aeb67a44cca37b38b8c47500d/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all%25255B%2527int_rate%2527%25255D%252520%25253D%252520df_all%25255B%2527int_rate%2527%25255D.apply%2528int_rate_to_float%2529%25250Adf_all%25255B%2527revol_util%2527%25255D%252520%25253D%252520df_all%25255B%2527revol_util%2527%25255D.apply%2528int_rate_to_float%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all%2525255B%252527int_rate%252527%2525255D%25252520%2525253D%25252520df_all%2525255B%252527int_rate%252527%2525255D.apply%252528int_rate_to_float%252529%2525250Adf_all%2525255B%252527revol_util%252527%2525255D%25252520%2525253D%25252520df_all%2525255B%252527revol_util%252527%2525255D.apply%252528int_rate_to_float%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/149c32b490ee48780d05f02776124dca/href">https://medium.com/media/149c32b490ee48780d05f02776124dca/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddef%252520emp_length_to_int%2528x%2529%25253A%252520%252520%252520%252520%25250A%252520%252520%252520%252520x%252520%25253D%252520str%2528x%2529.replace%2528%2527%25253C%252520%2527%25252C%252520%2527%2527%2529%252520%252520%252520%252520%25250A%252509%252520%252520x%252520%25253D%252520x.replace%2528%2527%25252B%2527%25252C%252520%2527%2527%2529%252520%252520%252520%252520%25250A%252509%252520%252520x%252520%25253D%252520x.split%2528%2527%252520%2527%2529%252520%252520%252520%252520%25250A%252509%252520%252520if%252520len%2528x%2529%252520%25253E%2525201%25253A%252520%252520%252520%252520%252520%252520%252520%252520%25250A%252509%252509%252520%252520%252520%252520x%252520%25253D%252520int%2528x%25255B0%25255D%2529%252520%252520%252520%252520%25250A%252509%252520%252520else%25253A%252520%252520%252520%252520%252520%252520%252520%252520%25250A%252509%252509%252520%252520%252520%252520x%252520%25253D%252520None%252520%252520%252520%252520%25250A%252520%252520%252520%252520return%252520x&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddef%25252520emp_length_to_int%252528x%252529%2525253A%25252520%25252520%25252520%25252520%2525250A%25252520%25252520%25252520%25252520x%25252520%2525253D%25252520str%252528x%252529.replace%252528%252527%2525253C%25252520%252527%2525252C%25252520%252527%252527%252529%25252520%25252520%25252520%25252520%2525250A%25252509%25252520%25252520x%25252520%2525253D%25252520x.replace%252528%252527%2525252B%252527%2525252C%25252520%252527%252527%252529%25252520%25252520%25252520%25252520%2525250A%25252509%25252520%25252520x%25252520%2525253D%25252520x.split%252528%252527%25252520%252527%252529%25252520%25252520%25252520%25252520%2525250A%25252509%25252520%25252520if%25252520len%252528x%252529%25252520%2525253E%252525201%2525253A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%2525250A%25252509%25252509%25252520%25252520%25252520%25252520x%25252520%2525253D%25252520int%252528x%2525255B0%2525255D%252529%25252520%25252520%25252520%25252520%2525250A%25252509%25252520%25252520else%2525253A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%2525250A%25252509%25252509%25252520%25252520%25252520%25252520x%25252520%2525253D%25252520None%25252520%25252520%25252520%25252520%2525250A%25252520%25252520%25252520%25252520return%25252520x&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/34ba02c0934a153728d9a8e607e98344/href">https://medium.com/media/34ba02c0934a153728d9a8e607e98344/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dprint_apply_results%2528col_name%25253D%2527emp_length%2527%25252C%252520func%25253Demp_length_to_int%25252C%252520df%25253Ddf_all%2529%25250A%25253E%2525202%252520years%252520--%25253E%2525202%25250A3%252520years%252520--%25253E%2525203%25250A%25253C%2525201%252520year%252520--%25253E%2525201%25250A10%25252B%252520years%252520--%25253E%25252010%25250A9%252520years%252520--%25253E%2525209%25250A5%252520years%252520--%25253E%2525205%25250A7%252520years%252520--%25253E%2525207%25250A4%252520years%252520--%25253E%2525204%25250Anan%252520--%25253E%252520None%25250A1%252520year%252520--%25253E%2525201%25250A8%252520years%252520--%25253E%2525208%25250A6%252520years%252520--%25253E%2525206&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dprint_apply_results%252528col_name%2525253D%252527emp_length%252527%2525252C%25252520func%2525253Demp_length_to_int%2525252C%25252520df%2525253Ddf_all%252529%2525250A%2525253E%252525202%25252520years%25252520--%2525253E%252525202%2525250A3%25252520years%25252520--%2525253E%252525203%2525250A%2525253C%252525201%25252520year%25252520--%2525253E%252525201%2525250A10%2525252B%25252520years%25252520--%2525253E%2525252010%2525250A9%25252520years%25252520--%2525253E%252525209%2525250A5%25252520years%25252520--%2525253E%252525205%2525250A7%25252520years%25252520--%2525253E%252525207%2525250A4%25252520years%25252520--%2525253E%252525204%2525250Anan%25252520--%2525253E%25252520None%2525250A1%25252520year%25252520--%2525253E%252525201%2525250A8%25252520years%25252520--%2525253E%252525208%2525250A6%25252520years%25252520--%2525253E%252525206&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/637934abe72a256a0bfa0bc267314aaa/href">https://medium.com/media/637934abe72a256a0bfa0bc267314aaa/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all%25255B%2527emp_length%2527%25255D%252520%25253D%252520df_all%25255B%2527emp_length%2527%25255D.apply%2528emp_length_to_int%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all%2525255B%252527emp_length%252527%2525255D%25252520%2525253D%25252520df_all%2525255B%252527emp_length%252527%2525255D.apply%252528emp_length_to_int%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/1b774bd006df9d7ca4315846a01e995f/href">https://medium.com/media/1b774bd006df9d7ca4315846a01e995f/href</a></iframe><p>What about all the other columns that contain string values? We can see their cardinality (that’s the number of unique values in a categorical column) by chaining together the following methods:</p><ul><li><a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.select_dtypes.html">pandas.DataFrame.select_dtypes()</a></li><li><a href="https://pandas.pydata.org/docs/reference/api/pandas.Series.nunique.html">pandas.Series.nunique()]</a></li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all.select_dtypes%2528include%25253D%25255B%2527object%2527%25255D%2529.nunique%2528%2529%25250A%25253E%252520grade%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%2525207%25250Asub_grade%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%25252035%25250Aemp_title%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520129449%25250Ahome_ownership%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%2525204%25250Averification_status%252520%252520%252520%252520%252520%252520%252520%252520%252520%2525203%25250Aloan_status%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%2525207%25250Apymnt_plan%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%2525201%25250Apurpose%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%25252013%25250Atitle%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%25252012%25250Azip_code%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520897%25250Aaddr_state%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%25252050%25250Ainitial_list_status%252520%252520%252520%252520%252520%252520%252520%252520%252520%2525202%25250Aapplication_type%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%2525202%25250Ahardship_flag%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%2525202%25250Adebt_settlement_flag%252520%252520%252520%252520%252520%252520%252520%252520%2525202%25250Adtype%25253A%252520int64&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all.select_dtypes%252528include%2525253D%2525255B%252527object%252527%2525255D%252529.nunique%252528%252529%2525250A%2525253E%25252520grade%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252525207%2525250Asub_grade%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%2525252035%2525250Aemp_title%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520129449%2525250Ahome_ownership%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252525204%2525250Averification_status%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252525203%2525250Aloan_status%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252525207%2525250Apymnt_plan%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252525201%2525250Apurpose%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%2525252013%2525250Atitle%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%2525252012%2525250Azip_code%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520897%2525250Aaddr_state%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%2525252050%2525250Ainitial_list_status%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252525202%2525250Aapplication_type%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252525202%2525250Ahardship_flag%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252525202%2525250Adebt_settlement_flag%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252525202%2525250Adtype%2525253A%25252520int64&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/3c2df1de4735aeb275e22fd5fedccfe8/href">https://medium.com/media/3c2df1de4735aeb275e22fd5fedccfe8/href</a></iframe><p>It looks like all columns except for emp_title have a very low number of unique values compared to the total number of observations in our dataset. There’s not much we can do here — in today’s world, the number of different occupations is very large. We did, however, notice that differences in capitalization (e.g. teacher vs Teacher) contributed to the high cardinality of the emp_title column. One thing we can do is normalize the values by making them all lowercase.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddef%252520normalize_emp_title%2528x%2529%25253A%252520%252520%252520%252520%25250A%252520%252520%252520%252520if%252520isinstance%2528x%25252C%252520str%2529%25253A%252520%252520%252520%252520%252520%252520%252520%252520%25250A%252509%252509%252520%252520%252520%252520x%252520%25253D%252520x.strip%2528%2529.lower%2528%2529%252520%252520%252520%252520%25250A%252509%252520%252520else%25253A%252520%252520%252520%252520%252520%252520%252520%252520%25250A%252509%252509%252520%252520%252520%252520x%252520%25253D%252520None%252520%252520%252520%252520%25250A%252509%252520%252520return%252520x&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddef%25252520normalize_emp_title%252528x%252529%2525253A%25252520%25252520%25252520%25252520%2525250A%25252520%25252520%25252520%25252520if%25252520isinstance%252528x%2525252C%25252520str%252529%2525253A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%2525250A%25252509%25252509%25252520%25252520%25252520%25252520x%25252520%2525253D%25252520x.strip%252528%252529.lower%252528%252529%25252520%25252520%25252520%25252520%2525250A%25252509%25252520%25252520else%2525253A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%2525250A%25252509%25252509%25252520%25252520%25252520%25252520x%25252520%2525253D%25252520None%25252520%25252520%25252520%25252520%2525250A%25252509%25252520%25252520return%25252520x&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/06fad5ff64cd30ba556d13309f578550/href">https://medium.com/media/06fad5ff64cd30ba556d13309f578550/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all%25255B%2527emp_title%2527%25255D%252520%25253D%252520df_all%25255B%2527emp_title%2527%25255D.apply%2528normalize_emp_title%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all%2525255B%252527emp_title%252527%2525255D%25252520%2525253D%25252520df_all%2525255B%252527emp_title%252527%2525255D.apply%252528normalize_emp_title%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/9b2e1f157b0ee14907a975c61b24aac1/href">https://medium.com/media/9b2e1f157b0ee14907a975c61b24aac1/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all.select_dtypes%2528include%25253D%25255B%2527object%2527%25255D%2529.nunique%2528%2529%25250A%25253E%252520grade%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%2525207%25250Asub_grade%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%25252035%25250Aemp_title%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520100434%25250Ahome_ownership%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%2525204%25250Averification_status%252520%252520%252520%252520%252520%252520%252520%252520%252520%2525203%25250Aloan_status%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%2525207%25250Apymnt_plan%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%2525201%25250Apurpose%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%25252013%25250Atitle%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%25252012%25250Azip_code%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520897%25250Aaddr_state%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%25252050%25250Ainitial_list_status%252520%252520%252520%252520%252520%252520%252520%252520%252520%2525202%25250Aapplication_type%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%2525202%25250Ahardship_flag%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%2525202%25250Adebt_settlement_flag%252520%252520%252520%252520%252520%252520%252520%252520%2525202%25250Adtype%25253A%252520int64&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all.select_dtypes%252528include%2525253D%2525255B%252527object%252527%2525255D%252529.nunique%252528%252529%2525250A%2525253E%25252520grade%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252525207%2525250Asub_grade%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%2525252035%2525250Aemp_title%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520100434%2525250Ahome_ownership%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252525204%2525250Averification_status%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252525203%2525250Aloan_status%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252525207%2525250Apymnt_plan%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252525201%2525250Apurpose%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%2525252013%2525250Atitle%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%2525252012%2525250Azip_code%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520897%2525250Aaddr_state%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%2525252050%2525250Ainitial_list_status%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252525202%2525250Aapplication_type%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252525202%2525250Ahardship_flag%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252525202%2525250Adebt_settlement_flag%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252525202%2525250Adtype%2525253A%25252520int64&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/48d63eb27496bb020eb790c492471c90/href">https://medium.com/media/48d63eb27496bb020eb790c492471c90/href</a></iframe><p>As for the remaining variables containing strings, we’ll convert them to pandas’ <a href="https://pandas.pydata.org/docs/user_guide/categorical.html">categorical</a> data type.</p><blockquote><em>The categorical data type is useful when:</em></blockquote><blockquote>- We have a string variable consisting of only a few different values. Converting such a string variable to a categorical variable will save some memory</blockquote><blockquote>- The lexical order of a variable is not the same as the logical order (&#39;one&#39;, &#39;two&#39;, &#39;three&#39;). By converting to a categorical and specifying the order of the categories, sorting and min/max will use the logical order instead of the lexical order</blockquote><blockquote>- As a signal to other Python libraries that this column should be treated as a categorical variable (e.g. to use suitable statistical methods or plot types)</blockquote><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3D%252523%252520first%25252C%252520we%252520explicitly%252520exclude%252520the%252520%2527emp_title%2527%252520column%25250Adf_to_cat%252520%25253D%252520df_all.select_dtypes%2528include%25253D%25255B%2527object%2527%25255D%2529.drop%2528columns%25253D%2527emp_title%2527%2529%25250Adf_to_cat.head%2528%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3D%25252523%25252520first%2525252C%25252520we%25252520explicitly%25252520exclude%25252520the%25252520%252527emp_title%252527%25252520column%2525250Adf_to_cat%25252520%2525253D%25252520df_all.select_dtypes%252528include%2525253D%2525255B%252527object%252527%2525255D%252529.drop%252528columns%2525253D%252527emp_title%252527%252529%2525250Adf_to_cat.head%252528%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/cfa3293e1479c7e396cf8b2bdd8211e7/href">https://medium.com/media/cfa3293e1479c7e396cf8b2bdd8211e7/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*63g20OpnQarfdO_VUyWcJw.png" /></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_to_cat.columns%25250A%25253E%252520Index%2528%25255B%2527grade%2527%25252C%252520%2527sub_grade%2527%25252C%252520%2527home_ownership%2527%25252C%252520%2527verification_status%2527%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%2527loan_status%2527%25252C%252520%2527pymnt_plan%2527%25252C%252520%2527purpose%2527%25252C%252520%2527title%2527%25252C%252520%2527zip_code%2527%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%2527addr_state%2527%25252C%252520%2527initial_list_status%2527%25252C%252520%2527application_type%2527%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%2527hardship_flag%2527%25252C%252520%2527debt_settlement_flag%2527%25255D%25252C%25250A%252520%252520%252520%252520%252520%252520dtype%25253D%2527object%2527%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_to_cat.columns%2525250A%2525253E%25252520Index%252528%2525255B%252527grade%252527%2525252C%25252520%252527sub_grade%252527%2525252C%25252520%252527home_ownership%252527%2525252C%25252520%252527verification_status%252527%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252527loan_status%252527%2525252C%25252520%252527pymnt_plan%252527%2525252C%25252520%252527purpose%252527%2525252C%25252520%252527title%252527%2525252C%25252520%252527zip_code%252527%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252527addr_state%252527%2525252C%25252520%252527initial_list_status%252527%2525252C%25252520%252527application_type%252527%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252527hardship_flag%252527%2525252C%25252520%252527debt_settlement_flag%252527%2525255D%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520dtype%2525253D%252527object%252527%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/22d570772b4bf89575b4d5101fd814a2/href">https://medium.com/media/22d570772b4bf89575b4d5101fd814a2/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all%25255Bdf_to_cat.columns%25255D%252520%25253D%252520df_to_cat.astype%2528%252522category%252522%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all%2525255Bdf_to_cat.columns%2525255D%25252520%2525253D%25252520df_to_cat.astype%252528%25252522category%25252522%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/3dfde4661b29de678bab91293ada550d/href">https://medium.com/media/3dfde4661b29de678bab91293ada550d/href</a></iframe><p>Let’s see what datatypes we have now. The only object datatype we expect to see is for the &#39;emp_title&#39; column.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all.dtypes.astype%2528str%2529.value_counts%2528%2529%25250A%25253E%252520int64%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%25252052%25250Afloat64%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%25252029%25250Acategory%252520%252520%252520%252520%252520%252520%252520%252520%252520%25252014%25250Adatetime64%25255Bns%25255D%252520%252520%252520%252520%2525204%25250Aobject%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%2525201%25250Adtype%25253A%252520int64&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all.dtypes.astype%252528str%252529.value_counts%252528%252529%2525250A%2525253E%25252520int64%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%2525252052%2525250Afloat64%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%2525252029%2525250Acategory%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%2525252014%2525250Adatetime64%2525255Bns%2525255D%25252520%25252520%25252520%25252520%252525204%2525250Aobject%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252525201%2525250Adtype%2525253A%25252520int64&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/28834c4025f3cf3e2cf9b34a0bba2b82/href">https://medium.com/media/28834c4025f3cf3e2cf9b34a0bba2b82/href</a></iframe><p>This concludes our feature engineering efforts.</p><h3>Duplicates</h3><p>Now, we’ll quickly check for the presence of duplicate values in the dataset by chaining together the DataFrame method <a href="https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.duplicated.html">duplicated()</a>and the Series method <a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.sum.html">sum()</a>.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all.duplicated%2528%2529.sum%2528%2529%25250A%25253E%2525200&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all.duplicated%252528%252529.sum%252528%252529%2525250A%2525253E%252525200&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/20ed7bd55eea84d5460df1809efffde4/href">https://medium.com/media/20ed7bd55eea84d5460df1809efffde4/href</a></iframe><p>No duplicates.</p><h3>Checkpointing the results</h3><p>It’s a good idea to occasionally create checkpoints of your results by saving intermediate DataFrames. The main goal is to allow yourself to pause and return to your work later without having to redo earlier, potentially time-consuming steps. The most common way of doing this is to use <a href="https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_csv.html">pandas.DataFrame.to_csv()</a> which, as the name suggests, saves your DataFrame as a CSV file. This approach has its pros and cons.</p><p>Pros:</p><ul><li>Human-readable</li><li>Can be opened by many software tools like Excel, as well as other programming languages</li></ul><p>Cons:</p><ul><li>Files are typically much larger than the information they store (because everything is stored in plain text)</li><li>The files don’t store information about data types. That means all your dates/times, categorical variables, etc. will be loaded as simple strings</li></ul><p>If you are planning to share your results with someone else, CSV may be the preferred format.</p><p>Here, though, since we’re just checkpointing, we’ll save the clean DataFrame in the Feather file format using the <a href="https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_feather.html">pandas.DataFrame.to_feather</a> method.</p><p>Pros of Feather:</p><ul><li>It has high read and write performance</li><li>It can be read and written in Python, Rm and a few other programming languages</li><li>It stores information about DataFrame data types</li></ul><p>Cons:</p><ul><li>Unfortunately, unlike CSV files, Feather files are binary files that cannot be viewed in text editors</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all.to_feather%2528Path%2528%2527data%2527%2529%25252F%2527df_clean.feather%2527%2529&amp;display_name=Carbon&amp;url=http%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all.to_feather%252528Path%252528%252527data%252527%252529%2525252F%252527df_clean.feather%252527%252529&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/b38bad526c10916773729aca13d084ca/href">https://medium.com/media/b38bad526c10916773729aca13d084ca/href</a></iframe><p>Let’s load the Feather file back in and check the data types.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all%252520%25253D%252520pd.read_feather%2528Path%2528%2527data%2527%2529%25252F%2527df_clean.feather%2527%2529%25250A%252523%252520confirm%252520that%252520data%252520types%252520were%252520preserved%25250Adf_all.dtypes.astype%2528str%2529.value_counts%2528%2529%25250A%25253E%252520int64%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%25252052%25250Afloat64%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%25252029%25250Acategory%252520%252520%252520%252520%252520%252520%252520%252520%252520%25252014%25250Adatetime64%25255Bns%25255D%252520%252520%252520%252520%2525204%25250Aobject%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%2525201%25250Adtype%25253A%252520int64&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf_all%25252520%2525253D%25252520pd.read_feather%252528Path%252528%252527data%252527%252529%2525252F%252527df_clean.feather%252527%252529%2525250A%25252523%25252520confirm%25252520that%25252520data%25252520types%25252520were%25252520preserved%2525250Adf_all.dtypes.astype%252528str%252529.value_counts%252528%252529%2525250A%2525253E%25252520int64%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%2525252052%2525250Afloat64%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%2525252029%2525250Acategory%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%2525252014%2525250Adatetime64%2525255Bns%2525255D%25252520%25252520%25252520%25252520%252525204%2525250Aobject%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252525201%2525250Adtype%2525253A%25252520int64&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/f637ad42e2baf9a987b7f34325ae987d/href">https://medium.com/media/f637ad42e2baf9a987b7f34325ae987d/href</a></iframe><p>As expected, the Feather format has preserved the data types, and they are indeed the same as what we had earlier.</p><p>Now our data is ready for further exploration.</p><h3>What’s next?</h3><p>In <a href="https://medium.com/practicum-by-yandex/first-things-first-a-brief-guide-to-data-loading-cleaning-and-exploration-562cbc666c3d">Part 3</a>, we’ll take a look at a couple of Python libraries that can help us expedite the exploration stage.</p><p>Learn more about the data science bootcamp and other courses by checking out <a href="https://tripleten.com/">TripleTen</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=847a20a1432f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tripleten-bootcamp/first-things-first-a-brief-guide-to-data-loading-cleaning-and-exploration-847a20a1432f">First Things First: A Brief Guide to Data Loading, Cleaning, and Exploration</a> was originally published in <a href="https://medium.com/tripleten-bootcamp">TripleTen Career Learning Platform</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[First Things First: A Brief Guide to Data Loading, Cleaning, and Exploration]]></title>
            <link>https://medium.com/tripleten-bootcamp/first-things-first-a-brief-guide-to-data-loading-cleaning-and-exploration-56f7222c0e0b?source=rss-997422ae1232------2</link>
            <guid isPermaLink="false">https://medium.com/p/56f7222c0e0b</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[codingbootcamp]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analytics]]></category>
            <dc:creator><![CDATA[TripleTen]]></dc:creator>
            <pubDate>Thu, 19 Aug 2021 12:51:09 GMT</pubDate>
            <atom:updated>2026-01-12T19:42:54.750Z</atom:updated>
            <content:encoded><![CDATA[<h4>Part 1</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-6dCpAsNE8VCgXZwVh0tBA.png" /><figcaption>by Alex Kim, Consultant at TripleTen <a href="https://twitter.com/alex000kim">Twitter</a> | <a href="https://www.linkedin.com/in/alex000kim/">LinkedIn</a></figcaption></figure><p>Note: This is Part 1 of a three-part article.</p><p>If you’ve ever worked with real-world datasets, you know how rare it is for them to be analysis-ready. Inevitably, there will be missing values, inexplicable outliers, unspecified time zones, unclear formatting. Some issues will be specific to the work you’re doing; others will be fairly common across a wide variety of datasets.</p><p>In this guide, we’ll focus on common problems and show you how to deal with them. The goal is to get a clean version of the dataset that’s ready for further data exploration and/or predictive modeling. In Part 3, we’ll show you two libraries that can expedite the exploration stage.</p><h3>Our dataset</h3><p>The dataset we will be using comes from <a href="https://www.lendingclub.com/">LendingClub</a>, a US-based fintech company that facilitates peer-to-peer lending. A brief description by Wikipedia states:</p><blockquote><em>LendingClub enables borrowers to create unsecured personal loans between $1,000 and $40,000. The standard loan period is three years. Investors are able to search and browse the loan listings on the LendingClub website and select loans that they want to invest in based on the information supplied about the borrower, amount of loan, loan grade, and loan purpose. Institutional investors make money from the interest on these loans. LendingClub makes money by charging borrowers an origination fee and investors a service fee.</em></blockquote><p>Each observation in this dataset represents one of the millions of loans made through the LendingClub platform. For detailed descriptions of each column in the dataset, please refer to this <a href="https://github.com/alex000kim/lending_club_eda/blob/main/LCDataDictionary.xlsx">Excel file</a>.</p><p>All the code snippets to follow can also be found in <a href="https://github.com/alex000kim/data_cleaning_guide/blob/main/DataCleaningGuide.ipynb">this Jupyter notebook</a>.</p><h3>Importing libraries</h3><p>The pandas library will help us do all the data manipulation operations we need to clean our data. Additionally, we’ll use:</p><ul><li>urllib and zipfile to download and unzip LendingClub’s data</li><li>pathlib for convenient operations on file system paths</li><li>tqdm for displaying progress bars for long-running tasks</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dimport%252520urllib.request%25250Aimport%252520zipfile%25250Aimport%252520pandas%252520as%252520pd%25250Afrom%252520pathlib%252520import%252520Path%25250Afrom%252520tqdm.auto%252520import%252520tqdm&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dimport%25252520urllib.request%2525250Aimport%25252520zipfile%2525250Aimport%25252520pandas%25252520as%25252520pd%2525250Afrom%25252520pathlib%25252520import%25252520Path%2525250Afrom%25252520tqdm.auto%25252520import%25252520tqdm&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/700ed5d0ded3bce94f9358ef6c941848/href">https://medium.com/media/700ed5d0ded3bce94f9358ef6c941848/href</a></iframe><h3>Data loading</h3><p>Thankfully, LendingClub’s data is available publicly. LendingClub provides data accessible at various URLs with this pattern:</p><p><a href="https://resources.lendingclub.com/LoanStats_"><strong>https://resources.lendingclub.com/LoanStats_</strong></a><strong>&lt;YEAR_QUARTER&gt;.csv.zip</strong></p><p>where &lt;YEAR_QUARTER&gt; needs to be replaced with values indicating the year and the quarter of the year corresponding to each batch of data: 2017Q1, 2017Q2, …, 2020Q2, etc.</p><p>We simply need to download the compresse archives (one per quarter), unzip them, and load the resulting CSV files into pandas DataFrames.</p><p>First, we create a list of years and quarters we are interested in:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dyears%252520%25253D%252520%25255B2018%25255D%252520%252523%252520we%252520can%252520add%252520multiple%252520values%252520to%252520this%252520list%25252C%252520starting%252520with%2525202007%25250Aquarters%252520%25253D%252520%25255B1%25252C%2525202%25252C%2525203%25252C%2525204%25255D%252520%252523%252520below%252520we%252520are%252520using%252520a%252520double%252520for-loop%252520in%252520a%252520list%252520comprehension%252520expression%25250Ayear_quarter_list%252520%25253D%252520%25255Bf%2527%25257Byear%25257DQ%25257Bquarter%25257D%2527%252520for%252520year%252520in%252520years%252520for%252520quarter%252520in%252520quarters%25255D%25250Ayear_quarter_list%25250A%25253E%252520%25255B%25272018Q1%2527%25252C%252520%25272018Q2%2527%25252C%252520%25272018Q3%2527%25252C%252520%25272018Q4%2527%25255D&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dyears%25252520%2525253D%25252520%2525255B2018%2525255D%25252520%25252523%25252520we%25252520can%25252520add%25252520multiple%25252520values%25252520to%25252520this%25252520list%2525252C%25252520starting%25252520with%252525202007%2525250Aquarters%25252520%2525253D%25252520%2525255B1%2525252C%252525202%2525252C%252525203%2525252C%252525204%2525255D%25252520%25252523%25252520below%25252520we%25252520are%25252520using%25252520a%25252520double%25252520for-loop%25252520in%25252520a%25252520list%25252520comprehension%25252520expression%2525250Ayear_quarter_list%25252520%2525253D%25252520%2525255Bf%252527%2525257Byear%2525257DQ%2525257Bquarter%2525257D%252527%25252520for%25252520year%25252520in%25252520years%25252520for%25252520quarter%25252520in%25252520quarters%2525255D%2525250Ayear_quarter_list%2525250A%2525253E%25252520%2525255B%2525272018Q1%252527%2525252C%25252520%2525272018Q2%252527%2525252C%25252520%2525272018Q3%252527%2525252C%25252520%2525272018Q4%252527%2525255D&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/7622b33b30b347b084a29e639a4e09e9/href">https://medium.com/media/7622b33b30b347b084a29e639a4e09e9/href</a></iframe><p>Then we create a corresponding list of URLs:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Durl_list%252520%25253D%252520%25255Bf%2527https%25253A%25252F%25252Fresources.lendingclub.com%25252FLoanStats_%25257Bi%25257D.csv.zip%2527%252520for%252520i%252520in%252520year_quarter_list%25255D%25250Aurl_list%25250A%25253E%252520%25255B%2527https%25253A%25252F%25252Fresources.lendingclub.com%25252FLoanStats_2018Q1.csv.zip%2527%25252C%25250A%252520%2527https%25253A%25252F%25252Fresources.lendingclub.com%25252FLoanStats_2018Q2.csv.zip%2527%25252C%25250A%252520%2527https%25253A%25252F%25252Fresources.lendingclub.com%25252FLoanStats_2018Q3.csv.zip%2527%25252C%25250A%252520%2527https%25253A%25252F%25252Fresources.lendingclub.com%25252FLoanStats_2018Q4.csv.zip%2527%25255D&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Durl_list%25252520%2525253D%25252520%2525255Bf%252527https%2525253A%2525252F%2525252Fresources.lendingclub.com%2525252FLoanStats_%2525257Bi%2525257D.csv.zip%252527%25252520for%25252520i%25252520in%25252520year_quarter_list%2525255D%2525250Aurl_list%2525250A%2525253E%25252520%2525255B%252527https%2525253A%2525252F%2525252Fresources.lendingclub.com%2525252FLoanStats_2018Q1.csv.zip%252527%2525252C%2525250A%25252520%252527https%2525253A%2525252F%2525252Fresources.lendingclub.com%2525252FLoanStats_2018Q2.csv.zip%252527%2525252C%2525250A%25252520%252527https%2525253A%2525252F%2525252Fresources.lendingclub.com%2525252FLoanStats_2018Q3.csv.zip%252527%2525252C%2525250A%25252520%252527https%2525253A%2525252F%2525252Fresources.lendingclub.com%2525252FLoanStats_2018Q4.csv.zip%252527%2525255D&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/cacc44e2376e857edb3bc193ed51e9fc/href">https://medium.com/media/cacc44e2376e857edb3bc193ed51e9fc/href</a></iframe><p>We’ll store our data in its own directory called data. Let’s create it:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3DPath%2528%2527data%2527%2529.mkdir%2528exist_ok%25253DTrue%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3DPath%252528%252527data%252527%252529.mkdir%252528exist_ok%2525253DTrue%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/103dcfc33f26eb4325a6f3a76a394e23/href">https://medium.com/media/103dcfc33f26eb4325a6f3a76a394e23/href</a></iframe><p>Next, we’ll download each zip file from a URL list and unzip it into the data directory:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dfor%252520url%252520in%252520tqdm%2528url_list%2529%25253A%252520%252520%252520%252520%25250A%252509out_fpath%252520%25253D%252520Path%2528%2527data%2527%2529%25252FPath%2528url%2529.name%252520%252520%252520%252520%25250A%252509%252523%252520download%252520the%252520file%252520if%252520it%252520doesn%2527t%252520exist%25250A%252509if%252520not%252520out_fpath.exists%2528%2529%25253A%252520%252520%252520%252520%252520%252520%252520%252520%25250A%252509%252509urllib.request.urlretrieve%2528url%25252C%252520str%2528out_fpath%2529%2529%252520%25250A%252509%252523%252520unzip%252520the%252520downloaded%252520file%25250A%252509with%252520zipfile.ZipFile%2528out_fpath%25252C%252520%252522r%252522%2529%252520as%252520zip_ref%25253A%25250A%252509%252509zip_ref.extractall%2528Path%2528%2527data%2527%2529%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dfor%25252520url%25252520in%25252520tqdm%252528url_list%252529%2525253A%25252520%25252520%25252520%25252520%2525250A%25252509out_fpath%25252520%2525253D%25252520Path%252528%252527data%252527%252529%2525252FPath%252528url%252529.name%25252520%25252520%25252520%25252520%2525250A%25252509%25252523%25252520download%25252520the%25252520file%25252520if%25252520it%25252520doesn%252527t%25252520exist%2525250A%25252509if%25252520not%25252520out_fpath.exists%252528%252529%2525253A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%2525250A%25252509%25252509urllib.request.urlretrieve%252528url%2525252C%25252520str%252528out_fpath%252529%252529%25252520%2525250A%25252509%25252523%25252520unzip%25252520the%25252520downloaded%25252520file%2525250A%25252509with%25252520zipfile.ZipFile%252528out_fpath%2525252C%25252520%25252522r%25252522%252529%25252520as%25252520zip_ref%2525253A%2525250A%25252509%25252509zip_ref.extractall%252528Path%252528%252527data%252527%252529%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/c12f3652e8f0d5eabe6ce8c7114bfb4d/href">https://medium.com/media/c12f3652e8f0d5eabe6ce8c7114bfb4d/href</a></iframe><p>Now let’s collect all files with a .csv extension in the data directory:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dfpath_list%252520%25253D%252520list%2528Path%2528%2527data%2527%2529.glob%2528%2527%252A.csv%2527%2529%2529%25250A%252523%252520glob%252520searches%252520for%252520file%252520paths%252520that%252520match%252520the%252520regular%252520expression%252520in%252520the%252520parentheses%25250Afpath_list%25250A%25253E%252520%25255BPosixPath%2528%2527data%25252FLoanStats_2018Q1.csv%2527%2529%25252C%25250A%252520PosixPath%2528%2527data%25252FLoanStats_2018Q2.csv%2527%2529%25252C%25250A%252520PosixPath%2528%2527data%25252FLoanStats_2018Q3.csv%2527%2529%25252C%25250A%252520PosixPath%2528%2527data%25252FLoanStats_2018Q4.csv%2527%2529%25255D&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dfpath_list%25252520%2525253D%25252520list%252528Path%252528%252527data%252527%252529.glob%252528%252527%25252A.csv%252527%252529%252529%2525250A%25252523%25252520glob%25252520searches%25252520for%25252520file%25252520paths%25252520that%25252520match%25252520the%25252520regular%25252520expression%25252520in%25252520the%25252520parentheses%2525250Afpath_list%2525250A%2525253E%25252520%2525255BPosixPath%252528%252527data%2525252FLoanStats_2018Q1.csv%252527%252529%2525252C%2525250A%25252520PosixPath%252528%252527data%2525252FLoanStats_2018Q2.csv%252527%252529%2525252C%2525250A%25252520PosixPath%252528%252527data%2525252FLoanStats_2018Q3.csv%252527%252529%2525252C%2525250A%25252520PosixPath%252528%252527data%2525252FLoanStats_2018Q4.csv%252527%252529%2525255D&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/d714e26b26544a33eacc048f51e7d62a/href">https://medium.com/media/d714e26b26544a33eacc048f51e7d62a/href</a></iframe><p>We now have all the files we need on our local file system!</p><h3>Coming up next…</h3><p>In Part 2, we’ll uncover various issues that are polluting this data and demonstrate a few techniques for dealing with them.</p><p>Learn more about the data science bootcamp and other courses by checking out <a href="https://tripleten.com/">TripleTen</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=56f7222c0e0b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tripleten-bootcamp/first-things-first-a-brief-guide-to-data-loading-cleaning-and-exploration-56f7222c0e0b">First Things First: A Brief Guide to Data Loading, Cleaning, and Exploration</a> was originally published in <a href="https://medium.com/tripleten-bootcamp">TripleTen Career Learning Platform</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying Data Science Web Apps to the Cloud]]></title>
            <link>https://medium.com/tripleten-bootcamp/deploying-data-science-web-apps-to-the-cloud-9d8967d7fc34?source=rss-997422ae1232------2</link>
            <guid isPermaLink="false">https://medium.com/p/9d8967d7fc34</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[web-app-development]]></category>
            <category><![CDATA[codingbootcamp]]></category>
            <dc:creator><![CDATA[TripleTen]]></dc:creator>
            <pubDate>Wed, 11 Aug 2021 15:36:35 GMT</pubDate>
            <atom:updated>2026-01-13T17:16:38.542Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*td3l4t6fklSSLeYyZiuwqA.png" /><figcaption>by Alex Kim, Consultant at TripleTen <a href="https://twitter.com/alex000kim">Twitter</a> | <a href="https://www.linkedin.com/in/alex000kim/">LinkedIn</a></figcaption></figure><h3>What you’ll learn in this post</h3><p>Do you want to learn how to build beautiful and interactive web apps with zero web development experience?</p><p>In this post, we’ll provide step-by-step instructions on how to:</p><ol><li>Quickly build an interactive web application using <a href="https://streamlit.io/">Streamlit</a></li><li>Deploy this web application to the cloud using <a href="https://www.heroku.com/">Heroku</a>, so that it’s accessible to everyone</li></ol><p>In the first two sections, we’ll discuss why we think you, as a data scientist, should learn a few things about web development and deployment, and we’ll do a quick overview of available web frameworks and cloud providers.</p><p>If you want to skip the prose, feel free to jump right into the tutorial part, starting with the “<strong>Project description and setup</strong>” section.</p><h3>Why is being familiar with web development important for a data scientist?</h3><p>As a data scientist, you’ll be expected to conduct sophisticated statistical analyses and build advanced machine learning models. However, this work will always be a part of a bigger project that involves multiple stakeholders, many of whom won’t have the technical skills, time, or desire to dive deep into your code and the nuances of your work. So now the question is: how do you effectively communicate the results of your work and sell your ideas to the broader team? That’s where web applications come in — an interactive format for communicating the results of your work to other users or project stakeholders.</p><p>Additionally, web applications make an excellent addition to your project portfolio, significantly improving your chances of getting hired.</p><p>Until recently, to create even a simple web application, you had to learn a whole slew of web technologies such as HTML, JavaScript, CSS and relational databases. On top of that, you had to learn Linux administration, networking, security, etc.</p><p>In the last few years, there’s been a rise in web application frameworks explicitly designed for data professionals. These frameworks allow you to quickly build interactive data-centric web applications with zero web development knowledge.</p><p>And on the deployment side, there are quite a few platforms that take care of all the system administration tasks, allowing you to focus on your application.</p><h3>Overview of web frameworks and cloud platforms</h3><p>There are several Python-based web frameworks that are well-suited for data science applications.</p><p>The most popular ones are:</p><ul><li><a href="https://flask.palletsprojects.com/en/2.0.x/">Flask</a> — a general-purpose micro web framework</li><li><a href="https://plotly.com/dash/">Dash</a> — a framework built on top of Flask, Plotly, and ReactJS</li><li><a href="https://github.com/voila-dashboards/voila">Voila</a> — an extension to Jupyter that allows you to convert a Jupyter notebook into an interactive dashboard</li><li><a href="https://streamlit.io/">Streamlit</a> — a dashboarding framework built on top of <a href="https://www.tornadoweb.org/en/stable/">Tornado</a>. (This is the one we’ll be focusing on.)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fSxLvt8pb2pCi-5-nhMuew.jpeg" /><figcaption>GitHub star trends of most popular web frameworks used by data scientists. Source: <a href="https://star-history.t9t.io/">https://star-history.t9t.io/</a></figcaption></figure><h3>IaaS vs. PaaS</h3><p>Now let’s quickly discuss the differences between the two types of cloud services: Infrastructure as a Service (IaaS) and Platform as a Service (PaaS).</p><p>PaaS delivers a platform for developers that they can build upon and use to create customized applications. All servers, storage, and networking are managed by the provider.</p><p>IaaS delivers cloud computing infrastructure, including servers, networks, operating systems, and storage, through virtualization technology.</p><p>The table below summarizes the differences between IaaS and PaaS services and lists some examples of cloud vendors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4cV3j1oaYvbjLPrtobUnNw.png" /><figcaption>Summary of the differences between IaaS and PaaS services and examples of cloud vendors.</figcaption></figure><h3>Project description and setup</h3><p>In this project, we’ll build a simple application to explore some properties of a used car dataset (available <a href="https://raw.githubusercontent.com/alex000kim/simple-streamlit-datavis-app/main/vehicles_us.csv">here</a>). You can see the final result in the video below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/888/1*IQImdvDvobMjYNDZz40zPw.gif" /><figcaption>Final web application</figcaption></figure><p>We’ll be using <a href="https://pypi.org/project/pipenv/">pipenv</a> to manage the virtual environment for this project, so make sure to install it into your development environment. Once that’s done, we’ll need to perform the following steps:</p><ul><li><a href="https://docs.github.com/en/get-started/quickstart/create-a-repo">Create a GitHub repository</a></li><li><a href="https://docs.github.com/en/github/creating-cloning-and-archiving-repositories/cloning-a-repository-from-github/cloning-a-repository">Clone the repository locally</a></li><li>Run the following commands inside the repository directory:</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dapplication%252Fx-sh%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dpipenv%252520shell%25250Apipenv%252520install%252520pandas%252520streamlit%252520plotly-express&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dapplication%25252Fx-sh%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dpipenv%25252520shell%2525250Apipenv%25252520install%25252520pandas%25252520streamlit%25252520plotly-express&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/28f772284c1a10c425914755b2b6b4f4/href">https://medium.com/media/28f772284c1a10c425914755b2b6b4f4/href</a></iframe><ul><li>Download the <a href="https://raw.githubusercontent.com/alex000kim/simple-streamlit-datavis-app/main/vehicles_us.csv">dataset</a> into the repository folder</li></ul><p>At this stage, your repository folder should contain the following files: Pipfile, Pipfile.lock, vehicles_us.csv, and some other optional files like README.md, .gitignore, LICENSE, etc. depending on what options you selected when creating your repository.</p><h3>Building and testing our web application</h3><p>All the code for our application will be contained in one file: app.py.</p><p>First, let’s import all the libraries we installed earlier:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dimport%252520streamlit%252520as%252520st%25250Aimport%252520pandas%252520as%252520pd%25250Aimport%252520plotly_express%252520as%252520px&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dimport%25252520streamlit%25252520as%25252520st%2525250Aimport%25252520pandas%25252520as%25252520pd%2525250Aimport%25252520plotly_express%25252520as%25252520px&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ca781a70433ebc951d25bd8e019062d1/href">https://medium.com/media/ca781a70433ebc951d25bd8e019062d1/href</a></iframe><p>Next, we load our data and create a new column manufacturer by getting the first word from the model column:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf%252520%25253D%252520pd.read_csv%2528%2527vehicles_us.csv%2527%2529%25250Adf%25255B%2527manufacturer%2527%25255D%252520%25253D%252520df%25255B%2527model%2527%25255D.apply%2528lambda%252520x%25253A%252520x.split%2528%2529%25255B0%25255D%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddf%25252520%2525253D%25252520pd.read_csv%252528%252527vehicles_us.csv%252527%252529%2525250Adf%2525255B%252527manufacturer%252527%2525255D%25252520%2525253D%25252520df%2525255B%252527model%252527%2525255D.apply%252528lambda%25252520x%2525253A%25252520x.split%252528%252529%2525255B0%2525255D%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/a2ee8d8cd31d92b2108e1f7633ea7c08/href">https://medium.com/media/a2ee8d8cd31d92b2108e1f7633ea7c08/href</a></iframe><p>Now we can display our DataFrame with Streamlit by simply writing:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3D%252523%252520create%252520a%252520text%252520header%252520above%252520the%252520dataframe%25250Ast.header%2528%2527Data%252520viewer%2527%2529%252520%25250A%252523%252520display%252520the%252520dataframe%252520with%252520streamlit%25250Ast.dataframe%2528df%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3D%25252523%25252520create%25252520a%25252520text%25252520header%25252520above%25252520the%25252520dataframe%2525250Ast.header%252528%252527Data%25252520viewer%252527%252529%25252520%2525250A%25252523%25252520display%25252520the%25252520dataframe%25252520with%25252520streamlit%2525250Ast.dataframe%252528df%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/e2546092882706b14fbe6e962d2af389/href">https://medium.com/media/e2546092882706b14fbe6e962d2af389/href</a></iframe><h3>Vehicle types by manufacturer</h3><p>To see the distribution of vehicle types by the manufacturer (e.g. how many Ford sedans vs. trucks are in this dataset), we can create a simple Plotly histogram and display it with Streamlit like this:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dst.header%2528%2527Vehicle%252520types%252520by%252520manufacturer%2527%2529%25250A%252523%252520create%252520a%252520plotly%252520histogram%252520figure%25250Afig%252520%25253D%252520px.histogram%2528df%25252C%252520x%25253D%2527manufacturer%2527%25252C%252520color%25253D%2527type%2527%2529%25250A%252523%252520display%252520the%252520figure%252520with%252520streamlit%25250Ast.write%2528fig%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dst.header%252528%252527Vehicle%25252520types%25252520by%25252520manufacturer%252527%252529%2525250A%25252523%25252520create%25252520a%25252520plotly%25252520histogram%25252520figure%2525250Afig%25252520%2525253D%25252520px.histogram%252528df%2525252C%25252520x%2525253D%252527manufacturer%252527%2525252C%25252520color%2525253D%252527type%252527%252529%2525250A%25252523%25252520display%25252520the%25252520figure%25252520with%25252520streamlit%2525250Ast.write%252528fig%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/e54fd76eaceb849a602db06182c1ac6a/href">https://medium.com/media/e54fd76eaceb849a602db06182c1ac6a/href</a></iframe><h3>Histogram of condition vs. model_year</h3><p>Similarly, if we wanted to explore the relationship between condition and model_year, we&#39;d use the following:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dst.header%2528%2527Histogram%252520of%252520%252560condition%252560%252520vs%252520%252560model_year%252560%2527%2529%25250Afig%252520%25253D%252520px.histogram%2528df%25252C%252520x%25253D%2527model_year%2527%25252C%252520color%25253D%2527condition%2527%2529%25250Ast.write%2528fig%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dst.header%252528%252527Histogram%25252520of%25252520%25252560condition%25252560%25252520vs%25252520%25252560model_year%25252560%252527%252529%2525250Afig%25252520%2525253D%25252520px.histogram%252528df%2525252C%25252520x%2525253D%252527model_year%252527%2525252C%25252520color%2525253D%252527condition%252527%252529%2525250Ast.write%252528fig%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/46d8c571a95b71550219c79d48f539df/href">https://medium.com/media/46d8c571a95b71550219c79d48f539df/href</a></iframe><h3>Compare price distribution between manufacturers</h3><p>Let’s say we want to look into price distribution between a pair of manufacturers.</p><p>First, we’d need to get the user’s input for the first and second manufacturer’s names, then filter our DataFrame to contain only these two manufacturers.</p><p>Then we can create a Plotly histogram with a few parameters:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dst.header%2528%2527Compare%252520price%252520distribution%252520between%252520manufacturers%2527%2529%25250A%252523%252520get%252520a%252520list%252520of%252520car%252520manufacturers%25250Amanufac_list%252520%25253D%252520sorted%2528df%25255B%2527manufacturer%2527%25255D.unique%2528%2529%2529%25250A%252523%252520get%252520user%2527s%252520inputs%252520from%252520a%252520dropdown%252520menu%25250Amanufacturer_1%252520%25253D%252520st.selectbox%2528%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520label%25253D%2527Select%252520manufacturer%2525201%2527%25252C%252520%252523%252520title%252520of%252520the%252520select%252520box%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520options%25253Dmanufac_list%25252C%252520%252523%252520options%252520listed%252520in%252520the%252520select%252520box%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520index%25253Dmanufac_list.index%2528%2527chevrolet%2527%2529%252520%252523%252520default%252520pre-selected%252520option%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%2529%25250A%252523%252520repeat%252520for%252520the%252520second%252520dropdown%252520menu%25250Amanufacturer_2%252520%25253D%252520st.selectbox%2528%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520label%25253D%2527Select%252520manufacturer%2525202%2527%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520options%25253Dmanufac_list%25252C%252520%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520index%25253Dmanufac_list.index%2528%2527hyundai%2527%2529%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%2529%25250A%252523%252520filter%252520the%252520dataframe%252520%25250Amask_filter%252520%25253D%252520%2528df%25255B%2527manufacturer%2527%25255D%252520%25253D%25253D%252520manufacturer_1%2529%252520%25257C%252520%2528df%25255B%2527manufacturer%2527%25255D%252520%25253D%25253D%252520manufacturer_2%2529%25250Adf_filtered%252520%25253D%252520df%25255Bmask_filter%25255D%25250A%25250A%252523%252520add%252520a%252520checkbox%252520if%252520a%252520user%252520wants%252520to%252520normalize%252520the%252520histogram%25250Anormalize%252520%25253D%252520st.checkbox%2528%2527Normalize%252520histogram%2527%25252C%252520value%25253DTrue%2529%25250Aif%252520normalize%25253A%25250A%252520%252520%252520%252520histnorm%252520%25253D%252520%2527percent%2527%25250Aelse%25253A%25250A%252520%252520%252520%252520histnorm%252520%25253D%252520None%25250A%25250A%252523%252520create%252520a%252520plotly%252520histogram%252520figure%25250Afig%252520%25253D%252520px.histogram%2528df_filtered%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520x%25253D%2527price%2527%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520nbins%25253D30%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520color%25253D%2527manufacturer%2527%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520histnorm%25253Dhistnorm%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520barmode%25253D%2527overlay%2527%2529%25250A%252523%252520display%252520the%252520figure%252520with%252520streamlit%25250Ast.write%2528fig%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dpython%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dst.header%252528%252527Compare%25252520price%25252520distribution%25252520between%25252520manufacturers%252527%252529%2525250A%25252523%25252520get%25252520a%25252520list%25252520of%25252520car%25252520manufacturers%2525250Amanufac_list%25252520%2525253D%25252520sorted%252528df%2525255B%252527manufacturer%252527%2525255D.unique%252528%252529%252529%2525250A%25252523%25252520get%25252520user%252527s%25252520inputs%25252520from%25252520a%25252520dropdown%25252520menu%2525250Amanufacturer_1%25252520%2525253D%25252520st.selectbox%252528%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520label%2525253D%252527Select%25252520manufacturer%252525201%252527%2525252C%25252520%25252523%25252520title%25252520of%25252520the%25252520select%25252520box%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520options%2525253Dmanufac_list%2525252C%25252520%25252523%25252520options%25252520listed%25252520in%25252520the%25252520select%25252520box%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520index%2525253Dmanufac_list.index%252528%252527chevrolet%252527%252529%25252520%25252523%25252520default%25252520pre-selected%25252520option%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252529%2525250A%25252523%25252520repeat%25252520for%25252520the%25252520second%25252520dropdown%25252520menu%2525250Amanufacturer_2%25252520%2525253D%25252520st.selectbox%252528%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520label%2525253D%252527Select%25252520manufacturer%252525202%252527%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520options%2525253Dmanufac_list%2525252C%25252520%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520index%2525253Dmanufac_list.index%252528%252527hyundai%252527%252529%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252529%2525250A%25252523%25252520filter%25252520the%25252520dataframe%25252520%2525250Amask_filter%25252520%2525253D%25252520%252528df%2525255B%252527manufacturer%252527%2525255D%25252520%2525253D%2525253D%25252520manufacturer_1%252529%25252520%2525257C%25252520%252528df%2525255B%252527manufacturer%252527%2525255D%25252520%2525253D%2525253D%25252520manufacturer_2%252529%2525250Adf_filtered%25252520%2525253D%25252520df%2525255Bmask_filter%2525255D%2525250A%2525250A%25252523%25252520add%25252520a%25252520checkbox%25252520if%25252520a%25252520user%25252520wants%25252520to%25252520normalize%25252520the%25252520histogram%2525250Anormalize%25252520%2525253D%25252520st.checkbox%252528%252527Normalize%25252520histogram%252527%2525252C%25252520value%2525253DTrue%252529%2525250Aif%25252520normalize%2525253A%2525250A%25252520%25252520%25252520%25252520histnorm%25252520%2525253D%25252520%252527percent%252527%2525250Aelse%2525253A%2525250A%25252520%25252520%25252520%25252520histnorm%25252520%2525253D%25252520None%2525250A%2525250A%25252523%25252520create%25252520a%25252520plotly%25252520histogram%25252520figure%2525250Afig%25252520%2525253D%25252520px.histogram%252528df_filtered%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520x%2525253D%252527price%252527%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520nbins%2525253D30%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520color%2525253D%252527manufacturer%252527%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520histnorm%2525253Dhistnorm%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520barmode%2525253D%252527overlay%252527%252529%2525250A%25252523%25252520display%25252520the%25252520figure%25252520with%25252520streamlit%2525250Ast.write%252528fig%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/1a540da05bafcf393e7c52326e286cf7/href">https://medium.com/media/1a540da05bafcf393e7c52326e286cf7/href</a></iframe><p>That’s it. With these few lines of code, we’ve built a simple but perfectly functional application.</p><p>You can find the complete app.py file <a href="https://github.com/alex000kim/simple-streamlit-datavis-app/blob/main/app.py">here</a>. We hope that you get the main idea and that later you&#39;ll be able to make your applications as sophisticated as you want them to be.</p><p>Let’s quickly test the application by running it locally first.</p><p>In your terminal, type the following command:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dapplication%252Fx-sh%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dstreamlit%252520run%252520app.py&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dapplication%25252Fx-sh%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dstreamlit%25252520run%25252520app.py&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/9ae03a904ec6c0128a6b06598236357b/href">https://medium.com/media/9ae03a904ec6c0128a6b06598236357b/href</a></iframe><p>It will start a Streamlit web server that you can access in your browser at <a href="http://localhost:8501/">http://localhost:8501/</a></p><p>If there’s anything that you want to fix/add to your application, edit the app.py file and refresh the browser to see the changes.</p><p>If you are satisfied with the local version of the application, let’s commit our edits back to GitHub:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dapplication%252Fx-sh%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3D%252523%252520add%252520all%252520changes%252520in%252520the%252520working%252520directory%252520to%252520the%252520git%252520staging%252520area%25250Agit%252520add%252520%252A%252520%25250A%252523%252520%252522save%252522%252520a%252520snapshot%252520of%252520the%252520project%2527s%252520currently%252520staged%252520changes%25250Agit%252520commit%252520-m%252520%252522initial%252520commit%252522%25250A%252523%252520upload%252520local%252520repository%252520content%252520to%252520a%252520remote%252520repository%25250A%252523%252520right%252520now%25252C%252520we%252520only%252520have%252520one%252520remote%252520repository%252520-%252520github%25250Agit%252520push&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dapplication%25252Fx-sh%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3D%25252523%25252520add%25252520all%25252520changes%25252520in%25252520the%25252520working%25252520directory%25252520to%25252520the%25252520git%25252520staging%25252520area%2525250Agit%25252520add%25252520%25252A%25252520%2525250A%25252523%25252520%25252522save%25252522%25252520a%25252520snapshot%25252520of%25252520the%25252520project%252527s%25252520currently%25252520staged%25252520changes%2525250Agit%25252520commit%25252520-m%25252520%25252522initial%25252520commit%25252522%2525250A%25252523%25252520upload%25252520local%25252520repository%25252520content%25252520to%25252520a%25252520remote%25252520repository%2525250A%25252523%25252520right%25252520now%2525252C%25252520we%25252520only%25252520have%25252520one%25252520remote%25252520repository%25252520-%25252520github%2525250Agit%25252520push&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/d4a4140639b3a2169415f96b63b470d1/href">https://medium.com/media/d4a4140639b3a2169415f96b63b470d1/href</a></iframe><h3>Deploying our application</h3><p>Now it’s time to deploy our application to the cloud:</p><ul><li>Add a file named Procfile to the root of your repository (i.e. in the same directory with other files like Pipfile and Pipfile.lock)</li><li>Write the following line to Procfile: web: streamlit run --server.enableCORS false --server.port $PORT <br>This tells Heroku what kind of application we&#39;ll be running (i.e. web), and what specific command to run when launching your application for the first time.</li><li>Create an account on <a href="https://heroku.com/">heroku.com</a> and install <a href="https://devcenter.heroku.com/articles/heroku-cli">heroku cli</a></li><li>Create a new app and give it a name</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*URQrZL8YSiz2pys5rMdhkg.png" /></figure><ul><li>Add heroku repository:</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dapplication%252Fx-sh%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dheroku%252520git%25253Aremote%252520-a%252520%25253CMY_HEROKU_APP_NAME%25253E&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dapplication%25252Fx-sh%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dheroku%25252520git%2525253Aremote%25252520-a%25252520%2525253CMY_HEROKU_APP_NAME%2525253E&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/3831e802d01c1a80cdc80db33c1a629f/href">https://medium.com/media/3831e802d01c1a80cdc80db33c1a629f/href</a></iframe><ul><li>Deploy to Heroku:</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528255%252C255%252C255%252C1%2529%26t%3Dvscode%26wt%3Dbw%26l%3Dapplication%252Fx-sh%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dgit%252520push%252520heroku%252520HEAD%25253Amaster&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528255%25252C255%25252C255%25252C1%252529%26t%3Dvscode%26wt%3Dbw%26l%3Dapplication%25252Fx-sh%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dfalse%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dgit%25252520push%25252520heroku%25252520HEAD%2525253Amaster&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/dc19aaf9f47a403af3319b0302c3f6e3/href">https://medium.com/media/dc19aaf9f47a403af3319b0302c3f6e3/href</a></iframe><p>This last command pushes your code to Heroku and triggers the provisioning of resources (i.e. a small virtual machine that will host our application). It also takes care of installing all required Python libraries. Heroku automatically detects Pipfile and Pipfile.lock , and will install everything we put there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M_h8cmejNbxbgamNcfvjSA.png" /></figure><p>Finally, after a couple of minutes, you’ll see the URL of your deployed application. It’ll look like this:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528171%252C%2B184%252C%2B195%252C%2B1%2529%26t%3Dseti%26wt%3Dnone%26l%3Dpython%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dhttps%25253A%25252F%25252F%25253CMY_HEROKU_APP_NAME%25253E.herokuapp.com&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528171%25252C%2B184%25252C%2B195%25252C%2B1%252529%26t%3Dseti%26wt%3Dnone%26l%3Dpython%26ds%3Dtrue%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dhttps%2525253A%2525252F%2525252F%2525253CMY_HEROKU_APP_NAME%2525253E.herokuapp.com&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/2097bf3d96e2f021694a3f9cea2c8b7b/href">https://medium.com/media/2097bf3d96e2f021694a3f9cea2c8b7b/href</a></iframe><p>Visit this URL to make sure everything is working as expected.</p><h3>Wrap up</h3><p>Congratulations! Feel free to add this URL to your portfolio and share it with your friends and family.</p><p>Streamlit is an excellent framework for those getting started with web development, and we’ve only scratched the surface of what it can do. Visit Streamlit’s <a href="https://streamlit.io/gallery">application gallery</a> to see more incredible examples of what’s possible to build with it.</p><p>We hope to have convinced you that adding skills like web development to your toolkit is a worthwhile and enjoyable way to boost your chances of landing a data scientist job or getting ahead in your current role.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d8967d7fc34" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tripleten-bootcamp/deploying-data-science-web-apps-to-the-cloud-9d8967d7fc34">Deploying Data Science Web Apps to the Cloud</a> was originally published in <a href="https://medium.com/tripleten-bootcamp">TripleTen Career Learning Platform</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Science Portfolio: Making the Most Out of GitHub]]></title>
            <link>https://medium.com/tripleten-bootcamp/data-science-portfolio-making-the-most-out-of-github-dac98c536ffc?source=rss-997422ae1232------2</link>
            <guid isPermaLink="false">https://medium.com/p/dac98c536ffc</guid>
            <category><![CDATA[portfolio]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[codingbootcamp]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[TripleTen]]></dc:creator>
            <pubDate>Tue, 15 Jun 2021 13:39:04 GMT</pubDate>
            <atom:updated>2026-07-13T17:06:55.235Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jw1qj94J3yH5H3H6suQUWg.png" /><figcaption>by <strong>Alex Kim, </strong>Consultant at TripleTen <a href="https://twitter.com/alex000kim">Twitter</a> | <a href="https://www.linkedin.com/in/alex000kim/">LinkedIn</a></figcaption></figure><p>In an earlier post on “<a href="https://medium.com/tripleten-bootcamp/how-to-create-an-outstanding-data-science-portfolio-34f96a66cf4c">How to Create an Outstanding Data Science Portfolio</a>,” we discussed different ideas on approaching your portfolio and maximizing the chances of finding your first job as a data scientist. One of the best ways to highlight your technical skills is to have a strong presence on GitHub. What would this presence entail?</p><p>For this post, we will focus on the four areas where GitHub can help you strengthen your portfolio:</p><ul><li>Project repositories</li><li>Profile README</li><li>Open-source collaboration</li><li>GitHub Pages</li></ul><p>Everything below assumes that you have basic familiarity with git and GitHub terminology such as repository, branch, commit, pull request, etc. If not, go ahead and check out this <a href="https://guides.github.com/introduction/git-handbook/">guide</a> and come back for this post.</p><h3>Project repositories</h3><p>As a data scientist, you can make your project repository focused on one or more of the following:</p><ul><li><strong>Dataset collection</strong> — it could be a set of useful shell and/or python scripts to collect interesting data from a website or a web API. These types of projects are very often overlooked by data scientists building their portfolios. It’s a shame because there’s a real shortage of suitable datasets to work with. For this reason, sharing an interesting dataset (or at least a way to collect it) could be your shortcut to building a good reputation among your peers in the data science community.</li><li><strong>Data storytelling and visualization</strong> — a project where you thoroughly analyze a particular dataset, uncover unique insights, and build elegant visualizations. These projects are amazing for practicing one of the most valuable skills for a data scientist — communicating your findings to other people with little to no programming skills or statistical knowledge.</li><li><strong>Machine learning</strong> — this type of project would contain code that performs data cleaning, feature engineering, and ML modeling. Here you can either 1) train a model on a unique and interesting dataset that no one has approached before you, or 2) try to use novel techniques and ML algorithms to achieve state-of-the-art performance on some benchmark datasets. Unless your ultimate goal is to have a research-oriented position, we recommend the former approach.</li></ul><p>Regardless of what type of project you decide to work on, you need to make sure it’s well-structured. While a project structure may differ depending on what you are trying to achieve as well as your personal preference, there are some conventions:</p><ul><li>Keeping the source code separate from your jupyter notebooks and unit tests</li><li>Having a separate directory for all static files and media assets (images, audio files, etc.)</li><li>Having all project configs (such as virtual environment files) at the root of the project</li></ul><p>As a good starting point, check out this good (although a little opinionated), “<a href="https://github.com/drivendata/cookiecutter-data-science">Cookiecutter Data Science</a>” library that can help you set up a logical project structure in minutes.</p><p>Another important consideration for your project is code quality. You need to make sure that your code follows best practices when it comes to modularity, variable naming, formatting, clear comments, and documentation, etc. For example, if you are using Python, there’s either a <a href="https://www.python.org/dev/peps/pep-0008/">PEP 8 Style Guide</a> or its alternative <a href="https://google.github.io/styleguide/pyguide.html">Google Python Style Guide</a>. Whichever one you chose is a matter of taste, but make sure that your code editor/IDE (e.g. Visual Studio Code or PyCharm) automates as many code quality checks as possible (via one of the many available linting libraries such as Pylint, Pyflakes, Pydocstyle, etc.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/0*iC8cw_tCucaaXgc8.png" /><figcaption>Automatic Python code linting in Visual Studio Code.</figcaption></figure><p>And finally, you must have a clear and well-structured README file at the root of your repository. At the bare minimum, you should specify the project name, description, and setup/usage instructions. Additionally, depending on the project type, you might need to add contribution instructions, credits, table of contents, etc.</p><p>To properly format your README file, you’ll need to familiarize yourself with some Markdown syntax. Thankfully, there are many Markdown cheat sheets available online, like this <a href="https://guides.github.com/pdfs/markdown-cheatsheet-online.pdf">one</a>. Many IDEs support WYSIWYG-style Markdown editing. Alternatively, you can install a dedicated Markdown editor like <a href="https://typora.io/">Typora</a> or <a href="https://marktext.app/">Mark Text</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XkvvX91Fecgs0eEwswGi6A.png" /><figcaption>Markdown editing and preview in Visual Studio Code.</figcaption></figure><h3>Profile README</h3><p>There’s one very special <a href="https://docs.github.com/en/github/setting-up-and-managing-your-github-profile/customizing-your-profile/managing-your-profile-readme">Profile README</a> file that you can create to give the visitors of your GitHub profile a quick overview of yourself, your skills and various projects you are working on.</p><p>If you create a repository with the same name as your username and add a README file to it, GitHub will prominently display this README file above all your pinned repositories.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KCSU7JmHoe2dGeYc.png" /><figcaption>Create a repository with the same name as your username</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UssAEJxm25VoVkrB.png" /><figcaption>Example of what a super simple Profile README looks like.</figcaption></figure><p>After that, you can edit this file just like you would any other project README. The only difference is this README is all about you, so fill it with your interests and achievements, and style it according to your personality.</p><p>You can pick a header image to be displayed at the top or use emojis, icons, animated GIFs, and videos. <a href="https://github.com/anuraghazra/github-readme-stats">GitHub Stats Card</a> can generate a card with your GitHub statistics (like number of stars, commits, PRs, etc.) that you can embed into your README.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b9dj2OjMBnb517cIpuhwPg.png" /><figcaption>Example of a card with GitHub stats.</figcaption></figure><p>If you want to get started quickly without manually editing your Profile README file, you can use one of several Profile README generators such as:</p><ul><li><a href="https://github-profile-readme-creator.web.app/">https://github-profile-readme-creator.web.app/</a></li><li><a href="https://rahuldkjain.github.io/gh-profile-readme-generator/">https://rahuldkjain.github.io/gh-profile-readme-generator/</a></li><li><a href="https://profilinator.rishav.dev/">https://profilinator.rishav.dev/</a></li></ul><p>Just enter the info you want, download the generated README and add it to your profile — that’s it!</p><h3>Open source contributions</h3><p>Another great way to hone your skills is to contribute to other GitHub projects. It could be a small project that your friend is trying to get off the ground, or a big open-source project, like scikit-learn or pandas, that has been around for many years and has an active community of collaborators.</p><p>There are many different ways to do that. Of course, the most common form of contribution is code that either fixes a bug or adds a feature. However, you can contribute in many other ways: add or clarify documentation, report an issue, create a tutorial, etc.</p><p>It may even be recommended for beginners to start with non-code contributions to get to know the project and its community a little better. Over time you may even find a few mentors who will guide you and provide feedback on your work. If open-source contributions are something you are interested in, check out this excellent “<a href="https://opensource.guide/how-to-contribute/">How to Contribute to Open Source</a>” guide.</p><h3>Personal blog on GitHub Pages</h3><p>Have you been thinking about creating your own data science blog?</p><p>Then GitHub Pages is the service for you! GitHub Pages allows hosting static websites.</p><p>A static website only serves HTML, CSS, and JavaScript client-side code as opposed to a dynamic website. It doesn’t have to process any external database or authentification for website users. While this might seem like a limitation, there are many benefits to it, such as simplicity of development/deployment, as well as the latency of serving the website to the end-users.</p><p>To take advantage of GitHub Pages, create a repository named &lt;your_username&gt;<a href="http://username.github.io/">.github.io</a> and start adding your HTML, CSS, and JavaScript files to it. Then, your blog will be accessible through the same URL as the repository name to everyone in the world.</p><p>Of course, as a data scientist, you might not be an expert in web development — that’s where static site generators come to the rescue. These generators are essentially opinionated frameworks that come with “batteries included” (if by batteries we mean design themes and layouts). So all you need to do is focus on the data science content.</p><p>Some of the most popular static site generators are:</p><ul><li><a href="https://gohugo.io/">Hugo</a></li><li><a href="https://jekyllrb.com/">Jekyll</a></li><li><a href="https://www.gatsbyjs.com/">Gatsby</a></li><li><a href="https://blog.getpelican.com/">Pelican</a></li></ul><h3>Summary</h3><p>This post covers all the four major services and products available on GitHub that will help you boost your portfolio. If you are new to GitHub and maybe only recently created an account there, we would suggest starting with creating your Profile README, as it would allow you to learn Markdown syntax and get familiar with GitHub’s user interface. After that, you might want to clean up your data science projects that have been living on your personal computer and upload them to GitHub. Don’t forget about code quality, documentation, and a README. Next, set up your blog, try out a few different static site generators and design themes, then start posting! Or maybe fixing that pesky bug in a python library that you’ve been using is something that’s more up your alley. So go ahead, fix it and create a pull request. Hopefully, we’ve convinced you that establishing your presence on GitHub is one of the best ways to boost your portfolio.</p><p>Learn more about the data science bootcamp and other courses by checking out <a href="https://tripleten.com/">TripleTen</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dac98c536ffc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tripleten-bootcamp/data-science-portfolio-making-the-most-out-of-github-dac98c536ffc">Data Science Portfolio: Making the Most Out of GitHub</a> was originally published in <a href="https://medium.com/tripleten-bootcamp">TripleTen Career Learning Platform</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>