<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Wei-Ting Yap on Medium]]></title>
        <description><![CDATA[Stories by Wei-Ting Yap on Medium]]></description>
        <link>https://medium.com/@weitingyp?source=rss-94b2766fcfe9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*1C5wWVqitzCZchOPGHxXzw.jpeg</url>
            <title>Stories by Wei-Ting Yap on Medium</title>
            <link>https://medium.com/@weitingyp?source=rss-94b2766fcfe9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 05:00:09 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@weitingyp/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How I landed an offer as an Uber APM — the recruiting process]]></title>
            <link>https://weitingyp.medium.com/uber-apm-recruiting-process-aa0476736df9?source=rss-94b2766fcfe9------2</link>
            <guid isPermaLink="false">https://medium.com/p/aa0476736df9</guid>
            <category><![CDATA[professional-development]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[apm]]></category>
            <category><![CDATA[tech-recruiting]]></category>
            <dc:creator><![CDATA[Wei-Ting Yap]]></dc:creator>
            <pubDate>Mon, 19 Jul 2021 17:33:52 GMT</pubDate>
            <atom:updated>2021-07-19T17:33:52.833Z</atom:updated>
            <content:encoded><![CDATA[<h4>By an incoming Uber APM</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*P48vB53DjHo9fvOv" /><figcaption>Photo by <a href="https://unsplash.com/@thisisengineering?utm_source=medium&amp;utm_medium=referral">ThisisEngineering RAEng</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>Part 2 of a two-part series:</strong></p><ul><li><a href="https://weitingyp.medium.com/preparing-for-apm-recruiting-5b22ecde830a">Part 1: How to prepare for APM recruiting</a></li><li><strong>Part 2: How I landed an offer as an Uber APM — the recruiting process</strong></li></ul><p><strong>Beyond these…</strong></p><p>Subscribe to <a href="https://breakintotech.substack.com/">Break Into Tech, a newsletter where I publish weekly actionable tips for professional development</a>.</p><h3>The Process</h3><p>Uber was one of the first few APM programs to open applications. Their applications started in mid-August (this year, <a href="https://www.uber.com/us/en/careers/apm/">they’ll open this month</a>). I applied with a referral and nervously went through the ups and downs of the recruiting process.</p><p>Two months later, after a grueling 5-hour virtual onsite, the recruiter called me to tell me I landed an offer — at midnight in London time. I was stoked! I screamed in joy and celebrated the good news with my biggest supporters.</p><p>I have come a long way to become an incoming APM at Uber. I’m writing this article to share my experience with current aspiring PMs like you, hoping to pass on the kind mentorship of those who have helped me.</p><h3>Contents</h3><ul><li><a href="#fccd"><strong>How I stood out — positive feedback received</strong></a></li><li><a href="#f8eb"><strong>Overview of the timeline</strong></a></li><li><a href="#68b2"><strong>Research and personal branding</strong></a></li><li><a href="http://ee93/"><strong>Take-home assignment</strong></a></li><li><a href="http://5c0d/"><strong>Interviews</strong></a></li><li><a href="http://edae/"><strong>Virtual onsite</strong></a></li><li><a href="http://6d31/"><strong>Contemplating the offer</strong></a></li></ul><h3>How I stood out — positive feedback received</h3><p>After I landed an offer, I asked my recruiter about the feedback I received from my interviewers. I stood out as a candidate for the following factors:</p><ol><li><strong>Contagious enthusiasm and excitement.</strong> My excitement, positive attitude and humility made me an enjoyable interviewee. <strong>Present your most sincere and authentic self</strong>, and you will achieve the same. As a person, I tend to be curious and enthusiastic; I let this shine through as a strength, rather than toning it down in the name of professionalism. I also tended toward issues close to my heart, identifying users such as single mothers or elderly (I’m passionate about tech for women and tech accessibility).</li><li><strong>Systematic problem-solving.</strong> During my product case interviews, I bolstered my decisions with reference to user and business goals. This demonstrated user-centricity and goal-orientation (<a href="https://weitingyp.medium.com/preparing-for-apm-recruiting-5b22ecde830a">as discussed in an earlier article on preparing for recruiting</a>).</li><li><strong>Ability to add a global perspective to the cohort: international work and living experience.</strong> As an undergraduate student who has lived in 7 different countries, I have seen a wide variety of ride sharing competitors, markets and regulations. This helped me stand out, since Uber PMs navigate global markets.</li><li><strong>Relevant but different skills and experiences. </strong>Aside from my past product management stints, I also have a considerable amount of experience in public relations. I am able to write and communicate clearly and strategically, consider things from a branding perspective, manage complex projects and negotiate priorities with stakeholders along the vertical hierarchy. This helped me stand out.</li></ol><p>These points may be specific to me and my experiences, but it took time and effort to craft a cohesive personal narrative that best represented me.</p><blockquote>Consider experiences you are proud of, string them together to show growth, and brainstorm how these will help you be a better product manager.</blockquote><p>To do this, consider experiences you are proud of, string them together to show a projection of growth, and brainstorm how these will help you be a better product manager.</p><h3>Overview of the timeline</h3><p>Uber APM applications for the 2021 intake opened on 14 August 2020. With a referral, I applied within a week. Over the next two months, I had the following recruiting tasks:</p><ul><li>Take-home assignment</li><li>45-minute product interview #1</li><li>45-minute product interview #2</li><li>Virtual (due to COVID-19) onsite: 4 x 45-minute interviews</li></ul><h3>Research and personal branding</h3><p>Before my application, I researched Uber through online resources and coffee chats. After conversing with a software engineer and two product managers, I understood that <strong>Uber’s product approach is systematic, with a heavy emphasis on data-centric decision making</strong>.</p><blockquote>By reviewing and highlighted favorable traits in the job description, I also drafted an Uber APM hiring “rubric.” This rubric served as a checklist in all opportunities to present myself, from my resume to my interview responses.</blockquote><p>By reviewing and highlighted favorable traits in the job description, I also drafted an Uber APM hiring “rubric.” This rubric served as a checklist in all opportunities to present myself, from my resume to my interview responses.</p><p>I revised my resume to include keywords and experiences that showcase these traits, and when the time came, I practiced my ‘tell me about yourself’ response to flag the same traits.</p><h3>Take-home assignment</h3><p>Each candidate had a week from when they opened the take-home assignment to submit a product proposal in the form of slides. The assignment instructions and tips (to-do’s and not to-do’s) that Uber attached were highly informative, highlighting evaluation criteria and critical components to include (MVP, prioritization, Go-To-Market plan, for example).</p><p>In completing my submission, I played to my strengths and employed the CIRCLES framework with an additional metrics prioritization step. In deciding and presenting the MVP, I stuck to paper-sketched wireframes with annotations on how the design tailored to user needs (but remains to be tested).</p><h3>Interviews</h3><p>I was invited to two 45-minute interviews, each comprised a 5-min behavioral questions component, a 35-minute product case, and a 5-min section for questions.</p><p>To prepare for the interviews, I would look up my interviewers on LinkedIn to understand commonalities, interests, and personality types. If we had any common experiences or interests, I highlighted them to build rapport (please do not fabricate anything — be sincere). Reading through their profiles also humanized my interviewers and made me feel less nervous about my interviewers.</p><p>During the interviews, I kept a bullet point checklist of the APM hiring rubric (based on my research) on hand to make sure I showcase as many relevant traits as I could. When it came to the question and answer section, I asked for advice, tips to prepare if I were to move forward, and specific questions relating to their work.</p><p>After each of the interviews, I sent a follow-up email thanking them for their time (personally, if possible, otherwise to the recruiter with a request for it to be forwarded)</p><h3>Virtual onsite</h3><p>The onsite interview day comprised four 45-minute interviews, with a 1-hour lunch break between each two. Of the four interviews, one was with an engineer, another was with a UX designer, and the remaining were with product managers.</p><p>It was the first onsite in my whole recruiting cycle — I was nervous!</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2F13Cmju3maIjStW%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2F13Cmju3maIjStW&amp;image=https%3A%2F%2Fmedia0.giphy.com%2Fmedia%2F13Cmju3maIjStW%2Fgiphy.gif%3Fcid%3D790b76117612ee941bb4ba7d94bfb6909493e1b95a39f099%26rid%3Dgiphy.gif%26ct%3Dg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="324" frameborder="0" scrolling="no"><a href="https://medium.com/media/7d01d1e3ab5fe232ecad261dddfa44b0/href">https://medium.com/media/7d01d1e3ab5fe232ecad261dddfa44b0/href</a></iframe><p>However, all my interviewers were extremely approachable and collaborative. It was a pleasure to interview with them! In this section, I’ll only recount my experiences with the engineer and the product designer.</p><h4>System design interview</h4><p>My interview with the engineer was a system design one, where I prioritized showcasing the ability to collaborate with technical teammates. As a PM candidate, you are not expected to formulate the entire response on your own. By using ‘we’ language and checking in often (‘do you feel what I have proposed thus far is reasonable?’), you engage the interviewer and show the ability to seek and integrate feedback.</p><h4>Product design (behavioral) interview</h4><p>My interview with the UX designer comprised behavioral questions. The questions prompted anecdotal or hypothetical responses about collaboration with designers. I had coffee chats with PMs and UX designers asking specifically about their PM-UX collaboration experiences and preferences to prepare for this. In my experience, treating this interview like a discussion and listening closely and curiously to your interviewer shows that you’ll be a great listener and collaborator on the job.</p><h3>Contemplating the offer</h3><p>When I received my offer, I was elated! I learned a lot about Uber and myself through the recruiting process and was confident that we were a great fit. While at this stage, I reached out to existing and previous APMs and connected with my prospective cohort-mates to confirm interest.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2F12XDYvMJNcmLgQ%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2F12XDYvMJNcmLgQ%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2F12XDYvMJNcmLgQ%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="325" frameborder="0" scrolling="no"><a href="https://medium.com/media/3036ebf1372b812578b622f81b6c5de0/href">https://medium.com/media/3036ebf1372b812578b622f81b6c5de0/href</a></iframe><p>The APM recruiting process can be hectic, especially once the school semester starts. If you are reading this article and regularly practicing your interviews, you are already a step ahead! I wish you the best of luck.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aa0476736df9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Prepare for APM recruiting — lessons and resources]]></title>
            <link>https://weitingyp.medium.com/preparing-for-apm-recruiting-5b22ecde830a?source=rss-94b2766fcfe9------2</link>
            <guid isPermaLink="false">https://medium.com/p/5b22ecde830a</guid>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[professional-development]]></category>
            <category><![CDATA[tech-recruiting]]></category>
            <category><![CDATA[tech-internship]]></category>
            <category><![CDATA[tech-interview]]></category>
            <dc:creator><![CDATA[Wei-Ting Yap]]></dc:creator>
            <pubDate>Fri, 16 Jul 2021 19:06:57 GMT</pubDate>
            <atom:updated>2021-07-25T19:49:21.682Z</atom:updated>
            <content:encoded><![CDATA[<h3>Prepare for APM recruiting — lessons and resources</h3><h4>By an incoming Uber APM</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pRhx78RLV2Mi4GgF" /><figcaption>Photo by <a href="https://unsplash.com/@kaleidico?utm_source=medium&amp;utm_medium=referral">Kaleidico</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>Part 1 of a two-part series:</strong></p><ul><li><strong>Part 1: Prepare for APM recruiting</strong></li><li><a href="https://medium.com/@weitingyp/uber-apm-recruiting-process-aa0476736df9">Part 2: How I landed an offer as an Uber APM — the recruiting process</a></li></ul><p><strong>Beyond these…</strong></p><p>Subscribe to <a href="https://breakintotech.substack.com/">Break Into Tech, a newsletter where I publish weekly actionable tips for professional development</a>.</p><p>This time last year, my time was packed with research, coffee chats, mock interviews, work, and job applications.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/390/1*nkY8lz3dxMPuL6U9dw3nKg.png" /><figcaption>Hi, it’s me! (Posing awkwardly at my graduation photoshoot, by the way.)</figcaption></figure><p>I am a first-generation, international and female-identifying college graduate. With no prior big tech internships and no mentors in the industry, I had to prepare for Associate Product Management (APM) recruiting primarily on my own. Given the fragmented information in the space, it wasn’t easy.</p><p>I have come a long way to become an incoming APM at Uber. I’m writing this article to share my experience with current aspiring PMs like you, hoping to pass on the kind mentorship of those who have helped me.</p><h3>Contents</h3><ul><li><a href="#d164"><strong>Research and network</strong></a></li><li><a href="#c0be"><strong>Personal branding</strong></a></li><li><a href="#613f"><strong>Interview preparation</strong></a></li><li><a href="#1402"><strong>During the interview</strong></a></li></ul><p>Each section ends with a list of recommended resources!</p><p>*Affiliate links are marked, if you appreciate my article and are planning to purchase the books, I’d be grateful if you did it through the link.</p><h3>Research and network</h3><p>At the very beginning of my process, I researched what product managers’ job scopes across different environments. Environmental factors to consider include company maturity, product type, team structure or collaborators, company culture.</p><p>Reading the first few chapters of Decode and Conquer and Cracking the PM Interview gave me a basic idea of the role to build on.</p><p>Then, I reached out to product managers and product collaborators across multiple teams and companies to understand their <em>PM philosophy</em> and day-to-day tasks. Reflecting on my coffee chats, I identified common language used by product managers to describe their motivations and work, and prioritized companies according to fit. The network that I built through coffee chats also became supportive and guiding mentors that advocated for me through referrals.</p><p><strong>Recommended resources</strong></p><ul><li><a href="https://www.notion.so/The-College-Product-Manager-Guide-f0b7c36363c04cc69cb7d0f1e8dc954d">How do I ask for a coffee chat?</a> comes with informational interview request templates (email, LinkedIn request) that worked for me</li><li><a href="https://www.notion.so/PM-Handbook-7a17a9126f6b4961ba1766cff4fefb2c">PM Handbook</a> to get a general overview of product management</li><li>The first few chapters of <a href="https://www.amazon.com/gp/product/0998120499/ref=as_li_tl?ie=UTF8&amp;camp=1789&amp;creative=9325&amp;creativeASIN=0998120499&amp;linkCode=as2&amp;tag=wty00-20&amp;linkId=d77c452e970f4dc0c3dd104f3c369b4c">Decode and Conquer: Answers to Product Management Interviews</a>*</li><li>The first few chapters of <a href="https://www.amazon.com/gp/product/0984782818/ref=as_li_tl?ie=UTF8&amp;camp=1789&amp;creative=9325&amp;creativeASIN=0984782818&amp;linkCode=as2&amp;tag=wty00-20&amp;linkId=d67568ac9e87eeae79717067fad5c0bc">Cracking the PM Interview: How to Land a Product Manager Job in Technology</a>*</li><li>Exponent’s company-specific i<a href="https://www.tryexponent.com/guides">nterview guides</a> talk through recruiting stages and (sometimes) hiring criteria</li></ul><h3>Personal branding</h3><p>This stage of preparation includes iterating on your resume, your personal introduction, and possibly your portfolio.</p><p><strong>It is crucial to tailor your resume to each company and job posting. </strong>Recruiters work through many resumes for these highly competitive APM roles, and mirroring the language of the job post and highlighting relevant experiences can only work in your favor.</p><p>My critical tips for these are:</p><ul><li>Write each bullet point using <a href="https://www.inc.com/bill-murphy-jr/google-recruiters-say-these-5-resume-tips-including-x-y-z-formula-will-improve-your-odds-of-getting-hired-at-google.html">Google’s X-Y-Z formula</a>: “accomplished X, as measured by Y, by doing Z.” This works great because it shows results-orientation. Ensure you quantify each bullet point.</li><li>Eliminate ruthlessly. There will be experiences you’re incredibly proud of but aren’t relevant — delete them. Recruiters only spend about 30 seconds per resume, so make it succinct.</li><li>Bold key experiences that mirror the language of the job requirements.</li><li>Iterate, iterate, and iterate. I worked through my resume 4–5 times, gathering feedback from my aspiring PM peers and professional contacts.</li></ul><p>I also referenced the resumes of existing PMs, if available, through their personal websites.</p><p><strong>Recommended resource</strong></p><ul><li>CodePath’s <a href="https://books.codepath.org/student-handbook/internship-search/student-resume-guide">Student Resume Guide</a> contains a comprehensive list of tips, and discusses how to present online courses and projects</li></ul><h3>Interview preparation</h3><p>The product management interview tests for four key competencies using corresponding question types. Behavioral questions test for culture fit and leadership skills; product design and strategy questions judge user-centricity and problem-solving; analytical questions aim to evaluate data-driven reasoning; and technical and system design questions assess technical competency.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/919/1*1WcnVm_M3BhK6uOxt-MQwQ.png" /><figcaption>Question types in a PM interview and the traits they evaluate</figcaption></figure><h4>Behavioral questions</h4><p>Behavioral questions judge culture fit, communication skills, and leadership skills.</p><blockquote>When you’re with any interviewer, the first consideration that comes to mind is: would I enjoy working with this person?</blockquote><p>When you’re with any interviewer, the first consideration that comes to mind is: would I enjoy working with this person? Present yourself as humble, sincere, teachable, and relatable to ace these.</p><ol><li><strong>Tell me about yourself: a 2-minute personal narrative</strong></li></ol><p>One often overlooked question at interviews is <strong>‘</strong>Tell me about yourself.<strong>’ </strong>The challenge is to pitch yourself as a favorable teammate and candidate without boring the interviewer. <strong>Keep it short — 2.5 minutes at most — and highlight only the most relevant experiences in a compelling narrative</strong>.</p><p>My suggested structure:</p><ul><li>Okay, first of all, I’d like to thank you for taking the time to speak with me today. I’m excited about our conversation ahead.</li><li>I’m [name] and I am a [rising senior/etc.] at [school]. I major in [your major].</li><li>I’m passionate about [your interests and passions].</li><li>Over the past few years, I’ve gained experience across [functions/company stages/product types/industry].</li><li>2–4 key experiences: I started as a [role] at [company (short description if needed)], where I achieved [outcome] as measured by [metric].</li><li>In my free time, I like to [1–2 hobbies].</li></ul><p>The most valuable resources I used to fine-tune my ‘Tell me about yourself’ response are my resume and my mock interview partners.</p><p><strong>2. Tell me about a time where…: building a story toolbox</strong></p><p>After highlighting the company or role’s criteria to 3–5 traits, build a story toolbox. A story toolbox comprises 5–8 stories which you can repurpose for any behavioral question. Note the traits (strengths and weaknesses) that each anecdote showcases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*eos0enkiDztcvuyEKbRMLQ.png" /><figcaption>Situation-Action-Results structure for behavioral questions</figcaption></figure><p>I use the <strong>SAR (Situation-Action-Results) structure</strong> to answer these questions and aim to answer <strong>in under 4 minutes</strong>.</p><p>A mistake I made at the start was over-focusing on describing situations in which I made mistakes. <strong>Your focus should be on the action component </strong>— interviewers want to see how you respond to situations.</p><p>Finally, be sure to <strong>strategically use keywords to flag the traits</strong> they are looking for in the candidate.</p><p><strong>Recommended resources</strong></p><ol><li>The last 3 minutes of <a href="https://youtu.be/IV30jAw7dxA?t=256">How to Answer Behavioral Interview Questions Sample Answers</a> helped me understand how much detail is (un)necessary in an answer, to stay concise</li><li><a href="https://www.youtube.com/watch?v=xj-8YmjkOuk&amp;t=0s">The Powerful Formula to Answer the Most Common Interview Questions (Story Toolbox)</a> talks through building a story toolbox</li></ol><h4>Product design/strategy questions</h4><p>Product design and strategy questions aim to judge your product sense, which entails user-centricity and problem-solving. These questions include ‘Design a complementary product to Amazon’s Twitch’ or ‘How would you improve Uber’s food pick-up feature?’</p><blockquote>The key to doing well here is to explicit figure out the user and/or business goals at the start of the response, and to religiously use it in your reasoning.</blockquote><p>The key to doing well here is to explicit figure out the user and/or business goals at the start of the response, and to religiously use it in your reasoning. Through this, you show user-centricity and goal orientation.</p><p>How I prepared:</p><ol><li>Speed through Decode and Conquer and Cracking the PM Interview to understand the gist of product questions and responses</li><li>Watched 1–2 recorded simulated PM interviews to see how it’s done</li><li>Practice, practice and practice. I did about 50 interviews, through which I:<br>a) honed user-centricity and problem solving skills, by always <strong>relating back to the goals</strong>, <br>b) aimed for clarity and concision (<strong>under 30 minutes</strong> leaves time for follow-up questions), and <br>c) eliminated any bad speaking habits (‘um’, repeating ‘following which’ as a transition, etc.).</li><li>Once I was confident, I stayed on top of my game by writing down bullet point responses to a product question each day, and still did mock interviews once a week.</li></ol><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FQJEBUOX4r32sX2A2lN%2Fvideo&amp;display_name=GIPHY&amp;url=https%3A%2F%2Fgiphy.com%2Fclips%2Fonepeloton-peloton-emma-lovewell-QJEBUOX4r32sX2A2lN&amp;image=https%3A%2F%2Fmedia0.giphy.com%2Fmedia%2FQJEBUOX4r32sX2A2lN%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="480" height="270" frameborder="0" scrolling="no"><a href="https://medium.com/media/c345f8d01c853c0c349669e749959d82/href">https://medium.com/media/c345f8d01c853c0c349669e749959d82/href</a></iframe><p><strong>Recommended resources</strong></p><ul><li>The <strong>CIRCLES framework</strong> and example responses in <a href="https://www.amazon.com/gp/product/0998120499/ref=as_li_tl?ie=UTF8&amp;camp=1789&amp;creative=9325&amp;creativeASIN=0998120499&amp;linkCode=as2&amp;tag=wty00-20&amp;linkId=d77c452e970f4dc0c3dd104f3c369b4c">Decode and Conquer: Answers to Product Management Interviews</a>*</li><li>The example responses in <a href="https://www.amazon.com/gp/product/0984782818/ref=as_li_tl?ie=UTF8&amp;camp=1789&amp;creative=9325&amp;creativeASIN=0984782818&amp;linkCode=as2&amp;tag=wty00-20&amp;linkId=d67568ac9e87eeae79717067fad5c0bc">Cracking the PM Interview: How to Land a Product Manager Job in Technology</a>*</li><li><a href="https://dailyproductprep.com/">Daily Product Prep</a>, which sends you a PM question a day for practice, and (if you subscribe to Pro Monthly, which I did), the solution the next day. I enjoy their variety of questions across companies and industries. Their model answers also suggest practical frameworks (for example, TROPIC for metric investigation questions).</li><li>Watching actual PMs answer product questions on <a href="https://www.youtube.com/channel/UCjm_qVkCPjOVDz9BWjNqO9A">Exponent YouTube channel</a></li><li><a href="https://www.pramp.com/#/">Pramp</a>, which matches you with other people for mock interviews</li><li>Any communities with mock interview calendars and sheets (Ladder, Product Buds, etc.)</li></ul><h4>Analytical questions</h4><p>Analytical questions pertain specifically to how you would measure and evaluate product success. There are a whole host of metrics you can choose for any product or feature, depending on the user and business goals. Common questions include ‘How would you evaluate the success of xxx feature?’ and ‘What would you choose as the north star metric and why?’.</p><p>To prepare for this, I read the Medium article listed below and practiced metrics questions in roughly 1 out of 5 of my mock interviews.</p><p><strong>Recommended resource</strong></p><ul><li><a href="https://medium.com/hackernoon/metrics-game-framework-5e3dce1be8ac">GAME framework (Goal — Action — Metric — Evaluation)</a>, a Medium article which was a game changer for me. It presents a clear structure for choosing a metric and understanding the trade-offs between the metrics of choice.</li></ul><h4>Technical/system design questions</h4><p>Lastly, technical and system design questions are asked to judge technical competency, which is crucial to empathizing, communicating, and collaborating with developers.</p><p>Technical questions tend to be very high level, such as: ‘How does an API work?’ and ‘How does the browser display a webpage?’. System design questions relate to system design, with questions such as ‘How does Google search work?’ and ‘How would you design the system for feature XYZ?’.</p><p>When I found out I had to prepare for these, I crammed the following resources in a short 36 hours and practiced with two mock interviews. As I was short on time, I wrote bullet points responses to about five questions.</p><p><strong>Recommended resources</strong></p><ol><li>This <a href="https://github.com/donnemartin/system-design-primer">open-source System Design primer</a> which is comprehensive enough or;</li><li><a href="https://www.educative.io/courses/grokking-the-system-design-interview?aff=BPQn">Grokking the System Design Interview</a>* (course) which is also comprehensive, but skips on extraneous details (which you do not have time to delve into during the interview) — this is what I used because I didn’t know of the open source one above.</li><li>Gaurav Sen’s <a href="https://www.youtube.com/playlist?list=PLMCXHnjXnTnvo6alSjVkgxV-VH6EPyvoX">System Design playlist</a>, where you can watch him respond to various system design questions (with white-boarding)</li></ol><p>Given that system design questions tend to be asked only later in the process, I would de-prioritize system design preparation.</p><h3>During the interview</h3><p>Congratulations! You have been invited to an interview with a company.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2Fely3apij36BJhoZ234%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2Fgood-job-congratulations-otter-ely3apij36BJhoZ234&amp;image=https%3A%2F%2Fmedia4.giphy.com%2Fmedia%2Fely3apij36BJhoZ234%2Fgiphy-downsized-large.gif%3Fcid%3D790b76116b161872a7eb66d48d4365d653622f68958fdeb2%26rid%3Dgiphy-downsized-large.gif%26ct%3Dg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="435" frameborder="0" scrolling="no"><a href="https://medium.com/media/29df1e369241dcf559ec47af5900db9a/href">https://medium.com/media/29df1e369241dcf559ec47af5900db9a/href</a></iframe><blockquote><strong>Before each interview, I would distill my research down into a rough hiring rubric or criteria. The rubric should be specific to what it means to be a product manager at the specific company. </strong>Throughout the interview, I highlighted the traits or experiences the company mentioned in the job description or otherwise.</blockquote><p>Before each interview, I would distill my research down into a rough hiring rubric or criteria. The rubric should be specific to what it means to be a product manager at the specific company. Throughout the interview, I highlighted the traits or experiences the company mentioned in the job description or otherwise.</p><p>For example, during one of my product design cases, I knew I was running out of time, but I wanted to show that I could evaluate trade-offs between features. I explained to my interviewer that I wanted to showcase this but was stuck because I over-narrowed the scope of the design task. My interviewer helped reframe the question so that I can demonstrate the skill.</p><blockquote><strong>When unconfident or running out of time, tell the interviewer that you are stuck or are mindful of the time and suggest an area of focus that works to your advantage.</strong></blockquote><p>When unconfident or running out of time, tell the interviewer that you are stuck or are mindful of the time and suggest an area of focus.</p><p>During my system design interview, I was unconfident about finishing a thorough response within 45-minutes. My suggestion to my interviewer: I want to be mindful of time. Can I propose focusing only on designing the APIs and scoping out the necessary system resources for this feature? He accepted, and I was able to limit the scope of the response to something more manageable.</p><h3>Final resources</h3><ul><li><a href="https://weiting109.notion.site/Breaking-into-PM-Resources-48d1edd89bfd43fcb100c3c114e6c489">Breaking into PM resources</a></li><li><a href="https://weiting109.notion.site/PM-communities-8979ba19f61b4ab492c3dc7075b31286">List of PM communities</a></li></ul><p>Thank you for reading thus far, and I hope that this article was helpful for you! If it was, please let me know. I am wishing you all the best!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5b22ecde830a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pre-processing should extract context-specific features.]]></title>
            <link>https://medium.com/data-science/pre-processing-should-extract-context-specific-features-4d01f6669a7e?source=rss-94b2766fcfe9------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d01f6669a7e</guid>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[getting-started]]></category>
            <category><![CDATA[spacy]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Wei-Ting Yap]]></dc:creator>
            <pubDate>Tue, 15 Dec 2020 23:55:25 GMT</pubDate>
            <atom:updated>2020-12-23T15:24:02.333Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/getting-started">Getting Started</a></h4><h3>Context-specific Pre-processing for NLP with spaCy: Tweets</h3><h4>Pre-processing should extract context-specific features. With tweets, these include mentions, hashtags and URLs.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SUeRKgtD9b1-ZhdI" /><figcaption>Photo by <a href="https://unsplash.com/@visuals?utm_source=medium&amp;utm_medium=referral">visuals</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Natural Language Processing is a field of machine learning concerned with understanding human language. As opposed to numerical data, NLP works primarily with text. Exploring and pre-processing text data requires different techniques and libraries, and this tutorial demonstrates the basics.</p><p>However, pre-processing is not an algorithmic procedure. With data science tasks, often <strong>the context of the data determines what aspects of the data are valuable, and what aspects are irrelevant or unreliable</strong>. In this tutorial, we explore text pre-processing in the context of tweets, or more generally, social media.</p><p>Kaggle’s 9-year-old Getting Started <a href="https://www.kaggle.com/c/nlp-getting-started">Real or Not? NLP with Disaster Tweets</a> competition presents a reasonably-sized dataset (around 7500 tweets in the training set) for practice. The challenge is to classify tweets, given their text, keyword and location, into whether they are really about disasters or not.</p><p>The code for this tutorial can be followed at this <a href="https://github.com/weiting109/disaster-tweets-classifier/blob/main/nb.ipynb">notebook and repository</a>.</p><p>Before we get started, download the nlp-getting-started data from Kaggle. In my project directory, I put train.csv, test.csv, and sample_submission.csv under a data subdirectory.</p><h3>Data Exploration</h3><p>Let’s start by importing typical and useful data science libraries and creating a dataframe out of train.csv. I won’t delve into the details of libraries that are not NLP-specific.</p><pre>import pandas as pd<br>import numpy as np<br>import matplotlib.pyplot as plt</pre><pre>data = pd.read_csv(&#39;data/train.csv&#39;, index_col=&#39;id&#39;)<br>data.head()</pre><pre>keyword	location	text	target<br>id				<br>1	NaN	NaN	Our Deeds are the Reason of this #earthquake M...	1<br>4	NaN	NaN	Forest fire near La Ronge Sask. Canada	1<br>5	NaN	NaN	All residents asked to &#39;shelter in place&#39; are ...	1<br>6	NaN	NaN	13,000 people receive #wildfires evacuation or...	1<br>7	NaN	NaN	Just got sent this photo from Ruby #Alaska as ...</pre><p>Our data comprises 4 columns, keyword, location, text and target. Quoting the <a href="https://www.kaggle.com/c/nlp-getting-started/data">data description on Kaggle</a>:</p><ul><li>id - a unique identifier for each tweet</li><li>text - the text of the tweet</li><li>location - the location the tweet was sent from (may be blank)</li><li>keyword - a particular keyword from the tweet (may be blank)</li><li>target - in <strong>train.csv</strong> only, this denotes whether a tweet is about a real disaster (1) or not (0)</li></ul><p>To ensure integrity with the number of rows and columns in our dataset, as well as make judgements on the generalizability of our training set, let’s understand the size of our training data.</p><pre>data.shape</pre><pre>(7613, 4)</pre><p>Examining closer, we find there are 52 duplicate rows (different id’s, but same keyword, location, text and target.</p><pre>np.sum(data.duplicated()) # 52 duplicates</pre><p>So let’s drop the duplicate rows. The index (set as id) remains intact. After removing duplicate rows, we are left with 7561 tweets (integrity check, as mentioned earlier), a manageable amount for this tutorial.</p><p>However, 7561 data points is still relatively little for NLP, especially if we are using deep learning models. Given that there are probably close to a million tweets each day, I doubt a model trained on just 7561 data points is generalizable enough.</p><pre># let&#39;s drop the duplicates! <br>df = data<br>df = data.drop_duplicates()<br>df.shape</pre><p>Our output:</p><pre>(7561, 4)</pre><p>Aside from training size, the balance of classes (target) in training set is also important. A training set with all target value = 0 will leave the model classifying every tweet as not about a disaster. The vice versa situation is also true. Ideally, the training set should have all classes represented equally.</p><p>We can use panda’s dataframe value_counts() method to count the number of rows for each class. With 4322 tweets not about disasters (target=0) and 3239 tweets about disasters (target=1), we have a 4:3 class balance. That&#39;s not perfectly balanced, but it&#39;s not disastrously imbalanced.</p><pre># target 1 refers to disaster tweet, 0 is not a disaster tweet<br>df[&#39;target&#39;].value_counts()</pre><p>Our output:</p><pre>0    4322<br>1    3239<br>Name: target, dtype: int64</pre><p>Let’s also look into the completeness of data. Using we can sum up the series returned by panda’s dataframe isna() method to count the number of na entries for each column.</p><pre># checking for completeness of data<br>print(f&quot;{np.sum(df[&#39;keyword&#39;].isna())} rows have no keywords&quot;)<br>print(f&quot;{np.sum(df[&#39;location&#39;].isna())} rows have no location&quot;)<br>print(f&quot;{np.sum(df[&#39;text&#39;].isna())} rows have no text&quot;)<br>print(f&quot;{np.sum(df[&#39;text&#39;].isna())} rows have no target&quot;)</pre><p>Our output:</p><pre>61 rows have no keywords <br>2533 rows have no location <br>0 rows have no text <br>0 rows have no target</pre><p>Ideally, we would further characterize and explore the data by analyzing the word lengths, sentence lengths, word frequencies and more. While that’s out of scope for this tutorial, you can learn more about it <a href="https://neptune.ai/blog/exploratory-data-analysis-natural-language-processing-tools">here</a>.</p><h3>NLP Pre-processing concepts</h3><p>Now that we have explored the data, let’s pre-process the tweets and represent them in a form our models can take as input.</p><p>The most common numerical representation for texts is the bag-of-words representation.</p><h4>Bag of words</h4><p>Bag of words is a way to represent text data numerically. Text data is essentially split into words (or more accurately, tokens), which are features. The frequency of each word in each text data is are the corresponding feature values. For example, we might represent &quot;I love cake very very much&quot; as a bag of words dictionary:</p><pre>{<br>&#39;I&#39;:1,<br>&#39;love&#39;:1,<br>&#39;cake&#39;:1,<br>&#39;very&#39;:2,<br>&#39;much&#39;:1<br>}</pre><h4>Tokenization</h4><p>Tokenization breaks text data up into its tokens at the level of NLP (word, phrase, sentence). The lowest (and most common) is words, which fits perfectly with our bag of words representation. However, these tokens could also include punctuation, stop words, and other custom tokens. We’ll consider these in the context of tweets and the challenge in the next session.</p><h4>Stemming</h4><p>Stemming refers to truncating words of their affixes (prefixes or suffixes) to approximate them to their root form. This is often done with a lookup dictionary of prefixes and suffixes, making it computationally fast.</p><p>However, there is a performance trade-off. In the English language, some affixes change the meaning of the word completely, resulting in inaccurate feature representation.</p><h4>Lemmatization</h4><p>The alternative to stemming is lemmatization, where words are reduced to their lemmas, or root form. This is done using a lookup dictionary of words and their lemmas, hence resulting in it being more computationally expensive. However, performance is often better, since features are represented more accurately.</p><p>Given the relatively smaller size of our dataset, we will use lemmatization.</p><h3>In the context of tweets</h3><p>Getting from tweets to their bag of words representation is less straightforward. What about:</p><ul><li>words of different cases <em>e.g. cake vs Cake,</em></li><li>punctuation,</li><li>stopwords,</li><li>numbers,</li><li>mentions,</li><li>hashtags, and</li><li>URLs?</li></ul><p>When deciding what we want to do with these elements, we have to consider the context of the data and reconcile that with the challenge.</p><h4>Words of different cases — standardize to lower-case</h4><p>In internet lingo, different cases could communicate different sentiments (<em>e.g. danger vs DANGER!)</em> or different parts of speech (<em>e.g. start of the sentence, or a pronoun like The Fight Club</em>). By changing all tokens to upper- or lower- case, we could be losing data helpful for classification.</p><p>However, since we have a small dataset (7500 tweets), there is unlikely to be sufficient data of each upper-/lower-case variant, let’s go with lower-case.</p><h4>Punctuation</h4><p>Tweets are undoubtedly going to contain punctuations, which can convey different sentiments or emotions too. Consider, in internet lingo, the difference between:</p><ul><li><em>Help needed?</em></li><li><em>Help needed!</em></li></ul><p>We’ll consider punctuations each as their own tokens, with special cases like ‘…’ being a separate token from ‘.’. So we don’t lose data, we can disregard them (and even tune which punctuations to ignore) when tuning our hyper-parameters.</p><h4>Stop words</h4><p>Stop words are essentially words so common that they have little significant contribution to the meaning of the text. These words include articles (the, a, that) and other commonly used words (what, how, many).</p><p>Stop word tokens are often ignored in NLP processing. Plus, the character limit of a tweet (280 characters) often results in grammatically incorrect tweets, where articles are missed.</p><p>However, rather than ignore stop words from the start, let’s disregard them (and even tune which stop words to ignore) when <a href="https://machinelearningmastery.com/hyperparameters-for-classification-machine-learning-algorithms/">tuning our hyper-parameters</a> so we don’t lose data.</p><h4>Numbers</h4><p>Numbers in tweets can convey the quantity of literal objects, but can also convey the scale of something (e.g. <em>7.9 Richter scale earthquake</em>) or the year (e.g. <em>2005 Hurricane Katrina</em>).</p><p>In the latter two cases, such numerical information may be valuable depending on the level of NLP we choose to do later down the road (word-level vs phrase- or sentence- level), or if we want to filter tweets about historical disasters vs current disasters.</p><p>As such, we will retain numbers as tokens, with the option of ignoring them (or even only counting numbers that are years) when tuning our hyper-parameters.</p><h4>Mentions</h4><p>On Twitter, mentions allows users to address each other through a tweet. While mentions between personal accounts may be less significant, mentions to authorities to alert them of disasters (consider @<em>policeauthorities, gun shooting down brick lane right now!</em>).</p><p>Let’s tokenize mentions along with their usernames, but also count the number of mentions, which could convey a conversation.</p><h4>Hashtags</h4><p>Hashtags on Twitter allow users to discover content related to a specific theme or topic. When it comes to natural disasters, hashtags like <em>#prayforCountryX</em> and <em>#RIPxyzShootings</em> can differentiate tweets about disasters from everyday tweets.</p><p>As such, let’s tokenize hashtags with their content, but also count the number of hashtags. The number of hashtags could flag sensationalized social media marketing tweets (e.g. <em>This beat drop is the bomb! #edm #music #dubstep #newrelease</em>) that use disaster keywords.</p><h4>URLs</h4><p>Disaster tweets could include URLs to news articles, relief efforts, or images. However, the same can be said of everyday tweets. Since we’re unsure if disaster tweets are more likely to have URLs or a certain type of URL, let’s keep URLs as tokens and the number of URLs as a feature.</p><p>This challenge’s dataset features tweets with Twitter-shortened URLs (<a href="http://t.co/"><em>http://t.co</em></a>), but more current tweet data could include the domains, which can then be extracted (I imagine the red cross domain would be highly correlated with disaster tweets). For more complicated algorithms, one can also consider visiting the shortened URL and scraping web page elements.</p><h3>NLP with the spaCy library</h3><p><a href="https://spacy.io/">spaCy</a> is an open-source python library for natural language processing. It integrates well with the rest of the python machine learning libraries (scikit-learn, TensorFlow, PyTorch) and more, and uses a <strong>object-oriented approach</strong> to keep its interface readable and easy to use. Notably, it its model returns Document type data, which consists of tokens with various useful annotations (e.g. its lemma, whether it&#39;s a stopword) as attributes.</p><p>Let’s import spaCy, download the model for American English, and load the relevant spaCy model.</p><pre># download spaCy model for American English<br>!python3 -m spacy download en_core_web_sm</pre><pre>import spacy <br>import en_core_web_sm<br>nlp = en_core_web_sm.load()</pre><h4>How well does out-of-the-box spaCy do with tweets?</h4><p>Before we customize spaCy, we can take a look at how the out-of-the-box model tokenizes tweets with its default rules. I created a tweet that included a number, a contraction, a hashtag, a mention and a link.</p><p>As shown below, out-of-the-box spaCy already breaks up contractions and gives us the relevant lemmas. It also recognizes numbers, mentions and URLs as their own tokens according to the default rules. That leaves us with hashtags, which are split into a ‘#’ punctuation and hashtag content, instead of it staying as a whole token.</p><pre># Let&#39;s see what spaCy does with numbers, contractions, #hashtags, @mentions and URLs<br>s = &quot;2020 can&#39;t get any worse #ihate2020 @bestfriend &lt;https://t.co&gt;&quot;<br>doc = nlp(s)</pre><pre># Let&#39;s look at the lemmas and is stopword of each token<br>print(f&quot;Token\\t\\tLemma\\t\\tStopword&quot;)<br>print(&quot;=&quot;*40)<br>for token in doc:<br>    print(f&quot;{token}\\t\\t{token.lemma_}\\t\\t{token.is_stop}&quot;</pre><p>This prints:</p><pre>Token		Lemma		Stopword<br>========================================<br>2020		2020		False<br>ca		can		True<br>n&#39;t		not		True<br>get		get		True<br>any		any		True<br>worse		bad		False<br>#		#		False<br>ihate2020		ihate2020		False<br>@bestfriend		@bestfriend		False<br>&lt;https://t.co&gt;		&lt;https://t.co&gt;		False</pre><h4>Modifying spaCy</h4><p>We can modify spaCy’s model to recognize hashtags as entire tokens.</p><p>spaCy’s tokenizer can be modified (you can also build a custom tokenizer if you want!) by redefining its default rules. spaCy’s tokenizer prioritizes rules in the following order: token match pattern, prefix, suffix, infixes, URL, special cases (see <a href="https://spacy.io/usage/linguistic-features#how-tokenizer-works">How spaCy’s Tokenizer Works</a>).</p><p>In our case, we’ll modify the tokenizer’s pattern matching regex pattern (read more about regex here: <a href="https://towardsdatascience.com/a-simple-intro-to-regex-with-python-14d23a34d170#:~:text=Regular%20expressions%20are%20used%20to,projects%20that%20involve%20text%20analytics.">A simple intro to Regex with Python</a>) by appending &#39;#\\w+&#39;, which is a pattern for the hash symbol and a word.</p><pre># We want to also keep #hashtags as a token, so we will modify the spaCy model&#39;s token_match</pre><pre>import re</pre><pre># Retrieve the default token-matching regex pattern<br>re_token_match = spacy.tokenizer._get_regex_pattern(nlp.Defaults.token_match)</pre><pre># Add #hashtag pattern<br>re_token_match = f&quot;({re_token_match}|#\\w+)&quot;<br>nlp.tokenizer.token_match = re.compile(re_token_match).match</pre><pre># Now let&#39;s try again<br>s = &quot;2020 can&#39;t get any worse #ihate2020 @bestfriend &lt;https://t.co&gt;&quot;<br>doc = nlp(s)</pre><pre># Let&#39;s look at the lemmas and is stopword of each token<br>print(f&quot;Token\\t\\tLemma\\t\\tStopword&quot;)<br>print(&quot;=&quot;*40)<br>for token in doc:<br>    print(f&quot;{token}\\t\\t{token.lemma_}\\t\\t{token.is_stop}&quot;)</pre><p>Our code prints:</p><pre>Token		Lemma		Stopword<br>========================================<br>2020		2020		False<br>ca		can		True<br>n&#39;t		not		True<br>get		get		True<br>any		any		True<br>worse		bad		False<br>#ihate2020		#ihate2020		False<br>@bestfriend		@bestfriend		False<br>&lt;https://t.co&gt;		&lt;https://t.co&gt;		False</pre><h4>Pre-processing algorithm</h4><p>We can then proceed to create a preprocessing algorithm, and put it in a function so it can be called on every tweet in the training set. In the following preprocess function, each tweet:</p><ul><li>Is changed to lower case</li><li>Is tokenized with our modified spaCy model</li><li>Has its lemmas of tokens set unioned with our features set</li><li>Has its bag-of-words representation constructed in a dictionary freq</li><li>Has its hashtags, mentions and URLs counted</li></ul><pre># Create a pre-process function for each tweet<br>def preprocess(s, nlp, features):<br>    &quot;&quot;&quot;<br>    Given string s, spaCy model nlp, and set features (lemmas encountered),<br>    pre-process s and return updated features and bag-of-words representation dict freq<br>    - changes s to lower-case<br>    - tokenize s using nlp to create a doc<br>    - update features with lemmas encountered in s<br>    - create bag-of-words representation in dict type freq, including counts for hashtags, mentions and URLs<br>    &quot;&quot;&quot;</pre><pre># To lowercase<br>    s = s.lower()</pre><pre># Creating a doc with spaCy<br>    doc = nlp(s)<br></pre><pre>    lemmas = []<br>    for token in doc:<br>        lemmas.append(token.lemma_)</pre><pre># Union between lemmas and our features set<br>    features |= set(lemmas)</pre><pre># Constructing a bag of words for the tweet<br>    freq = {&#39;#&#39;:0,&#39;@&#39;:0,&#39;URL&#39;:0}<br>    for word in lemmas:<br>        freq[str(word)] = 0<br>    for token in doc: <br>        if &#39;#&#39; in str(token): freq[&#39;#&#39;] += 1 # Count number of hashtags, regardless of hashtag<br>        if &#39;@&#39; in str(token): freq[&#39;@&#39;] += 1 # Count number of mentions, regardless of mention<br>        if &#39;http://&#39; in str(token): freq[&#39;URL&#39;] += 1 # Count number of URLs, regardless of URL<br>        freq[str(token.lemma_)] += 1<br>        <br>    return features, freq</pre><h4>Setting it up</h4><p>We’ll create a copy our de-duplicated data as a best practice, so that any pre-processing changes does not affect the original state of our training data. Then, we will initialize a python set features, which will contain all features of each tweet. In addition to all lemmas encountered via tokenization of each tweet, features will include number of hashtags (#), number of mentions (@), and number of URLs (URL).</p><pre>preprocess_df = df # Duplicate for preprocessing<br>features = set({&#39;#&#39;,&#39;@&#39;,&#39;URL&#39;}) # Using set feature to contain all words (lemmas) seen</pre><p>Using our preprocess function, we&#39;ll preprocess every tweet, updating features each time with new lemmas seen. With each tweet, the tweet&#39;s bag of words representation freq is also appended to an array of bag of words for each tweet (bow_array).</p><pre># Array bow_array of bow representations for each tweet;<br># bow_array[i] is the bow representation for tweet id (i+1)<br>bow_array = [] <br>for i in range(len(preprocess_df)):<br>    features, freq = preprocess(preprocess_df.iloc[i][&#39;text&#39;],nlp,features)<br>    bow_array.append(freq)<br>len(bow_array) # 7561</pre><p>With all lemmas encountered across all tweets collected in features, we can create a dataframe bow to represent the features of all the tweets.</p><pre># Create dataframe for bag of words representation for each tweet<br>bow = pd.DataFrame(&#39;0&#39;, columns=features,index=range(len(preprocess_df)))<br>bow[&#39;id&#39;]=preprocess_df.index<br>bow.set_index(&#39;id&#39;,drop=True,inplace=True)</pre><p>Now, let’s update our dataframe with the feature values of each tweet.</p><pre># Update bow[i] with bag-of-words freq of the tweet id (i+1)<br>for i in range(len(preprocess_df)):<br>    freq = bow_array[i]<br>    for f in freq:<br>        bow.loc[i+1,f]=freq[f]</pre><p>Finally, we will join our training data dataframe with our bag-of-words. pandas Dataframe’s join method allows us to add columns from one dataframe to another for rows where the index matches. Note that we append &#39;_data&#39; as a suffix to all columns from the &#39;left&#39; dataframe, which is preprocess_df. This is to avoid conflicts between the keyword given as part of the training data, and &#39;keyword&#39; as a lemma-token-feature. Remember to save the preprocessed .csv file for easier next steps!</p><pre># Join bag-of-words representation to train dataframe<br># Append _data suffix to &#39;keyword&#39;,&#39;location&#39;,&#39;text&#39;,&#39;target&#39; for features that are not lemma tokens<br>preprocess_df = preprocess_df.join(bow,lsuffix=&#39;_data&#39;)</pre><pre># Saving bag-of-words representation for collaborators<br>preprocess_df.to_csv(&quot;data/train_preprocessed.csv&quot;,index=True,index_label=&#39;id&#39;)</pre><h3>Splitting into training and validation data</h3><p>Now that we’ve pre-processed our data, there’s just one last step before we can jump into using it to train our model of choice. We have split it, stratified according to the distribution of classes, into training and validation sets. Using train_test_split from sklearn.model_selection:</p><pre>from sklearn.model_selection import train_test_split</pre><pre># stratify=y creates a balanced validation set<br>y = preprocess_df[&#39;target_data&#39;]</pre><pre>df_train, df_val = train_test_split(preprocess_df, test_size=0.10, random_state=101, stratify=y)</pre><pre># Saving csv files for collaborators<br>df_train.to_csv(&quot;data/train_preprocessed_split.csv&quot;,index=True)<br>df_val.to_csv(&quot;data/val_preprocessed_split.csv&quot;,index=True)</pre><pre>print(df_train.shape, df_val.shape)</pre><pre>(6851, 21330) (762, 21330)</pre><p>Just to be sure, we can check our balance.</p><pre># Checking balance<br>print(f&quot;&quot;&quot;<br>Ratio of target=1 to target=0 tweets in:\\n <br>Original data set = {np.sum(preprocess_df[&#39;target_data&#39;]==1)/np.sum(preprocess_df[&#39;target_data&#39;]==0)},</pre><pre>\\n<br>Training data set = {np.sum(df_train[&#39;target_data&#39;]==1)/np.sum(df_train[&#39;target_data&#39;]==0)},</pre><pre>\\n<br>Validation data set = {np.sum(df_val[&#39;target_data&#39;]==1)/np.sum(df_val[&#39;target_data&#39;]==0)}&quot;&quot;&quot;)</pre><p>This prints:</p><pre>Ratio of target=1 to target=0 tweets in:<br> <br>Original data set = 0.7533394748963611,</pre><pre>Training data set = 0.7535193242897363,</pre><pre>Validation data set = 0.7517241379310344</pre><h3>Ready, set, go!</h3><h4>Pre-processing vs Tuning hyperparameters</h4><p>If you have seen other NLP pre-processing tutorials, you’ll find that a lot of their steps have been included as considerations, but not implemented here. These include removing punctuations, numbers, and stop words. However, our training dataset is small, and these steps could remove information valuable in the context of tweets and the challenge. Hence, rather than eliminate these data at the pre-processing stage, we have left them as possible ways to <a href="https://machinelearningmastery.com/hyperparameters-for-classification-machine-learning-algorithms/">tune the hyper-parameters</a> of our model.</p><h4>Possible extensions</h4><p>Through this tutorial, we have pre-processed tweets into their bag-of-words representation. However, you may choose to go a few steps further with <a href="https://towardsdatascience.com/tf-idf-for-document-ranking-from-scratch-in-python-on-real-world-dataset-796d339a4089">Term Frequency — Inverse Document Frequency (TFIDF)</a>. TF-IDF stands for “Term,Information Retrieval and Text Mining, which penalizes terms that appear too often (as they become less discriminatory as features), or <a href="https://www.kaggle.com/matleonard/word-vectors">word vectors</a>, which also numerically account for the word’s context and semantics. Word vectors encoding will result in better performance than TFIDF encoding, which will result in better performance than bag-of-words encoding.</p><p>We have ignored location and keyword in this tutorial, focusing entirely on tweets. You could consider encoding location by similarity, accounting for different spellings of the same place (<em>e.g. USA vs U.S.</em>), and missing values. You can also weight keywords heavier and see how that affects the performance of your model.</p><p>Lastly, there may be valuable information in the URLs that we are missing out. Given that they are in shortened form, we are unable to extract the domain name or page content from the text data alone. You could consider building an algorithm to visit the site and extract the domain name, as well as scrape relevant elements on the page (<em>e.g. page title</em>).</p><h4>Next steps</h4><p>Now that we have performed basic pre-processing our dataset, we can move forward in two possible directions. You can either continue on to <a href="https://emma-stiefel.medium.com/standardizing-spelling-and-locations-with-python-pyspellchecker-and-mordecai-8b0fa7d44feb">advanced pre-processing by performing spell checks and correction with Pyspellchecker and Mordecai</a>, or try out and evaluate candidate machine learning models! Possible models for such classification problems include logistic regression, neural networks, and SVMs.</p><h3>References</h3><p>[1] Kaggle, <a href="https://www.kaggle.com/c/nlp-getting-started">Disaster tweets classification challenge on Kaggle</a> (2020), Kaggle</p><p>[2] D. Becker and M. Leonard, <a href="https://www.kaggle.com/matleonard/intro-to-nlp">Intro to NLP</a> (n.d.), Natural Language Processing Course on Kaggle</p><p>[3] D. Becker and M. Leonard, <a href="https://www.kaggle.com/matleonard/text-classification">Text Classification with SpaCy </a>(n.d.), Natural Language Processing Course on Kaggle</p><p>[4] Yse, D. L. <a href="https://towardsdatascience.com/your-guide-to-natural-language-processing-nlp-48ea2511f6e1">Your Guide to Natural Language Processing</a> (2019), Towards Data Science</p><p>[5] Explosion AI, <a href="https://spacy.io/usage/spacy-101">spaCy’s 101: Everything you need to know</a> (n.d.), spaCy</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d01f6669a7e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/pre-processing-should-extract-context-specific-features-4d01f6669a7e">Pre-processing should extract context-specific features.</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>