<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by John Micah Reid on Medium]]></title>
        <description><![CDATA[Stories by John Micah Reid on Medium]]></description>
        <link>https://medium.com/@johnmicahreid?source=rss-4514eefa6d8b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*4vMHtGc4fcZakPOwoHAXXA.jpeg</url>
            <title>Stories by John Micah Reid on Medium</title>
            <link>https://medium.com/@johnmicahreid?source=rss-4514eefa6d8b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 12:04:21 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@johnmicahreid/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to prioritize bugs and feature requests in real-time]]></title>
            <link>https://johnmicahreid.medium.com/how-to-prioritize-bugs-and-feature-requests-in-real-time-181d8e429bed?source=rss-4514eefa6d8b------2</link>
            <guid isPermaLink="false">https://medium.com/p/181d8e429bed</guid>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[prioritization]]></category>
            <dc:creator><![CDATA[John Micah Reid]]></dc:creator>
            <pubDate>Wed, 04 Sep 2024 11:38:05 GMT</pubDate>
            <atom:updated>2024-09-09T15:21:50.750Z</atom:updated>
            <content:encoded><![CDATA[<p>Reading Product Manager job descriptions, you’d be forgiven for thinking that the job is just sitting around a conference table with post-it notes and whiteboards, ‘defining strategy’ and especially ‘prioritizing roadmaps’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yskTkr10ns-DVHJm.png" /><figcaption>this would never happen in real life right?</figcaption></figure><p>For the uninitiated, this is a regular exercise (e.g. quarterly) where you gather a long list of ideas from stakeholders and users, then sort them into prioritized buckets such as Now/Next/Later using a framework like Reach-Impact-Confidence-Effort. Here’s one of my previous articles if you want to learn more:</p><p><a href="https://johnmicahreid.medium.com/how-to-prioritize-your-product-roadmap-1e8dc79200e">How to prioritize your product roadmap</a></p><p>However, this is actually only a small part of the prioritization work. Most of the time you’ll be receiving a constant stream of ad-hoc incoming issues, for example:</p><ul><li>Crash reports are up 20% on Android after the new release</li><li>Marketing needs a new demo built</li><li>Several users have asked for a new download file format</li></ul><p>You need to decide what to do with each of them one at a time. This is a harder problem than the quarterly roadmap exercise because you can’t just wait to gather them in a batch and compare them against each other — you don’t know what else might come up in the near future. You need to prioritize in “real-time” (days) rather than in “batch” (months).</p><p>(Tangent: there’s a whole field of research around this called <a href="https://en.wikipedia.org/wiki/Secretary_problem">stopping theory</a> with intriguing names like the <em>secretary problem</em>, <em>marriage problem</em> and <em>sultan’s dowry problem</em> where you try to pick the best option from a sequence without knowing how many other options you’ll see)</p><p>It’s tempting to follow one of the extreme approaches:</p><ul><li><strong>Say no to everything — </strong>you stick to your guns and never fix critical bugs or address small, easy feature requests that end up having a disproportionate impact on user satisfaction.</li><li><strong>Take on all new issues</strong>— you add half-baked fixes to the product that clutter it up and never make progress on your main objectives.</li></ul><p>Most PMs muddle around in the middle, saying yes to some requests and no to others depending on the day of the week. Those around you get frustrated by your exercise of petty authority, which can lead to distrust.</p><p>In this article we’ll discuss an alternative system for real-time prioritization. I’ve used this successfully in my current role to speed up the process and provide more transparency to everyone around me.</p><h3>The system</h3><p>The idea is to score each incoming issue by several criteria, then prioritize based on the final score.</p><p>The criteria are as follows:</p><ol><li>Is it a bug or feature request? (bug ⬆️)</li><li>Are multiple users reporting an issue? (more users ⬆️)</li><li>What % of our user base is potentially affected, as measured by the % who use this feature? (more users ⬆️)</li><li>What is the business impact of the issue? (more severe ⬆️)</li><li>What is the estimated time to implement a change? (lower and more certain ⬆️)</li></ol><p>These generate a score between 0 and 100, which result in the following priority levels:</p><ul><li>70–100 → 🟥 <strong>Highest</strong>: Take into current sprint</li><li>40–70 → 🟧 <strong>High:</strong> Estimate for next sprint</li><li>20–40 → 🟨 <strong>Low/Medium:</strong> Put in backlog</li><li>0–20 → 🟦: No action</li></ul><p>I’ve defined the exact options and weights for each question inside a Google Sheets calculator which you can try here (remember to make a copy first so you can edit it):</p><p><a href="https://docs.google.com/spreadsheets/d/1nKU1poIWRZcVThFyHpobnczBJsUT64i8QKeOyng8biQ/edit?usp=sharing">Prioritization Calculator</a></p><p>The scoring system has a couple of advantages:</p><ul><li><strong>Transparent:</strong> the reasons why a ticket is prioritized or deprioritized are clearly shown by the weighting of different criteria — this will help you to explain to stakeholders why you will/won’t be working on their request</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fd20Q782a_1O6PzK.jpg" /><figcaption>okay this was unrelated but hilarious</figcaption></figure><ul><li><strong>Reproducible &amp; Consistent:</strong> the process can be followed by anyone, not just members of the product team, and the answers should be the same each time</li><li><strong>Fast:</strong> No need to spend time debating about where a given issue fits in</li></ul><h3>Notes on using it</h3><p>In terms of scheduling, you’ll need to reserve some capacity each sprint to actually work on these issues — 20% of engineering time is a good benchmark. If nothing more important comes up, you can use that extra time to work on tech debt and prevent your engineers from losing too much hair.</p><p>Some of the questions require more information to score e.g. for Question 3 you can link to a feature usage dashboard, and Question 5 might require getting an estimate from an engineer.</p><p>You’ll also want to adapt the questions and weighting for your industry and use-case. For example in B2B, you might want to weight incoming tickets from the top 20% of your customers more highly.</p><h3>Worked Examples</h3><p>Let’s go through a few of those previous examples to see how they end up getting prioritized — you can follow along on the sheet if you want to see the exact calculations:</p><p><strong>Crash reports are up 20% on Android after the new release</strong></p><ul><li>Bug: +15</li><li>Multiple users reporting an issue: +15</li><li>20-50% of userbase affected: +20</li><li>Impact is High (data loss): +30</li><li>Time to fix is Unknown: 0</li></ul><p>The overall score here is 80, which means it is Highest Priority and you should work on it in your current sprint.</p><p><strong>Marketing needs a new demo built</strong></p><ul><li>Feature: 0</li><li>No users reporting an issue: 0</li><li>10–20% of userbase affected: +10 (assuming that’s the number of prospects who will see it)</li><li>Impact is low (cosmetic): 0</li><li>Time to implement is more than a week: 0</li></ul><p>The overall score is 10, meaning you do not even put it in your backlog. This might sound harsh, but there are probably better ways to spend your team’s time.</p><p><strong>Several users have asked for a new download file format</strong></p><ul><li>Feature: 0</li><li>Multiple users reporting an issue: +15</li><li>&lt;10% of users affected: 0</li><li>Impact is medium (workaround required): +15</li><li>Time to implement is less than a day: +20</li></ul><p>In this case the score is 50, meaning you should estimate it for the next sprint.</p><h4>Closing Thoughts</h4><p>You’ll notice that some scores result in not taking any action and essentially ignoring the issue 😱 — this might seem harsh but there’s no point putting something in the backlog that you’ll never work on because it’s so low priority. This is exactly the point of prioritization: choosing what you’re going to focus on so you can actually make progress.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/1*spGhw80SGRBvlA-3BMpxdA.gif" /></figure><p>Using this framework you can prioritize incoming issues one-at-a-time, without knowing the relative importance of everything else. This is a useful counterpoint to the batch-style prioritization that is more commonly known, because you are trading a bit of accuracy for a quicker decision. It’s also really helpful for staying flexible when your best-made quarterly roadmaps become out of date as they encounter reality.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=181d8e429bed" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Replacing myself with an AI assistant for a day]]></title>
            <link>https://johnmicahreid.medium.com/replacing-myself-with-an-ai-assistant-for-a-day-8f2284cc7be6?source=rss-4514eefa6d8b------2</link>
            <guid isPermaLink="false">https://medium.com/p/8f2284cc7be6</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[product-management]]></category>
            <dc:creator><![CDATA[John Micah Reid]]></dc:creator>
            <pubDate>Mon, 10 Jun 2024 08:33:50 GMT</pubDate>
            <atom:updated>2024-06-10T08:46:55.483Z</atom:updated>
            <content:encoded><![CDATA[<h4>A product manager’s experiment</h4><p>Have you wondered recently how replaceable you are in your job? I’m not talking run-of-the-mill imposter syndrome but the vaguely ominous sensation you get as you read how AI is ‘transforming industries’ i.e. people losing their jobs. What if you’re just a trained monkey, a ‘stochastic parrot’, and a reasonably well-tuned set of prompts could basically do your job?</p><p>I decided to confront this question head on by instructing a virtual assistant to replace me as a product manager at Snowplow Analytics for a day. If they did great then I could automate some of my work while I prepared to leave tech to make furniture (in the time-honoured tradition). If they weren’t up to scratch, I’d still have a job, and at least have learned some new tips and techniques for making me more productive at work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_fOMWn2voAlmSinnGTZV6Q.jpeg" /><figcaption>cue obligatory cutesy/contrived AI-generated image for lazy article writers</figcaption></figure><p>This ended up being a hilarious social experiment, and it seems I still have a job for a while yet.</p><h3>The Setup</h3><p>To get interesting results, I had to set some parameters:</p><ul><li>Ran the experiment on a Friday during our AI Hackathon week, where we were given some extra time to explore projects and learning around AI, hoping this would cut me some slack with colleagues.</li><li>Asked ChatGPT to role-play as a virtual assistant called <a href="http://john.ai/">John.ai</a>. All my interactions with others had to be in <a href="http://john.ai/">John.ai</a>’s words — I could not communicate directly with anyone. I could regenerate the outputs until I was happy, but could not edit the final text.</li><li>Communicated to colleagues by updating my Slack name and status, and sending a message to everyone I interacted with on that day to give them a heads-up. For video meetings, I turned my camera off, raised my hand when I wanted to “speak”, and played audio responses from <a href="http://john.ai/">John.ai</a>.</li><li>Tried a variety of tools (paid trials so no sensitive data was shared): ChatGPT, Gemini, Notion AI and ElevenLabs (for text-to-speech output)</li><li>Sent out a survey at the end of the day to everyone I communicated with asking them about their experience interacting with me. While there’s an interesting debate to be had about how you can actually measure the short-term output of knowledge workers, particularly in a fuzzy role like product management, I picked satisfaction scores as my main success metric.</li></ul><h3>The Day</h3><ul><li>9:14am: Started work on a quiet Friday morning. Got some muted techno beats going with the help of Spotify’s recommendation algo (this is one AI I will swear by). Spent a while catching up on messages and email.</li><li>9:34am: Tried to generate some wireframes for a new screen to be added to our hackathon project. The results were amusingly bad, and did not improve with more prompting. In the end I just resorted to writing a very detailed prompt explaining what I wanted, then sending the response to my designer for him to figure out.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pvBl570yEJAy9gsCvMdpiw.jpeg" /><figcaption>Do you want to exceute or uccegute the quuiry?</figcaption></figure><ul><li>10am: Hackathon standup. I asked <a href="http://john.ai/">John.ai</a> to generate a message explaining the experiment, and then copied it over to ElevenLabs for a more natural-sounding voice, but quickly realised that lag was an issue. Ended up sticking with ChatGPT entirely, although the chirpy female American voice didn’t quite feel like me.</li><li>10:20am: Got deep into a new idea around generating user features. This was a very productive session — I asked <a href="http://john.ai/">John.ai</a> to come up with 10 features, then a JSON schema for them, then the SQL for them, then hacked and poked around and identified the gaps with our current spec. I felt for the first time a real sense of teamwork and collaboration — that my assistant was helping me automate the boring bits and evaluate different tradeoffs.</li><li>11:30am: Attended an update meeting about a new initiative with 4 senior engineers. Got a bit of a laugh when <a href="http://john.ai/">John.ai</a> introduced himself, but it was a bit hard to contribute to the conversation due to the time lag. Someone would ask a question, I’d type a prompt for <a href="http://john.ai/">John.ai</a>, receive a text response, press play and then hear the audio — each of these loops took about 5–10 seconds which killed the flow of conversation. I resorted to pre-generating some responses and waiting for the right time to press play, but inevitably what I said would be slightly out of sync.</li><li>12pm: Lunch came around and I breathed a sigh of relief. I hadn’t realised how exhausting it was not being able to <strong>talk</strong> to people directly — it was like taking a vow of silence for the day.</li><li>1pm: Got stuck into replying to some messages. Here the cracks really started to show. A lot of my work is high-context, short communication to help keep my team on track. I need to tell a specific person a very specific thing, and it took a lot of time to write that out exactly. I ended up typing out exactly what I wanted to say then getting ChatGPT to repeat it back to me — not exactly adding much value:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-ymcvTa894DEUASlUdmZ7Q.jpeg" /><figcaption>wow, so deep</figcaption></figure><ul><li>1:30pm: Had a 1:1 with our cofounder to talk about our H2 roadmap. This was the big test for the day — if <a href="http://john.ai/">John.ai</a> could represent me well here then I’d be sorted. But after two minutes the cofounder said bluntly, “Sorry <a href="http://john.ai/">John.ai</a>, can I talk to the real John? This isn’t working.” So I turned my camera on and we had a real, productive conversation for about 45 minutes where we sketched out some ideas, stress-tested them, and came to a nuanced conclusion about how to present these changes to the company.</li><li>2:30pm: Organised and prioritised some feedback from internal testing of a new feature. Here the AI did a reasonable job. With minimal prompting, it organised about 6 pages of feedback into a neatly categorised list, then helped me generate tickets and acceptance criteria for the most high-priority ones. A theme was becoming clear: it works great on constrained problems where it can follow best practice, but isn’t so good at deciding what to do next.</li><li>4:27pm: Sent out a survey asking people their opinions on how the experiment went. <a href="http://john.ai/">John.ai</a> actually did pretty well at this, even generating some funky emojis for the different responses.</li></ul><h3>The Results</h3><p>So after all that, how did it go? The results were reassuring — with an average rating of 2/5, I’m afraid to say <a href="http://john.ai/">John.ai</a> wouldn’t have passed his probation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mMkdRqfy5ZZBnVwwlxU9dw.png" /></figure><p>Overall people didn’t seem to like interacting with <a href="http://john.ai/">John.ai</a> very much — it’s that feeling of mounting frustration you get when you’re on a website trying to cut through the chatbot’s decision tree and talk to a real person. Accuracy seemed to be okay, but it made me appreciate just how finely tuned our speech is to small non-verbal cues, in-jokes, interruptions and body language. We’re normally a camera-on culture, and turning the camera off was also perceived as a bit hostile.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OPuLYX-UnydNDUVvJ8DSgQ.png" /></figure><p>I think <a href="http://john.ai/">John.ai</a> came across better in text than on calls, especially when I prompted it (him?) to speak more clearly and casually.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VUXWRh527GrBEx2K0nUW3w.png" /></figure><p>The first response to this question was actually quite sweet — good to know when you’re appreciated at work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4GqbodMpmoScui1h0k1i-A.png" /></figure><h3>The Takeaway</h3><p>So in summary, what didn’t go so well? While <a href="http://john.ai/">John.ai</a> did well on individual tasks involving generating examples or sorting data, he was not a general replacement for me, for a couple of reasons:</p><ul><li><strong>Lag</strong> — the response time delay was tiring and killed any sense of conversational flow — it was like having a phone call with a crappy connection. Maybe the results would be different with ChatGPT 4o’s new realtime voice model, but I suspect there would still be a delay, and I wouldn’t be comfortable sitting out the calls and delegating all my responses to <a href="http://john.ai/">John.ai</a>.</li><li><strong>Context length vs output length </strong>— the times I got most value out of <a href="http://john.ai/">John.ai</a> was when I asked a relatively short question and received a long response that wasn’t just boilerplate. A lot of my communication is short and specific however, and I often spent way longer explaining what I wanted than the time it would have taken to reply/act directly.</li><li><strong>Tool switching </strong>— I hadn’t appreciated just how much I hop around between different tools and platforms. A simple task might involve logging into and checking 3–5 different platforms, and communicating across multiple mediums, and there’s no way an AI could currently a) have all that context, and b) be able to make changes across all those platforms.</li></ul><p>On a deeper level though, what was missing with <a href="http://john.ai/">John.ai</a>’s responses was a deep contextual understanding of these particular people and systems — the thousands of tiny data points, social cues, in-jokes and experiences in the company that allow me to tune my response to that exact person and situation. Maybe this applies particularly to product managers where so much of the job is about relationships and communication, and our impact is measured by the success of the team around us, but I’d like to think that we all use these adaptive skills in our daily work.</p><p>This well of context also allows me to act proactively: to look at a situation in its entirety, figure out what is missing then give a small nudge to get the ball rolling. And being forced to explain this process of ‘intuition’ to an eager but clueless AI assistant was a valuable exercise that made me appreciate just how hard it is.</p><p>So I’m not going to be replaced by <a href="http://john.ai/">John.ai</a> anytime soon. Maybe I’ll get him an internship though.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f2284cc7be6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The 5 Dark Arts of Product Management]]></title>
            <link>https://johnmicahreid.medium.com/the-dark-arts-of-product-management-d2371179e363?source=rss-4514eefa6d8b------2</link>
            <guid isPermaLink="false">https://medium.com/p/d2371179e363</guid>
            <category><![CDATA[product-management]]></category>
            <dc:creator><![CDATA[John Micah Reid]]></dc:creator>
            <pubDate>Wed, 07 Feb 2024 06:51:21 GMT</pubDate>
            <atom:updated>2024-02-08T11:44:04.123Z</atom:updated>
            <content:encoded><![CDATA[<h3>What do you mean “not best practice”?</h3><p>As a product manager, you will be judged on whether you make a meaningful impact to the business, which depends on building the right solutions to solve the right problems. You may be tempted to cut corners to achieve your goal, but need to understand the risks and long-term consequences of doing so.</p><p>In this article we will be discussing 5 ‘dark arts’ of product management, which we’ll define as risky or questionable techniques to achieve a worthy goal, as well as alternative ways to achieve the same goal. We’ll focus on practices that may be harmful overall but serve some benefit to your team, and are specific to product management i.e. not just general toxic workplace behaviours.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/662/0*i27I8mF2yi15xDaa.jpg" /><figcaption>Yes yes, I know I’m mixing up my Harry Potter and Star Wars references</figcaption></figure><p>I’ve witnessed or used all of these techniques in the past. Mostly, they cause a lot of harm. Occasionally, they can be useful. Ultimately, you need to learn judgment in balancing short-term vs long-term benefits, or ends vs means. As that luxury watch quote goes, <em>To break the rules you must first master them.</em></p><h3>🕵️‍♂️ Deception</h3><p>This technique involves telling people different versions of the same story, often saying something that you know isn’t actually true (i.e. lying). For example, you could tell a random stakeholder that you’ll work on their feature request next month, when you know it probably won’t happen.</p><p>This technique can be helpful for buying some breathing room for your team while you figure out some details. Telling people what they want to hear is a great way to get them to like you, or at least get them to do something for you. And tailoring our communication to different people is beneficial at a mild level — you shouldn’t communicate the same detail on a proposal to your CEO vs a customer vs an engineer.</p><p>It can go so wrong though. You have to keep track of exactly which stories you have told to which people so you don’t lose consistency. And if you constantly say things you don’t mean you’ll eventually get caught out and break trust, which is a death sentence for such a relational job role.</p><p>The alternative is harder, but sets you up much better for the long-term, and it’s to be honest with people even when it causes discomfort. In this example, you could simply tell the stakeholder that you won’t work on their feature request because it doesn’t fit in with the product strategy. There are of course better and worse ways to communicate truthfulness — talking respectfully, giving constructive feedback, offering alternatives, not making it personal — basically, try not to be an asshole about it. (My favourite book on communication is called <em>Difficult Conversations</em> by Bruce Patton, Sheila Heen, and Douglas Stone — <a href="https://www.goodreads.com/book/show/774088.Difficult_Conversations">check it out</a>).</p><p>And if you receive strong negative pushback when telling the truth, try to remember that you are not your role, that people aren’t upset or angry at you in your innermost being/deepest childhood self — they are upset that a Product Manager has told them something that causes them difficulty at work. Being deceptive can lead to much worse reactions in the long term, because being lied to feels more personally hurtful than being told an uncomfortable truth.</p><h3>🎲 Gaming your metrics</h3><p>This technique involves picking and optimizing a short-term metric to win praise for yourself or your team (min-maxing for the geeks). It’s more common at larger companies, especially those which have bought into a framework like Objectives &amp; Key Results, and are sufficiently data-mature to actually measure success.</p><p>A few examples here:</p><ul><li>If you are measured by input metrics like number of features delivered or tickets completed, you can split your projects into smaller and smaller increments to make it look like you are busier than you are.</li><li>If you are measured by an output metric like checkout step conversion, you can optimize it by making another part of the process worse i.e. add a hard signup requirement or relax fraud requirements.</li></ul><p>It’s key here that you don’t do or share more detailed data analysis that would disrupt the nice neat narrative you want to tell. Those around you will also want to hear about successes, because the alternative is harder and sounds like more work. This technique is helpful for making your team and yourself look good — after all nobody can argue with results, right? It may buy you more autonomy because you are seen to be performing. And sometimes it really does result in a short-term revenue increase for the company.</p><p>The consequences should be familiar to us all — products get worse over time. The user experience becomes bad, and internal tech debt and bureaucracy pile up. Cory Doctorow talks about this process as<a href="https://www.wired.com/story/tiktok-platforms-cory-doctorow/"> enshittification</a> — here’s a quote from his Wired interview on the topic of platforms:</p><blockquote>Here is how platforms die: first, they are good to their users; then they abuse their users to make things better for their business customers; finally, they abuse those business customers to claw back all the value for themselves. Then, they die. I call this enshittification, and it is a seemingly inevitable consequence arising from the combination of the ease of changing how a platform allocates value, combined with the nature of a “two sided market”, where a platform sits between buyers and sellers, hold each hostage to the other, raking off an ever-larger share of the value that passes between them.</blockquote><p>It’s actually a hidden consequence of the overall positive shift to data-driven decisionmaking — that unless you’re careful the data will tell you what you want to hear. The alternative involves investing more time and effort in your data — pick thoughtful metrics (and counter-metrics), set up careful experiments, and report the results accurately. Drive a culture of healthy discussion about results, and don’t tie performance evaluations purely to project success metrics.</p><p>And sometimes, you need to acknowledge that some things cannot be easily measured but nevertheless have value. In the absence of (gamifiable) data, you’ll need to rely on fuzzier principles, like <em>empathy</em>, <em>product vision</em> or <em>rational arguments</em> (eww, I know it’s gross). For a fuller treatment on this topic, read <a href="https://twitchard.github.io/posts/2022-08-26-metrics-schmetrics.html">Be good-argument-driven, not data-driven</a> by Richard Marmorstein.</p><h3>💣 Shipping a broken product</h3><p>This involves deliberately shipping a product with known bugs or untested functionality. Sometimes it’s an edge case that could cause a crash, or even a button that doesn’t actually work which you hope nobody presses.</p><p>This is a tricky one because a core skill of product management is scoping features down into smaller increments, which means leaving out some requirements that aren’t essential. It’s part of the fixed time variable scope principle of Agile, which says that you should commit to whatever you can deliver within a timeframe, rather than committing to the full set of requirements and taking as long as you need to achieve it. You should be trying to ship small meaningful increments often so that you are getting feedback on whether you are building the right thing, which is often called the minimum viable product. The key word here is “viable”, i.e. it has to work.</p><p>The consequences for shipping broken stuff can be catastrophic and reflect really badly on you/your team. You could cause users to churn, an outage in production, or cripple the long-term adoption of the feature because you created a bad first impression. It’s a sickening feeling releasing a new feature and then watching the bug reports come in, and leads to a lot of stress for your engineers as they scramble to reverse the rollout or release a hotfix to prod.</p><p>The alternative of not releasing anything that isn’t perfect is a bit too strict, because velocity will slow down and projects will stretch out. Instead, you should strive for a balanced approach, with a few rules in place:</p><ol><li><strong>Define minimum release standards</strong> — write down a set of criteria for release and if your feature doesn’t meet all of them then halt rollout until you have fixed the problem. Some things to include here are: unit tests, integration tests, X days on a staging environment, security flaws, and bugs on the critical user journey. (You can ship with some cosmetic stuff not fixed, but <em>make sure you know about the issues through testing first.)</em></li><li><strong>Document the limitations</strong> — make sure all the potential failure cases are written down and communicated internally in advance, and share this with users if you can.</li><li><strong>Communicate level of certainty</strong> on the feature i.e. <em>private preview/public preview/general availability </em>— this helps to align expectations on the quality of the software being released.</li></ol><h3>🗡️ Sabotage</h3><p>Ah, this is a classic. Let’s say your team is responsible for a certain system within the company, e.g. the communications platform, and that occasionally you get requests from other teams to make a change to the system to support one of their projects. You can simply refuse all such outside requests so that your team doesn’t get distracted, no matter how important or beneficial the request would be. Bonus points for: poaching engineers from other teams, sowing distrust for other product managers and withholding key information that would be helpful to another team. I wish I could say it wasn’t like this, but organizations can be just like kids’ playgrounds, replete with tantrums, bullies and petty feuds. This pattern is more common in a company with a lot of autonomous teams and poor/disengaged leadership.</p><p>As with all these points, a low amount of this can be healthy. You need to fight to keep your team free from distractions, and you need to prioritize your own projects so you can actually achieve your goals. Not all feature requests are reasonable, especially if they come from a non-technical stakeholder where a simpler solution could suffice. Don’t try to avoid all conflict by saying yes to everything (see my previous piece <a href="https://medium.com/@johnmicahreid/3-things-to-avoid-saying-as-a-product-manager-4b0d37e9e56">3 things to avoid saying as a PM</a>).</p><p>However, if you take this approach to the extreme, two things could happen in the long-term:</p><ul><li><strong>Silos </strong>— other teams learn not to ask you for anything, and everyone starts to develop their own workarounds rather than integrate with an existing system. Thus, standards and systems proliferate and the product becomes a confusing mess.</li><li><strong>Reciprocity </strong>— You might need that other team to make a change for you in the future, and if you’ve burned your bridges then good luck getting that feature shipped.</li></ul><p>The alternative is to have a more grown-up conversation about tradeoffs and priorities across the company — it really helps if you have invested in relationships with other teams so you can bargain from a place of trust and mutual respect. If it is a recurring issue, then escalate to higher management and ask them to prioritize. Ultimately a lot of this cross-team conflict can be due to Conway’s Law, where the structure of the teams is not aligned to the projects you are delivering. In this case, the company should consider a reorganization to reduce dependencies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*j_cGNeivUql7kDoW" /></figure><h3>🔥 Being a Hero</h3><p>Being a hero involves stepping in to do tasks for other people, or going above and beyond the call of duty. You work longer hours than everyone else, you never say no to a request for help or support, you get involved in every possible meeting and initiative.</p><p>This may seem like a strange example to include — isn’t it a good thing to work hard? But my definition of a dart art involves gaining a short-term benefit at the expense of a person or a group of people. So who is harmed in each case?</p><ul><li>Deception: Everybody</li><li>Gaming your metrics: The company</li><li>Shipping a broken product: Users</li><li>Sabotage: Other teams</li><li>Being a hero: <strong>yourself</strong></li></ul><p>If you’re working continuously at maximum pace, you are setting yourself up for failure in the long-term, with three major consequences:</p><ul><li><strong>Burnout</strong> — we’re not made to work flat-out without a break. Our ancestors were unparalleled<a href="https://en.wikipedia.org/wiki/Persistence_hunting"> endurance hunters</a>, jogging after prey for many hours until the target would literally collapse from exhaustion. We weren’t the fastest kids on the savannah, but we knew how to set a steady, sustainable pace to achieve a goal.</li><li><strong>Reduced productivity</strong> — knowledge work is not shift work. More time doesn’t necessarily equal better results, in fact you may be doing more harm than good by staying at your desk. Studies of East Asian working cultures show that<a href="https://www.cnbc.com/2018/06/01/japan-has-some-of-the-longest-working-hours-in-the-world-its-trying-to-change.html"> long working hours are correlated with low productivity</a>. Ask yourself if you’re still doing your best work after 10+ hours at your desk, or if it can wait until the morning.</li><li><strong>Learned helplessness </strong>— the more you step in for others, the more you encourage them to ask you again in future and not learn to help themselves. Beware the saying “If you need something done, ask a busy person to do it”.</li></ul><p>The alternative is to learn to say no, which is a useful signalling mechanism for other people to reflect and change their behavior. And if you have a lot on your plate, delegate tasks (graciously) to others, knowing that they might do work differently to how you would have done it. It’s hard — I believe as product managers we often have a natural inclination to take responsibility and get involved, but the better long-term strategy is to build up a team and a system capable of delivering results, rather than relying on your own heroic efforts. Plus, it’s more scalable.</p><p>So these are the dark arts of product management. Hopefully you’ve learned why they are useful, what can go wrong, and how you can achieve the same result through safer / more sustainable means. I’ll leave you with a quote from our boy Aristotle discussing the <a href="https://en.wikipedia.org/wiki/Golden_mean_(philosophy)">Golden Mean</a>, where he argued that “the virtuous course of action is always the mean between two extremes of excess (too much) and deficiency (too little).”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*PeDObKTG51ZlTsEo5kKttQ.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d2371179e363" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to prioritize your product roadmap]]></title>
            <link>https://johnmicahreid.medium.com/how-to-prioritize-your-product-roadmap-1e8dc79200e?source=rss-4514eefa6d8b------2</link>
            <guid isPermaLink="false">https://medium.com/p/1e8dc79200e</guid>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[prioritization]]></category>
            <dc:creator><![CDATA[John Micah Reid]]></dc:creator>
            <pubDate>Thu, 27 Apr 2023 15:25:12 GMT</pubDate>
            <atom:updated>2024-02-08T11:51:44.470Z</atom:updated>
            <content:encoded><![CDATA[<p>As a product manager, one of your most important responsibilities is prioritization — you need to give your team a clear view of <strong>what to work on next. </strong>And since there are always more projects to work on then capacity to do them, you need to reject or deprioritize some ideas: focus is key to success.</p><p>But what does this process actually look like? Imagine you’re a Product Manager at a cat grooming marketplace, responsible for growing the supply of new cat groomers. Every time you have an idea, or receive a piece of feedback, you write it down somewhere on a sticky note. The goal of the PM is to take this messy board of potential options and turn it into a neat prioritized list of projects:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/674/1*j9Uj9qIH0v-lpy3gKnoeTw.png" /></figure><p>While there are tons of techniques you can use to break down these ideas into smaller chunks, then test/validate etc., we’ll leave that out and focus on 4 main prioritization techniques.</p><h3>Stakeholder Driven Prioritization</h3><p><em>a.k.a. what your boss wants</em></p><p>In this approach you simply ask your boss what they want done. Every time they have a new idea, you put it to the top of the stack and deprioritize the rest (alt: do this every time anyone requests something of you). This is very common in smaller, founder-led businesses, or when you’re starting out as a junior PM.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/541/0*DzRpLaS2SKYNkAlt.PNG" /><figcaption>just listen to this guy basically</figcaption></figure><p>In defense of this strategy, you can say that it ensures the product roadmap aligns with the overall company strategy. It also helps to build a better relationship with the stakeholders by showing that their input is valued. Cynically, it can be good for your career to be currying favour with your boss.</p><p>However, it also has significant downsides. You’re basically outsourcing the prioritization question to somebody else, and that someone else may not have an accurate picture of your team’s capacity. You often end up implementing half-baked ideas that aren’t validated by user input or data, or you spend all your team’s time on requests from other stakeholders and don’t actually ship anything meaningful.</p><h3>User Driven Prioritization</h3><p><em>a.k.a what your users want</em></p><p>User-driven prioritization means listening to the needs and wants of the end-users and implementing the most popular ideas. Imagine you send out a survey to all your top cat groomers asking them what the main issues and feature requests are. You group them together, then prioritize by the most popular requests downwards. You can repeat this process for any source of user feedback e.g. customer support issues/app store reviews/social media comments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/425/0*rhgAmvf4JO91_fmh.gif" /><figcaption>listen to whoever is yowling loudest</figcaption></figure><p>This is a somewhat better approach than the previous one, because there is at least some validation of the problem. However, it is important to note that users may not always know what they want. Sometimes, they may request features that are not feasible or are not aligned with the product strategy. In the worst-case scenario you end up building something that everyone asks for but nobody ends up using.</p><p>While this can be mitigated to some extent by clever survey design and good UX research (<a href="https://www.amazon.co.uk/Mom-Test-customers-business-everyone/dp/1492180742">The Mom Test</a> is a fantastic book about this), you still need to exercise some skepticism when asking users directly what they want.</p><h3>Data Driven Prioritization</h3><p><em>a.k.a what the data says</em></p><p>Data-driven prioritization means weighing projects based on quantitative data of their cost/impact. For our example, it could mean analysing the signup funnel to detect where the most significant drops in conversion are, then coming up with projects to improve those steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/0*YZ7Oy3uxkVyAwhoV.png" /><figcaption>this is the best I can do for a cat graph. Source: XKCD</figcaption></figure><p>By trying to quantify the impact based on some external business metric, you will make more objective and therefore impactful decisions. It’s a big upgrade on the previous methods because data doesn’t lie to you to make you feel better.</p><p>However, it is important to consider the limitations of the data. Data analysis tells you what your users are doing, but not why they are doing it. It’s great for evaluating the success of something already released, but can’t help you prioritize an experimental feature that doesn’t exist yet.</p><p>As William Bruce Cameron said:</p><blockquote>“Not everything that can be counted counts.<br>Not everything that counts can be counted.”</blockquote><p>Basically, while it is super important to be rigorous in your analysis, don’t fall into the trap of just doing what you can easily measure.</p><h3>Cost-Benefit Analysis Prioritization</h3><p><em>a.k.a. what the business needs</em></p><p>Cost-benefit analysis prioritization means evaluating the potential return on investment (ROI) for each project. This involves looking at the costs associated with implementing a project (such as development time and resources) and the potential benefits (such as increased revenue or user retention). It incorporates elements of all the previous methods, except you try to quantify the size of the benefit/problem solved, the cost and the uncertainty.</p><p>A classic way to do this is the RICE method — Reach x Impact x Confidence / Effort.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/999/0*9AybIMCsmbQ46hLl.png" /></figure><p>For each project, you assign a value to each term, multiply them together and use the resulting RICE score to sort the overall list. In our case, we could say Video Onboarding might score an 8 for Reach, 5 for Impact, 3 for Confidence and 6 for effort for a total score of 20. We’d then do this for all the others and see which one had the highest score.</p><p>This is the most grown-up method and, done well, can provide a rigorous assessment of each project’s potential. It helps you make semi-objective tradeoffs between different types of projects e.g. maintenance work (high reach, low impact, medium effort) vs risky experiments (low reach, high impact, high uncertainty).</p><p>However, in my experience you still end up needing to make a lot of subjective assumptions. The results can feel counter-intuitive or unsatisfying, and it’s easy to fall into the trap of massaging the numbers to favour your pet projects. Effort is <strong>always underestimated</strong>, in part because there are usually edge-cases, tech debt or switching costs you haven’t considered. Also, if you’re doing this properly it gets harder over time, because you solve all the obvious problems first.</p><h3>Bonus: True Wisdom™</h3><p>While the RICE method is best if you have the time and resources, it doesn’t actually contradict the other methods. Rather than sticking to one approach rigorously, I like to think of product prioritization like playing a complex boardgame, where you have an overall strategy but deviate from it if you spot opportunities for small gains.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bUKEkvU7ZnM3mZLp.jpg" /><figcaption>This game is Terraforming Mars, 10/10 would recommend!</figcaption></figure><p>Ultimately you need to be both <strong>flexible </strong>and<strong> greedy (</strong>for product wins). Prioritization work can never be finished — it can only be managed for a time. Don’t get attached to your roadmap, and be ready to discard previous assumptions based on new data.</p><p>In summary, there are 4 main prioritization techniques in increasing order of rigour and effectiveness:</p><ul><li>Stakeholder-driven</li><li>User-driven</li><li>Data-driven</li><li>Cost-benefit (RICE)</li></ul><p>In the next article, we’ll cover some sneaky product management dark arts to cheat the game — once you know the rules you can break them!</p><p><a href="https://johnmicahreid.medium.com/the-dark-arts-of-product-management-d2371179e363">The 5 Dark Arts of Product Management</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1e8dc79200e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why you shouldn’t be yourself at work]]></title>
            <link>https://johnmicahreid.medium.com/why-you-shouldnt-be-yourself-at-work-23a67be53386?source=rss-4514eefa6d8b------2</link>
            <guid isPermaLink="false">https://medium.com/p/23a67be53386</guid>
            <dc:creator><![CDATA[John Micah Reid]]></dc:creator>
            <pubDate>Tue, 28 Feb 2023 19:57:48 GMT</pubDate>
            <atom:updated>2023-03-06T09:42:48.066Z</atom:updated>
            <content:encoded><![CDATA[<p>When I started working, I thought I needed to be my authentic, ‘real’ self at work. I tried to be friends with all my colleagues, putting effort into arranging dinners and hangouts after work and sharing deep vulnerabilities. And yet I found many of these interactions exhausting and unsatisfying — they didn’t really ‘go’ anywhere or lead to deeper connections.</p><p>I also poured myself into my projects, trying to complete each task with creativity and technical brilliance. As a result, it was really tough receiving criticism. When somebody called my idea bad or pointed out a mistake, I took that personally and got hurt. I would try to justify myself, getting into arguments and trying to refute every piece of criticism with further data and modifications to my original idea.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/513/0*_2fe8Mzrz6OueACh.jpg" /></figure><p>The hard-won truth is that work is ultimately a transactional relationship and applying the same rules from your personal life and relationships will lead to more stress and anxiety in the long term.</p><p>So what is a better way to act at work? We often hear the term ‘being professional’, but it’s somewhat hard to define — you know it when you see it. I’ve now come to think of working as like playing a <strong>role</strong> that comes with certain expectations and responsibilities. Netflix talks about a company being a <a href="https://www.slideshare.net/reed2001/culture-1798664/24-Were_a_team_not_a">sports team rather than a family</a>, but arguably it’s more like a theatre group putting on a performance — everyone has a role to play and needs to work together to create an experience for the audience (your users/clients). In this scenario it’s essential to be inauthentic i.e. acting according to a script of what is expected.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Ftenor.com%2Fembed%2F21054892&amp;display_name=Tenor&amp;url=https%3A%2F%2Ftenor.com%2Fen-GB%2Fview%2Fcats-gif-21054892&amp;image=https%3A%2F%2Fmedia.tenor.com%2FYE2Ys0lgT8IAAAAC%2Fcats.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=tenor" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/c4d4d76aca8667be12d07586bbcc7ee8/href">https://medium.com/media/c4d4d76aca8667be12d07586bbcc7ee8/href</a></iframe><p>Here’s how I’ve learned to separate out my professional and personal selves:</p><ul><li>In my Product Manager role at work, I put more effort into communicating clearly and simply. I am quite pushy and goal-directed, and I’m not afraid to speak out if I think something is a bad idea. I am more analytical, thinking about tradeoffs and how much to invest in any given project or conversation.</li><li>Outside of work, I am sillier and more distracted, I like to do things inefficiently, like cooking elaborate meals. I’m more indirect and nuanced in conversation — I talk less and ask more questions. I̵ ̵p̵o̵s̵t̵ ̵a̵r̵t̵i̵c̵l̵e̵s̵ ̵o̵n̵ ̵M̵e̵d̵i̵u̵m̵ ̵w̵i̵t̵h̵ ̵l̵o̵t̵s̵ ̵o̵f̵ ̵c̵a̵t̵ ̵g̵i̵f̵s̵.̵</li></ul><p>Once you understand the role you need to play at work, you start realizing it isn’t necessarily a repressive system. Sometimes it can be freeing to come to the office (or Zoom call) and be someone different to who you are in your personal life — it’s a mini-game with a different set of rules and rewards, and this often leads to growth. One of my most satisfying experiences as a manager was mentoring a data analytics intern. She came in super shy and reserved, but loved digging into data and trying out new AI models for a complex problem. At the end of her internship she prepared a presentation for the top management on what she’d done, and had become so invested in her project and the need to communicate her results that she blew everyone away with her subject matter authority. She had developed a new persona through her work, one in which I felt like the intern and her the expert.</p><p>Another benefit of playing a role is learning to not take things personally. If someone criticizes a project I’ve spent months working on, they are criticizing the project that a product manager called John has worked on, not ‘me’ and my innermost hopes and desires. Arguably they would still have criticized it if a different person in my role had worked on it instead. Paradoxically, having some distance makes you more effective. You get better at avoiding petty arguments, and can focus on getting stuff done faster. By focusing on the task and not your own ego, you learn to critique your own ideas and make more objective decisions.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Ftenor.com%2Fembed%2F24487716&amp;display_name=Tenor&amp;url=https%3A%2F%2Ftenor.com%2Fen-GB%2Fview%2Fcoolcat-gif-24487716&amp;image=https%3A%2F%2Fmedia.tenor.com%2F8iRH-hhK7nEAAAAM%2Fcoolcat.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=tenor" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/f8040ae5679233c3ed2468acd275861d/href">https://medium.com/media/f8040ae5679233c3ed2468acd275861d/href</a></iframe><p>It also leads to more satisfying relationships with your colleagues. When you’re clear about which level you’re relating on — personal or professional — both states become more enjoyable because you’re not trying to be everything to everyone. You learn to enjoy the interactions for what they are, taking joy in the small moments of connection. These may lead to a deeper friendship, or they may not, and both of these outcomes are fine.</p><p>As with all things, this compartmentalisation can be harmful when taken to extremes. Nobody wants to work with a sociopath — being professional still means treating people with dignity and respect. Playing a role that is completely out of alignment with your normal baseline will probably be exhausting, and you still need to draw on elements of yourself to play your role effectively.</p><p>Or maybe — as many philosophies argue — there isn’t even such a thing as a stable self, and we’re just moments of consciousness arranged in a sequence over time. If that is so, then even more reason to enjoy playing your part in the show!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s07Pr8jO7iCsrQlY8Nseog.jpeg" /><figcaption>our cat being ‘professional’</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=23a67be53386" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[14 questions to ask as a Product Manager]]></title>
            <link>https://medium.com/design-bootcamp/14-questions-to-ask-as-a-product-manager-bf7382c75928?source=rss-4514eefa6d8b------2</link>
            <guid isPermaLink="false">https://medium.com/p/bf7382c75928</guid>
            <category><![CDATA[product-management]]></category>
            <dc:creator><![CDATA[John Micah Reid]]></dc:creator>
            <pubDate>Mon, 28 Nov 2022 16:58:54 GMT</pubDate>
            <atom:updated>2024-02-08T11:54:47.989Z</atom:updated>
            <content:encoded><![CDATA[<p>There are no technical requirements to be a product manager. You don’t need a fancy degree, or to pass a certification exam that tests for deep domain knowledge. People become PMs from many different backgrounds, often because they enjoy working with people and solving problems.</p><p>Yet product management can be tough at the start because it often feels so messy. You need to make hard decisions with imperfect information. You have limited time to gather and communicate complex requirements. You must create structure so that other people can actually get the work done, and if things don’t fit together, that’s on you.</p><p>Luckily there’s this <em>one weird trick</em> which will make your life so much easier — asking questions!</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2Fl0Iy8hSJalxmgTOF2%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2Fl0Iy8hSJalxmgTOF2%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2Fl0Iy8hSJalxmgTOF2%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="244" frameborder="0" scrolling="no"><a href="https://medium.com/media/ce9c5cc2dc740701ff040867cb9b7e3f/href">https://medium.com/media/ce9c5cc2dc740701ff040867cb9b7e3f/href</a></iframe><p>The right question can bring clarity to any situation and help you progress in the right direction. In this article we’ll encounter a fictional product problem, and ask 14 questions, from gathering requirements to proposing a solution, and ending with execution and followup.</p><p>I use these exact questions on a daily basis as a PM; hopefully you’ll find them useful too!</p><h3>Gathering Requirements</h3><p>Let’s set the scene. You’re a new PM at a hip tech startup that delivers on-demand cat food (that has to be a thing, right?). It’s a Monday morning and you’ve got some banging lo-fi beats playing. You hear the ‘ding’ of a new Slack notification and open it up to a question from someone in Operations.</p><blockquote>hey can we chat</blockquote><blockquote>we’re getting a lot of rider complaints about an app thing and I wanted to talk to you about it.</blockquote><p>Sure, you think. I’ve got time.</p><blockquote>of course — I’ll send you an invite for this afternoon!</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*-Z1_pR2KtrvAqKlL.jpg" /></figure><p>[Side note: while we start with an initial conversation, these questions are directed at different people over time. The exact wording can also be adapted for whomever you’re talking to.]</p><h4>1. What is the problem?</h4><p>It’s helpful to start off with an open-ended question that encourages people to give you as much information as possible in their own words. Ideally you want to be taking notes (notebook or laptop both work) while they are speaking. Let’s see how it goes:</p><p><em>Me: Hey, tell me what the problem is?</em></p><p><em>Operations: We had this guy crash on his bike last night and he said it was because his phone was too bright. He was switching from Google Maps (which is dark) to look at his order details and lost focus. He’s fine now, but this has happened a few times. Also riders complain a lot about our app and they say they can’t see where they need to go.</em></p><h4>2. What is the current state and desired state?</h4><p>You’ll get a lot of info with the previous question, and the next task is to untangle and simplify it. One useful way to do it is to establish a baseline for how things work right now, and what change somebody is requesting. Later on, you’ll want to describe this as a user flow, i.e. a sequence of actions.</p><p><em>Me: So how do our riders use the app at night, and what do you think we should change?</em></p><p><em>Ops: It seems that when it gets dark our riders usually turn the brightness of their screens way down, but then they can’t see their order details. What they’d like is a ‘dark mode’ where they can still see the important stuff but not be blinded.</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FY4JE51QwzefMLfknfq%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FY4JE51QwzefMLfknfq%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FY4JE51QwzefMLfknfq%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="244" frameborder="0" scrolling="no"><a href="https://medium.com/media/289da6445103fe18844c559e6cbeb255/href">https://medium.com/media/289da6445103fe18844c559e6cbeb255/href</a></iframe><h4>3. Can I check if I understand you correctly?</h4><p>After listening to someone explain a complex problem, it’s helpful to repeat back what you understood. This lets you check your assumptions and make sure you’re not missing anything, and also gives you practice in explaining the problem in your own words (hot tip: this works great in relationships too!). Later on you’ll be presenting this issue and a potential solution to a lot of other people so you’ll want to start getting a handle on how to communicate it.</p><p><em>Me: Can I check if I understand you correctly? You’re saying that drivers find it hard to use our app at night because the screen is too bright, and this is leading to accidents. What you’d like is some kind of dark mode where they can see their order details when they are on the go.</em></p><p><em>Operations: Pretty much, and it also needs to work well when they switch between our app and a navigation app.</em></p><h4>4. What is the impact of the problem?</h4><p>With this question you want to understand if this is an isolated incident or a recurring pattern. Ideally you also want to quantify the seriousness of the problem in terms of lost users, revenue or some other north star metric.</p><p><em>Me: What evidence do we have of the problem, like is it losing us riders or costing us?</em></p><p><em>Operations: Well, we pay our drivers’ insurance and hospital bills and it can get quite expensive. Also, we did a survey of driver habits and 30% of them said they switch to another app in the evenings. When we followed up, they said it was because they felt our app was unsafe.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/509/0*V9B3Eb3pdRFSQIKm.jpg" /></figure><p>[Side note: you’ll want to validate all this info later on with some data analysis.]</p><h4>5. Is this the most important problem we could be solving right now?</h4><p>Once you’ve understood the problem, you need to prioritize it against everything else you’re doing. Bonus points if it has a clear and serious impact, and ties into a broader company objective. You’ll also have to pitch this to your stakeholders and the rest of the company, and be prepared to justify the tradeoffs. This is a good question to ask yourself first.</p><p><em>Me: Is this the most important problem we could be solving right now?</em></p><p><em>Mirror-me: Good question, me. We need to finish this quarter’s projects first. After that though, our backlog is mostly tech debt and I’d say this takes priority especially since it fits right in with next quarter’s OKR of increasing online hours of riders. Let’s use the next 4 weeks to scope out a solution and then start development after that.</em></p><h3>Proposing a Solution</h3><p>Once you have a good understanding of the problem (ideally supplemented by data analysis, surveys and/or UX testing), it’s time to work on a solution. As you start crafting a pitch, you’ll want to answer some critical questions with the help of your team.</p><h4>6. What is the ideal user experience we could create?</h4><p>Use some blue-sky thinking here and put yourself in the shoes of your users to imagine the best possible experience for them. Don’t think about technical constraints or cost yet — there’s plenty of time to scope it down later on.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2F11thnyggFkrmmc%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2F11thnyggFkrmmc%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2F11thnyggFkrmmc%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="244" frameborder="0" scrolling="no"><a href="https://medium.com/media/245d6876be92d0f007181bb59afcf645/href">https://medium.com/media/245d6876be92d0f007181bb59afcf645/href</a></iframe><p>At this stage it might be helpful to start writing a <a href="https://medium.com/agileinsider/press-releases-for-product-managers-everything-you-need-to-know-942485961e31">PR/FAQ</a> (Press Release &amp; Frequently Asked Questions). This is a format popularized by Amazon where you write an imaginary press release describing your feature to your end user.</p><p><em>Me: What is the ideal user experience we could create?</em></p><p><em>Designer: Imagine the rider’s phone sensed when it was getting dark and automatically switched to dark mode, like in a car. That could be the default, but the rider would also be able to choose a manual mode, or following the system default. We’d probably want to allow riders to set their choice when they sign up and also highlight this feature on our website.</em></p><h4>7. How will we measure success in solving the problem?</h4><p>Next, you’ll need to pick a single, currently measurable metric that correlates with the problem you’re trying to solve and can’t be easily gamed. Later on you’ll use this to determine whether the feature was successful or not.</p><p><em>Me: How will we know we’ve solved the problem? How will we track success?</em></p><p><em>Analyst: Our main goal should be to increase online hours in the evenings, since if drivers feel more safe using the app they are less likely to switch to competitors. We’ll also keep an eye on what percentage of riders enable this feature, and monthly satisfaction survey scores. We should probably run this as an A/B test to be sure of the result.</em></p><h4>8. What options do we have to implement this feature?</h4><p>This is a conversation to have with the whole team, especially your designers and engineering managers. Use the MECE principle (mutually exclusive, collectively exhaustive) to generate a full list of options, and spend the time to discuss pros and cons. Often it’ll become pretty clear which one (or two) to focus on.</p><p><em>Me: What options do we have to implement this feature?</em></p><p><em>Tech lead: I think there are several ways to do it: <br>1. We could just change our whole app’s colour scheme to dark mode and not allow users to switch — this would be quickest but isn’t the best UX.</em></p><p><em>2. We could use some automated library that automatically inverts our colours.</em></p><p><em>3. We could define two colour schemes and use the system settings to determine if we use the light mode or dark mode.</em></p><p><em>4. [etc etc]</em></p><p>[Side note: these are just examples, I haven’t actually researched how you would implement dark mode]</p><h4>9. How do users interact with our potential solution?</h4><p>This is a good question to ask, if you have the time and resources. Once you have decided on an initial approach, put together a prototype on Figma or another design tool, let users click around and see where they get confused. Or run it by other people in the company, especially those who have close contact with your users.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*P9WIBSJqI2eJ-D8bEuuMKg.jpeg" /><figcaption>btw this is a terrible question to ask users because they’ll only tell you what you want to hear</figcaption></figure><p><em>Me: What do users think of our potential solution?</em></p><p><em>UX Researcher: We found that they really liked having dark mode in the main navigation menu, but not on the profile page because they want to see all their badges nicely displayed. Most users are fine with the app automatically switching to dark mode when it gets dark.</em></p><h4>10. How long will our solution take, and are there ways to implement it more cheaply?</h4><p>Now that you’ve decided on an overall approach and drafted the product spec, it’s time to check the price. Ask your engineering manager for a high-level estimate of how long it will take. You don’t need granular task-by-task estimates, just an answer on the order of weeks/months.</p><p>If the initial implementation will take too long, use the <a href="https://asana.com/resources/pareto-principle-80-20-rule">80–20 rule</a> to be ruthless in cutting scope. Most of the value can probably be delivered by a minimal solution, with nice-to-have requirements left for a future version.</p><p><em>Me: How long will our solution take, and are there ways to implement it more cheaply?</em></p><p><em>Tech lead: If we did the full thing, it would take about 6–8 weeks with regression testing and app store release. If we didn’t implement automatic nighttime detection, that would probably cut it down to 3–4 weeks.</em></p><h3>Execution &amp; Reflection</h3><p>Once you have a project nicely designed and scoped out, it’s time to start building!</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2F3ohjUWwFJ9y81RO69q%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2F3ohjUWwFJ9y81RO69q%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2F3ohjUWwFJ9y81RO69q%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="244" frameborder="0" scrolling="no"><a href="https://medium.com/media/382d8892b965ffa042b0723679a98d11/href">https://medium.com/media/382d8892b965ffa042b0723679a98d11/href</a></iframe><h4>11. Does everyone understand what they need to do?</h4><p>It’s helpful to have a quick kickoff meeting with your team at the start of development for a new feature where you present the spec and discuss outstanding questions. It’s really important here to <strong>check for understanding,</strong> and can also be helpful to appoint a feature owner, usually the dev with the most tasks for the feature.</p><p><em>Me: Does everyone understand what they need to do?</em></p><p><em>Mobile dev: Actually there’s an edge case we need to consider — what if the user receives a system alert in light mode?</em></p><h4>12. How will we communicate this feature to our colleagues and users?</h4><p>Some user-facing changes will require you to communicate this change to your users — that PR/FAQ you wrote for Question 6 might come in handy. It could be as simple as a paragraph in a monthly email, or a full-on website rebuild. You should also let internal teams who will be affected by your release know well in advance.</p><p><em>Me: How will we let people know about this feature?</em></p><p><em>Marketing: It would be cool to put some of these screenshots on our rider landing page, and we can also add it to our monthly product update email.</em></p><h4>13. What does the initial data say?</h4><p>After you’ve finished development and (hopefully) released the feature without too much pain, check the stats right away. Often there won’t be any because you’ve forgotten to add some crucial event, or encounter a weird bug. Once you’re sure it’s working correctly, ease off until you have a meaningful volume of data (If you’re running an A/B test, this will be once you’ve reached statistical significance).</p><p><em>Me: What does the initial data say?</em></p><p><em>Analyst: Uptake on iOS has been super positive, half our drivers are using it! Android is about 10% lower, seems we might have missed something in our design?</em></p><h4>14. What are the next steps? How can we make this better?</h4><p>This is the easy part! With the answer to the previous question, you’ve usually got a pretty clear idea of how to improve things.</p><p><em>Me: Good job everyone! Let’s think about the next version now, how do want to make it better?</em></p><p><em>Designer: I want to change the highlight colour.</em></p><p><em>Analyst: We need to fix a bug with Android.</em></p><p><em>Engineer: There are some tech debt tasks we should finish.</em></p><p>Aaaaaaaaand Cut!</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2Fl2SqcHB29t1OuwjCw%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2Fl2SqcHB29t1OuwjCw%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2Fl2SqcHB29t1OuwjCw%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="244" frameborder="0" scrolling="no"><a href="https://medium.com/media/89884d8215d658da4aeac829d2b7c48a/href">https://medium.com/media/89884d8215d658da4aeac829d2b7c48a/href</a></iframe><h3>Parting thoughts</h3><p>In this article we’ve seen how these 14 questions and the answers they elicited have brought structure to a messy problem and helped to define a working solution.</p><p>As we’ve seen, it’s more important to ask the right question than to have the right answer. You’ll often discover hidden assumptions or gaps in your knowledge that are way more significant than you initially thought, and this learning can itself be a joy.</p><p>Asking a question is also a crucial tool for collaboration — by admitting you don’t know something, you’re inviting others to help you work together towards a solution. It helps people feel that you’re not micromanaging them and that their expertise is valued.</p><p>So, what did you learn from this piece?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bf7382c75928" width="1" height="1" alt=""><hr><p><a href="https://medium.com/design-bootcamp/14-questions-to-ask-as-a-product-manager-bf7382c75928">14 questions to ask as a Product Manager</a> was originally published in <a href="https://medium.com/design-bootcamp">Bootcamp</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting sh*t done as a manager]]></title>
            <link>https://johnmicahreid.medium.com/getting-sh-t-done-as-a-manager-87dde617225a?source=rss-4514eefa6d8b------2</link>
            <guid isPermaLink="false">https://medium.com/p/87dde617225a</guid>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[communication]]></category>
            <category><![CDATA[task-management]]></category>
            <dc:creator><![CDATA[John Micah Reid]]></dc:creator>
            <pubDate>Mon, 29 Aug 2022 18:42:30 GMT</pubDate>
            <atom:updated>2022-09-13T10:28:53.922Z</atom:updated>
            <content:encoded><![CDATA[<p>A personal organization system for managing myself and my team</p><h3>The problem</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9tigj9c9sCN2WmSXhQOHvw.png" /><figcaption>I made this in Powerpoint for you</figcaption></figure><p>As a new product manager, I felt constantly overwhelmed by the flood of incoming requests, ideas, tasks, meetings, messages, and more. I’d end every day completely exhausted, and inevitably would have forgotten to finish something, or reply to someone. The admin felt relentless, and left me feeling stressed and ineffective, like I wasn’t doing any ‘real’ work.</p><p>In a way, this is the system working as intended — it’s a manager’s job to bear the <a href="https://english.emmaclit.com/2017/05/20/you-shouldve-asked/">mental load</a> of being aware of everyone’s tasks and responsibilities and handling distractions so that your team can get on with the work of actually building features/fixing bugs etc. (Paul Graham has a great write-up on this in his article <a href="http://www.paulgraham.com/makersschedule.html">Maker’s Schedule, Manager’s Schedule</a>).</p><p>But in order to manage your team well, you first need to manage yourself. If you don’t have a system to do this, you’ll find yourself spread too thin and get burned out. And that doesn’t help anyone.</p><h3>A solution</h3><p>In this article we’ll go through a simple organization system to keep track of daily tasks. Conceptually, it’s a very simplified version of the <a href="https://en.wikipedia.org/wiki/Getting_Things_Done">Getting Things Done</a> framework, and it can be summarized in three basic principles:</p><ol><li>Write things down</li><li>Prioritize</li><li>Keep it simple</li></ol><p>The primary goal is to reduce the number of things to remember, because we can only hold <a href="https://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two">5–7 things in our short term memory</a> at once. Anyone who claims to be a brilliant multi-tasker is <a href="http://who claims to be a brilliant multi-tasker">deceiving themselves</a>, and context switching from task to task<strong> </strong>is one of the fastest ways to turn you into a drooling zombie staring at the same screen for minutes on end.</p><h3>How it works</h3><p>I use an outliner called <a href="http://workflowy.com/">Workflowy</a>, which is an infinite bullet/toggle list (I have no affiliation with them, just love their product). You can use any similar note-taking app. Here’s the overall structure:</p><pre>- Year<br>   - Month<br>      - Day<br>         - Scheduled Meetings<br>         - Tasks<br>      - Later</pre><p>I start each workday by creating a new bullet point for that day and put in all my meetings and tasks (hint: meetings count as tasks!). This is a great way to center myself at the beginning of the day, see what is coming up and decide on task priorities.</p><p>Here’s how it looks first thing in the morning:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1eIKzB6-otfVMP0durcQyw.png" /></figure><p>As the day progresses, and incoming tasks/requests come in, I mark items complete, add/delete items, or move them around. This is halfway through my day:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*awuF3ENSMnw3p_lBlvI_Aw.png" /></figure><p>You can see that an extra meeting has been added, some tasks were completed, and new tasks were added. At the end of the day, I close off the previous day and move over tasks to the next day (Workflowy makes this super easy by allowing you to collapse each bullet point).</p><p>Each item crossed off provides a little dopamine hit that keeps me going, and it’s an amazing feeling to get to the end of the day and have everything ticked off. I usually get through about 80% on a normal day, being careful not to get into <a href="https://paperform.co/blog/toxic-productivity/">toxic productivity</a> (creating and finishing tasks just for the sake of it).</p><h3>An explanation of the principles</h3><p>Now that we’ve seen the basic idea, let’s dive deeper into the principles behind it:</p><h4><strong>1) Write things down</strong></h4><p>The scarcest resource for most knowledge workers is not time but attention. You only have a finite amount of high-quality attention to spend each day, and you don’t want to waste it on trying to remember random tasks to do later.</p><p>We can only actually focus on one thing at a time, so writing down incoming requests/ideas/tasks frees up your short-term memory to focus on your immediate task. Therefore, write down anything that you’d spend effort on remembering, even if it’s replying to an email or Slack message.</p><p>Tasks should also not to be too big and should take max 4 hours to complete. If it’s part of a bigger project e.g. Write product spec for new feature , then I tend to break it into several separate tasks Start product specand Finish prodoct spec . I also put personal errands in e.g. Call mom because otherwise I’d need to spend brain space to remember it.</p><p>Once it’s written down, you can review the list later and decide what to work on next.</p><h4><strong>2. Prioritize</strong></h4><p>Usually I’ll look at my daily tasks and realize that not everything needs to be done today. I’ll work on the shortest tasks first just to clear the board, which just leaves a few big, nebulous tasks like “Send team thoughts on next year’s roadmap”.</p><p>If I know I don’t have capacity to work on something right away, I’ll create a <strong>reminder loop — </strong>basically anything that will resurface the task for me at a future date, such as:</p><ul><li>Creating tomorrow’s bullet point in the to-do list and moving the task there</li><li>Sending a message/meeting invite to talk about it with someone else</li><li>Jotting down the task/idea in a long-running backlog elsewhere</li></ul><p>That’ll let me clear the task knowing that I have a plan to deal with it later. Of course, you can’t just keep kicking the can of important-but-not-urgent things down the road, so make sure to get around to it on a quieter day.</p><h4><strong>3) Keep it simple</strong></h4><p>Whatever system you use needs to be as simple as possible — for me that means I’m able to record a task in mid-conversation without interrupting the flow, within 3–5 seconds. I also need to have one source of truth across all my work systems (calendar, email, Slack, docs and conversations) and personal life, because this personal org system needs to last between projects, roles and companies.</p><p>I’ve used digital tools like Trello, Asana, and JIRA in the past, and all of these have their place. I often use them when collaborating with other people on large complex projects but I find them too heavyweight for my personal tasks. The simplicity of Workflowy is actually really helpful, because I don’t see any other people, reminders, other systems etc. It’s just my daily to-do list.</p><h3>In Conclusion</h3><p>Having a simple system like this has helped me to relax at work, knowing that I don’t need to be a superhuman keeping perfectly on top of everything.</p><p>The deeper lesson I’ve learned through this is to <strong>be more honest and kind to myself. </strong>We’re all human (just ‘monkeys in suits’ as a colleague once told me), and we all have our physical and mental limitations. Recognizing that I can’t remember everything just in my head, and I can’t work on more than one thing at a time is not a weakness but a strength. Paradoxically, that’s also made me way more productive and impactful.</p><p>Thanks for reading, and I hope this has been helpful!</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2F0ghvTK7d0nWF4EthHo%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2F0ghvTK7d0nWF4EthHo%2Fgiphy-downsized-large.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2F0ghvTK7d0nWF4EthHo%2Fgiphy-downsized-large.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="308" frameborder="0" scrolling="no"><a href="https://medium.com/media/08b014c8293ba0d37655a2d072b248d3/href">https://medium.com/media/08b014c8293ba0d37655a2d072b248d3/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=87dde617225a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[3 things to avoid saying as a Product Manager]]></title>
            <link>https://johnmicahreid.medium.com/3-things-to-avoid-saying-as-a-product-manager-4b0d37e9e56?source=rss-4514eefa6d8b------2</link>
            <guid isPermaLink="false">https://medium.com/p/4b0d37e9e56</guid>
            <category><![CDATA[how-to]]></category>
            <category><![CDATA[communication]]></category>
            <category><![CDATA[product-management]]></category>
            <dc:creator><![CDATA[John Micah Reid]]></dc:creator>
            <pubDate>Mon, 09 May 2022 12:56:23 GMT</pubDate>
            <atom:updated>2022-05-09T12:56:23.685Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/375/0*R0c0WHv2WV0V0ADA.jpg" /><figcaption><a href="https://www.memecreator.org/meme/i-cant-believe-i-just-said-that">Image</a> from MemeCreator</figcaption></figure><p>Do you ever lie awake at night thinking of something dumb you said and wishing you could take it back? I’ve done this plenty of time, often in a work context and especially as a junior product manager.</p><p>In this article we’ll go over three scenarios where I made communication mistakes and then review how they could have gone better to help you avoid some common PM pitfalls. Let’s get to it!</p><h3>1. Saying Yes straightaway</h3><p>Here’s a conversation I remember from my first month as a new product manager:</p><blockquote>CEO: Hey John, I was thinking recently about our onboarding funnel, and I think all our new signups should watch a video so they understand how to use our product. Can you work on this for me?</blockquote><blockquote>Me: *oh my gosh, it’s the CEO* Yes, that sounds like a great feature. I’ll get to work on it right now and will hopefully have something for you to try in 2 weeks.</blockquote><p>After the conversation, I told my tech team to drop everything they were working on and start building a video onboarding solution. 2 weeks passed and my team was getting frustrated that we had made zero progress, I realized it would take another few months to get something actually working. I went back to the CEO to tell him this, and he said “Oh, you didn’t have to build it now. I just wanted to tell you my idea.” 🙄</p><p>What should I have said instead? Here’s how I would replay the conversation with a couple more years of experience:</p><blockquote>CEO: Hey John, I was thinking recently about our onboarding funnel, and I think all our new signups should watch a video so they understand how to use our product. Can you work on this for me?</blockquote><blockquote>Me: That’s a great idea! Can I just check, do we have any evidence that our current onboarding is a problem? Also, just to inform you, our team is currently working on Feature X - would you like us to drop that and prioritize this?</blockquote><blockquote>CEO: Well, I actually haven’t heard any complaints, I just thought it was a cool idea. Feature X is more important, definitely finish that first.</blockquote><p>What I’ve done here is a) acknowledge the idea, b) dig deeper into the context and problem, and c) ask the CEO to prioritize. It turns out that this isn’t actually an important problem to solve right now and our team can get on with completing our existing work.</p><p>Takeaway: don’t say yes to anything right away before you’ve had a chance to <strong>validate if it’s solving an actual problem</strong>. Even if it’s coming from the CEO.</p><h3>2. Saying No straightaway</h3><p>Remembering my previous mistake, I spent the next few months focusing very intensely on our roadmap, leading to this conversation:</p><blockquote>Operations team member: Hey, I wrote this doc about an idea to improve user retention, basically we can send emails to —</blockquote><blockquote>Me: (interrupting) Sorry, we have too much on our roadmap right now and besides, emails aren’t really my team’s responsibility.</blockquote><blockquote>Ops: Oh, okay. Sorry to bother you.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/406/0*6twd5XsFxvmOkn1f.jpg" /><figcaption>Image from MemeGenerator</figcaption></figure><p>A few months later, user retention surfaced as a major issue in the company. I spent some time researching the problem and sketching out a solution, and happened to check the doc my team member had sent me. She described almost the exact system that I was proposing. In this case, the consequences weren’t as drastic (focus is almost always a good thing), but it did come with a relationship cost and opportunity cost.</p><p>In the two examples I’ve given so far, you’ll notice that my initial response ended the conversation. This is often a mistake, as you miss out valuable details and ideas. Asking questions is a great way to clarify meaning and extend the conversation — let’s see how it could work in this scenario.</p><blockquote>Operations team member: Hey, I wrote this doc about an idea to improve user retention - basically we can send emails to users who haven’t been active for a certain time.</blockquote><blockquote>Me: That’s a really interesting idea, why do you think it’ll make a difference?</blockquote><blockquote>Ops: We’ve already tried with this one email manually and found that it led to a 5% increase in user retention. To roll it out fully, we’ll need support from the product team to send these out automatically.</blockquote><blockquote>Me: Those are pretty good results, and I’ll make sure to review your doc. Just a heads-up that my team is busy with several other projects right now, but I’ll discuss this with the other product teams and let you know.</blockquote><p>Note that I still haven’t said yes to this idea, but I have acknowledged it and now have the opportunity to review it at leisure. By asking questions and being open to ideas, you build a much deeper understanding of your field and are more likely to come up with effective solutions. This is also a good way to build relationships across your organization, which always come in handy — as a PM you’re constantly relying on other people to get things done.</p><h3>3. Giving tasks, not context</h3><p>A final mistake is one I often still make, especially when I feel busy or rushed for time. In this case I was trying to find out where our new signups were coming from:</p><blockquote>Me: Hey analyst, can you estimate what percentage of our app signups found our website on Google Search?</blockquote><blockquote>Analyst: Mmm, it’s tricky because they are different data sources. What is the max amount of time from viewing the website to installing the app that we count as a valid conversion?</blockquote><blockquote>Me: I don’t know, 24 hours maybe?</blockquote><blockquote>Analyst: What if someone used different devices?</blockquote><blockquote>Me: *gosh, this is getting complicated* …</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*X1_WX7NYknomqlyi.jpg" /><figcaption>Image from <a href="https://makeameme.org/meme/prepare-yourselves-tasks">makeameme</a></figcaption></figure><p>We continued back and forth for several minutes and came up with a wonky but working definition. Presenting this data to my manager later, he said, “But why did you come up with this whole new estimate? We just ask new users where they came from during signup.”</p><p>This is what is known as an <a href="https://en.wikipedia.org/wiki/XY_problem">XY problem</a>, meaning the end user asks for a solution to problem X, which is actually caused by problem Y. In this case, I didn’t share <strong>why I needed the data, or what problem I was trying to solve, </strong>meaning we wasted time on the wrong thing.</p><p>Giving context upfront makes it easier for others to help you because they spend less effort figuring out what you really want. Let’s include that and see how the conversation goes:</p><blockquote>Me: Hey analyst, I’m trying to figure out where our users are coming from so we can help our marketing team prioritize their budget across different channels. Could you tell me what information we have on app signup sources and how we could estimate it?</blockquote><blockquote>Analyst: Sure, we ask people for that information directly during signup — here’s an existing dashboard on it.</blockquote><p>In this case, I’ve gotten what I need more quickly because I shared more about my underlying goal. Making the request open-ended and asking the other person to think of potential solutions also taps into their creativity. If you’re only ever a task-maker, your team will be limited by your own understanding and imagination.</p><h3>Conclusion</h3><p>What unites a lot of these examples is that they were caused by a<strong> lack of thoughtfulness </strong>in communication. As normal people, we communicate in a hurried way all the time and it’s fine. However, the bar for product managers is a lot higher, because communication is literally 90% of the job. Good communication requires taking the time to write (or say) a thoughtful response that considers the other person and the situation at hand. And it can turn a confused set of individuals building random features, into an energized team solving real problems.</p><p>I’ve found it helpful to think of communication as a <a href="https://www.thebalancecareers.com/hard-skills-vs-soft-skills-2063780">hard skill</a> for PMs , meaning that a) it can be quantified, and b) it can be taught. By not saying yes to anything straightaway, asking questions, being open to ideas and sharing context, you’re well on your way to becoming a more effective product manager.</p><h4>Bonus: Recommended Reading</h4><p>If you’re interested in learning how to improve your communication skills, see my two favourite books on the topic below. As with all great books, their lessons can be applied in many areas of your life.</p><ol><li><a href="https://www.amazon.co.uk/How-Win-Friends-Influence-People/dp/0091906814">How to Win Friends and Influence People</a> by Dale Carnegie</li><li><a href="https://www.amazon.co.uk/Difficult-Conversations-Discuss-What-Matters/dp/0670921343/r">Difficult Conversations</a> by Douglas Stone and Sheila Heen</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4b0d37e9e56" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Sentiment Analysis report using NLTK and Altair]]></title>
            <link>https://medium.com/data-science/building-a-sentiment-analysis-interactive-report-using-nltk-and-altair-83cb9fcb36fe?source=rss-4514eefa6d8b------2</link>
            <guid isPermaLink="false">https://medium.com/p/83cb9fcb36fe</guid>
            <category><![CDATA[report]]></category>
            <category><![CDATA[sentiment-analysis]]></category>
            <category><![CDATA[nltk]]></category>
            <category><![CDATA[news]]></category>
            <category><![CDATA[altair]]></category>
            <dc:creator><![CDATA[John Micah Reid]]></dc:creator>
            <pubDate>Fri, 12 Nov 2021 14:00:05 GMT</pubDate>
            <atom:updated>2021-11-15T10:31:41.207Z</atom:updated>
            <content:encoded><![CDATA[<h4>Using the UCI News dataset we’ll calculate positivity of news headlines, visualize the results and package them into an interactive report</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1SmNYd1MtrW5TtRno5ruQg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@glencarrie">Glen Carrie</a> on <a href="https://unsplash.com/photos/oHoBIbDj7lo">Unsplash</a></figcaption></figure><p><em>(Disclaimer: I work as a product manager at Datapane)</em></p><p>Data analysis often starts with <strong>structured data </strong>that’s<strong> </strong>already stored as numbers, dates, categories etc. However, unstructured data can yield crucial insights if you use appropriate techniques. In this tutorial, we&#39;ll run sentiment analysis on a textual dataset to calculate positive/negative sentiment, and turn the results into an interactive report.</p><h3>Running sentiment analysis</h3><p>Let’s imagine we’re a data scientist working for a news company and we’re trying to figure out how ‘positive’ our news headlines are in comparison to the industry. We’ll start with the <a href="https://www.kaggle.com/uciml/news-aggregator-dataset">UCI News Aggregator dataset</a> (<a href="https://creativecommons.org/publicdomain/zero/1.0/">CC0: Public Domain</a>) [1] which is a collection of news headlines from different publications in 2014. This is a fun dataset because it has articles from a wide range of publishers and contains useful metadata.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1f94d9ff94825ce48bf13c781c9addbd/href">https://medium.com/media/1f94d9ff94825ce48bf13c781c9addbd/href</a></iframe><p>After downloading and cleaning up the data, we get the following result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6xK9VG0egNnCx_WERYyGdw.png" /><figcaption>Image by author</figcaption></figure><p>We have 8 columns and about 400k rows. We’ll use the ‘Title’ for the actual sentiment analysis, and group the results by ‘Publisher’, ‘Category’ and ‘Timestamp’.</p><h4><strong>Classifying the headlines</strong></h4><p>Through the magic of open-source, we can use someone else’s hard-earned knowledge in our analysis — in this case a pretrained model called the Vader Sentiment Intensity Analyser from the popular <a href="https://www.nltk.org/index.html">NLTK</a> library.</p><p>To build the model, the authors gathered a list of common words and then asked a panel of human testers to rate each one on <strong>valence </strong>i.e. positive or negative, and <strong>intensity </strong>i.e. how strong the sentiment is. As the <a href="http://comp.social.gatech.edu/papers/icwsm14.vader.hutto.pdf">original paper</a> says: :</p><blockquote>[After stripping out irrelevant words] this left us with just over 7,500 lexical features with validated valence scores that indicated both the sentiment polarity (positive/negative), and the sentiment intensity on a scale from –4 to +4. For example, the word “okay” has a positive valence of 0.9, “good” is 1.9, and “great” is 3.1, whereas “horrible” is –2.5, the frowning emoticon “:(” is –2.2, and “sucks” and “sux” are both –1.5.</blockquote><p>To classify a piece of text, the model calculates the valence score for each word, applies some grammatical rules e.g. distinguishing between ‘great’ and ‘not great’, and then sums up the result.</p><p>Interestingly, this simple lexicon-based approach has equal or better accuracy compared to machine-learning approaches, and is much faster. Let’s see how it works!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b6af1aef222ed3f00a2c5ceb04799a90/href">https://medium.com/media/b6af1aef222ed3f00a2c5ceb04799a90/href</a></iframe><p>In this code we import the library, classify each title in our dataset then append the results to our original dataframe. We have added 4 new columns:</p><ul><li><strong>pos</strong>: positive score component</li><li><strong>neu</strong>: neutral score component</li><li><strong>neg</strong>: negative score component</li><li><strong>compound</strong>: the sum of the three score components</li></ul><p>As a sanity check, let’s take a look at the most positive, neutral and negative headline in the text by using pandas idxmax :</p><pre>negative = df.iloc[df.neg.idxmax()]<br>neutral = df.iloc[df.neu.idxmax()]<br>positive = df.iloc[df.pos.idxmax()]</pre><pre>print(f&#39;Most negative: {negative.TITLE} ({negative.PUBLISHER})&#39;)<br>print(f&#39;Most neutral: {neutral.TITLE} ({neutral.PUBLISHER})&#39;)<br>print(f&#39;Most positive: {positive.TITLE} ({positive.PUBLISHER})&#39;)</pre><p>Running that code gives us the following result:</p><pre>Most negative: I hate cancer (Las Vegas Review-Journal \(blog\))<br>Most neutral: Fed&#39;s Charles Plosser sees high bar for change in pace of tapering (Livemint)<br>Most positive: THANK HEAVENS (Daily Beast)</pre><p>Fair enough — ‘THANKS HEAVENS’ is a lot more positive than ‘I hate cancer’!</p><h3>Visualizing the results</h3><p>What does the distribution of our scores look like? Let’s visualize this in a couple of ways using the interactive plotting library <a href="https://altair-viz.github.io/index.html">Altair</a>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d6eb747ac63dca5b916195121e77c35d/href">https://medium.com/media/d6eb747ac63dca5b916195121e77c35d/href</a></iframe><p>Here we’re showing both a histogram for the overall distribution, as well as a 100% stacked bar chart grouped by category. Running that code, we get the following result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cM8fnztmpLYn4DHGMGtluA.png" /><figcaption>Image by Author</figcaption></figure><p>Seems like most headlines are neutral, and health has overall more negative articles than the other categories.</p><p>To give more insight into how our model is classifying the articles, we can create two more plots, one showing a sample of how the model classifies particular headlines, and another showing the average sentiment score for our largest publishers over time:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f6d06113db0d39d1af2b5854b7504d31/href">https://medium.com/media/f6d06113db0d39d1af2b5854b7504d31/href</a></iframe><p>This is where the declarative syntax of Altair really shines — we just change a few of the keywords e.g. mark_bar to mark_pointand we get a completely different yet still meaningful result:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fdatapane.com%2Fu%2Fjohnmicahreid%2Freports%2F0keR9XA%2Fscatter-and-line-plots%2Fembed%2F&amp;display_name=Datapane&amp;url=https%3A%2F%2Fdatapane.com%2Fu%2Fjohnmicahreid%2Freports%2F0keR9XA%2Fscatter-and-line-plots%2F&amp;image=https%3A%2F%2Fdatapane.com%2Fstatic%2Fpython_report_illustration.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=datapane" width="960" height="700" frameborder="0" scrolling="no"><a href="https://medium.com/media/d865042313a61fe9a93dbaed2a160626/href">https://medium.com/media/d865042313a61fe9a93dbaed2a160626/href</a></iframe><p>By creating interactive visualizations, you enable viewers to explore the data directly. They’ll be much more likely to trust your overall conclusions if they can drill down to the original datapoints.</p><p>Looking at the publishers chart it seems that HuffPost is consistently more negative and RTT more positive. Hmmm, seems like they have different editorial strategies…</p><h3>Creating a report</h3><p>The final step is to package the results into an interactive report. As data scientists, we often forget to <strong>communicate our results effectively</strong>. I’ve often made the mistake of spending hours analysing data to answer somebody’s question then sending over a screenshot of a chart with a one-line explanation. My viewer then doesn’t understand the results and may not use them to actually make decisions.</p><p>Rule #1— always assume someone looking at your work has zero background and needs to understand from scratch. It’s always worth spending extra time and effort to write up the <strong>context and implications </strong>of your work.</p><p>For this tutorial we’ll use a library called <a href="https://datapane.com/"><strong>Datapane</strong></a><strong> </strong>to create a shareable report from these interactive visualizations. To do this, we’ll first need to create an account on Datapane, wrap our charts inside Datapane blocks and then upload the report.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/19815ff7cdb3ed71bb385e22acaa0f65/href">https://medium.com/media/19815ff7cdb3ed71bb385e22acaa0f65/href</a></iframe><p>We write chunks of text in Markdown format to give context, interspersed with our actual plots and data. You can see an embedded version of the chart with interactive visualizations here:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fdatapane.com%2Fu%2Fjohnmicahreid%2Freports%2FOkp4JXk%2Fdistribution-of-sentiment%2Fembed%2F&amp;display_name=Datapane&amp;url=https%3A%2F%2Fdatapane.com%2Fu%2Fjohnmicahreid%2Freports%2FOkp4JXk%2Fdistribution-of-sentiment%2F&amp;image=https%3A%2F%2Fdatapane.com%2Fstatic%2Fpython_report_illustration.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=datapane" width="960" height="700" frameborder="0" scrolling="no"><a href="https://medium.com/media/c27c35e8ea0b60da938fd43fddd3ac84/href">https://medium.com/media/c27c35e8ea0b60da938fd43fddd3ac84/href</a></iframe><p>This is a minimal report that you could share with stakeholders (your boss, your mom etc) to show overall sentiment across the media landscape. From here you can explore comparisons across time, industry and publishers, with the goal of making recommendations for how your organization can improve.</p><h3>Conclusion</h3><p>We can summarize what we’ve learned in this tutorial through three points:</p><ol><li>Use <strong>sentiment analysis</strong> to extract value from unstructured text</li><li>Build charts in <strong>interactive libraries</strong> like Altair<strong> </strong>so your viewers can explore the data themselves, reinforcing your overall conclusions</li><li>Spend extra time and effort writing up your results into a <strong>report with context</strong></li></ol><p>Now go get em!</p><p>[1] Lichman, M. (2013). UCI Machine Learning Repository [<a href="http://archive.ics.uci.edu/ml]">http://archive.ics.uci.edu/ml]</a>. Irvine, CA: University of California, School of Information and Computer Science.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=83cb9fcb36fe" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/building-a-sentiment-analysis-interactive-report-using-nltk-and-altair-83cb9fcb36fe">Building a Sentiment Analysis report using NLTK and Altair</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[13 ways to access data in Python]]></title>
            <link>https://medium.com/data-science/13-ways-to-access-data-in-python-bac5683e0063?source=rss-4514eefa6d8b------2</link>
            <guid isPermaLink="false">https://medium.com/p/bac5683e0063</guid>
            <category><![CDATA[dataset]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[John Micah Reid]]></dc:creator>
            <pubDate>Thu, 14 Oct 2021 13:29:23 GMT</pubDate>
            <atom:updated>2021-10-14T13:29:23.541Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to get data from local files, databases, APIs and data access libraries into your Python environment</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tYWM6La6f02Sh5AOTQMAmw.jpeg" /><figcaption>Image by Lance Grandahl on <a href="https://unsplash.com/photos/nShLC-WruxQ">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>Most Python analysis starts by importing data into your environment. But what if that data is stuck in a database? Or behind an API? Or in a bunch of tiny files?</p><p>Luckily, Python is incredibly flexible and has a lot of open-source libraries for accessing and processing data. In this tutorial we’ll look at 13 methods for getting data into a pandas Dataframe, after which it can be cleaned, analysed and visualized. We can group the methods into 4 main categories:</p><ol><li>Local files</li><li>Databases</li><li>APIs</li><li>Dataset access libraries</li></ol><p>The only major requirement is installing the pandas library:</p><pre>$ pip install pandas</pre><p>With that, let’s get cracking!</p><h3>📁 Local files</h3><p>Often the data you need is stored in a local file on your computer. Depending on where you’re running your Python environment, you can either specify the filename as a <a href="https://www.earthdatascience.org/courses/intro-to-earth-data-science/python-code-fundamentals/work-with-files-directories-paths-in-python/">relative or absolute path</a>:</p><pre># Absolute path<br>file1 = &quot;~/Users/johnreid/Documents/my_project/data/example.csv&quot;</pre><pre># Relative path, assuming current working directory is my_project<br>file2 = &quot;./data/example.csv&quot;</pre><h4>1. CSV files</h4><p>CSVs are a popular choice for storing tabular data, and the simplest way to get started. Let’s assume you’ve downloaded this population dataset from<a href="https://ourworldindata.org/grapher/population-by-country"> Our World in Data</a>:</p><pre>import pandas as pd</pre><pre>csv_file = &quot;/Users/johnreid/Downloads/population-by-country.csv&quot;<br>df_from_csv = pd.read_csv(csv_file)<br>df_from_csv.info()</pre><p>After importing the data, it’s helpful to run df.info() to understand how your data is structured e.g. how many rows, columns and non-null values you have. Running that code gives us the following output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*d4s8_fQhg1dGrbUt" /></figure><p>This method also works for files accessible by URLs, like a public Google Sheet or CSV file in a public Github repo. Also, if you keep getting aFileNotFoundError then try renaming your filename to replace spaces with underscores e.g. &quot;Financial Sample.xlsx&quot; becomes &quot;Financial_Sample.xlsx&quot;.</p><h4>2. Excel files</h4><p>You need to be a bit more cautious with Excel files, because they may contain more than one sheet of data and complex visual formatting e.g. extra header rows. Otherwise the syntax is pretty similar — here’s a <a href="https://go.microsoft.com/fwlink/?LinkID=521962">financial data</a> example:</p><pre>import pandas as pd</pre><pre>excel_file = &quot;/Users/johnreid/Downloads/Financial_Sample.xlsx&quot;<br>df_from_excel = pd.read_excel(excel_file, sheet_name = &quot;Sheet1&quot;)<br>df_from_excel.info()</pre><h4>3. Text files</h4><p>Text files often need more data processing — start by looking at how the data is stored and how you’d like to represent it in Python. From there, you can write code to transform textual input into a dataframe. Let’s use a shopping list example, with each line containing an item and a quantity:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FMBg56mcfWMB1t8p" /></figure><p>To convert that to a dataframe, you can run the following:</p><pre>shopping_list = &quot;/Users/johnreid/Downloads/shopping_list.txt&quot;</pre><pre>results = []</pre><pre>with open(shopping_list) as f:<br>    line = f.readline()</pre><pre>    while line:<br>        results.append(line.strip().split(&quot; &quot;))<br>        line = f.readline()</pre><pre>f.close()</pre><pre>df_from_textfile = pd.DataFrame(results, columns = [&quot;Item&quot;, &quot;Quantity&quot;])</pre><p>We read the lines one-by-one, strip extra whitespaces and split the line into two parts. When we create a dataframe, we also need to assign column names.</p><h4>4. Multiple files / folders</h4><p>What happens if you need to extract data from multiple stored files? Let’s combine a couple of things that we’ve learned to extract data from the <a href="http://mlg.ucd.ie/datasets/bbc.html">BBC Sport text dataset</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aBaD94EclppmxPA4" /></figure><p>We have 5 subfolders, each with around 100 files. Each file starts with a headline, followed by the body of the article. Our goal will be to combine all these files into a single dataframe with ‘Title’, ‘Subtitle’, ‘Body’ and ‘Genre’ columns. The glob library comes really in handy here to list all possible filenames:</p><pre>import glob<br>import pandas as pd</pre><pre>base_path = &quot;/Users/johnreid/Downloads/bbcsport/&quot;<br>genres = [&quot;athletics&quot;, &quot;cricket&quot;, &quot;football&quot;, &quot;rugby&quot;, &quot;tennis&quot;]</pre><pre>def read_and_split_file(filename):<br>    with open(filename, &#39;r&#39;, encoding=&quot;latin-1&quot;) as f:<br>        lines = f.readlines() # Get lines as a list of strings<br>        lines = list(map(str.strip, lines)) # Remove /n characters<br>        lines = list(filter(None, lines)) # Remove empty strings</pre><pre>    return lines</pre><pre>def get_df_from_genre(path, genre):<br>    files = glob.glob(path + genre + &quot;/*.txt&quot;)<br>    titles = []<br>    subtitles = []<br>    bodies = []</pre><pre>    for f in files:<br>        lines = read_and_split_file(f)<br>        titles.append(lines[0]) # First line is the title<br>        subtitles.append(lines[1]) # Second line is the subtitle<br>        bodies.append(&#39; &#39;.join(lines[2:])) # Combine all the rest</pre><pre>    return(pd.DataFrame({<br>        &#39;genre&#39;: genre,<br>        &#39;title&#39;: titles,<br>        &#39;subtitle&#39;: subtitles,<br>        &#39;body&#39;: bodies<br>        })<br>    )</pre><pre>final_df = pd.concat([get_df_from_genre(base_path, g) for g in genres])</pre><pre>final_df</pre><p>We use the * operator with glob to get all possible filenames ending in .txt. Note that you can concatenate multiple dataframes together using pd.concat. Running that code gives us the following output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DMVCrpt-MdsotIxv" /></figure><h3>🗄 Databases</h3><p>Most organizations store their business-critical data in a <a href="https://en.wikipedia.org/wiki/Relational_database">relational database</a> like Postgres or MySQL, and you’ll need to know <strong>S</strong>tructured <strong>Q</strong>uery <strong>L</strong>anguage (SQL) to access or update the data stored there. Databases have a number of advantages, like data normaliza</p><h4>5. SQLite</h4><p>SQLite is an embedded database that is stored as a single file, so it’s a great place to start testing out queries. Here we’ll show an example of connecting to a SQLite file of the <a href="https://github.com/lerocha/chinook-database">Chinook</a> database:</p><pre>import pandas as pd<br>import sqlite3 as sql</pre><pre>conn = sql.connect(&#39;/Users/johnreid/Downloads/chinook.db&#39;)</pre><pre># First pattern - turn query directly into dataframe:<br>df1 = pd.read_sql_query(&quot;SELECT * FROM invoice&quot;, conn)</pre><pre># Second pattern - get row-level data, but no column names<br>cur = conn.cursor()<br>results = cur.execute(&quot;SELECT * FROM invoice LIMIT 5&quot;).fetchall()<br>df2 = pd.DataFrame(results)</pre><p>If you’re curious, read my full tutorial on building an interactive dashboard using SQL here:</p><p><a href="https://towardsdatascience.com/building-an-interactive-python-dashboard-using-sql-and-datapane-46bd92294fd3">Building an interactive Python dashboard using SQL and Datapane</a></p><h4>6. Remote databases</h4><p>Connecting to a remote database like Postgres, Redshift, or SQLServer uses mostly the same syntax but requires access credentials. For security reasons, it’s best to store these credentials in a config file and load them into your Python script. You can create a separate .py file with the following info:</p><pre>host = &quot;localhost&quot;<br>database= &quot;suppliers&quot;<br>user = &quot;postgres&quot;<br>password = &quot;SecurePas$1&quot;</pre><p>and then import it into your Python script as follows (you’ll also need the psychopg2 library):</p><pre>import psycopg2<br>import config</pre><pre>conn = psycopg2.connect(<br>    host=config.host,<br>    database=config.database,<br>    user=config.user,<br>    password=config.password)</pre><pre>df1 = pd.read_sql_query(&quot;SELECT * FROM invoice&quot;, conn)</pre><p>Make sure to keep your config.py file safe and don&#39;t upload it elsewhere - you can add it to your .gitignore to make sure it doesn&#39;t get included in git commits.</p><h4>7. SQLAlchemy</h4><p>If you want a more ‘pythonic’ way of querying a database, try the <a href="https://www.sqlalchemy.org/">SQLAlchemy</a> library, which is an Object-Relational-Mapper. It’s typically used for applications so that developers don’t have to write pure SQL to update their database, but you can use it for querying data too!</p><p>Here’s an example using the same Chinook music store database:</p><pre>import sqlalchemy as db</pre><pre>engine = db.create_engine(&#39;sqlite:///chinook.db&#39;)<br>connection = engine.connect()<br>metadata = db.MetaData()</pre><pre>invoice = db.Table(&#39;invoice&#39;, metadata, autoload=True, autoload_with=engine)</pre><pre># Get the first 10 invoices from the USA<br>query = (db.select([invoice])<br>    .filter_by(billing_country = &#39;USA&#39;)<br>    .limit(10)<br>        )</pre><pre>df = pd.read_sql(query, engine)</pre><p>In this code we connect to the database, then set up some tables &amp; metadata in SQLAlchemy. Once that’s defined, we can write a query in a more ‘pythonic’ way and read the results directly to a Pandas dataframe. Running that code gives the following output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ny184a5Zzl-foqOheejl1g.png" /></figure><h3>📶 APIs</h3><p>Sometimes you’ll need to access data from a particular platform your company uses, like Hubspot, Twitter or Trello. These platforms often have a public API that you can pull data from, directly inside your Python environment.</p><p>The basic idea is you send a request (which may include query parameters and access credentials) to an endpoint. That endpoint will return a response code plus the data you asked for (hopefully). You’ll need to look at the API documentation to understand what data fields are available. The data will usually be returned in JSON format, which allows for deeply-nested data.</p><h4><strong>8. Without credentials</strong></h4><p>Let’s do a minimal example using the <a href="http://open-notify.org/Open-Notify-API/People-In-Space/">OpenNotify</a> API, which tracks all the people currently in space:</p><pre>import requests</pre><pre>response = requests.get(&quot;http://api.open-notify.org/astros.json&quot;)</pre><pre>print(response.status_code)<br>print(response.json())</pre><pre>res = pd.DataFrame(response.json()[&quot;people&quot;])</pre><pre>res.head()</pre><p>Running that code gives us the following output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2xCpCjNjI4IFWm1r" /></figure><p>The response code tells you the result of your API call — according to <a href="https://www.dataquest.io/blog/python-api-tutorial/">Dataquest</a> the most common are:</p><ul><li>200: Everything went okay, and the result has been returned (if any).</li><li>301: The server is redirecting you to a different endpoint. This can happen when a company switches domain names, or an endpoint name is changed.</li><li>400: The server thinks you made a bad request. This can happen when you don’t send along the right data, among other things.</li><li>403: The resource you’re trying to access is forbidden: you don’t have the right permissions to see it.</li><li>404: The resource you tried to access wasn’t found on the server.</li><li>503: The server is not ready to handle the request.</li></ul><h4>9. With credentials &amp; query parameters</h4><p>Sometimes you may need more specific information from the API, or have to authenticate. There are several ways to do this, however one of the most common is adding URL parameters to your request.</p><p>Let’s assume we have a config.pyfile with our API key in it:</p><pre>personal_api_key = &quot;wouldntyouliketoknow&quot;</pre><p>Then we create a dictionary for all the parameters (this is a made-up example) and pass it in:</p><pre>import config<br>import pandas as pd<br>import requests</pre><pre>parameters = {<br>    &quot;personal_api_key&quot;: config.personal_api_key, <br>    &quot;date&quot;: &quot;2021-09-22&quot;<br>}</pre><pre>response = requests.get(url, params = parameters)<br>    <br>print(response.status_code)<br>print(response.json())<br>    <br>res = pd.DataFrame(response.json()[&quot;people&quot;])<br>res.head()</pre><p>If you don’t want to deal with JSON you can try searching for a Python library for that API — these are usually open-source and maintained by the company or third parties.</p><h3>📚 Dataset access libraries</h3><p>What if you need some reference data for a comparison or adding context? There are a bunch of libraries for downloading public datasets straight into your environment — think of it as pulling from APIs without having to manage all the extra complexity.</p><h4>10. Pandas_datareader</h4><p><a href="https://pandas-datareader.readthedocs.io/en/latest/index.html">Pandas_datareader</a> is a great way to pull data from the internet into your Python environment. It is particularly suited to financial data, but also has some World Bank datasources. To get Zoom’s daily share price over the past few years, try the following:</p><pre>from pandas_datareader import data<br>import datetime as dt</pre><pre>zm = data.DataReader(<br>    &quot;ZM&quot;,<br>    start=&#39;2019-1-1&#39;,<br>    end=dt.datetime.today(),<br>    data_source=&#39;yahoo&#39;<br>).reset_index()</pre><pre>zm.head()</pre><p>Running that code gives us the following output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wJaAtxjh1n9wMxVT" /></figure><h4>11. DataCommons</h4><p><a href="https://datacommons.org/">Datacommons</a> is a project by Google providing access to standardized and cleaned public datasets. The underlying data is represented in a graph format, making it really easy to <a href="https://towardsdatascience.com/exploring-datacommons-the-api-powering-google-search-afc366ec242b">query and join data</a> from many different datasources e.g. the US Census, World Bank, Wikipedia, Centre for Disease Control and more. Here’s a basic example:</p><pre>!pip install datacommons datacommons_pandas --upgrade --quiet</pre><pre>import datacommons_pandas as dc<br>import pandas as pd</pre><pre>city_dcids = dc.get_property_values([&quot;CDC500_City&quot;], &quot;member&quot;, limit=500)[&quot;CDC500_City&quot;]</pre><pre>cdc500_df = dc.build_multivariate_dataframe(<br>    city_dcids,<br>    [&quot;Percent_Person_Obesity&quot;,  # Prevalence of obesity from CDC<br>    &quot;Median_Income_Person&quot;,<br>    &quot;Median_Age_Person&quot;,<br>    &quot;UnemploymentRate_Person&quot;,  # Unemployment rate from BLS<br>    &quot;Count_Person_BelowPovertyLevelInThePast12Months&quot;,  # Persons living below the poverty line from Census<br>    &quot;Count_Person&quot;,  # Total population from Census<br>    ],<br>)</pre><pre>cdc500_df.info()</pre><p>Running that code gives us the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LS0A_CNMmPEpBxJl" /></figure><p>If you want to learn how to use DataCommons, read my full tutorial here:</p><p><a href="https://towardsdatascience.com/exploring-datacommons-the-api-powering-google-search-afc366ec242b">Exploring DataCommons — the API powering Google Search</a></p><h4>12. PyTrends (Google Trends)</h4><p><a href="https://github.com/GeneralMills/pytrends#interest-over-time">PyTrends</a> is an unofficial but useful library for querying <a href="https://trends.google.com/trends/explore?date=today%205-y&amp;q=oat%20milk,soy%20milk,almond%20milk">Google Trends</a> data — here’s a simple example:</p><pre>import pandas as pd</pre><pre>from pytrends.request import TrendReq</pre><pre>pytrends = TrendReq()<br>keywords = [&quot;oat milk&quot;, &quot;soy milk&quot;, &quot;almond milk&quot;]</pre><pre>pytrends.build_payload(keywords, cat=0, geo=&#39;&#39;, gprop=&#39;&#39;) # Get data from the last 5 years</pre><pre>top_queries = pytrends.interest_over_time()[keywords]</pre><pre>top_queries.head()</pre><p>Running that code gives us the following output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/0*WuUEWgBmU6n2EcRo" /></figure><h4>13. Kaggle</h4><p>Kaggle is a data science community that hosts a lot of datasets and competitions for learning Python. You can download some of these datasets to play around with through their command-line interface (note: you’ll need to sign up for a Kaggle account). For example, say we want to download some <a href="https://www.kaggle.com/zillow/zecon">Zillow economics data</a>, we can run the following commands in our terminal (Jupyter users: replace the $ with ! in your Python code:</p><pre>$ pip install kaggle<br>$ export KAGGLE_USERNAME=datadinosaur<br>$ export KAGGLE_KEY=xxxxxxxxxxxxxx<br>$ kaggle datasets download zillow/zecon<br>$ unzip zecon.zip</pre><p>This will download a zipped file of the datasets, and then uncompress them. From there, you can open them as local files with Pandas:</p><pre>import pandas as pd</pre><pre>csv_file = &quot;/Users/johnreid/Downloads/Zip_time_series.csv&quot;<br>df_from_csv = pd.read_csv(csv_file)<br>df_from_csv.info()</pre><p>To learn more, check out the <a href="https://github.com/Kaggle/kaggle-api">Kaggle API documentation</a>.</p><h3><strong>Conclusion</strong></h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FD7z8JfNANqahW%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FD7z8JfNANqahW%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FD7z8JfNANqahW%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="290" frameborder="0" scrolling="no"><a href="https://medium.com/media/d1b52ddb8afb9d5c21acdf2b6d0821db/href">https://medium.com/media/d1b52ddb8afb9d5c21acdf2b6d0821db/href</a></iframe><p>You made it! Now you can use your newfound powers to access multiple datasources and join them together withpd.merge or pd.concat, then visualize them with an interactive library like Altair, Pandas or Folium.</p><p>Are there any methods I missed? Let me know in the comments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bac5683e0063" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/13-ways-to-access-data-in-python-bac5683e0063">13 ways to access data in Python</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>