<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nicola Paganelli on Medium]]></title>
        <description><![CDATA[Stories by Nicola Paganelli on Medium]]></description>
        <link>https://medium.com/@nicolapaganelli?source=rss-62040d379640------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*saO3uvtOrAB9XSx0.</url>
            <title>Stories by Nicola Paganelli on Medium</title>
            <link>https://medium.com/@nicolapaganelli?source=rss-62040d379640------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 05:57:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nicolapaganelli/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to Interview a Product Manager]]></title>
            <link>https://medium.com/@nicolapaganelli/how-to-interview-a-product-manager-7076279ce718?source=rss-62040d379640------2</link>
            <guid isPermaLink="false">https://medium.com/p/7076279ce718</guid>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[hiring-tips]]></category>
            <category><![CDATA[interviewing]]></category>
            <category><![CDATA[hiring]]></category>
            <dc:creator><![CDATA[Nicola Paganelli]]></dc:creator>
            <pubDate>Wed, 29 Nov 2017 15:01:03 GMT</pubDate>
            <atom:updated>2017-12-01T19:18:14.174Z</atom:updated>
            <content:encoded><![CDATA[<p>After interviewing product managers for the last 4 years and always having the question “<em>how do I understand if this guy is a good PM?” </em>popping up in my mind, until a few minutes before the actual interview, I decided to try out a methodology for assessment.</p><blockquote>Disclaimer 1 — this is not to asses ‘product leads’, meaning people who manage product people, or fresh out of college candidates.<br>Disclaimer 2 — this is my MVP, will iterate whenever flaws or problems arise, with the goal of ultimately eliminating bias, and ‘good storytelling’ from the candidate (aka, not telling the truth).</blockquote><blockquote><strong>Part 1 <br>General Assessment.</strong></blockquote><p>There are 4 questions, which give you a feel of who you are talking to, that I (try to) ask, actually in this order.</p><p><strong><em>1- Which one is a good product?</em></strong><em><br>Does she (the interviewed PM) understand what a product is? </em><br>This might sound silly, but many confuse a product with a feature or with an enhancement.<br><em>Can she tell you why this product is good?<br></em>Generally the answer lies in one of these cases<br>a) It solves a problem for the customer / Solves people’s problem<br>b) Made the business more efficient [good for internal products at scale]<br>c) Pivoted towards a new/larger reach (customer basis) [good for growth hackers)<br>d) It is revolutionary -&gt; humanity can do something that was not possible before [please let me know first if you have such a product, I’ll be your humble servant] — this can also play into (a)</p><p>If the candidate tells you that the iPhone is a good product… well, thank you very much, what of it exactly?</p><p><strong><em>2- Tell me about a good feature</em></strong><em>.<br></em>Also here, does he know what a feature is?<br>Can she tell what contribution to the product it brought?<br>- was it value capturing? for example, the user can do something faster, the user spent a bit more money<br>- was it value creation? the user can do something that was not possible before and is willing to pay extra money.</p><p><strong><em>3- What is a good (product development) process?</em></strong><em><br></em>This one is a tough one, as the actual question is <em>“how do you do what you do?”. <br></em>Giving a solid answer is very hard, not just for a PM but for any professional, as it is like asking Tommy Emmanuel, how does he compose such amazing acoustic guitar songs, and is so good at performing (virtuoso like) in public.<br>However, if the answer is sound, this is (in my opinion) one of the strongest indicator of ‘success replication’ (remember that a 30% success in terms of products and features is currently an amazing rate!) and fit to your company.</p><p><strong><em>4- Who is a good PM? (somebody you know personally)</em></strong><em><br></em>This question does one thing, makes the person not tell about her perception of herself, and makes her feel not under investigation (talking about others is easier than talking about yourself)<br>The second thing it does, is letting you understand the inclinations of this person (PM) — will come to this in a few lines.<br>The third thing it does is, gives you (the interviewer) a chance to dig deeper in finding out if her profile fits the level of expertise you need in the areas you believe to be important to do the job.</p><p>At this point, you might have already built an opinion about the candidate, and a great moment where you can decide if you want to go forward or not.</p><p>In case you think your and her time will lead you further, then another part of the interview follows.</p><p>Now, one needs to plot the answer(s) the candidate gives to the 4th question, and here is a possible way.</p><blockquote>A PM has several areas he should be good at, but some are more pronounced for candidate A rather than B.</blockquote><p>(Keep in mind, we are assessing a PM, not a PHD candidate)</p><p>These areas are:</p><ol><li>Networking</li><li>Business Savviness</li><li>Technical Proficiency</li><li>Scientific Approach</li><li>Customer Focus</li><li>Learning Ability</li><li>Process Savviness</li><li>Design Skills</li></ol><p>At this point, the trick is to translate ‘how good’ into quantifiable numbers , the higher the better — hence</p><p><strong>Networking — this is really importand to get stuff through <br></strong>1- Tells that is important to listen more than to talk, and somehow reflects this in your talk<br>2- States that in any company it is important to keep healthy relationships and know who to talk to.<br>3- Is able to clearly and concisely express an idea to a given interlocutor (e.g. an engineer, or the CEO of your business)<br>4- Is able to present in front of more than 100 people. Ideally he’s actually done this (meet-ups, presentations…)<br>5- Tells how important it is to make each other successful (team/culture healthiness)<br>6- Aims and has some notion of Success beyond Success (learn the book please!)</p><p><strong>Business Savviness<br></strong>1- Knows what a KPI is<br>2- Can define what top and bottom line KPIs management is looking at for the business she’s applying for<br>3- Can define industry trends indicators (how the market looks like, and how to act in it)<br>4- Can define your product KPIs (What she’s applying for)<br>5- Can come up with new KPIs to tell if your development and iterations are going in the right directions (OKRs is an example). You need an example or case for her here.<br>6- Can come up with KPIs for your discovery iterations. I think this is harder than having actual data from a machine in front of your eyes, as you are measuring product market fit with a few people.</p><p><strong>Technical Proficiency<br></strong>1. Knows what scrum/kanban/agile is.<br>2. Can code a little, in at least 1 language — ask if his software has ever been used, to validate. Also, knows what an API is.<br>3. Can write proper user stories, or at least has a sound structure for them.<br>4. Can write technical narratives (ask for one case)<br>5. Can actually create a working-prototype of the future software (ask for one case)<br>6. Knows how to make the prototype production ready software (here we get really technical)</p><p><strong>Scientific Approach<br></strong>1. Knows what a test is (an A/B for example) <br>2. Knows how to formulate sound hypothesis (that actually verify something) and tell you what the difference between correlation and causation is, with examples.<br>3. Knows how to assess a test validity (statistically)<br>4. Knows how to build a report for a product, in terms of metrics, funnels or analysises for your tests. Ask for examples <br>5. She is outstandingly structured when describing how to do the previous points (a scientists follow structures, processes, and methodologies while not jumping left and right)<br>6. Knows how to train a model for your AI (:))</p><p><strong>Customer Focus<br></strong>1. Repeats several times ‘focusing on people problems’ in your talk<br>2. Tells that validation is needed by talking/meeting with a user (ask for cases)<br>3. Asks several times ‘who are we doing this for’ in any of your case studies.<br>4. Knows how to set up a user test — you need an example from him, or a case.<br>5. Has no shame in going out or around your company to validate anything with actual user feedback (where possible) — get an example if he says he does this.<br>6. Looks and can assess if user behaviour actually changes with his solution. Discards the enhancement/product/feature, if not.</p><p><strong>Learning Ability<br></strong>1. Reads some blogs (work related)<br>2. Goes to events relevant to her job, and actually participates (it’s easy to ask and verify if she’s actually active in the community).<br>3. Reads books related or giving insights about product related areas (e.g. econometrics, behavioural psychology, machine learning…)<br>4. Captures ideas or processes from other industries — ask for an example of how he (tried) to implement what he thought was to take<br>5. Brings/Applies learnings to her product<br>6. Shares his knowledge with others, and contributes actively to the community.</p><p><em>All of the above, needs repetition to actually be a translated into learning — once a year, is not enough!</em></p><p><strong>Process Savviness<br></strong>1. Knows what an MVP is — really. (this needs to match your company’s stage, understanding, and budget for an MVP)<br>2. Can tell you how to groom a backlog<br>3. Can define what discovery&gt;delivery is, firmly.<br>4. Can tell you what type of artifact is good to validate ideas at a certain product (lifecycle) stage. You can make a case here and ask how he’d go about it. Could he validate it as cheaply as you would?<br>5. Can tell about a process that promotes iterative learnings and/or delivery, that she implemented successfully (or why it didn’t work out)<br>6. Can tell you how to structure and coodinate a product development team + stakeholders around it; and you are learning.</p><p><strong>Design skills<br></strong>1. Knows what design thinking is, and can tell you what it is.<br>2. Can tell, swiftly, what the <a href="http://www.openlawlab.com/wp-content/uploads/2013/03/What-is-User-Experience-Design.png">elements</a> of UX are <br>3. Has closely collaborated with a professional designer and can talk about how they interacted. Was it a collaboration between ‘equals’ or a factory production like interaction.<br>4. Can actually design something (Balsamiq, Photoshop) and have good hypothesis behind these sketches — as for an example.<br>5. Has a portfolio online where you can look at and discuss the purposes of these designs<br>6. Has designed, or can design a user experience — you should have a case here.</p><p>Now that you have your numbers, you can assess this candidate also in comparison to others, and visualize it with a nice spider web (makes it easier to communicate your assesment).</p><p>I personally tend to <strong>not</strong> go forward with anybody who has scores <strong>below 3 in any of the areas</strong> and look if this PM meets what she’s going to have to do once she joins the team. She should be strong there, otherwise she’s going to struggle to get accepted, and be successful (it’s better for her and for the company to say “no” in this case”).</p><h4>Ultimately — are you excited about the idea of working with him, so much that you want him in your team/company?</h4><p>If yes, this is usually a good sign — be his sponsor!</p><p>To simplify my day (and yours hopefully), you can use <a href="https://docs.google.com/spreadsheets/d/10Q0tRfu77bXq-qrDqqpnS3HbVKZOsesv-O5iZv7iUZk/edit?usp=sharing">this sheet</a> (by making a copy) to go through you numerical assessment.</p><p>I’d be honoured to get any feedback to this, please write or comment, and let’s iterate on a way to get product from a ‘function’ to a ‘discipline’.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7076279ce718" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[#1 — Lessons learnt on A/B Testing]]></title>
            <link>https://medium.com/@nicolapaganelli/day-1-thoughts-on-a-b-testing-53e9968cea23?source=rss-62040d379640------2</link>
            <guid isPermaLink="false">https://medium.com/p/53e9968cea23</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[product]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[best-practices]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Nicola Paganelli]]></dc:creator>
            <pubDate>Wed, 19 Apr 2017 14:37:52 GMT</pubDate>
            <atom:updated>2017-04-19T21:36:27.580Z</atom:updated>
            <content:encoded><![CDATA[<p>In the last months, we’ve integrated our communication platform for transactional messages with a propietary A/B testing tool, making it possible to test the hell out of every message we send to our customers.</p><p>The journey has been refreshing (testing is something I’ve been doing extensively in the past) and there are some learnings we got, which are worth sharing (I guess &amp; hope).</p><p>So, here they are:</p><p>When you start interacting with an analyst (or an entire function) who is going to take care of the tests and data, or when you do it yourself:</p><p>1 —<strong> Identify your core KPIs</strong>. This is damn hard, but focus and language is key to actually start moving the needle.</p><p><strong>2- Identify the (product) scope.</strong> What is it that your product is handling, what is its purpose?</p><p>3- <strong>Frame the product possibilities</strong>, and its next steps: what can we currently manage and do?</p><p>4- <strong>Make a baseline analysis</strong>; also damn hard, but it will show you and whoever you are talking to, where you stand, with data.</p><p>5- <strong>Define your biggest opportunities </strong>from point (4), and balance them with (3), talking about (1).</p><p>6- <strong>Set up (shared) reporting</strong>; what is your source of data truth? this can sometimes take weeks, but hey, starting somewhere is better than nothing (and Google tools are pretty good at this point).</p><p>7- <strong>Set up reporting standards for your A/B tests: </strong>ctrl+c, ctrl+v is easier than discussing how things should look time each time.</p><p>8- <strong>Make a roadmap for your tests</strong>: roadmap!! we all hate this word in product, but you got to, sometimes. Keep it high level and flexible.</p><p>9- <strong>Set clear A/B testing goals/KPIs</strong> (1), and before starting, come on, check if you can actually observe a statistically valid change in the customer behaviour (yes, data reflects the customer behavior, like in ‘The Matrix’).</p><p>10- <strong>Test 1, I repeat, 1 thing at a time</strong>. If you test more, you are testing pears with apples (unless you do multi variant tests, making it more complex, have fun).</p><p>11- <strong>Use ‘a’ tool for sharing information</strong> and planning that multiple stakeholders can relate to — I like git-hub, it’s flexible, and guess what, also your developers will be happy to look into it.</p><p>12- <strong>Triple check the set up of your test</strong>: don’t take anything for granted, especially around what the user sees. Once we had different wording in a CTA and it messed up the whole test (2 weeks!!).</p><p>13- <strong>Watch the test running</strong>: if you are able to do this, it will be cool for your team and cut down on time to spotting errors, issues, and eventually make you excited about where you are heading to (fail or win, doesn’t matter, you are still learning), (actually matters, you need to make more of something ‘good’ at the end of the day, but hey, no pressure).</p><p><strong>14- wrap up</strong> when the time is due and move on, using (11) to document what you learned (MISS/KISS) and the next steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*ah3R-I-kG21P9JX_6R5Eng.gif" /><figcaption>Break Time</figcaption></figure><p>I have some more tips that are not about the testing process but what’s around it, if you still want to hear.</p><p><strong>a)</strong> If you want to test something, start small, but not irrelevant, wait a little longer for the feature/change you want to test to be ready. For example, if you want to test if adding a visual makes the customer behaviour change, make sure that the visual is relevant for the customer, and not just a place holder. In the latter case you’ll learn how to waste some of your time in the best case :D.</p><p><strong>b) </strong>Consider possible opposite behaviors/results when you construct your hypothesis. This happens quite some times. For example, changing the headline to telling the customer how much price reduction an item got (from a general headline), is it goint to increase or decrease your open rate. Please test and let me know!</p><p><strong>c)</strong> Observe wider changes in the customer behavior: here (1) is key. Avoid looking at a micro conversion only, see what else happens in a broader set of data. The risk, is making one step ahead and 2 back.</p><p>Last 2 points, promise, and this is about <strong><em>people</em></strong>:</p><p><strong>i)</strong> Get clear and real with your stakeholders: what benefit will they get from letting you be the ‘owner’ of the testing on ‘their’ products, what is their role, and what decisions they will be asked to take. Handle people at once, in the tech world, it could help dedicating time to things you like seeing happening (progress) rather than bla bla…</p><p><strong>ii)</strong> Don’t, ever, sing-up, for $$$ (money). The purpose of testing is finding out if the idea you had was good; if you already knew that (and be serious here, you don’t), why are you testing?</p><p>Cheers</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=53e9968cea23" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>