<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Yuzheng Sun on Medium]]></title>
        <description><![CDATA[Stories by Yuzheng Sun on Medium]]></description>
        <link>https://medium.com/@yz_79011?source=rss-8959cd2ea97e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Fs5onGg8a0XH4Fa8ISHE0g.jpeg</url>
            <title>Stories by Yuzheng Sun on Medium</title>
            <link>https://medium.com/@yz_79011?source=rss-8959cd2ea97e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 09:29:31 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@yz_79011/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Why is aha moment wrong?]]></title>
            <link>https://medium.com/pragmatic-data-scientists/why-is-aha-moment-wrong-ed96ae493a01?source=rss-8959cd2ea97e------2</link>
            <guid isPermaLink="false">https://medium.com/p/ed96ae493a01</guid>
            <category><![CDATA[growth]]></category>
            <category><![CDATA[product]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Yuzheng Sun]]></dc:creator>
            <pubDate>Wed, 04 Oct 2023 23:36:28 GMT</pubDate>
            <atom:updated>2023-10-05T20:30:30.629Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gcjQzpE5eYOhm8CJFMKkig.png" /></figure><p>In the realm of product growth, the concept of ‘Aha Moments’ has gained significant traction. Inspired by notable examples from Facebook and Dropbox, professionals have come to identify these moments as the pivotal junctures in a user’s journey with a product. It’s the instant they comprehend the product’s value proposition. For instance, Facebook’s aha moment occurs when users connect with more than seven friends, making them realize its potential as a connectivity tool.</p><p>Aha moments are appealing because they represent a harmonious blend of quantitative and qualitative insights into user behavior. These events can be delineated and monitored on dashboards, allowing teams to channel resources into amplifying these moments. However, a critical oversight exists in this approach.</p><h3>The Issue with Aha Moments</h3><p>The common belief is that aha moments catalyze users to transition from being passive to active. This assumption justifies the vast resources dedicated to promoting such moments. The presumption is simple: expose users to these moments, and they’ll shift from passivity to activity.</p><p>But, more often than not, this isn’t the case. Active users experience these aha moments primarily because they’re already engaged. For instance, individuals predisposed to using Facebook will inevitably connect with friends quicker. This correlation extends to monetization. Users who exhibit certain behaviors might seem more inclined to become paid users. In reality, those with a higher intent to pay often demonstrate these purchase indicators from the outset.</p><p>Therefore, resources invested in fostering these moments might not yield the anticipated ROI. Essentially, businesses might merely be targeting users already predisposed to engagement, inundating them with notifications, marketing tactics, and incentives. This doesn’t genuinely add substantial value to the business.</p><h3>The Inherent Flaws</h3><p>Two inherent problems make this mistake nearly unavoidable:</p><ol><li><strong>They are users’ choice:</strong> These moments are essentially user behaviors, stemming from user choices laden with latent variables. One dominant latent variable is the user’s inherent inclination to be active. Hence, high-intent users will naturally gravitate towards these moments, while their low-intent counterparts might not.</li><li><strong>High Correlation with Desired Outcomes:</strong> A strong correlation usually exists between aha moments and the desired user activity. Given that these moments arise from active user choices, an endogeneity issue arises, making it challenging to establish a definitive causal relationship.</li></ol><h3>Rethinking the Approach</h3><p>Given this knowledge, how should businesses proceed?</p><ol><li><strong>Use Aha Moments as the proxies they are</strong>: Accept aha moments for what they truly are — indicators of user intent. They can serve as early signals of a user’s potential activity or inclination towards monetization. When viewed as indicators or predictors, these moments retain their utility. For instance, they can be employed to train machine learning models, allowing them to achieve scale sooner.</li><li><strong>Establishing Causality</strong>: If the goal is to establish a causal relationship, then the treatment needs to be exogenous. This requires placing certain features or experiences under the purview of the business, rather than leaving them to user discretion. One could randomly assign users to specific features or incentive programs, then compare long-term performance metrics with those lacking access. This methodology underpins AB testing and online experimentation.</li></ol><h3>In Conclusion</h3><p>When evaluating aha moments:</p><ul><li>Recognize that if they’re based on user choice, causality can’t be definitively established.</li><li>View them as indicators or predictors and leverage them accordingly.</li><li>To draw causal connections, ensure the treatment is exogenous.</li></ul><p>As the digital landscape evolves, it’s crucial for stakeholders to understand the true nature of aha moments. They might not be the game-changers they’re perceived to be, but when approached with a discerning eye, they can still offer valuable insights.</p><p>For further discussions or queries, please reach out. — <em>Pragmatic Data Scientists</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ed96ae493a01" width="1" height="1" alt=""><hr><p><a href="https://medium.com/pragmatic-data-scientists/why-is-aha-moment-wrong-ed96ae493a01">Why is aha moment wrong?</a> was originally published in <a href="https://medium.com/pragmatic-data-scientists">Pragmatic Data Scientists</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[THE clear definition of Retention, and why yours is ambiguous]]></title>
            <link>https://medium.com/pragmatic-data-scientists/the-clear-definition-of-retention-and-why-yours-is-ambiguous-3b7512629093?source=rss-8959cd2ea97e------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b7512629093</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[product]]></category>
            <dc:creator><![CDATA[Yuzheng Sun]]></dc:creator>
            <pubDate>Wed, 04 Oct 2023 19:41:53 GMT</pubDate>
            <atom:updated>2023-10-26T18:03:39.669Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D0e3FCicT3_WDHc1t2Y3kw.png" /></figure><h3>TLDR</h3><p>Retention is a concept where junior Data Scientists think they know, but senior Data Scientists become more and more careful with it. Why?</p><ul><li>Each retention has three durations — starting, ending, and the gap. Without a clear definition, you will have mistakes and waste a lot of time in alignment</li><li>We provide a complete definition, a unified framework, and a succinct version for common use</li><li>We provide examples for how to apply this definition</li></ul><h3>A hypothetical example</h3><p>Suppose your colleague asked you: <em>“What’s the weekly retention for active users of ChatGPT?”</em></p><p>The question is actually impossible to answer, because for different “cohorts” of users that started on different dates, each “cohort” can have different retentions, and their retentions change over time. Your colleague needs to be more exact in her ask.</p><p>You asked your colleague to be more exact, by giving you the start date and the end date. She returns:</p><p>“<em>For users who first became active on ChatGPT </em><strong><em>during 9/1–9/8</em></strong><em>, what’s their weekly retention </em><strong><em>on 9/15</em></strong><em>?”</em></p><p>This seemed clear, and you went and fetched a query.</p><p>But did you know that there may be four “correct” ways to interpret her ask?:</p><ol><li>For each daily cohort of 9/1 to 9/8, calculate the % of users who are weekly active till 9/15, then weighted average across cohorts</li><li>For each daily cohort of 9/1 to 9/8, (calculate the % of users who are daily active for 7 days prior to 9/15, take the sum) then weighted average across cohorts</li><li>For the weekly cohort of 9/1 to 9/8, calculate the % of users who are weekly active till 9/15</li><li>For the weekly cohort of 9/1 to 9/8, calculate the % of users who are daily active for 7 days prior to 9/15, take the sum</li></ol><p>You can have more ambiguities if you are not sure if the ending date should be 9/8, 9/15, or 9/22. Let’s not go there yet.</p><p>This kind of ambiguity is doomed to happen because there are <strong>three durations</strong> in any retention definition:</p><ol><li><strong>Starting duration</strong> — Users did some activities here to be included in the denominator</li><li><strong>Ending duration</strong>, and — Users who are in the denominator, did some activities here, to be included in the numerator</li><li><strong>The Gap in between</strong> — Retention is only meaningful when we give some time for users to churn</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SyZm2rKRnuY1T4ew5iBm7A.png" /></figure><p>We can’t just say “weekly” and use “common sense” to figure out which duration “weekly” is correspond to, because these definitions are all valid and can serve different purposes.</p><h3>More knowledge can lead to more ambiguity</h3><p>We see ambiguous definitions of retentions like this everywhere often because people don’t know the other possibilities. They thought their definition is the only way, when in fact, it is only a special case without good notation. It is natural that inexperienced data scientists being confident in their definition of retention, while more experienced data scientists became much more careful.</p><p>This ambiguity became obvious to me during my Facebook career. Back then Facebook was doing Commerce across Marketplace, Facebook Shops, and Instagram Shopping. We started to collaborate and try to compare our metrics. Turns out everyone has their own definition of retention, and we had to spend a lot of time to align on our definitions. If I remember correctly, in 2020, there was an internal post from a high level DS at Facebook, just to address this issue and have a uniform definition across the company. If Facebook, with all of its DS talents, had the problem until 2020, chances are, you do as well.</p><p>This post will help you solve this problem by breaking it down the the most fundamental elements, then reassemble. And it will be very clear. Every time you see a different definition of retention, you will be able to put your finger on where the difference arises and reconcile.</p><h3>How to define it clearly</h3><p>Retention is one word that contains four elements:</p><p><strong># of (Users with Action A during T0, and did Action B during T1) / # of Users with Action A during T0</strong></p><p>We need to be clear on:</p><ul><li><strong>T0, the starting duration</strong>: This is a duration of initial activities. It is often a day, but can often be a weekly, especially when you have a day-of-week effect, or a small sample size.</li><li><strong>T1, the ending duration</strong>: This is your observation window. Again, can be day or week.</li></ul><p>Remember that both T0 and T1 have two parameters — the length of the duration, and the date of the duration. From T0 and T1, we get:</p><ul><li><strong>T1 — T0, the “Gap”</strong>: Gap should be <strong>(end of T1 — end of T0)</strong>. This is where a lot of discrepancies in definitions happen as well. We should use end of durations, to avoid data truncation issues. I’ll explain later in the future post of this series.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gHp0tGyHm2LZYOh4pF2b8Q.png" /></figure><p>In addition, we often implicitly ignored:</p><ul><li>Action A: The initial activity, such as register, download, or <strong>active</strong>.</li><li>Action B: The returning activity, such as <strong>active</strong>, pay, or some meaningful usage.</li><li>Subset relationship: The numerator is a subset of the denominator</li></ul><p><strong>This definition gives you a single data point</strong></p><p>It is natural to think about retention and start thinking about J curve or triangle charts. We will talk about them later. But we need to have a clear definition of points, before we can define a curve well. The above definition is for a single data point. You can draw the J-curve by fixing the T0 and have different gaps.</p><h3>A more succinct version for “active retention”</h3><p>The most common application of retention is for Action A = First Time Active, and Action B = Active. Often, we are interested in T0 = daily, and T1 either = Daily or Weekly. We can create a succinct special case for this definition: <strong><em>DAU or WAU, @GAP</em>.</strong> Where</p><ul><li>D is daily, W is weekly, which correspond to the length of T1</li><li>Active User(AU) corresponds to the user’s status at T1</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AumTkqb6yCrQMyFqa4ldbQ.png" /></figure><p>So for example, if we use WAU@D14</p><ul><li>T1 is from D8 to D14</li><li>A user who is only active @D9 is considered a WAU@D14</li><li>A user who is only active @D6 is considered not a WAU@D14</li></ul><p>In practice, we need to specify T0 as well. The complete sentence is — For the X # of users who did action A between T0_start and T0_end, their WAU @GAP is XXX.</p><p><strong>Two pro tips:</strong></p><ul><li><strong>Retention vs. Stickiness:</strong> Note that Action A is first time active. If we remove “first time” and have Action A = Active, this becomes a common “stickiness” measurement. Under this definition, stickiness is only a special case of retention</li><li><strong>T1 ends on D13 intead of D14:</strong> You may want to consider using D0 as T0, and use D7 to D13 as T1, for two benefits:<br>- You fetch data on D14. If you define T1 as D7-D13, your data will be available.<br>- Suppose D14 is a Friday, your D0 will also be a Friday. So you are looking at Friday cohort on Friday. Anecdotally, some retention metrics spike at 7 day intervals so this is important to capture.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ckVyS25WMGKDzSCbWrFzWw.png" /></figure><h3>Apply the definition</h3><p>Applying the definition, let’s define the above ambiguous example again.</p><p>Suppose our colleague asks us her question because she wants to get a sense of how many users keep using ChatGPT at least once per week, after 2 weeks of first activity. Because there is a day-of-week effect in terms of when a user is registered, she also wants a weekly average of starting date.</p><p>Then we should</p><ol><li>Use WAU @ D14.</li><li>For T0s, use 7 daily cohorts across a week and average across them</li></ol><p>With our definition, <em># of (Users with Action A during T0, that did Action B during T1) / # of Users with Action A during T0</em>, it is clear:</p><ul><li>We have 7 different T0s:<br>- Length is daily<br>- (Beginning and )Ending dates are 9/1/2023–9/7/2023</li><li>Gap: 14 days</li><li>We have 7 different T1s:<br>- Length is weekly<br>- Ending dates are 9/14/2023–9/20/2023<br>- For the cohort that starts on 9/1, the ending date is 9/14; for the cohort that starts on 9/7, the ending date is 9/20</li><li>Action A: First Time Active, subject to the product team’s definition. I’ll talk about this later.</li><li>Action B: Active</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ysqysk7sKLK4zr95I1VAxg.png" /></figure><p>We get 7 numbers and do a weighted (based on denominator) average. For example, the first number is:</p><p>(The number of users who first became active on 9/1/2023, and were active during 9/8/2023–9/14/2023) / (The number of user who first became active on 9/1/2023)</p><p>It may take a while at the beginning. But using this definition will force you and your colleagues to think clearly on what exact retentions you want to study. After a while it will be natural and you will have a better understanding of your users.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b7512629093" width="1" height="1" alt=""><hr><p><a href="https://medium.com/pragmatic-data-scientists/the-clear-definition-of-retention-and-why-yours-is-ambiguous-3b7512629093">THE clear definition of Retention, and why yours is ambiguous</a> was originally published in <a href="https://medium.com/pragmatic-data-scientists">Pragmatic Data Scientists</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Surviving the high-stakes game of product sense interviews]]></title>
            <link>https://medium.com/pragmatic-data-scientists/surviving-the-high-stakes-game-of-product-sense-interviews-2d960e116e57?source=rss-8959cd2ea97e------2</link>
            <guid isPermaLink="false">https://medium.com/p/2d960e116e57</guid>
            <category><![CDATA[interview]]></category>
            <category><![CDATA[interview-tips]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Yuzheng Sun]]></dc:creator>
            <pubDate>Fri, 11 Aug 2023 23:28:44 GMT</pubDate>
            <atom:updated>2023-08-11T23:28:44.458Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*844pRhKAEoPApXYzYmyWrg.png" /></figure><p>Imagine stepping into a dark forest. Each pathway leads to different outcomes; some to treasure, others to dead ends. Your task is to choose your path wisely, armed only with your knowledge and intuition. In many ways, answering open-ended questions in product sense interviews feels exactly like this: a high-stakes game of exploration and decision-making.</p><p>Over countless mock interviews, I’ve watched many fellow explorers, armed with their technical prowess and eagerness to impress, rushing headfirst down these pathways. They scramble to find a definitive answer, the “treasure” at the end of the path. Unfortunately, they often find themselves at a dead end, realizing too late that they’ve missed the essence of the game. But fear not, for I’m here to guide you through this challenging yet rewarding forest.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*wabDEFaR9Hinf6Kc.jpg" /></figure><h3>Principle 1: A two-way conversation</h3><p>Imagine that each time you step onto a new path, the forest whispers hints to you. The forest here is the interviewer, guiding you subtly toward your destination. To hear these whispers, you must be willing to engage in a two-way conversation, maintaining an optimal talk time ratio of approximately 2:1 or 3:1, with you talking the most. Don’t be the explorer who forges ahead without pausing to listen!</p><h3>Principle 2: Decipher the ‘what’ and ‘why’ before the ‘how’</h3><p>When a path splits in two, do you randomly choose a direction, or do you first try to understand why one path might be preferable over the other? Jumping headlong into problem-solving without grasping the problem is like choosing a path at random. Understand the ‘what’ and ‘why’ before you decide ‘how’ to proceed.</p><h3>Principle 3: The realm of trade-offs</h3><p>The forest doesn’t hold one single treasure, but multiple ones, each with its pros and cons. Similarly, open-ended questions have many answers, each with its own set of trade-offs. You’re not searching for the ‘right’ answer but assessing which solution fits best given the conditions.</p><p>Let’s journey through two hypothetical scenarios:</p><h3>Scenario 1: A gastronomic adventure</h3><p>You’ve been tasked with planning a dinner. The forest offers you various paths: a fancy restaurant here, a homely diner there. Before you rush to recommend the fanciest option, consider your party’s needs. What’s their budget? How many people are attending? What’s the purpose of the dinner?</p><h3>Scenario 2: The fake news conundrum</h3><p>As we approach the thorny problem of assessing the percentage of fake news on Facebook, many explorers may rush to define fake news and propose a detection method. However, the forest asks you a crucial question: why do you need to understand the ratio of fake news?</p><p>Tackling this problem requires a careful exploration of the paths ahead, taking into consideration your available resources, and adjusting your definition of fake news accordingly. Through an open dialogue with the forest (your interviewer), you can show your understanding of the issue at hand and your willingness to work collaboratively towards a solution.</p><p>Let’s illustrate how to approach this challenge. Begin by asking the forest about the purpose of the question, the available resources, and the accessible data. Then, propose a strategy like random sampling and manual tagging, considering resource constraints.</p><p>After presenting your initial plan, engage in a deeper discussion about implementing machine learning to further refine your strategy. This approach not only showcases your technical skills but also exhibits your strategic thinking and adaptability.</p><p>Surviving the forest of product sense interviews involves much more than presenting a definitive answer. It’s about communicating effectively, understanding the underlying business problem, and recognizing potential trade-offs. You’re an explorer, navigating through a rich and complex forest, not merely a treasure hunter seeking a single prize.</p><p>Have any thoughts or questions? Let’s start a conversation in the comments below. I’m always excited to learn from you and to make future posts even more beneficial for our collective journey. Happy exploring!</p><p><em>This post is an edited transcript of </em><a href="https://youtu.be/722vLIY5EJk"><em>a video on product sense interviews</em></a><em>, adapted for a blog format.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2d960e116e57" width="1" height="1" alt=""><hr><p><a href="https://medium.com/pragmatic-data-scientists/surviving-the-high-stakes-game-of-product-sense-interviews-2d960e116e57">Surviving the high-stakes game of product sense interviews</a> was originally published in <a href="https://medium.com/pragmatic-data-scientists">Pragmatic Data Scientists</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>