<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Piotr Zawieja on Medium]]></title>
        <description><![CDATA[Stories by Piotr Zawieja on Medium]]></description>
        <link>https://medium.com/@p.r.zawieja?source=rss-5309f9bf426e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*Ah00cIYtU3LDg6Sf</url>
            <title>Stories by Piotr Zawieja on Medium</title>
            <link>https://medium.com/@p.r.zawieja?source=rss-5309f9bf426e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 16:05:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@p.r.zawieja/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Secret Sauce of dbt: Macros That Make Your Data Pipeline Smarter]]></title>
            <link>https://medium.com/@p.r.zawieja/the-secret-sauce-of-dbt-macros-that-make-your-data-pipeline-smarter-91af8a819685?source=rss-5309f9bf426e------2</link>
            <guid isPermaLink="false">https://medium.com/p/91af8a819685</guid>
            <category><![CDATA[analytics-engineering]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[data-modeling]]></category>
            <category><![CDATA[macro]]></category>
            <dc:creator><![CDATA[Piotr Zawieja]]></dc:creator>
            <pubDate>Sun, 09 Mar 2025 12:20:31 GMT</pubDate>
            <atom:updated>2025-03-09T12:20:31.926Z</atom:updated>
            <content:encoded><![CDATA[<p>If you’ve ever caught yourself copying and pasting SQL logic across multiple dbt models, it’s time to stop. There’s a better way — <strong>macros.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BrbhckSZ4zohMDsl" /></figure><h3>Why Macros in dbt Are a Game-Changer</h3><p>If you’ve ever caught yourself copying and pasting SQL logic across multiple dbt models, it’s time to stop. There’s a better way — <strong>macros.</strong></p><p>Macros in dbt are like SQL on steroids: they let you <strong>automate repetitive tasks, enforce best practices, and keep your code DRY (Don’t Repeat Yourself).</strong></p><p>Instead of writing the same SQL logic over and over (and over), you can create a macro once and reuse it everywhere. Not only does this save time, but it also <strong>reduces errors and makes your dbt project scalable</strong> as your data models grow.</p><p>In this post, we’ll explore:</p><p>✅ What macros are and how they work in dbt<br>✅ Common use cases where macros <strong>save time and effort</strong><br>✅ Advanced examples to take your dbt workflow to the next level</p><h3>What Are Macros in dbt?</h3><p>Macros in dbt are <strong>Jinja-powered functions</strong> that allow you to write reusable SQL code. They work similarly to functions in programming languages — accepting arguments and returning dynamic SQL statements.</p><h4>Example: A Simple Macro to Convert to CET Date</h4><p>Instead of writing repetitive date conversions in your SQL models, use a macro:</p><p>Define the Macro (macros/date_conversion.sql):</p><pre>{% macro convert_to_cet(column_name) %}<br>    CONVERT_TIMEZONE(&#39;UTC&#39;, &#39;CET&#39;, {{ column_name }})<br>{% endmacro %}</pre><p>Use It in a Model (models/stg_orders.sql):</p><pre>SELECT<br>    order_id,<br>    {{ convert_to_cet(&#39;order_timestamp&#39;) }} AS order_date_cet<br>FROM {{ ref(&#39;raw_orders&#39;) }}</pre><p>📌 <strong>Why it’s useful?</strong> Ensures all timestamps are consistently converted to CET (Berlin), avoiding discrepancies in time zone calculations across reports.</p><h3>Practical Use Cases for Macros</h3><h4>1. Enforcing Naming Conventions</h4><p>Ever had column names formatted inconsistently? Some use camelCase, some use snake_case? Define a macro to standardize column names.</p><p>Macro: Convert to Snake Case (macros/standardize_column_names.sql)</p><pre>{% macro snake_case(column_name) %}<br>    LOWER(REGEXP_REPLACE({{ column_name }}, &#39;[^a-zA-Z0-9]+&#39;, &#39;_&#39;))<br>{% endmacro %}</pre><p>Example Usage in a Model:</p><pre>SELECT<br>    {{ snake_case(&#39;customerName&#39;) }} AS customer_name<br>FROM {{ ref(&#39;customers&#39;) }}</pre><p>📌 <strong>Why it’s useful?</strong> Keeps naming conventions consistent across your entire dbt project.</p><h4>2. Generating Safe Division to Prevent Errors</h4><p>NULL values can break aggregations. Instead of handling them manually, create a macro that ensures safe defaults.</p><p>Macro: Safe Division (macros/safe_division.sql)</p><pre>{% macro safe_divide(numerator, denominator) %}<br>    CASE WHEN {{ denominator }} = 0 OR {{ denominator }} IS NULL THEN NULL<br>    ELSE {{ numerator }} / {{ denominator }}<br>    END<br>{% endmacro %}</pre><p>Example Usage in a Model:</p><pre>SELECT<br>    order_id,<br>    {{ safe_divide(&#39;revenue&#39;, &#39;units_sold&#39;) }} AS revenue_per_unit<br>FROM {{ ref(&#39;orders&#39;) }}</pre><p>📌 <strong>Why it’s useful?</strong> Ensures division operations never result in errors due to NULLs or zero values, preventing query failures in dashboards and reports.</p><h4>3. Automating Incremental Filters for Performance</h4><p>Sometimes you want to preview a model without selecting all columns manually. Create a macro that lets you dynamically select a subset of columns.</p><p>Macro: Apply Incremental Filter (macros/incremental_filter.sql)</p><pre>{% macro incremental_filter(date_column, days) %}<br>    WHERE {{ date_column }} &gt;= CURRENT_DATE - INTERVAL &#39;{{ days }}&#39; DAY<br>{% endmacro %}</pre><p>Usage in a Model:</p><pre>SELECT * FROM {{ ref(&#39;events&#39;) }} {{ incremental_filter(&#39;event_date&#39;, 30) }}</pre><p>📌 <strong>Why it’s useful?</strong> Ensures efficient incremental processing by filtering data dynamically, reducing unnecessary scans and improving query performance.</p><h4><strong>4. Dynamic Column Selection by Pattern</strong></h4><p>Sometimes you need to select multiple columns that share a common prefix, suffix, or keyword without listing them manually. A macro can help automate this selection dynamically.</p><p>Macro: Select Coulumns by Pattern (macros/select_columns_by_pattern.sql)</p><pre>{% macro select_columns_by_pattern(pattern, table_name) %}<br>    {%- set relation = ref(table_name) -%}<br>    {%- set columns = adapter.get_columns_in_relation(relation) -%}<br>    {%- set selected_columns = [] -%}<br><br>{%- for column in columns -%}<br>        {%- if pattern in column.name -%}<br>            {%- do selected_columns.append(column.name) -%}<br>        {%- endif -%}<br>    {%- endfor -%}<br>    {{ selected_columns | join(&#39;, &#39;) }}<br>{% endmacro %}</pre><p><strong>Usage in a Model:</strong></p><pre>SELECT <br>    {{ select_columns_by_pattern(&#39;revenue&#39;, &#39;sales_data&#39;) }} <br>FROM {{ ref(&#39;sales_data&#39;) }};</pre><p>If sales_data has columns like: id, revenue_usd, revenue_eur, total_cost, sales_date , the query dynamically expands to:</p><pre>SELECT revenue_usd, revenue_eur FROM sales_data;</pre><p><strong>📌 Why it’s useful?</strong></p><ul><li>Saves time by avoiding manual column selection.</li><li>Automatically adjusts when new relevant columns are added.</li><li>Reduces query errors by ensuring only the needed columns are selected.</li><li>Improves performance by selecting only relevant fields instead of SELECT *.</li></ul><h3>Final Thoughts: Why Macros Should Be in Your dbt Toolbox</h3><p>Macros take dbt from <strong>great to elite</strong> by helping you:</p><p>✅ Write cleaner, <strong>more maintainable SQL</strong><br>✅ Save time <strong>by automating repetitive logic</strong><br>✅ Ensure consistency <strong>across all models</strong><br>✅ Improve query performance with <strong>dynamic and optimized logic</strong></p><p>If you’re not using macros yet, start with <strong>simple transformations</strong> like date conversions or NULL handling. As your dbt project grows, you’ll <strong>quickly see the value in automating logic, enforcing standards, and making SQL reusable.</strong></p><p>🚀 <strong>Ready to level up?</strong> Try implementing a macro today, and let your dbt project scale with ease.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=91af8a819685" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stop Bad Data Before It Starts: Automating dbt Tests & Custom Validations]]></title>
            <link>https://medium.com/@p.r.zawieja/stop-bad-data-before-it-starts-automating-dbt-tests-custom-validations-c02ee5007189?source=rss-5309f9bf426e------2</link>
            <guid isPermaLink="false">https://medium.com/p/c02ee5007189</guid>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[data-modeling]]></category>
            <category><![CDATA[analytics-engineering]]></category>
            <category><![CDATA[data-build-tool]]></category>
            <dc:creator><![CDATA[Piotr Zawieja]]></dc:creator>
            <pubDate>Wed, 26 Feb 2025 20:48:18 GMT</pubDate>
            <atom:updated>2025-02-26T20:48:18.976Z</atom:updated>
            <content:encoded><![CDATA[<h4>Why Testing in dbt is Essential (and Why You Should Care)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VG3BCi4JMBmAP7fV5q3nNA.png" /></figure><p>Let’s be real — nobody likes debugging data issues in production. You build your beautiful <strong>dbt models</strong>, everything runs smoothly… until one day, numbers don’t match, stakeholders are asking questions, and you’re stuck digging through thousands of rows to figure out why there’s a duplicate from 2017 causing a $10 million discrepancy. Sound familiar?</p><p>This is exactly why automated testing in <strong>dbt (Data Build Tool)</strong> is a game changer. It saves you from embarrassing data mishaps, keeps your models trustworthy, and ensures data integrity. And no, it’s not just about adding unique and not null constraints and calling it a day—though those are a good start. In this article, we’ll cover how to:</p><ol><li>Use built-in dbt tests efficiently</li><li>Automate tests to catch issues before they escalate</li><li>Write custom data validations that go beyond the basics</li></ol><h3>Types of dbt Tests</h3><p>dbt offers <strong>three main types of tests:</strong></p><h4>1. Schema Tests (Pre-built in dbt)</h4><p>These are the most basic tests, added directly to schema.yml. They ensure that key constraints are met before data gets used in reports:</p><ul><li><strong>unique</strong>: Ensures that values in a column are not duplicated.</li><li><strong>not_null</strong>: Ensures that a column contains no null values.</li><li><strong>accepted_values</strong>: Ensures a column only contains predefined values.</li><li><strong>relationships</strong>: Ensures referential integrity between tables.</li></ul><p>Schema tests are great for quick validations but don’t catch deeper data quality issues.</p><h4>2. Custom Tests (Written in SQL)</h4><p>Sometimes, built-in tests won’t cut it. Maybe you need to check for negative revenues, missing business-critical fields, or unexpected data anomalies. Custom tests let you write <strong>SQL-based rules</strong> that flag specific issues.</p><h4>3. Generic Tests (Reusable SQL Logic)</h4><p>Instead of writing the same custom SQL test over and over for different models, you can <strong>parameterize</strong> it and apply it across multiple tables. This makes testing scalable and easier to maintain.</p><h3>Using Built-in Schema Tests</h3><p>Adding built-in tests in schema.yml is a great way to start testing immediately. Example:</p><pre>version: 2<br><br>models:<br>  - name: stg_orders<br>    description: &quot;Staging table for orders&quot;<br>    columns:<br>      - name: order_id<br>        tests:<br>          - unique<br>          - not_null<br>      - name: order_date<br>        tests:<br>          - not_null<br>      - name: status<br>        tests:<br>          - accepted_values:<br>              values: [&#39;pending&#39;, &#39;shipped&#39;, &#39;delivered&#39;, &#39;canceled&#39;]yml</pre><p>To run these tests, execute:</p><pre>dbt test</pre><p>If a test fails, dbt provides a detailed log, so you can fix the issue before it hits production.</p><h3>Writing a Custom SQL Test</h3><p>Let’s say your finance team is freaking out because order totals <strong>should never be negative</strong>. Built-in tests can’t check for that, but a custom test can.</p><p>Create tests/order_total_positive.sql:</p><pre>WITH errors AS (<br>    SELECT order_id<br>    FROM {{ ref(&#39;stg_orders&#39;) }}<br>    WHERE total_amount &lt; 0<br>)<br>SELECT * FROM errors;</pre><p>Then register it in schema.yml:</p><pre>models:<br>  - name: stg_orders<br>    tests:<br>      - order_total_positive</pre><p>Now, dbt will flag any negative order totals before they break our revenue numbers. Crisis averted! 🚨</p><h3>Creating a Generic Test (Reusable Across Models)</h3><h4>Example: Prevent Future Dates from Sneaking In</h4><p>Data pipelines can be unpredictable. What if your ingestion process mistakenly loads future dates? Your reports might suddenly predict that customers <strong>signed up in 2099</strong>. A generic test can prevent this.</p><p>Step 1: Create tests/no_future_dates.sql</p><pre>SELECT * <br>FROM {{ ref(model) }}<br>WHERE {{ column_name }} &gt; CURRENT_DATE;</pre><p>Step 2: Register in schema.yml</p><pre>tests:<br>  - name: no_future_dates<br>    description: &quot;Ensure no dates are in the future&quot;<br>    arguments:<br>      - name: column_name<br>        type: string</pre><p>Step 3: Apply Across Multiple Models</p><pre>models:<br>  - name: stg_orders<br>    columns:<br>      - name: order_date<br>        tests:<br>          - no_future_dates<br>  - name: stg_customers<br>    columns:<br>      - name: signup_date<br>        tests:<br>          - no_future_dates</pre><p>Now, <strong>any table with a date field can use this test</strong>. No more time-traveling customers!</p><h3>Automating Tests in dbt Cloud &amp; CI/CD</h3><p>Running tests manually is fine for development, but <strong>automated testing ensures data quality at scale</strong>. If something breaks, you want to know before it affects dashboards, not after.</p><p>If you use <strong>dbt Cloud</strong>, you can schedule dbt test to run automatically before deployments. Alternatively, integrate tests into your CI/CD pipeline with <strong>GitHub Actions</strong> or <strong>GitLab CI/CD</strong>.</p><p>Example GitHub Actions workflow (.github/workflows/dbt.yml):</p><pre>jobs:<br>  dbt-test:<br>    runs-on: ubuntu-latest<br>    steps:<br>      - name: Checkout repo<br>        uses: actions/checkout@v3<br>      - name: Set up dbt<br>        run: pip install dbt-core dbt-snowflake<br>      - name: Run dbt tests<br>        run: dbt test</pre><p>Now, <strong>every pull request will run dbt tests automatically</strong>. If a test fails, the pipeline stops, preventing broken data from reaching production.</p><h3>Final Thoughts</h3><p>Think of automated dbt testing like brushing your teeth — you don’t notice when it works, but you’ll <em>definitely</em> notice when it doesn’t. Nobody wants to spend hours sifting through broken reports, manually tracking down why a key metric suddenly plummeted. Good testing isn’t just about avoiding problems — it’s about building trust in your data so that when things do go wrong, you catch them before they turn into a full-blown crisis.</p><p>Let’s break it down one last time:</p><p>✅ <strong>Built-in schema tests</strong> give you easy, foundational checks.<br>✅ <strong>Custom SQL tests</strong> let you catch the weird, business-specific edge cases.<br>✅ <strong>Generic tests</strong> save you from rewriting the same logic 15 times.<br>✅ <strong>Automation</strong> ensures issues are flagged before your CEO asks, “Why is revenue negative?”</p><p>Set up your tests, automate them, and enjoy the confidence of knowing your data is as rock-solid as your best analytics insights. 🚀</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c02ee5007189" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering Python Coding Challenges: A Guide to Problem-Solving Efficiency]]></title>
            <link>https://medium.com/@p.r.zawieja/mastering-python-coding-challenges-a-guide-to-problem-solving-efficiency-fa0bc85041d9?source=rss-5309f9bf426e------2</link>
            <guid isPermaLink="false">https://medium.com/p/fa0bc85041d9</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[interview]]></category>
            <category><![CDATA[cheatsheet]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[interview-tips]]></category>
            <dc:creator><![CDATA[Piotr Zawieja]]></dc:creator>
            <pubDate>Wed, 12 Feb 2025 09:46:34 GMT</pubDate>
            <atom:updated>2025-02-13T21:38:52.630Z</atom:updated>
            <content:encoded><![CDATA[<p>Summary of my 30 days of tackling the coding challenges and cheat sheets that help you prepare for an interview.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*V-RJuvRjZNGkBLpd" /></figure><p>A month ago, I set a goal for myself:</p><blockquote><strong>“Solve 5 Python beginner and intermediate coding challenges daily for 30 days.” </strong>💡</blockquote><p>I asked ChatGPT to send me problems, aiming for <strong>150+ challenges</strong>. While I know Python well, algorithmic exercises always felt tricky. This structured practice helped me spot patterns, refine my approach, and improve efficiency.</p><p><strong>But let’s be real — coding challenges are becoming less relevant</strong> with no-code tools and AI-assisted development. In real-world jobs, system design and business impact often matter more. <strong>Yet, they’re still a major hurdle in tech interviews.</strong></p><p>If you need to pass one (or just want to sharpen your skills), this guide covers <strong>key insights, common pitfalls, and smart techniques</strong> to tackle them efficiently.</p><h3>Why Do Coding Challenges Feel Difficult?</h3><p>Even if you have programming experience, solving algorithmic problems is a different skill set.</p><p>Some common difficulties include:</p><ul><li><strong>Understanding the problem statement clearly</strong> (especially if it’s phrased ambiguously).</li><li><strong>Finding an optimal solution instead of brute force</strong>.</li><li><strong>Handling edge cases</strong> that might break your implementation.</li><li><strong>Managing time pressure</strong> in competitive programming or interviews.</li></ul><p>However, the more problems you solve, the more you recognize <strong>patterns and strategies</strong> that make challenges easier to approach.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Y7OIvga254qK7r0g" /><figcaption>Photo by <a href="https://unsplash.com/@glenncarstenspeters?utm_source=medium&amp;utm_medium=referral">Glenn Carstens-Peters</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Key Takeaways from 30 Days of Coding Challenges</h3><h3>1️⃣ Read and Analyze the Problem Before Coding</h3><p>Many errors occur because the problem is <strong>not fully understood before implementation</strong>.</p><ul><li><strong>Rephrase the problem</strong> in your own words.</li><li><strong>Identify the inputs and expected outputs</strong>.</li><li><strong>Consider edge cases upfront</strong>.</li><li><strong>Break the problem into logical steps</strong> before writing code.</li></ul><p><em>📌 </em><strong><em>Example Challenge: Find the second largest number in a list</em></strong><em><br></em>Instead of jumping straight to code, first break it down:</p><ol><li>Remove duplicate values.</li><li>Sort the list in descending order.</li><li>Return the second element.</li></ol><pre>def second_largest(lst):<br>    unique_numbers = sorted(set(lst), reverse=True)<br>    return unique_numbers[1] if len(unique_numbers) &gt; 1 else None<br><br>print(second_largest([5, 1, 9, 3, 9, 7]))  # Output: 7</pre><p><strong>Edge cases considered:</strong></p><ul><li>List with all elements being the same.</li><li>List with fewer than two elements.</li></ul><h3>2️⃣ Use Python’s Built-in Functions to Simplify Code</h3><p>Python provides powerful built-in functions that can drastically <strong>reduce code complexity and improve readability</strong>.</p><p><strong>Common built-in functions that are useful in coding challenges:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FuNBzndpJFLZ9t8ZBphF9w.png" /></figure><p><em>📌 </em><strong><em>Challenge: Find the most frequent element in a list</em></strong></p><pre>from collections import Counter<br><br>def most_frequent(lst):<br>    return Counter(lst).most_common(1)[0][0]<br><br>print(most_frequent([1, 3, 3, 2, 3, 4]))  # Output: 3</pre><ul><li>Counter(lst) counts occurrences of each element.</li><li>.most_common(1) returns the most frequent element in <strong>O(n) time complexity</strong>. It means the function processes the list in a <strong>single scan </strong>(or two, but still linear time) and it doesn’t sort all elements, just finds the max count efficiently.</li></ul><h3>3️⃣ Recognizing Common Coding Patterns</h3><p>Many problems are variations of a <strong>few core patterns</strong>. Identifying these patterns helps solve problems <strong>faster and more confidently</strong>.</p><h4>Dictionary-Based Challenges</h4><p>Dictionaries (dict) are incredibly useful in problem-solving, especially when we need <strong>fast lookups</strong> or <strong>store key-value relationships</strong>.</p><p><em>📌 </em><strong><em>Challenge: Find the First Non-Repeating Character in a String</em></strong></p><pre>def first_unique_char(s):<br>    counts = {}<br>    <br>    # First pass: Count occurrences of each character<br>    for char in s:<br>        counts[char] = counts.get(char, 0) + 1<br>    <br>    # Second pass: Find the first character with a count of 1<br>    for char in s:<br>        if counts[char] == 1:<br>            return char<br>    <br>    # If no unique character is found, return None<br>    return None<br><br># Example usage<br>print(first_unique_char(&quot;swiss&quot;))  # Output: &quot;w&quot;<br>print(first_unique_char(&quot;aabb&quot;))   # Output: None</pre><p><strong>.get() method</strong> prevents KeyError by returning 0 if a character isn’t in counts yet.</p><p><strong>Two-pass approach</strong> ensures we maintain the order of appearance.</p><p><em>📌 </em><strong><em>Challenge: Find the Key with the Maximum Value in a Dictionary</em></strong></p><pre>def max_value_key(d):<br>    return max(d, key=d.get)<br><br># Example usage<br>scores = {&quot;Alice&quot;: 90, &quot;Bob&quot;: 85, &quot;Charlie&quot;: 95}<br>print(max_value_key(scores))  # Output: &quot;Charlie&quot;</pre><p>max(iterable, key=function) finds the key where d.get(key) is the highest.</p><ul><li>This method runs in <strong>O(n) time complexity</strong>, making it efficient.</li></ul><p><em>📌 </em><strong><em>Challenge: Sort Dictionary by Value</em></strong></p><pre>d = {&#39;apple&#39;: 3, &#39;banana&#39;: 1, &#39;cherry&#39;: 2}<br>sorted_dict = sorted(d.items(), key=lambda x: x[1])<br>print(sorted_dict)  # [(&#39;banana&#39;, 1), (&#39;cherry&#39;, 2), (&#39;apple&#39;, 3)]</pre><h3>4️⃣ Edge Cases Matter — Always Consider Them!</h3><p>Many bugs happen due to unexpected edge cases.</p><p>Common edge cases to check:</p><ul><li><strong>Empty input</strong> ([], &quot;&quot;, None)</li><li><strong>Single-element lists</strong> ([5], [&quot;a&quot;])</li><li><strong>Duplicate values</strong> ([1, 1, 1, 2, 2])</li><li><strong>Negative numbers</strong> ([-3, -1, -7])</li><li><strong>Very large values</strong> (handling efficiency for 10⁶ elements)</li></ul><p><em>📌 </em><strong><em>Challenge: Checking if a List is Sorted</em></strong></p><pre>def is_sorted(lst):<br>    return all(lst[i] &lt;= lst[i+1] for i in range(len(lst)-1))<br><br>print(is_sorted([1, 2, 3, 4]))  # Output: True<br>print(is_sorted([1, 3, 2, 4]))  # Output: False<br>print(is_sorted([]))  # Edge case: Empty list</pre><h3>Final Thoughts and Best Practices</h3><p>After 30 days of daily problem-solving, <strong>I feel significantly more confident</strong> when approaching coding challenges.</p><p>✔️ <strong>Key lessons from this journey:</strong></p><ul><li><strong>Breaking problems down</strong> makes implementation easier.</li><li><strong>Python’s built-in functions</strong> often simplify solutions.</li><li><strong>Recognizing common patterns</strong> speeds up problem-solving.</li><li><strong>Testing edge cases early</strong> prevents debugging issues later.</li><li><strong>Consistent practice is key</strong> — 5 challenges per day made a big difference.</li></ul><p>If you want to improve your problem-solving skills, <strong>set a challenge for yourself</strong> and practice daily. It won’t take long before patterns become second nature, and problem-solving feels <strong>structured and predictable rather than stressful.</strong></p><p><strong>What challenges do you find hardest?</strong> Let me know in the comments!</p><h3>Bonus: Cheat Sheets 🚀</h3><p>Enjoy this small yet comprehensive collection of the most common operations asked during the interviews.</p><h3>Strings:</h3><pre># The Ultimate Python String Cheat Sheet<br><br># 1. Creating Strings<br>s1 = &quot;Hello&quot;<br>s2 = &#39;World&#39;<br>multiline = &quot;&quot;&quot;This is<br>a multiline string&quot;&quot;&quot;<br><br># 2. Accessing Characters<br>print(s1[0])   # Output: &#39;H&#39; (first character)<br>print(s1[-1])  # Output: &#39;o&#39; (last character)<br>print(s1[1:4]) # Output: &#39;ell&#39; (substring from index 1 to 3)<br><br># 3. Modifying Strings (Strings are Immutable)<br># Strings cannot be modified directly, but you can create a new string<br>modified_s1 = s1.replace(&quot;H&quot;, &quot;J&quot;)  # Output: &#39;Jello&#39;<br><br># 4. String Concatenation<br>combined = s1 + &quot; &quot; + s2  # Output: &#39;Hello World&#39;<br><br># 5. String Multiplication<br>repeated = &quot;ha&quot; * 3  # Output: &#39;hahaha&#39;<br><br># 6. Checking Membership<br>print(&quot;H&quot; in s1)   # Output: True<br>print(&quot;Z&quot; not in s1)  # Output: True<br><br># 7. String Formatting<br>name = &quot;Alice&quot;<br>age = 25<br>formatted1 = f&quot;My name is {name} and I am {age} years old.&quot;  # f-strings (Recommended)<br>formatted2 = &quot;My name is {} and I am {} years old.&quot;.format(name, age)  # Using format()<br>formatted3 = &quot;My name is %s and I am %d years old.&quot; % (name, age)  # Old-style formatting<br><br># 8. Changing Case<br>lower_s = s1.lower()  # Output: &#39;hello&#39;<br>upper_s = s1.upper()  # Output: &#39;HELLO&#39;<br>title_s = &quot;hello world&quot;.title()  # Output: &#39;Hello World&#39;<br>capitalized_s = &quot;hello world&quot;.capitalize()  # Output: &#39;Hello world&#39;<br><br># 9. Trimming Whitespaces<br>s3 = &quot;  Hello World  &quot;<br>stripped = s3.strip()  # Output: &#39;Hello World&#39;<br>left_stripped = s3.lstrip()  # Output: &#39;Hello World  &#39;<br>right_stripped = s3.rstrip()  # Output: &#39;  Hello World&#39;<br><br># 10. Splitting and Joining Strings<br>words = &quot;apple,banana,cherry&quot;.split(&quot;,&quot;)  # Output: [&#39;apple&#39;, &#39;banana&#39;, &#39;cherry&#39;]<br>joined = &quot;-&quot;.join(words)  # Output: &#39;apple-banana-cherry&#39;<br><br># 11. Reversing a String<br>reversed_s = s1[::-1]  # Output: &#39;olleH&#39;<br><br># 12. Checking String Start/End<br>print(s1.startswith(&quot;He&quot;))  # Output: True<br>print(s1.endswith(&quot;lo&quot;))  # Output: True<br><br># 13. Finding and Replacing Substrings<br>index = s1.find(&quot;lo&quot;)  # Output: 3 (first occurrence)<br>index = s1.index(&quot;lo&quot;)  # Similar to find(), but raises error if not found<br>replaced = s1.replace(&quot;Hello&quot;, &quot;Hi&quot;)  # Output: &#39;Hi&#39;<br><br># 14. Counting Occurrences<br>count_l = s1.count(&quot;l&quot;)  # Output: 2<br><br># 15. Checking if String is Numeric, Alphabetic, or Alphanumeric<br>numeric = &quot;123&quot;.isdigit()  # Output: True<br>alpha = &quot;abc&quot;.isalpha()  # Output: True<br>alphanumeric = &quot;abc123&quot;.isalnum()  # Output: True<br><br># 16. Padding Strings (Useful for Formatting)<br>padded = &quot;42&quot;.zfill(5)  # Output: &#39;00042&#39;<br>left_padded = &quot;42&quot;.rjust(5, &quot;0&quot;)  # Output: &#39;00042&#39;<br>right_padded = &quot;42&quot;.ljust(5, &quot;0&quot;)  # Output: &#39;42000&#39;<br><br># 17. Removing Characters from a String<br>cleaned = &quot;Hello!!!&quot;.strip(&quot;!&quot;)  # Output: &#39;Hello&#39;<br><br># 18. Splitting a Multiline String<br>multiline_text = &quot;Hello\nWorld\nPython&quot;<br>lines = multiline_text.splitlines()  # Output: [&#39;Hello&#39;, &#39;World&#39;, &#39;Python&#39;]<br><br># 19. Checking Character Case<br>print(&quot;hello&quot;.islower())  # Output: True<br>print(&quot;HELLO&quot;.isupper())  # Output: True<br>print(&quot;Hello&quot;.istitle())  # Output: True<br><br># 20. Encoding and Decoding Strings<br>encoded = &quot;Hello&quot;.encode(&quot;utf-8&quot;)  # Converts string to bytes<br>decoded = encoded.decode(&quot;utf-8&quot;)  # Converts bytes back to string<br><br># 21. Using translate() for Character Mapping<br>table = str.maketrans(&quot;aeiou&quot;, &quot;12345&quot;)  # Mapping vowels to numbers<br>translated = &quot;hello world&quot;.translate(table)  # Output: &#39;h2ll4 w4rld&#39;<br><br># 22. Checking if a String is a Palindrome<br>def is_palindrome(s):<br>    return s == s[::-1]<br><br>print(is_palindrome(&quot;madam&quot;))  # Output: True<br>print(is_palindrome(&quot;hello&quot;))  # Output: False</pre><h3>Dictionaries:</h3><pre># The Ultimate Python Dictionary Cheat Sheet<br><br># 1. Creating a Dictionary<br>my_dict = {&quot;name&quot;: &quot;Alice&quot;, &quot;age&quot;: 25, &quot;city&quot;: &quot;New York&quot;}<br>empty_dict = {}<br>person = dict(name=&quot;Bob&quot;, age=30, job=&quot;Developer&quot;)<br><br># Creating a dictionary from lists (keys &amp; values)<br>keys = [&quot;name&quot;, &quot;age&quot;, &quot;city&quot;]<br>values = [&quot;Alice&quot;, 25, &quot;New York&quot;]<br>person_dict = dict(zip(keys, values))<br><br># Dictionary with default values<br>default_dict = dict.fromkeys([&quot;a&quot;, &quot;b&quot;, &quot;c&quot;], 0)<br><br># 2. Accessing Dictionary Values<br>print(my_dict[&quot;name&quot;])  # Output: Alice<br>print(my_dict.get(&quot;age&quot;))  # Output: 25<br>print(my_dict.get(&quot;salary&quot;, &quot;Not Found&quot;))  # Output: Not Found<br><br># Accessing keys, values, and items<br>print(my_dict.keys())   # Output: dict_keys([&#39;name&#39;, &#39;age&#39;, &#39;city&#39;])<br>print(my_dict.values()) # Output: dict_values([&#39;Alice&#39;, 25, &#39;New York&#39;])<br>print(my_dict.items())  # Output: dict_items([(&#39;name&#39;, &#39;Alice&#39;), (&#39;age&#39;, 25), (&#39;city&#39;, &#39;New York&#39;)])<br><br># 3. Adding &amp; Updating Dictionary Values<br>my_dict[&quot;job&quot;] = &quot;Engineer&quot;  # Add new key-value pair<br>my_dict[&quot;age&quot;] = 26  # Update existing key<br><br># Merging dictionaries<br>dict1 = {&quot;a&quot;: 1, &quot;b&quot;: 2}<br>dict2 = {&quot;b&quot;: 10, &quot;c&quot;: 3}<br>dict1.update(dict2)  # Modifies dict1<br>merged_dict = {**dict1, **dict2}  # Creates new dictionary<br><br># 4. Removing Dictionary Elements<br>del my_dict[&quot;city&quot;]  # Deletes key &quot;city&quot;<br>age = my_dict.pop(&quot;age&quot;)  # Removes and returns value of &quot;age&quot;<br>my_dict.clear()  # Removes all elements<br><br># 5. Dictionary Loops &amp; Comprehensions<br>for key, value in my_dict.items():<br>    print(f&quot;{key}: {value}&quot;)<br><br># Dictionary comprehension<br>squared_dict = {x: x**2 for x in range(5)}<br>print(squared_dict)  # Output: {0: 0, 1: 1, 2: 4, 3: 9, 4: 16}<br><br># 6. Sorting a Dictionary<br>sorted_dict = dict(sorted(my_dict.items()))  # Sort by keys<br>sorted_by_value = dict(sorted(my_dict.items(), key=lambda item: item[1]))  # Sort by values<br><br># Finding max &amp; min values in a dictionary<br>scores = {&quot;Alice&quot;: 90, &quot;Bob&quot;: 85, &quot;Charlie&quot;: 95}<br>max_key = max(scores, key=scores.get)  # Key with max value<br>min_key = min(scores, key=scores.get)  # Key with min value<br><br># 7. Advanced Dictionary Methods<br># Default values with .setdefault()<br>person = {&quot;name&quot;: &quot;Alice&quot;}<br>person.setdefault(&quot;age&quot;, 25)  # Adds &quot;age&quot; if it doesn&#39;t exist<br><br># Counting elements with Counter<br>from collections import Counter<br>word_counts = Counter(&quot;mississippi&quot;)<br><br># Grouping data using defaultdict<br>from collections import defaultdict<br>grouped = defaultdict(list)<br>data = [(&quot;Alice&quot;, &quot;Math&quot;), (&quot;Bob&quot;, &quot;Science&quot;), (&quot;Alice&quot;, &quot;Physics&quot;)]<br>for name, subject in data:<br>    grouped[name].append(subject)<br><br># Handling missing keys gracefully with defaultdict<br>word_count = defaultdict(int)  # Default value of int() is 0<br>word_count[&quot;hello&quot;] += 1  # No KeyError<br><br># Merging dictionaries using ChainMap<br>from collections import ChainMap<br>combined = ChainMap(dict1, dict2)<br>print(combined[&quot;b&quot;])  # Output: 2 (Takes value from dict1)</pre><h3>Lists:</h3><pre># The Ultimate Python List Cheat Sheet<br><br># 1. Creating a List<br>my_list = [1, 2, 3, 4, 5]<br>empty_list = []<br>mixed_list = [1, &quot;hello&quot;, 3.5, True]<br><br># Creating a list with repeated values<br>repeated_list = [0] * 5  # Output: [0, 0, 0, 0, 0]<br><br># Creating a list from a range<br>range_list = list(range(1, 6))  # Output: [1, 2, 3, 4, 5]<br><br># 2. Accessing Elements<br>print(my_list[0])   # Output: 1 (first element)<br>print(my_list[-1])  # Output: 5 (last element)<br>print(my_list[1:4]) # Output: [2, 3, 4] (slicing from index 1 to 3)<br><br># 3. Modifying a List<br>my_list.append(6)  # Adds 6 to the end<br>my_list.insert(2, 10)  # Inserts 10 at index 2<br>my_list[1] = 99  # Changes the second element to 99<br><br># 4. Removing Elements<br>my_list.remove(3)  # Removes first occurrence of 3<br>popped = my_list.pop()  # Removes and returns last element<br>del my_list[1]  # Deletes element at index 1<br>my_list.clear()  # Removes all elements<br><br># 5. List Operations<br>numbers = [1, 2, 3, 4, 5]<br>print(len(numbers))  # Output: 5 (number of elements)<br>print(sum(numbers))  # Output: 15 (sum of all elements)<br>print(min(numbers))  # Output: 1 (smallest element)<br>print(max(numbers))  # Output: 5 (largest element)<br><br># 6. Checking Membership<br>print(3 in numbers)   # Output: True<br>print(10 not in numbers)  # Output: True<br><br># 7. List Comprehensions<br>squared = [x**2 for x in range(5)]  # Output: [0, 1, 4, 9, 16]<br>evens = [x for x in range(10) if x % 2 == 0]  # Output: [0, 2, 4, 6, 8]<br><br># 8. Sorting a List<br>unsorted_list = [5, 3, 8, 1, 9]<br>unsorted_list.sort()  # Sorts list in-place (ascending)<br>unsorted_list.sort(reverse=True)  # Sorts list in descending order<br>sorted_list = sorted(unsorted_list)  # Returns a new sorted list<br><br># 9. Reversing a List<br>numbers.reverse()  # In-place reversal<br>reversed_numbers = numbers[::-1]  # Returns a reversed copy<br><br># 10. Finding the Index of an Element<br>index = numbers.index(3)  # Returns first occurrence of 3<br><br># 11. Counting Elements<br>count_ones = numbers.count(1)  # Counts occurrences of 1<br><br># 12. Extending a List<br>list1 = [1, 2, 3]<br>list2 = [4, 5, 6]<br>list1.extend(list2)  # Adds elements of list2 to list1<br>merged_list = list1 + list2  # Another way to merge lists<br><br># 13. Copying a List<br>copy1 = my_list.copy()  # Creates a shallow copy<br>copy2 = my_list[:]  # Another way to copy a list<br><br># 14. Looping Through a List<br>for item in numbers:<br>    print(item)<br><br># Looping with index<br>for index, value in enumerate(numbers):<br>    print(f&quot;Index {index}: Value {value}&quot;)<br><br># 15. Filtering a List<br>filtered_list = list(filter(lambda x: x % 2 == 0, numbers))  # Keeps only even numbers<br><br># 16. Flattening a Nested List<br>nested_list = [[1, 2], [3, 4], [5, 6]]<br>flattened = [item for sublist in nested_list for item in sublist]<br><br># 17. Using zip() to Pair Elements from Two Lists<br>names = [&quot;Alice&quot;, &quot;Bob&quot;, &quot;Charlie&quot;]<br>ages = [25, 30, 35]<br>paired = list(zip(names, ages))  # Output: [(&#39;Alice&#39;, 25), (&#39;Bob&#39;, 30), (&#39;Charlie&#39;, 35)]<br><br># 18. Using map() to Apply a Function to a List<br>squared_numbers = list(map(lambda x: x**2, numbers))  # Squares each element<br><br># 19. Using reduce() to Aggregate a List<br>from functools import reduce<br>product = reduce(lambda x, y: x * y, numbers)  # Multiplies all elements<br><br># 20. Using all() and any()<br>print(all(x &gt; 0 for x in numbers))  # Returns True if all elements are &gt; 0<br>print(any(x &gt; 100 for x in numbers))  # Returns True if at least one element &gt; 100</pre><h3>Slicing:</h3><pre># Python Slicing Cheatsheet<br><br># Basic Syntax: sequence[start:stop:step]<br><br># 1. Basic Slicing<br>my_list = [0, 1, 2, 3, 4, 5]<br>print(my_list[1:4])  # Output: [1, 2, 3] (start at index 1, stop before index 4)<br><br># 2. Omitting start or stop<br>print(my_list[:3])   # Output: [0, 1, 2] (start from the beginning)<br>print(my_list[3:])   # Output: [3, 4, 5] (go to the end)<br><br># 3. Using step<br>print(my_list[::2])  # Output: [0, 2, 4] (every 2nd element)<br>print(my_list[1:5:2]) # Output: [1, 3] (start at 1, stop before 5, step by 2)<br><br># 4. Reversing a sequence<br>print(my_list[::-1]) # Output: [5, 4, 3, 2, 1, 0] (step of -1 reverses the list)<br><br># 5. Negative indices<br>print(my_list[-3:])  # Output: [3, 4, 5] (last 3 elements)<br>print(my_list[:-2])  # Output: [0, 1, 2, 3] (exclude the last 2 elements)<br><br># 6. Removing first and last characters (strings)<br>my_string = &quot;hello&quot;<br>print(my_string[1:-1]) # Output: &quot;ell&quot; (exclude the first and last character)<br><br># 7. Extract every third character<br>text = &quot;abcdefghijkl&quot;<br>print(text[::3]) # Output: &quot;adgj&quot; (every 3rd character)<br><br># 8. Slice only the first `n` elements<br>fib = [0, 1, 1, 2, 3, 5, 8, 13]<br>n = 5<br>print(fib[:n]) # Output: [0, 1, 1, 2, 3] (first 5 elements)</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fa0bc85041d9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Decoding A/B Testing Results: A Guide to Understanding Key Metrics and Insights]]></title>
            <link>https://medium.com/@p.r.zawieja/decoding-a-b-testing-results-a-guide-to-understanding-key-metrics-and-insights-5344713dbc18?source=rss-5309f9bf426e------2</link>
            <guid isPermaLink="false">https://medium.com/p/5344713dbc18</guid>
            <category><![CDATA[product]]></category>
            <category><![CDATA[a-b-testing]]></category>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Piotr Zawieja]]></dc:creator>
            <pubDate>Mon, 20 Jan 2025 17:09:08 GMT</pubDate>
            <atom:updated>2025-01-28T16:46:29.844Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Decoding A/B Testing Results</strong></h3><h4>A Guide to Understanding Key Metrics and Insights</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fHqPSQxjGboY-sN7" /></figure><p>Imagine you’re managing an online store, and you’ve just launched a new product page design. You’re curious: will this design boost sales? To find out, you decide to run an A/B test. The results come in, and you see a slight increase in conversions. But is that enough to declare the new design a winner? This is where understanding A/B testing metrics becomes essential.</p><p>In this guide, we’ll break down how to interpret A/B test results in a way that’s both <strong>actionable</strong> and <strong>memorable</strong>. Whether you’re a seasoned analyst or an aspiring one, you’ll learn to confidently read the numbers and make data-driven decisions — without the guesswork.</p><h3>The Basics of A/B Testing</h3><h4>What is A/B Testing?</h4><p>At its core, A/B testing is like a controlled experiment. You show one group (the control) your current experience, while another group (the variation) sees a new version. By comparing how each group performs, you can determine which version works better.</p><h4>Key Components:</h4><ul><li><strong>Control vs. Variation:</strong> Control is the original version, and the variation is the new one you’re testing. Think of it as comparing a recipe you’ve always used (control) to a new twist (variation).</li><li><strong>Sample Size and Randomization:</strong> For reliable results, ensure your audience is randomly divided and that your sample size is large enough to detect meaningful differences. Use tools like power calculators to estimate the sample size needed based on your Minimum Detectable Effect (MDE).</li><li><strong>Hypotheses:</strong> Start with a clear hypothesis. For example: “The new design will increase conversions by 10%.”</li></ul><h3>Key Metrics to Monitor</h3><h4>Primary Metrics</h4><p>Primary metrics are the most critical indicators directly tied to your test’s goal. These are the metrics you define before starting the experiment, and they should represent the success criteria of your test.</p><p>🎯 <strong>Conversion Rate (CR):</strong> This metric indicates the percentage of users who complete a desired action, such as making a purchase, signing up for a newsletter, or clicking a button. It’s calculated as:</p><ul><li><strong><em>CR</em></strong><em> = (Number of Conversions / Total Visitors)​ ×100</em></li><li><strong>Example:</strong> If 1,000 users visit your website and 50 make a purchase, the CR is 5%.</li></ul><p>🎯 <strong>Lift:</strong> This measures the improvement observed in the variation compared to the control.</p><ul><li><strong><em>Absolute Lift </em></strong><em>= CR (Variation)−CR (Control)</em></li><li><strong><em>Relative Lift</em></strong><em> = ((CR (Variation)−CR (Control)) / CR (Control)) ​×100</em></li><li><strong>Example:</strong> If the control’s CR is 5% and the variation’s CR is 6%, the absolute lift is 1%, and the relative lift is 20%.</li></ul><h4>Secondary Metrics</h4><p>While secondary metrics aren’t the primary focus of your test, they <strong>provide context and help identify unintended consequences</strong> of your changes. Key secondary metrics include:</p><p>📊 <strong>Bounce Rate:</strong> The percentage of visitors who leave without interacting with the page. A sudden increase might indicate that your change confused or frustrated users.</p><p>📊 <strong>Average Order Value (AOV):</strong> This reflects the average amount spent per transaction and helps assess whether the change influenced purchasing behavior.</p><ul><li><strong><em>AOV</em></strong><em> = Total Revenue / Number of Transactions</em></li></ul><p>📊 <strong>Retention Rate:</strong> Measures how many users return to your platform after their initial visit. A decrease in retention could indicate long-term dissatisfaction with the variation.</p><h4>Guardrail Metrics</h4><p>Guardrail metrics act as safeguards to ensure that while optimizing primary metrics, <strong>other critical areas of the business are not negatively impacted</strong>. These metrics help maintain a balanced perspective.</p><p>👀 <strong>Revenue Per Visitor (RPV):</strong> Ensures that an increase in conversion rate doesn’t come at the cost of lower revenue. For example, offering heavy discounts might boost conversions but harm profitability.</p><p>👀 <strong>Customer Satisfaction (CSAT):</strong> Tracks user satisfaction levels through surveys or feedback mechanisms. A drop in CSAT might signal long-term dissatisfaction despite short-term gains.</p><p>👀 <strong>Infrastructure Metrics (e.g., Page Load Time):</strong> Monitors technical performance to ensure changes don’t slow down page loading, which could lead to user frustration and increased bounce rates.</p><p>👀 <strong>Churn Rate:</strong> Ensures that changes don’t inadvertently lead to higher churn by alienating existing customers.</p><p>Guardrail metrics provide a safety net and help you evaluate trade-offs more effectively, ensuring that positive changes don’t have unintended consequences.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*F2mIucSqgqJPmgxk" /></figure><h4>Statistical Significance Metrics</h4><p>Statistical significance ensures that the observed results are <strong>unlikely due to random chance.</strong></p><p><strong>P-value:</strong> The probability of obtaining results as extreme as the observed data if the null hypothesis is true. A commonly accepted threshold for significance is p&lt;0.05p &lt; 0.05.</p><ul><li><strong>Example:</strong> If the p-value is 0.03, there’s a 3% chance the results occurred by random chance, suggesting the variation’s effect is significant.</li></ul><p><strong>Confidence Interval (CI):</strong> The range within which the true value of the metric is likely to lie. A narrower CI indicates higher precision.</p><ul><li><strong>Example:</strong> If the CI is “4.8% to 5.2%,” it means the real value is probably between 4.8% and 5.2%.</li><li>A <strong>narrower range</strong> (e.g., 4.9% to 5.1%) means we are more precise.</li><li>A <strong>wider range</strong> (e.g., 4.5% to 5.5%) means less certainty.</li></ul><p>It’s like saying, “We’re confident the true value is somewhere in this range.”</p><h4>Minimum Detectable Effect (MDE)</h4><p>MDE represents the smallest change in your primary metric that you want to detect with confidence. It’s a critical input when calculating the required sample size for your test.</p><p><strong>Example:</strong> If your baseline CR is 5% and your MDE is 1%, you’re testing for a minimum improvement to 6%. Smaller MDE values require larger sample sizes.</p><h4>Statistical Power</h4><p>Statistical power is the probability of detecting a true effect if one exists. A standard power level is 80%, meaning there’s an 80% chance your test will correctly identify a real difference.</p><ul><li><strong>Tip:</strong> Low power increases the risk of Type II errors (failing to detect a true effect).</li></ul><h4>Sample Size</h4><p>The required sample size depends on the baseline metric, MDE, power, and significance level (commonly set at 5%). Calculating an adequate sample size ensures your test results are reliable and reduces the risk of false conclusions.</p><ul><li><strong>Formula Example:</strong> Use statistical software or online calculators to determine sample size based on inputs like baseline CR and desired MDE.</li><li><strong>Practical Note:</strong> Running tests with too small a sample size may lead to inconclusive or misleading results.</li></ul><h3>Effect Size and Coefficients</h3><p>Effect size shows <strong>how big the difference</strong> is between groups. It helps you understand the real impact of a change.</p><ul><li><strong>In A/B Testing:</strong> It measures how much the variation (like a new design) affects something like the conversion rate (CR).</li><li><strong>Using Regression:</strong> Coefficients help figure out the variation’s impact while considering other factors, like device type or traffic source.</li></ul><p>It’s like saying, “Here’s the size of the change, and here’s what caused it.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tLT9GRJYfJMvK1yM" /></figure><h3>Interpreting A/B Test Results</h3><h4>Step 1: Look at the Primary Metric</h4><p>Focus first on the primary metric you’re testing. Did the new design improve conversions?</p><ul><li>Example: Control group CR = 5%. Variation group CR = 6%. That’s a 1% absolute lift, or a 20% relative improvement.</li></ul><h4>Step 2: Check Statistical Significance</h4><p>Examine the p-value to determine if the result is statistically significant:</p><ul><li>If p &lt; 0.05: You can confidently say the change isn’t due to chance.</li><li>If p &gt; 0.05: Consider running the test longer or revising your hypothesis.</li></ul><h4>Step 3: Validate with Secondary Metrics</h4><p>Analyze secondary metrics to ensure the change doesn’t introduce trade-offs.</p><ul><li>Example: Your new design improved conversions but increased the bounce rate. This could indicate user frustration.</li></ul><h4>Step 4: Analyze the Broader Impact</h4><p>Always tie your results back to business objectives. For instance:</p><ul><li>Did the higher conversion rate result in more revenue? If not, was it due to a drop in AOV?</li></ul><h3>Common Pitfalls to Avoid 🔍</h3><h4>Stopping Tests Too Early</h4><p>It’s tempting to declare victory when results look promising, but stopping early can lead to false conclusions. Ensure you’ve reached your planned sample size.</p><ul><li>Tip: Think of it like baking a cake. If you take it out too early, it won’t be fully cooked.</li></ul><h4>Chasing Statistical Significance</h4><p>Statistical significance doesn’t always mean practical significance. A small but significant change might not justify the effort or cost.</p><ul><li>Example: A 0.1% lift in CR might not be worth implementing if it doesn’t impact revenue.</li></ul><h4>Ignoring Segment-Level Analysis</h4><p>Results may vary across different user segments (e.g., mobile vs. desktop users). Drill down to identify where the change has the most impact.</p><h4>Overlooking External Factors</h4><p>Consider external events that could skew results, such as holidays or marketing campaigns. Always run tests in a controlled environment.</p><h3>Real-World Example</h3><p>Let’s say you run an A/B test on your e-commerce site to test a new checkout button design:</p><p><strong>Hypothesis:</strong> A larger, brightly colored button will increase conversions by 5%.</p><p><strong>Results:</strong></p><ul><li>Control CR = 4.5% (45 purchases from 1,000 visitors)</li><li>Variation CR = 5.5% (55 purchases from 1,000 visitors)</li><li>MDE = 0.5%, and sample size met the required threshold.</li><li>P-value = 0.03, indicating statistical significance.</li><li>Bounce rate stayed the same, and AOV increased slightly. This suggests the change is positive overall.</li></ul><p><strong>Decision:</strong> Implement the new button design. 🚀</p><h3>Key Takeaways</h3><ul><li>Focus on the metrics that matter most for your goal.</li><li>Always validate your results with statistical and practical significance.</li><li>Avoid common pitfalls like stopping too early or ignoring segment-level insights.</li><li>Use MDE, power, and sample size calculations to ensure robust test designs.</li></ul><p>By mastering these steps, you’ll transform A/B testing results into actionable insights that drive real impact.</p><h3>Conclusion</h3><p>Interpreting A/B test results doesn’t have to be overwhelming. With a clear framework and a focus on the right metrics, you can make informed decisions that benefit your business. Whether you’re testing website changes, marketing campaigns, or product features, remember: good analysis leads to great outcomes.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5344713dbc18" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Build a Flexible A/B Testing Framework in Python]]></title>
            <link>https://medium.com/@p.r.zawieja/how-to-build-a-flexible-a-b-testing-framework-in-python-d60033e4d6e1?source=rss-5309f9bf426e------2</link>
            <guid isPermaLink="false">https://medium.com/p/d60033e4d6e1</guid>
            <category><![CDATA[ab-testing]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[learning]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Piotr Zawieja]]></dc:creator>
            <pubDate>Sun, 12 Jan 2025 16:38:40 GMT</pubDate>
            <atom:updated>2025-01-13T08:19:04.842Z</atom:updated>
            <content:encoded><![CDATA[<p>Unlock Meaningful Insights with Univariate A/B Testing</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*72wcZ2R_Irf3E0Tm" /></figure><h3>Introduction</h3><p>Imagine you are a Product Analyst at MyFitness, the team behind a popular workout tracker app. It’s January, and users are setting ambitious resolutions to stay consistent with their daily workouts. Your team developed a new feature called <strong>Daily Streak Tracker</strong> — a dashboard that motivates users to keep logging workouts by visually tracking their streaks.</p><p>This new year, you want to know: does this feature really boost engagement or is it just a nice-to-have?</p><p>That’s where <strong>A/B testing</strong> comes in! In this post, I’ll show you how to run an A/B test using Python — from planning how many users you need to interpreting the results with statistical confidence (don’t worry, I’ll explain what that means).</p><p>Let’s dive in!</p><h3>1. The Pre-Test Planning: Don’t Launch Without a Game Plan!</h3><h4>Why Do We Need to Plan the Sample Size?</h4><p>Think of sample size planning as making sure enough users interact with your feature to make a fair comparison. If only a small number of users try the new dashboard, you won’t know if the results are reliable. But if thousands try it, you’ll feel more confident in your conclusion.</p><p>Key ingredients for sample size planning:</p><ul><li><strong>Baseline conversion rate:</strong> The percentage of users who usually log a workout every day in the original version (let’s say 10%).</li><li><strong>Minimum detectable effect (MDE):</strong> The smallest improvement worth measuring (like a 2% increase).</li><li><strong>Significance level (α):</strong> The probability of wrongly declaring the new feature better when it’s not (usually set to 5%).</li><li><strong>Power:</strong> The probability of detecting a real improvement if it exists (often set to 80%).</li></ul><h3>2. The Pre-Test Sample Size Script (A Blueprint for Success)</h3><p>In this framework, the parameters for the A/B test are stored in a configuration dictionary:</p><pre># Configuration: A/B Test Parameters<br>AB_TEST_CONFIG = {<br>    &quot;file_path&quot;: &#39;ab_test_dataset.csv&#39;,<br>    &quot;alpha&quot;: 0.05,  # Significance level<br>    &quot;power&quot;: 0.8,   # Statistical power<br>    &quot;confidence_interval&quot;: 0.95,  # Confidence interval<br>    &quot;baseline_rate&quot;: 0.1,  # Expected baseline conversion rate<br>    &quot;mde&quot;: 0.02,  # Minimum detectable effect (e.g., 2% increase)<br>    &quot;n_samples&quot;: 1000,  # Total number of samples to generate (for synthetic data)<br>}</pre><p>We’ll use this configuration throughout the A/B testing process.</p><h4>Sample Size Calculation Script</h4><p>Here’s a Python script that calculates how many users you need for a fair test:</p><pre>import numpy as np<br>from statsmodels.stats.power import TTestIndPower<br><br>def calculate_required_sample_size(baseline_rate, mde, alpha=0.05, power=0.8):<br>    effect_size = mde / np.sqrt(baseline_rate * (1 - baseline_rate))<br>    required_sample_size = TTestIndPower().solve_power(effect_size=effect_size, alpha=alpha, power=power)</pre><h4>Running the script:</h4><p>To run the pre_test_summary function using values from AB_TEST_CONFIG:</p><pre>pre_test_summary(<br>    AB_TEST_CONFIG[&quot;baseline_rate&quot;],<br>    AB_TEST_CONFIG[&quot;mde&quot;],<br>    AB_TEST_CONFIG[&quot;alpha&quot;],<br>    AB_TEST_CONFIG[&quot;power&quot;]<br>)</pre><p>What it prints:</p><pre>Pre-Test Planning Summary:<br>----------------------------------------<br>Baseline conversion rate: 10.00%<br>Minimum detectable effect (MDE): 2.00%<br>Significance level (alpha): 0.05<br>Statistical power: 0.8<br>----------------------------------------<br>Required sample size per group: 3533 samples<br>Total required sample size (Control + Variant): 7066 samples<br>Ensure that both groups have at least 3533 participants to achieve the desired power.</pre><p>In simpler terms: You need <strong>3,533 users per group</strong> to confidently detect a 2% improvement in engagement. That’s a lot of users, but this way, you can be sure the results aren’t just due to random chance.</p><h3>3. Post-Test Analysis: What Do the Numbers Tell Us?</h3><p>Once you’ve gathered enough data, it’s time to analyze whether the Daily Streak Tracker is truly encouraging users to stick to their New Year’s fitness resolutions or if both dashboards perform about the same.</p><h4>The A/B Testing Framework Script: Let’s Crunch Some Numbers</h4><p>This Python script takes your dataset, runs a statistical test, and provides a friendly summary of the results.</p><p>This Python script takes your dataset, runs a statistical test, and provides a friendly summary of the results.</p><h4>Step 1: Load the Data</h4><p>In the run_ab_testing_framework, the load_data function is configured to take the file path directly from AB_TEST_CONFIG without needing to pass it separately. Here’s how to adjust the function to read the path directly:</p><pre>def load_data(config):<br>    file_path = config[&quot;file_path&quot;]<br>    n_samples = config[&quot;n_samples&quot;]<br><br>    if not os.path.exists(file_path):<br>        print(&quot;No valid file path provided or file not found. Using synthetic dataset.&quot;)<br>        return generate_synthetic_data(n_samples)<br><br>    print(f&quot;Loading data from {file_path}&quot;)<br>    df = pd.read_csv(file_path)<br>    print(f&quot;Loaded {len(df)} records from CSV.&quot;)<br><br>    if not REQUIRED_COLUMNS.issubset(df.columns):<br>        raise ValueError(f&quot;Missing required columns: {REQUIRED_COLUMNS - set(df.columns)}&quot;)<br>    return df</pre><p>This ensures that the file_path is always pulled from the configuration dictionary.</p><p>Here’s how our sample dataset looks like:</p><figure><img alt="Sample records from our dataset." src="https://cdn-images-1.medium.com/max/522/1*YqVcqhKGwOx9lkj7npELzg.png" /></figure><h4>Step 2: Perform the A/B Test</h4><pre>test_results = perform_ab_test(df, AB_TEST_CONFIG[&quot;alpha&quot;], AB_TEST_CONFIG[&quot;power&quot;], use_welch=True)</pre><p>This performs a Welch’s t-test (which doesn’t assume equal variance) and returns key metrics:</p><ul><li><strong>p-value:</strong> The chance you’d see this difference if both dashboards were equally effective. A p-value below 0.05 means the difference is statistically significant.</li><li><strong>Effect size (Cohen’s h):</strong> How big is the difference, scaled by variability (like comparing how consistent user engagement is between the versions).</li><li><strong>Post-hoc power:</strong> Did we have enough data to detect this difference reliably?</li></ul><h4>Step 3: Interpreting the Results</h4><p>Here’s what the script prints when analyzing a dataset with 4600 samples:</p><pre>Loading data from ab_test_dataset.csv<br>Loaded 4600 records from CSV.<br>Cleaned data: 4600 records (removed 0 records due to missing values).<br>Using Welch&#39;s t-test (equal_var=False) for unequal variances.<br><br>Conversion Rates Summary:<br>         Conversions  Total      Rate<br>Group                                <br>Control          254   2600  0.097692<br>Variant          267   2000  0.133500<br><br>Required sample size per group for detecting the MDE: 3533 samples<br>Observed required sample size per group (based on observed effect size): 1081 samples<br>Sample size per group used: 2600 samples<br>⚠️ Sample size is insufficient for detecting the MDE.<br>✅ Sample size is also sufficient based on the observed effect size.<br><br>Post-hoc power of the test: 0.98<br>Absolute difference in rates (Variant - Control): 0.0358<br>Standardized effect size (Cohen&#39;s h): 0.1206<br>p-value: 0.0002 (statistically significant)<br><br>Confidence Intervals:<br>Control: (0.08627260588580983, 0.10911200949880555)<br>Variant: (0.11858134585013583, 0.1484186541498642)</pre><p><em>In Plain English:</em></p><ol><li><strong>Observed vs Required Sample Size:</strong> Although we thought we’d need <strong>3,533 users</strong> per group, the actual difference in engagement between the dashboards was larger than expected, so a sample size of <strong>1,081</strong> users per group would have sufficed.</li><li><strong>Statistical Significance:</strong> The p-value of <strong>0.0002</strong> means there’s a 0.02% chance this difference is due to random noise.</li><li><strong>Post-Hoc Power:</strong> A power of <strong>0.98</strong> means we’re 98% confident we had enough data to detect a real difference.</li></ol><h3>4. Visualizing the Confidence Intervals</h3><p>The script also creates a bar chart showing the confidence intervals for both the control and variant groups:</p><pre>plot_confidence_intervals(confidence_intervals)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*os_aAiC4YIuflBsOi8spPQ.png" /></figure><h3>5. Lessons Learned (and Fitness Motivation)</h3><p>This A/B testing framework demonstrates a few key takeaways:</p><ol><li><strong>Plan your sample size:</strong> Don’t run an experiment without knowing how much data you need.</li><li><strong>Look beyond p-values:</strong> Effect sizes and post-hoc power help you understand if the difference is meaningful and if you had enough data.</li><li><strong>Data storytelling matters:</strong> Charts and clear summaries make it easier to communicate results.</li></ol><h3>6. Try It Yourself!</h3><p>To run your own A/B test:</p><ol><li>Check github for the <a href="https://github.com/PZawieja/Machine_Learning/tree/main/notebooks">notebook</a> and <a href="https://github.com/PZawieja/Machine_Learning/tree/main/datasets">dataset</a>.</li><li>Replace &#39;ab_test_dataset.csv&#39; with your own dataset path.</li><li>Run the full script.</li></ol><h3>Conclusion:</h3><p>A/B testing helps you make decisions based on data instead of guesswork. This Python framework not only helps you calculate the right sample size but also guides you through interpreting the results with confidence.</p><p>Whether you’re testing new app features or optimizing landing pages, these scripts have your back — so go ahead and experiment with confidence!</p><p>👉 <em>Check github for the </em><a href="https://github.com/PZawieja/Machine_Learning/tree/main/notebooks"><em>notebook</em></a><em> and </em><a href="https://github.com/PZawieja/Machine_Learning/tree/main/datasets"><em>dataset</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d60033e4d6e1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beyond Clicks: Measuring Long-Term Success in A/B Tests]]></title>
            <link>https://medium.com/@p.r.zawieja/beyond-clicks-measuring-long-term-success-in-a-b-tests-94906bedc49e?source=rss-5309f9bf426e------2</link>
            <guid isPermaLink="false">https://medium.com/p/94906bedc49e</guid>
            <category><![CDATA[a-b-testing]]></category>
            <category><![CDATA[metrics]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[growth]]></category>
            <dc:creator><![CDATA[Piotr Zawieja]]></dc:creator>
            <pubDate>Fri, 03 Jan 2025 19:54:58 GMT</pubDate>
            <atom:updated>2025-01-03T19:54:58.934Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Deep Dive into Metrics that Matter for Sustainable Growth</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2bdrFHWq6W7UaGGI" /><figcaption>Photo by <a href="https://unsplash.com/@ansleycreative?utm_source=medium&amp;utm_medium=referral">Matthew Ansley</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction: Why Short-Term Wins Can Be Misleading</h3><p>When you’re running an A/B test, it’s tempting to celebrate quick wins — <strong>click-through rates (CTR)</strong> spike, or <strong>conversion rates</strong> jump, and everyone’s ready to call the test a success. It feels great, right? But here’s the catch: those short-term boosts don’t always tell the whole story.</p><p>Imagine you roll out a new homepage design that increases sign-ups by 15%. Everyone’s high-fiving. But fast forward three months, and you realize those new users aren’t sticking around — they churn faster than before, and your <strong>customer lifetime value (CLV)</strong> takes a hit. Suddenly, that “win” doesn’t feel like such a win anymore.</p><p>This happens more often than you think. Short-term metrics are useful, but they’re just one chapter of the story. If you’re not thinking about the bigger picture — <strong>long-term customer retention</strong>, <strong>brand trust</strong>, <strong>profitability</strong> — you could be optimizing for vanity metrics that don’t move the needle where it really matters.</p><p>In this post, we’ll break down:</p><ol><li>Why focusing only on short-term outcomes can backfire.</li><li>How to define success metrics that align with your <strong>long-term goals</strong>.</li><li>Real examples where short-term wins didn’t lead to sustainable growth.</li><li>Practical tips for designing experiments with <strong>long-term impact</strong> in mind.</li></ol><p>Let’s dig in!</p><h3>The Problem with Short-Term Metrics</h3><p>Short-term metrics, like <strong>click-through rates</strong>, <strong>add-to-cart clicks</strong>, or <strong>conversion rates</strong>, are the low-hanging fruit of A/B testing. They’re easy to measure and provide quick feedback — perfect for showing immediate progress to stakeholders. But here’s the problem: <strong>what happens after the click?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wdPFKx-atbepol6g" /><figcaption>Photo by <a href="https://unsplash.com/@hostreviews?utm_source=medium&amp;utm_medium=referral">Stephen Phillips - Hostreviews.co.uk</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Short-term metrics are great for spotting immediate reactions, but they often miss the ripple effects that happen over time. Here are a few ways short-term wins can go sideways:</p><ol><li><strong>Increased Churn</strong>: Let’s say you run a discount offer A/B test, and purchases skyrocket. But what happens when those same customers only came for the discount and don’t return for full-priced purchases? You might accidentally train your audience to only buy when there’s a sale.</li><li><strong>Lower Customer Lifetime Value (CLV)</strong>: You launch a feature that drives more subscriptions in the short run, but it turns out people cancel after the first month because the onboarding experience wasn’t improved alongside it. Suddenly, you’re dealing with high cancellation rates and low retention.</li><li><strong>Brand Perception Issues</strong>: A flashy new design might boost engagement, but if it confuses or frustrates long-time users, you risk alienating your loyal customers — those who actually contribute most to your revenue.</li></ol><p>The truth is, A/B tests don’t just exist in a vacuum. Changes you make today can impact behavior weeks, months, or even years down the line. So, while short-term metrics might feel like a win, they’re just the <strong>tip of the iceberg</strong> when it comes to understanding the true impact of a change.</p><p><strong>Key takeaway:</strong> If you’re only measuring immediate behaviors, you could be blind to what’s happening in the long run. The goal isn’t just to move a single metric — it’s to drive sustainable, long-term growth.</p><h3>How to Identify Long-Term Success Metrics</h3><p>To make sure your A/B tests drive meaningful, lasting impact, you need to zoom out and choose metrics that reflect your long-term goals — not just short-term reactions. Here’s a step-by-step approach:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iTc6e99zOJjwThed" /><figcaption>Photo by <a href="https://unsplash.com/@theblowup?utm_source=medium&amp;utm_medium=referral">the blowup</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>1. Start with Your Business Objectives</h3><p>Your success metrics should align with your company’s <strong>bigger-picture goals</strong>. Are you focused on improving <strong>customer retention</strong>, increasing <strong>lifetime value</strong>, or growing <strong>brand loyalty</strong>? The metrics you choose should act as leading indicators for these outcomes.</p><p><strong>Example:</strong></p><ul><li>If your goal is <strong>higher retention</strong>, focus on metrics like <strong>repeat purchases</strong> or <strong>subscription renewal rates</strong>, rather than just sign-ups or first-time purchases.</li><li>If you’re aiming to boost <strong>lifetime value (CLV)</strong>, look at metrics like <strong>average order value (AOV)</strong> and <strong>customer tenure</strong> over time.</li></ul><h3>2. Balance Leading and Lagging Indicators</h3><p>Some long-term metrics, like <strong>annual churn rate</strong>, take months (or longer) to surface. Since you can’t wait that long for every A/B test, it helps to track <strong>leading indicators</strong> — signals that hint at future performance.</p><p><strong>Example:</strong></p><ul><li>Instead of waiting to measure <strong>churn rate</strong> after six months, look at <strong>day 30 retention</strong> or engagement trends over a few weeks as a proxy.</li></ul><p>However, leading indicators should be validated — they’re only useful if they reliably predict long-term outcomes. You don’t want to chase a metric that looks good early but fizzles out later.</p><h3>3. Consider Secondary Effects</h3><p>Some experiments might boost a primary metric but negatively affect others. For example:</p><ul><li>A new feature might increase <strong>click-through rates</strong>, but does it also increase <strong>support tickets</strong> due to user confusion?</li><li>A faster checkout process might drive <strong>short-term sales</strong>, but are customers skipping important upsells and leaving value on the table?</li></ul><p>Tracking <strong>secondary metrics</strong> alongside your primary goals can help you catch any unintended consequences early.</p><h3>4. Track Metrics Over Time</h3><p>It’s not enough to measure success during the A/B test window. You’ll need to revisit your data at key intervals — 30 days, 90 days, 6 months — to see how the changes are holding up. Did the initial win lead to sustainable growth, or did performance dip after rollout?</p><p><strong>Pro Tip:</strong> Create a <strong>“test-and-monitor” cadence</strong> where you plan for post-rollout reviews. This keeps your team accountable for evaluating the long-term effects of major experiments.</p><h3>5. Avoid Vanity Metrics</h3><p>Be cautious of metrics that look impressive but don’t really matter. For example:</p><ul><li>A spike in <strong>pageviews</strong> might not mean much if those visitors bounce immediately after landing on the page.</li><li>A bump in <strong>downloads</strong> might be meaningless if users never engage with your app after installing it.</li></ul><p>Always ask: <strong>Does this metric help predict a meaningful business outcome?</strong> If not, it’s probably not worth optimizing for.</p><h3>Real-World Examples: When Short-Term Wins Didn’t Lead to Long-Term Success</h3><p>Even the best data teams have been burned by tests that looked like wins but flopped later. Here are some real-world examples (and lessons) of where short-term success didn’t align with long-term outcomes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tswvkamP04lazbtm" /></figure><h3>1. The Discount Dilemma</h3><p><strong>Scenario:<br></strong> An e-commerce company ran an A/B test offering a 20% discount for first-time buyers. The test was a hit — conversion rates soared by 25%. The company rolled out the discount sitewide, expecting massive growth.</p><p><strong>What Happened Next:<br></strong> Three months in, overall profit margins tanked. Many of the new customers purchased only during discounts and didn’t return without another offer. Even worse, some loyal customers started “gaming” the system, creating new accounts to get first-time discounts repeatedly.</p><p><strong>Lesson Learned:<br></strong> Short-term spikes in sales are tempting, but they don’t always lead to loyal customers. Metrics like <strong>repeat purchase rate</strong>, <strong>customer acquisition cost (CAC)</strong>, and <strong>average order value (AOV)</strong> should be tracked alongside conversion rates to see if the changes build sustainable growth.</p><h3>2. The Feature Launch Fiasco</h3><p><strong>Scenario:<br></strong> A SaaS company tested a new onboarding flow that shortened the sign-up process by skipping optional customization steps. Sign-ups increased by 18% — a clear win!</p><p><strong>What Happened Next:<br></strong> Churn rates increased significantly. Users who skipped customization were more likely to abandon the product after a week because they didn’t get the “personalized” experience the old flow provided.</p><p><strong>Lesson Learned:<br></strong> Faster isn’t always better. The team should have tracked <strong>activation rate</strong> (the point where users experience core value) and <strong>day 30 retention</strong> to ensure the faster flow still delivered meaningful engagement.</p><h3>Designing A/B Tests for Long-Term Success</h3><p>To avoid the trap of short-term wins that don’t hold up, you need to approach A/B test design with <strong>long-term impact</strong> in mind. Here are some actionable strategies to help you build experiments that lead to sustainable growth:</p><ol><li><strong>Define Success Metrics That Reflect Long-Term Goals</strong></li><li><strong>Use Holdout Groups to Measure Post-Rollout Impact</strong></li><li><strong>Monitor Metrics After the Test Ends</strong></li><li><strong>Don’t Ignore Secondary Effects</strong></li><li><strong>Simulate Long-Term Scenarios with User Segments</strong></li><li><strong>Combine A/B Testing with Causal Inference Methods</strong></li><li><strong>Communicate Wins and Risks Transparently</strong></li></ol><h3><strong>Final Thought:</strong> Think Big, Measure Smart</h3><p>A/B testing isn’t just about quick wins — it’s a tool for building sustainable growth. By designing tests with long-term outcomes in mind, you’ll make better decisions, avoid costly missteps, and deliver changes that truly move the needle for your business.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=94906bedc49e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>