<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Agustin Figueroa on Medium]]></title>
        <description><![CDATA[Stories by Agustin Figueroa on Medium]]></description>
        <link>https://medium.com/@agusfigueroa?source=rss-4f6e9291789------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Y7E-nx-RqoIEz2BnSqHvJg.jpeg</url>
            <title>Stories by Agustin Figueroa on Medium</title>
            <link>https://medium.com/@agusfigueroa?source=rss-4f6e9291789------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 22:31:29 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@agusfigueroa/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Decisions]]></title>
            <link>https://medium.com/@agusfigueroa/decisions-85c163bbee33?source=rss-4f6e9291789------2</link>
            <guid isPermaLink="false">https://medium.com/p/85c163bbee33</guid>
            <dc:creator><![CDATA[Agustin Figueroa]]></dc:creator>
            <pubDate>Sun, 22 Dec 2024 22:53:01 GMT</pubDate>
            <atom:updated>2024-12-22T22:53:01.143Z</atom:updated>
            <content:encoded><![CDATA[<p>From a practical standpoint, there are two and only two types of phenomena in your life. There are the ones you may influence, and there are the ones you know you won’t. Influence by when? In the timeframe that is important for the decision you will make.</p><p>The result of this realisation is that it gives you a direct feeling of finiteness. There is a universe of things happening out there — the moment I acknowledge that I can’t really influence most of it , I accept that what I can do best is to use this information and decide based on that. <em>e.g. I may not be able to influence how my neighbourhood looks in the near future, but I can decide to go elsewhere, try to change it by enrolling in any sort of volunteering, or just choose to live with it the way it is.</em></p><p>But if you limit the decisions you will take to the phenomena you can influence, then you can ask yourself: what is it then worth my attention? which information do I need?<em> In the neighbourhood example: If I decide to leave it, why do I closely follow all what’s going on there on Instagram a few months after leaving? And most importantly, why do I emotionally engage with it? What is the point?</em></p><p>Where I am at: at identifying which phenomenon are important for me, focusing on the ones that I can influence, and bucketing the rest as non-relevant, at least for now.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=85c163bbee33" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part III: A/B Testing at HomeToGo — Running the whole A/B pipeline on Snowflake]]></title>
            <link>https://engineering.hometogo.com/part-iii-a-b-testing-at-hometogo-running-the-whole-a-b-pipeline-on-snowflake-70f0996b12e6?source=rss-4f6e9291789------2</link>
            <guid isPermaLink="false">https://medium.com/p/70f0996b12e6</guid>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[ab-testing]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Agustin Figueroa]]></dc:creator>
            <pubDate>Fri, 15 Dec 2023 06:59:21 GMT</pubDate>
            <atom:updated>2023-12-15T06:59:21.685Z</atom:updated>
            <content:encoded><![CDATA[<h3>Part III: A/B Testing at HomeToGo — Running the whole A/B pipeline on Snowflake</h3><p>A/B testing is the bread and butter and de facto standard of almost every digital company for experimentation.</p><p>This article is the third and last chapter of a three-part series (find links to the first chapter <a href="https://engineering.hometogo.com/a-b-testing-at-hometogo-when-and-why-we-do-it-52ef063eae08">here</a>, and the second chapter <a href="https://medium.com/p/d5111da31f59">here</a>). Throughout this collection, we aim to share our experience at HomeToGo with user testing in terms of fundamentals, governance, and the very specific details of how we perform calculations on Snowflake.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EIegmynFeWsHc8uJjRH5qA.jpeg" /></figure><p>If you are interested in user testing and work in the product/data space, stick around!</p><h3>Running the whole A/B pipeline on Snowflake</h3><p>In September 2022, we migrated our A/B testing raw tables from offline SQL scripts to dbt, which was a big step towards reducing complexity. However, the calculations were still performed by an R script that did not run in the data warehouse. This created significant overhead: We needed to generate the input for the statistics script, execute that script elsewhere, and then import the statistics back into the data warehouse for further analysis and reporting. To circumvent this, we opted to move the pipeline entirely into Snowflake. This enabled us to:</p><ul><li>Reduce complexity (e.g. get rid of importing/exporting data to and from the statistics calculator)</li><li>Create more transparency in terms of lineage (e.g. who consumes what) and stats calculation (e.g. how we calculate p-values and confidence intervals)</li><li>Centralize statistical function definitions by leveraging Python UDFs, which can be reused across the company.</li></ul><h4>Calculating p-values and confidence intervals for different types of metrics</h4><p>When we calculate test statistics, our base table is at a user level: We aggregate the data on a user grain, and build the rest upon that. As we mentioned before, users are our unit of randomization and are assumed to be iid <em>(Independent and Identically Distributed)</em>.</p><p>We deal with three types of metrics when A/B testing at HomeToGo:</p><ul><li>Binomial metrics: There are two possible outcomes per unit of randomization (in our case, per user) — true or false</li><li>Continuous metrics: Numeric calculations on the unit of randomization (in our case, we calculate the given metric for each user and apply a test to that)</li><li>Ratio metrics: Numeric calculations that only make sense when considering the whole group of users (calculating these on a user level would not make business sense)</li></ul><p>After removing outliers, we proceed to calculate the metrics as well as associated p-values and confidence intervals. As explained below, each type of metric needs to be treated differently.</p><h4>Binomial metrics</h4><p>An example of this metric is users with a booking. These types of metrics are the most robust ones, and are purely binomial (e.g. yes or no on a user level). We used <a href="https://github.com/hometogo/hometogo-data-code-snippets/blob/main/snowflake/user_defined_functions.yaml#L140">chi-squared tests to calculate p-values</a> and <a href="https://github.com/hometogo/hometogo-data-code-snippets/blob/main/snowflake/user_defined_functions.yaml#L75">estimate the confidence interval of the uplift</a>.</p><p>Calculating p-values and confidence interval estimates of the uplift analytically is the cheapest and most efficient option, simply because we do not need bootstrapping at all. Bootstrapping is time-consuming, computationally expensive, and relatively complex to maintain.</p><p>In other words, analytical calculations and estimations of p-values and confidence intervals help us not only reduce costs, but also lower the marginal cost of adding new metrics to the pipeline (which will likely be needed as the company grows and we launch new products or features). If we have a preference for robustness, influence of outliers, and the cost of calculating stats, binomial stats win across the board.</p><h4>Continuous metrics</h4><p>These metrics are still on a user grain, but we are interested in the continuous number (e.g.: revenue per user). Therefore, we use <a href="https://github.com/hometogo/hometogo-data-code-snippets/blob/main/snowflake/user_defined_functions.yaml#L49">t-tests</a> to calculate the p-values and bootstrapping for the confidence intervals. <a href="https://stats.stackexchange.com/questions/151663/normality-and-when-to-use-t-test-vs-mann-whitney-u-test">Bear in mind that you are allowed to use t-tests when the distribution of the variable is not normally distributed</a>, as long as the mean of the underlying variable is normally distributed (see also <a href="https://arxiv.org/abs/2305.16459">this</a> paper).</p><h4>Ratio metrics</h4><p>An example of a ratio metric is revenue per impression. We can only calculate this type of metric once per variation across all users. How can we derive a p-value and confidence intervals from this?</p><p>The answer for us is bootstrapping (also called sampling with replacement). Bootstrapping allows you to calculate p-values and confidence intervals for almost anything, but you need to consider that it’s the most expensive calculation path. It’s important to ensure that the benefits outweigh the costs.</p><p>In a nutshell, you draw a sample from the pool of users N times and calculate the uplift of the ratio metric each iteration. You can then calculate descriptive statistics based on the estimated distribution of the uplift that bootstrapping generated for you. See <a href="https://community.snowflake.com/s/article/Bootstrapping-at-Scale-in-Snowflake">this article</a> for a walkthrough of how to implement bootstrapping on Snowflake.</p><p>With a sufficiently large number of samples (in our case, N above 900), you can assume normality and use a t-test or <a href="https://github.com/hometogo/hometogo-data-code-snippets/blob/main/snowflake/user_defined_functions.yaml#L1">z-test</a> for the p-values. It’s worth noting the difference between a distribution tending to normality and being normal: If we can’t ensure that it is normal, using the t-test is the right option, even though the z-test will likely yield extremely similar results.</p><p>For the 95% confidence intervals, this becomes a bit easier compared to the p-value calculations: We generate the distribution of the difference between v0 and v1, and then simply calculate the 2.5th and 97.5th percentiles.</p><h3>Validity checks implementation</h3><h4>Penalizing p-values to reduce peeking &amp; false positives</h4><blockquote>What is peeking?</blockquote><blockquote><em>“Peeking”, or sometimes “unaccounted peeking” and more precisely “unaccounted peeking with intent to stop” is a term used to describe the poor practice of checking the results of an ongoing</em><a href="https://www.analytics-toolkit.com/glossary/a-b-test/"><em> A/B test</em></a><em> with the intent to stop it and make a decision or inference based on the observed outcome: value of the variable of interest, reaching a given</em><a href="https://www.analytics-toolkit.com/glossary/significance-threshold/"><em> significance threshold</em></a><em>, etc. Peeking is a significant threat to the validity of any</em><a href="https://www.analytics-toolkit.com/glossary/online-controlled-experiment/"><em> online controlled experiment</em></a><em> as it can uncontrollably increase the</em><a href="https://www.analytics-toolkit.com/glossary/type-i-error/"><em> type I error</em></a><em> rate and render any significance calculations or</em><a href="https://www.analytics-toolkit.com/glossary/confidence-intervals/"><em> confidence intervals</em></a><em> meaningless.</em></blockquote><blockquote><em>It is a poor practice since it causes a discrepancy between the</em><a href="https://www.analytics-toolkit.com/glossary/nominal-p-value/"><em> nominal p-value</em></a><em> you calculate and the</em><a href="https://www.analytics-toolkit.com/glossary/actual-p-value/"><em> actual p-value</em></a><em> which reflects the probability of observing the result you observed under the actual circumstances. With just a few looks at the data the actual significance can be orders of magnitude larger than the nominal.</em></blockquote><blockquote><em>Source: </em><a href="https://www.analytics-toolkit.com/glossary/peeking/"><em>Analytics-toolkit</em></a></blockquote><h4>What to do about peeking?</h4><p>We definitely recommend taking peeking into account when calculating and showing p-values. We do this via correcting the nominal p-values so that they get as close to the actual p-values as possible.</p><p>Let’s take a step back and think of experimentation in a digital product. When we are running tests we basically want two things:</p><ol><li>Their results to be rigorous</li><li>To spot early on variations that break the user experience</li></ol><p>As you may have realized, (b) looks somehow juxtaposed to (a). If you run a test for a month, until you reach the minimum sample size, your results will be rigorous (a), but what if a bug in the checkout was dragging conversion rate down by 10%? Then 50% of the traffic * 10% decrease = 5% total loss. On the other hand, if you always look at results and p-values, you would spot (b) but you are also peeking and the results will be non rigorous. It would also be hard to separate the signal from the noise. The answer must be something in between.</p><p>What has worked for us quite well, as well as for other companies in the industry, is penalizing p-values. The reason behind this is factoring in uncertainty: A p-value of 0.1 reflects a different underlying level of uncertainty if it is the result of a calculation with 1000 users or with one million users. P-values do not reflect this: By design in the process, a p-value is calculated only once, not multiple times in the course of a controlled trial.</p><p>The idea is that the corrected p-value converges to the classical p-value if the sample is large enough. In any other case, the corrected p-value will always be higher than the classical p-value (one can interpret this as “less significant”). How much you penalize your p-value in the beginning, and when you make it converge with the classical p-value, depends largely on your business, your data, and how complex you want this calculation to be.</p><p>We recommend building this off your rarest event (in our case, users who book) and adding a few parameters you can use to fine-tune sensitivity. Bear in mind that if a change across variations is very extreme (e.g. a drop of 80%), you still want to see it even if you only have a few users! What comes next is understanding how the p-value penalization affects power and significance in your data, which you can do through simulation. We simulated both A/A tests (to check the false positives ratio) and uplifts (to see power).</p><p>Simulation will help you iterate and find the hyperparameters that work for you. Running sanity checks on this and readjusting hyperparameters, particularly if the frequency of your rarest event changes, is highly recommended.</p><p>In case you want to read more, we recommend checking <a href="https://www.etsy.com/codeascraft/how-etsy-handles-peeking-in-a-b-testing/">Etsy’s</a> example and approach on the matter.</p><h4>Ensuring power of a given metric</h4><p>The idea of power is intrinsic to testing (you can check what power in testing is <a href="https://www.analytics-toolkit.com/glossary/power-analysis/">here</a>). In this post we want to focus on two situations we often encounter where the power concept comes in handy:</p><ol><li>Most of my tests come back neutral, why?</li><li>I ran this test for 3 days and it looks very significant, are we safe to release this?</li></ol><p>The situation in (a) suggests that you are either testing impactless material, or that you do not let your tests run long enough. Estimating the sample size can be painful — especially if the test shows you that, given your traffic and expected impact, you won’t be able to draw any conclusions in a reasonable time. But this is fine! It’sbetter to know this in advance than to blow up resources to only find it out later. Estimating the sample size in advance is needed and useful. However, sometimes the expected effect is different than what you see in the test. — Recalculating power is not perfect, but it gives you a rough idea about where you stand in terms of traffic that you need.</p><p>On the other hand, the situation in (b) is one where you can profit from your scepticism the most: If it looks too good to be true, it likely is. A metric can be at a given point in time <a href="https://www.evanmiller.org/ab-testing/chi-squared.html#!14/100;25/100@95">significant</a> yet <a href="https://www.evanmiller.org/ab-testing/sample-size.html#!14.000000000000002;80;5;13;0">unpowered</a>. When we encounter these cases, we think about them as“it looks like it’s going in a positive direction, yet we need to run this a bit longer to make a call on significance.”</p><p>To help end-consumers understand where their tests stand in terms of power, we show if a metric is powered next to the significance value. As the calculations are way more simple for binomial metrics than for continuous metrics, we only do it for the former (link to the code <a href="https://github.com/hometogo/hometogo-data-code-snippets/blob/dc75d86c9fe800fd92e313df091805c8505adc32/snowflake/user_defined_functions.yaml#L23">here</a>). If you know that a binomial and a non binomial metric are heavily linked (e.g. users with a booking and revenue per user in our business) you can check that the binomial one is powered. From here, you can infer that the revenue one would be too, or check how many users you would need. It’s not perfect but it’s good enough, and it’s easy to implement. In our case, this is enough.</p><h4>Binomial tests to check proper user assignment</h4><p>With the exception of canary tests, which are by design different (e.g. one may assign 99% traffic to control and 1% to treatment), we expect users to have equal chances to enter either variation. This is a fundamental assumption we make when calculating stats. When this does not hold, we don’t display any results, except for totals, to prevent misinterpretations.</p><p>To check if the assignment likely follows a binomial distribution, we calculate a <a href="https://en.wikipedia.org/wiki/Binomial_test">binomial test</a> with p=0.5. In our case, the trials are users in control or treatment, and our goal is to refute that users had different chances of falling into control or treatment. The test returns a p-value that, when below a certain threshold, can be interpreted in our case as “chances that the difference in user assignment we see is caused by randomness”.</p><pre>--implementing a test to check if the sampling between v0 and v1 looks like follows a p=0.5 binomial distribution https://en.wikipedia.org/wiki/Binomial_test<br>(case<br>when abs(metric_change) &gt;= 0.5<br>and abs(metric_change) &lt;= 70<br>and metric_value &gt;= 5000<br>then abs(metric_value - 0.5 * (metric_value + metric_value + metric_change_abs)) / sqrt(0.5 * 0.5 * (metric_value + metric_value + metric_change_abs))<br>else 0<br>end &gt;= 5)::int as bad_assignement</pre><p>When we first implemented this, we used to interpret the binomial test’s Z-Value&gt;1.96 as “95% chance that users did not have the same likelihood of falling into control or treatment”. However, when doing this, we became victims of <a href="https://www.analytics-toolkit.com/glossary/peeking/">peeking</a> and consequently experienced a flood of false positives. The adjustments we’ve made to its current implementation catches the most extreme cases and keeps noise quite low. While these conditions suit our needs, make sure you adjust them to cater to yours.</p><p>The conditions we meet so that the flag goes off include:</p><ul><li>Z-Value &gt; 5</li><li>User change &gt; 0.5%</li><li>User change &lt; 70% (as a heuristic to filter out canary tests)</li><li>At least 5000 users in control</li></ul><h3>A few notes on user assignment</h3><h4>The iid assumption we make when testing</h4><p>When we A/B test, we are making use of statistical devices that are purposeless if the right assumptions don’t hold (as your old Stats 101 book still points out).</p><p>We expect the unit of randomization to be <a href="https://en.wikipedia.org/wiki/Independent_and_identically_distributed_random_variables">iid</a> because we randomize on a user level. This has a few implications when we run the binomial test to check proper user assignment (section above):</p><ul><li>We want to check that the assignment to control or treatment is completely randomized. Except for specific cases, such as canary tests, the likelihood of falling into treatment of control is theoretically the same (with the most common traffic assignment being 50/50).</li><li>Our ultimate assumption is that what one user does is completely independent from what any other user does.</li></ul><h4>Considering only affected users (a.k.a. “Long tails”)</h4><p>We often come across situations where only a small portion of our website visitors are exposed to a change during a test. For example, think about a change in the payment selection process. Only a small percentage of our users actually reach this stage. Therefore, we only want to include a user in the test when their experience has been influenced by the changes we’re testing.</p><p>This concept of including only affected users in a test is what we refer to as “long-tails.” In other words, we consider a user and their interactions as part of a test only after their experience has been impacted by the changes we’re testing. For example, if we modify the way payment methods are displayed, we will only assign users to the test who have actually seen the payment methods.</p><p>However, having the ability to select which users participate in a test also comes with responsibility. We must ensure that our testing process remains rigorous and scientific.For this reason, we check for adequate user assignment systemically, as it helps us maintain fairness in user assignment to variation or control.</p><p>In our experience, one common reason for variations in user counts across different versions is the activation of “long-tails” at different stages. For example, in the case of changing payment method sorting, if one version merges previous checkout steps, it may appear that more users in that version are interacting with payment methods. In reality, this is because of the change in the stage being tested.</p><p>Last but not least, even when randomization is working as expected and the long-tail activation point is the same, we sometimes still observe significant differences in user counts between the control and treatment groups. What we have often seen is that the test may be affecting page load times, causing users who might have bounced in one version to stay in the test in another.</p><p>Currently, we haven’t found a solution to address this issue completely: Filtering out users who didn’t bounce is an option but it’s not a comprehensive one. Instead, we focus on looking at the overall impact on core metrics. One needs to bear in mind that all metrics that rely on users or sessions (e.g. impressions per user or impressions per session) are directly affected by this phenomenon. However, total metrics (e.g. total impressions) should not be as impacted because users who enter one version (and not the other) are typically low-intent users who should not significantly affect the total results.</p><p>Thanks Athene Cook, Stephan Claus, Hiep Minh Pham, Domenico Boris Salvati and Breanna Shepard for your feedback and reading drafts of the whole series.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=70f0996b12e6" width="1" height="1" alt=""><hr><p><a href="https://engineering.hometogo.com/part-iii-a-b-testing-at-hometogo-running-the-whole-a-b-pipeline-on-snowflake-70f0996b12e6">Part III: A/B Testing at HomeToGo — Running the whole A/B pipeline on Snowflake</a> was originally published in <a href="https://engineering.hometogo.com/">HomeToGo Data &amp; Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part II: A/B Testing at HomeToGo — Governance]]></title>
            <link>https://engineering.hometogo.com/part-ii-a-b-testing-at-hometogo-governance-d5111da31f59?source=rss-4f6e9291789------2</link>
            <guid isPermaLink="false">https://medium.com/p/d5111da31f59</guid>
            <category><![CDATA[ab-testing]]></category>
            <category><![CDATA[governance]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[product-management]]></category>
            <dc:creator><![CDATA[Agustin Figueroa]]></dc:creator>
            <pubDate>Thu, 16 Nov 2023 11:50:46 GMT</pubDate>
            <atom:updated>2023-11-16T13:55:12.459Z</atom:updated>
            <content:encoded><![CDATA[<h3>Part II: A/B Testing at HomeToGo — Governance</h3><p>A/B testing is the bread and butter and de facto standard of almost every digital company for experimentation.</p><p>This article is the second one of a three-part series (link to the first one <a href="https://engineering.hometogo.com/a-b-testing-at-hometogo-when-and-why-we-do-it-52ef063eae08">here</a>). Throughout this collection, we aim to share our experience at HomeToGo with user testing in terms of fundamentals, governance, and the very specific details of how we perform calculations on Snowflake.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q5-ZzWJMpKy0_bBXyswbmg.jpeg" /></figure><p>If you are interested in user testing and work in the product/data space, stick around!</p><h3>Ensuring validity of results at scale</h3><p>A key part of our successful A/B testing strategy at HomeToGo is finding a way to make our A/B test readouts (dashboards that contain results, p-values and confidence intervals) business-safe to be interpreted by anyone at any given point in time. To make sure we stick to this approach, our analysts remain involved in experiment design and test termination for sign-offs, as these two stages are critical and require specialized support.</p><p>Even though no solution is bulletproof, we’ve based our approach and actions on the learnings we’ve collected over the years. The below measures allow us to still be rigorous even while usage of the HomeToGo A/B testing platform rapidly grows. In the end, reducing the room for mistakes means reducing complexity for the decision makers, which fully aligns with one of our key leadership principles (<a href="https://www.hometogo.com/about/leadership-principles/"><em>Embrace change and reduce complexity</em></a>).</p><h4>Early detection of tests with bad user assignment</h4><p>When we are testing (with the exception of canary tests), we expect the user distribution to be even. In other words, a user should have the same probability of entering the control or treatment group. This is a core assumption of user testing and needs to be reflected in the numbers.</p><p>There are multiple reasons why this may not be the case in practice: Incorrect user assignment, different loading time, incompatibility across browsers, etc. Besides the root cause, it is important to identify these tests and spot them early on, so test owners can be notified as this condition invalidates almost all test results. We will elaborate more on user assignment in the next article.</p><p>We perform a binomial test to uncover such cases. If the A/B test turns out to be suspicious we:</p><ol><li>Do not display any results in our A/B testing results dashboard, except for totals (to avoid the risk that consumers report on invalid results) and instead display a call to action asking test owners to check user assignment</li><li>Send out a report on slack and email every week containing the tests with poor user assignment</li></ol><p>In our experience, not displaying results has been the quickest way to catch the attention of the relevant parties, and has saved us a lot of time. If you see this early on in the process, you can also correct it sooner rather than later and reiterate. If not, you spend time on funky tests that will likely yield no useful conclusions.</p><p>In the past, we displayed results for all tests along with a small disclaimer. This approach was not as powerful, as it was overcomplicating the A/B readout (we were basically saying “these are all the results the test <em>could</em> yield, but we also believe they are not true”) and leading to confusion.</p><h4>Only displaying penalized significance</h4><p>Due to peeking (further explanation and technical implementation will be found in the last article of this series) the significance calculations one gets straightaway can lead to overconfident results. People who work in the field daily know this, but from a process-perspective, it is quite inefficient to display a significance value that always needs to be adjusted. Moreover, we have seen that displaying overconfident results yields false positives (in the end, we work hard on great/accurate shipping features and we should not downplay confirmation bias).</p><p>What we have learned is that the best approach is to have a single source of truth for significance that is already corrected and share that across the organization. The A/B platform does the significance calculations for everyone so people can always see and discuss the same numbers.</p><h4>Our funnel is always visible</h4><p>Many product folks at HomeToGo focus on a particular part of the user journey: From search experience to landing page or checkout, each stage is quite different and there are teams built around this. However, due to the complexity of our product, a change in a given part of the journey can impact a subsequent part of the process, even unexpectedly!</p><p>Therefore, we have introduced a set of “core funnel metrics” which are at all times visible in our A/B testing dashboard. We always take a look at them before releasing a test as a sanity check — not only to make sure that numbers are positive (or neutral), but also to see if the relationship between all the figures makes sense.</p><p><em>Non-exhaustive list of funnel metrics</em></p><ol><li>Users</li><li>Sessions</li><li>Sessions per user</li><li>Searches per user</li><li>Impressions per user</li><li>Micro conversions per user</li><li>Users entering the checkout</li><li>Users with a booking</li></ol><h4>Showing minimum user count required for the seen effect</h4><p>A question that’s been asked very often is “does my test have enough users?”. Even though you can estimate this beforehand, the reality is that the change you see (e.g. uplift of 3%) may not be the one you expected upfront, making it all a bit more unclear.</p><p>While not theoretically perfect, our solution to this issue is to add a calculation of minimum sample size to our reporting, which gives us a ballpark estimate of how many users per variation we would need. We also include a binary “is_metric_powered” flag: If the flag is equal to 0, it suggests that more data is needed before a call can be reliably made.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fAneSKfD--G-WWhz" /></figure><p>The technical implementation will be found in our third article of this series.</p><h4>Adding a global impact estimator to the A/B testing dashboard</h4><p>Before running a test, we define the primary and secondary metrics which are usually connected to our company goals (e.g. users with a booking, revenue per user in a certain time period, etc.). Upon terminating a test, the question that follows is how that translates into our global goals (e.g. a test improved revenue per user by 10% — how would this uplift be reflected in our overall revenue per user?).</p><p>To solve this, we have added the key company goals metrics to show to what extent they have been impacted by the test.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fZG-L5do3bCKDsGL" /></figure><p>In this case, the reach of this test looks to be somewhere around 19% for metric A, 28% for metric B and 16.5% for metric C. This gives us a rough estimate and helps us know where we stand.</p><p>It is worth noting that the share factor is heavily affected by:</p><ul><li>Start/end date of the test</li><li>Changes to the traffic assignation (e.g. from 20/20 to 50/50)</li></ul><p>If you are interested in estimating long-term impact from A/B experiments, we recommend taking a look at <a href="https://conferences.oreilly.com/strata/strata-ca-2018/public/schedule/detail/63986.html">this</a> link.</p><h4>Calling out tests that have been running for longer than six months</h4><p>HomeToGo’s product is ever-changing. This means that if a test has been running for a long time, it has likely been through many different stages of our product and team setups. This makes the conclusions harder to extrapolate and it’s more difficult to learn from these tests.</p><p>In most cases, we don’t really need tests to run for that long. To generate visibility into this, we’ve set up a report that is fired into a Slack channel every week to show the tests that have been running for more than 180 days and are still active. This gives us our team the opportunity to clean up our testing environment and reduce technical debt down the line.</p><h3>Process: Interaction with Analysts</h3><h4>Analytics signs off every test that counts towards our goals</h4><p>At HomeToGo we measure improvements by A/B testing, which means that if a department claims that they improved a metric (e.g. Users with a Booking) by X% they need to have test results to back this up.</p><p>Once a test has concluded, the results display various metrics along with their respective significance and confidence interval values. In some cases, these test results may appear quite promising, and test owners are eager to make them count towards their department’s goals. As a final step, and as part of our commitment to accountability, a test must receive approval from Analytics before being officially designated as a “true success” that can count towards goals.</p><p>This translates into a process in which owners create a ticket for Analytics, who then run through a checklist to ensure all criteria of a valid test have been met:</p><ul><li>Whether the user assignment is balanced</li><li>Whether core funnel metrics are not adversely impacted</li><li>Whether the target metric is significant and powered</li></ul><p>This process is quick for tests with a cut and dry outcome, and also allows Analytics to take more time to analyze “borderline” tests in detail and document the rationale behind sign-off decisions (e.g. if significance has been around 93% in the past two weeks and funnel metrics look positive, what do we do?)</p><h4>A/B testing deep dives through “credits”</h4><p>We know that deep dives into A/B test outcomes beyond the standard sign-offs mentioned above require a prioritization framework, given they typically drive value but are time-intensive. We therefore introduced a number of credits (effectively a fixed number of tests to be evaluated each quarter) to decide which tests are worth investing in.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d5111da31f59" width="1" height="1" alt=""><hr><p><a href="https://engineering.hometogo.com/part-ii-a-b-testing-at-hometogo-governance-d5111da31f59">Part II: A/B Testing at HomeToGo — Governance</a> was originally published in <a href="https://engineering.hometogo.com/">HomeToGo Data &amp; Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A/B testing at HomeToGo: When and why we do it]]></title>
            <link>https://engineering.hometogo.com/a-b-testing-at-hometogo-when-and-why-we-do-it-52ef063eae08?source=rss-4f6e9291789------2</link>
            <guid isPermaLink="false">https://medium.com/p/52ef063eae08</guid>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[a-b-testing]]></category>
            <dc:creator><![CDATA[Agustin Figueroa]]></dc:creator>
            <pubDate>Thu, 02 Nov 2023 20:28:18 GMT</pubDate>
            <atom:updated>2023-11-02T20:28:18.492Z</atom:updated>
            <content:encoded><![CDATA[<p>A/B testing is the bread and butter and de facto standard of almost every digital company for experimentation.</p><p>This article kicks off a three-part series. Throughout this collection, we aim to share our experience at HomeToGo with user testing in terms of fundamentals, governance, and the very specific details of how we perform the calculations on Snowflake.</p><p>If you are interested in user testing and work in the product/data space, stick around!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OkLrqkECLuTYY-HL5JAcfQ.jpeg" /></figure><h3>Why do we A/B test?</h3><p>Even though it may seem obvious, it’s good to ask ourselves this fundamental question from time to time. Our answer: <strong>we test to learn if (and how) changes to our product impact users.</strong> We have hypotheses pre-formulated which we then accept or reject using the data we collect throughout the tests.</p><h4><em>Upsides of testing</em></h4><ul><li>We learn from our users (sometimes gaining insights that are quite surprising) and we can use this information to iterate</li><li>We may prove that something works</li><li>We may prove that something does not work</li></ul><h4><em>Downsides of testing</em></h4><ul><li>We may provide users with a worse experience, which can translate into revenue loss</li><li>It takes time and time is scarce. <em>On this note, it’s always worth considering if a running test is still “collecting the upsides” (learning or proving something) and if the benefits of testing outweigh the downsides.</em></li></ul><h3>When does it make sense to test?</h3><p>In an ideal world, we would test every change we release. But the real world differs in practice, as you need to factor in the downsides stated above.</p><p>At HomeToGo, we find it useful to separate this problem into three main questions:</p><ol><li><strong><em>What are the chances that the changes we test will degrade the user performance?</em></strong><em><br></em>A useful way around this is thinking of the core booking journey. If your change impacts any of the elements of the journey (e.g. calls to action, checkout) then it’s advisable to test</li><li><strong><em>How eager are you to learn about the unexpected outcomes of the test?</em></strong><em><br></em>Tests not only serve to check or refute hypotheses; they also show us the unexpected. For example: A test that improves map performance can make users interact much more with the map, which in principle is great. But this may also distract the user during the booking process. A/B tests also show the unexpected and enable us to learn from it.</li><li><strong><em>How big is the change itself and what is the likelihood of gathering sufficient data in a reasonable time period? Will the test be powered enough?</em></strong><em><br></em>Small changes being tested, or testing on a very small portion of the traffic being affected, makes it harder to gather sufficient data in a reasonable time period (this happens particularly often with new products). In such a case, testing may take longer than the product development cycle.</li></ol><p>There is no silver bullet, but we recommend thinking in terms of (1), (2) and (3) when deciding between testing vs. not testing.</p><h3>Classifying user related A/B tests</h3><p>There are different reasons and goals for running A/B tests. The list below is not exhaustive, but the majority of our tests can be categorized into the buckets shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*M9U8AGNph2vp31Uhr7ZUbg.png" /></figure><h3>The neutral opportunity cost of neutral tests and their infinite learning potential</h3><p>One situation we encounter quite often is if we should test minor changes that are not expected to impact relevant product metrics in a statistically significant way. Our answer in most of these cases is yes. Here’s why:</p><blockquote>Example: we want to introduce a design change in a component that blurs displayed background elements upon a given action. We believe this can make the experience better, but the change is so far upstream that the effect on users with a booking will be hard, if not impossible, to detect in a reasonable amount of time (e.g. waiting a year for results would simply be too long). <strong>Should we test this<em>?</em></strong></blockquote><p>If the test comes back neutral (where all funnel metrics look stable across variations), it will likely never reach significance. You may even have millions of users in the test, but because the change will trend to zero, the sample size for that Minimum Detectable Effect (MDE) will become incredibly large. Of course, we don’t need to run the test forever — in this case, you can define how risk averse you are and based on this calculate for how long you want to keep this running (<a href="https://blog.analytics-toolkit.com/2017/importance-statistical-power-online-ab-tests/">the concept of power and what you are able to see</a> is what matters here). If the test is neutral, it also means that during the test you are not facing losses in revenues, therefore running the test is virtually free.</p><p>And if the test does not come back neutral, then it means you are facing the unexpected! You can learn from it, refine your hypothesis, and, most importantly, running the test instead of simply releasing will have saved you from providing your users with a less performant platform in the mid/long run.</p><p>Bear in mind: If your goal is only to learn but not to release, it’s up to you to decide if you need to wait to reach a given sample size or if the hints you get before then are strong enough to work on an upcoming iteration. As we pointed out in the section above, it’s good practice to ask yourself if you may benefit from running the test longer or if the data you collected are sufficient.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=52ef063eae08" width="1" height="1" alt=""><hr><p><a href="https://engineering.hometogo.com/a-b-testing-at-hometogo-when-and-why-we-do-it-52ef063eae08">A/B testing at HomeToGo: When and why we do it</a> was originally published in <a href="https://engineering.hometogo.com/">HomeToGo Data &amp; Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How HomeToGo connected dbt and Superset to make metadata more accessible and reduce analytical…]]></title>
            <link>https://engineering.hometogo.com/how-hometogo-connected-dbt-and-superset-to-make-metadata-more-accessible-and-reduce-analytical-2223af539cc6?source=rss-4f6e9291789------2</link>
            <guid isPermaLink="false">https://medium.com/p/2223af539cc6</guid>
            <category><![CDATA[api]]></category>
            <category><![CDATA[superset]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[reporting]]></category>
            <dc:creator><![CDATA[Agustin Figueroa]]></dc:creator>
            <pubDate>Fri, 10 Jun 2022 08:44:00 GMT</pubDate>
            <atom:updated>2022-06-10T08:44:00.804Z</atom:updated>
            <content:encoded><![CDATA[<h3>How HomeToGo connected dbt and Superset to make metadata more accessible and reduce analytical overhead</h3><h3>Intro</h3><p>Back in December 2021, when we started implementing Superset at a larger scale in our organization, we faced challenges related to making dbt’s metadata easily accessible to data consumers and ensuring that our dbt reporting layer was fully in sync with Superset.</p><p>We solved these problems by writing a dbt&lt;&gt;Superset connector that leverages Superset’s API capabilities and dbt’s manifest. If this problem sounds familiar, and you believe you can benefit from our solution, give our latest data blog post a read below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/1*xWzH4jjnNJrLN91gvuPtmA.jpeg" /></figure><h3>Problems we encountered</h3><p>We needed to make sure that the Superset datasets and their metadata consistently came clean out of our highly-monitored dbt reporting layer. Plus, we needed this process to be automated!</p><h3>1. “What does this column represent?” Columns descriptions are not handy when building charts</h3><p>One of the amazing things about dbt is how close column descriptions sit to the models. This is great for the model author! However, accessing dbt docs means opening a new tab, which causes context switching and is very impractical.</p><p>A direct effect of this is that exploring new tables (that may contain the needed data how it needs to be consumed) requires a significant amount of energy for users. Consequently, they simply do it less. To learn more about the benefits of metadata being ever present, we recommend <a href="https://roundup.getdbt.com/p/everpresent-metadata-thin-semantic?token=eyJ1c2VyX2lkIjo0OTI4NzcyNiwicG9zdF9pZCI6NTMyNjM1MjYsIl8iOiJXUVJCMiIsImlhdCI6MTY1MjA0NjE4NCwiZXhwIjoxNjUyMDQ5Nzg0LCJpc3MiOiJwdWItMzcwOTg4Iiwic3ViIjoicG9zdC1yZWFjdGlvbiJ9.uLTmRLKU1ra00l5RgjmPBYEC3uLDm16FusKGH1jeEtw&amp;s=r">this read</a>.</p><h3>2. The datasets list isn’t automatically updated</h3><p>In a dynamic business environment, there are many moving parts — and this is certainly reflected in the data that needs to be transformed and analyzed! This means that the list of physical datasets changes when condensing models, creating new schemas, dropping some others, creating new tables, etc. These changes were not picked up automatically.</p><h3>3. Overlooking expected differences between physical and virtual datasets can lead to wrong business decisions</h3><p>In line with our belief in providing space to Superset users to fully explore the data, we allow them to create virtualized datasets. Of course, a side effect of this measure is heavily increasing the number of datasets present on Superset overall. This can lead to confusion when users need to select a dataset to work on — should they pick a virtualized one or a physical one? Is there any difference?</p><h3>How did we solve these challenges?</h3><p>In all cases, our first source of inspiration was the <a href="https://github.com/slidoapp/dbt-superset-lineage">dbt-superset-lineage</a> project. Combining that with the options that Superset’s API provides, we ended up either adjusting dbt-superset-lineage’s code or writing our own.</p><h3>1. Automating the sync of physical datasets</h3><p>Technically speaking, this means that there must be a process that:</p><ol><li>Adds the tables under the corresponding schemas, specifically if the tables are not on Superset already. In our case, the reporting layer consists of several schemas that start with the prefix “reporting” (e.g. reporting, reporting_marketing, reporting_product).</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*P6ITkpj3OP1mHk38tu5VQg.png" /></figure><ol><li>Drops any physical table that exists on Superset but does not exist on dbt anymore. The major use-case for this is when a dbt model is deprecated; this procedure ensures that Superset doesn’t display that dataset as available.</li><li>Automates the columns refresh of physical datasets, so they are always up to date.</li></ol><h3>2. Showing certifications for built-by-analytics models and warnings when applicable</h3><p>To specify to the end-users which datasets are business-safe, and create transparency around who takes care of them, we made the following decisions:</p><ol><li><strong>Add a checkmark for all the datasets that come out of dbt.</strong></li></ol><p>If a table comes straight from dbt, it means that it has an active owner in the Analytics domain and that it’s actively monitored. This sets high data-quality standards and makes its usage business-safe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/247/0*vcWi0wuOpdMToXUj" /><figcaption>This is how the white check mark looks like on the datasets list</figcaption></figure><p><strong>2. Pushed the business logic into the repository, as much as possible.</strong></p><p>Superset has the amazing option of adding calculated columns and measures to datasets (both physical and virtual). We started off counting with both of them to define business metrics, making our dbt models leaner. However, we then realized that such code was not version-controlled and it could get out of hand quickly!</p><p>In our current setup, people use calculated columns and metrics but at their own risk — model owners don’t count on them as tools complementary to dbt.</p><p><strong>3. Pushing dbt documentation onto Superset</strong></p><p>Revamping the code from the <a href="https://github.com/slidoapp/dbt-superset-lineage">dbt-superset-lineage</a> project, we pushed all dbt descriptions to the columns on Superset. Therefore, documentation is written alongside the models in the dbt’s YAML files, and it sits right next to where the data is consumed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/298/1*nzm5FoW-VYIwjs3WJB35oA.png" /><figcaption>dbt descriptions are displayed when users hover over a field on Superset</figcaption></figure><h3>Main foundations behind our solution</h3><h3>Using the dbt manifest as a source of truth</h3><p>One of the main benefits of dbt is that all the project’s resources (models, tests, macros, etc.) are represented in a single file: the <a href="https://docs.getdbt.com/reference/artifacts/manifest-json">dbt manifest</a>. From it, we get:</p><ul><li>The column descriptions</li><li>The models with their corresponding names, aliases, and schemata.</li></ul><h3>Leveraging Superset’s API</h3><p>While the API is documented quite nicely (<a href="https://superset.apache.org/docs/api/">link here</a>), we still had to do a good portion of reverse engineering mostly on which arguments can be passed on any API call and how. To accomplish this, the most useful command was</p><pre>“GET (…)_info?”</pre><p>as it shows all the options in detail.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Flpmin2oVXPydpf5" /><figcaption>Running a GET call on Postman. The ending (_info?) makes it return highly descriptive information that will allow you to leverage the API to its fullest.</figcaption></figure><p>Keep in mind, when using a client like Postman, the arguments need to follow the RISON structure (an article on what RISON is <a href="https://medium.com/@takezoe/rison-uri-friendly-strucured-data-representation-74cb1feca2a7">here</a>).</p><h3>Some other Superset API flows we developed</h3><p>The API flows became a great ally when changes that affect multiple elements (e.g. multiple charts, datasets, dashboards) needed to be performed. These tasks kept popping up, so investing time on writing them largely paid off by saving us from doing them by hand repeatedly.</p><p>Examples:</p><ul><li>Swapping the source for all charts that are linked to any particular dashboard or dataset. This is especially useful when a model is refactored, and it’s needed for charts to look at a different dataset as a source.</li><li>Swapping the underlying database for a (group of) dataset(s) — particularly useful when migrating databases.</li><li>Adding a bulk of saved queries and assigning them to users.</li></ul><p>If you want to share knowledge on the above topics, have questions, feedback, or simply want to start a conversation, please feel free to comment and to reach out to us directly in the Superset Slack Community!</p><p>Shoutout to <em>Donatas Jocas</em>, <em>Riccardo Frontini</em> and <em>Stephan Claus </em>for working on this topic. Also, big thanks to Breanna Shepard for reading drafts of this article.</p><p>May you want to know more about other topics that we are working on, you can find our data tech stack-related articles in our <a href="https://engineering.hometogo.com/">Hometogo publication</a>.</p><h3>Appendix</h3><p>You can find all the code in our repository, links below.</p><h3>Code 1 — keeping dbt and Superset in sync</h3><p><a href="https://github.com/hometogo/hometogo-data-code-snippets/blob/main/superset/dbt_connector/dbt_superset_connector.py">Link to dbt_superset_connector.py</a></p><h3>Code 2 — pushing metadata onto Superset</h3><p><a href="https://github.com/hometogo/hometogo-data-code-snippets/blob/main/superset/dbt_connector/push_descriptions.py">Link to push_descriptions.py</a></p><h3>Code 3 — API flows</h3><p><a href="https://github.com/hometogo/hometogo-data-code-snippets/blob/main/superset/api_flows/Superset_snippets.ipynb">Link to Superset_snippets.ipynb</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2223af539cc6" width="1" height="1" alt=""><hr><p><a href="https://engineering.hometogo.com/how-hometogo-connected-dbt-and-superset-to-make-metadata-more-accessible-and-reduce-analytical-2223af539cc6">How HomeToGo connected dbt and Superset to make metadata more accessible and reduce analytical…</a> was originally published in <a href="https://engineering.hometogo.com/">HomeToGo Data &amp; Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[2021, Week 41 — Flow is gold]]></title>
            <link>https://medium.com/@agusfigueroa/2021-week-41-flow-is-gold-36829586aa7b?source=rss-4f6e9291789------2</link>
            <guid isPermaLink="false">https://medium.com/p/36829586aa7b</guid>
            <category><![CDATA[berlin]]></category>
            <category><![CDATA[expat]]></category>
            <dc:creator><![CDATA[Agustin Figueroa]]></dc:creator>
            <pubDate>Sat, 23 Oct 2021 09:04:31 GMT</pubDate>
            <atom:updated>2021-10-23T09:04:31.265Z</atom:updated>
            <content:encoded><![CDATA[<h3>2021, Week 41 — Flow is gold</h3><p>This week’s post is inspired by the “flow time” concept, which is roughly speaking a block of time in which you<strong> only do one thing and “… have time to think deeply, get lost, get off track, find your way, and get uncomfortable.” </strong>(source <a href="https://betterwithdata.substack.com/p/deepworkfordatateams">here</a>, great article). There is vast literature exposing the benefits of flow time in the workspace, but as usual, I found those lessons apply up to a greater extent.</p><p>During my last trip to the Nordics, I reunited with 2 old friends from college, whom I have not met in the past 3 years. It was a blast to see each other together and spend long hours doing basic things like playing music or riding a bike without noticing that time was going by. Right before my departure flight, we hung out for the last time and I cooked lunch for them. We spent two hours only doing that, without checking phones or doing something else. The cooking was joyful and the food was delish.</p><p>On my way back home, I saw that what just happened to us was that we just spent flow time together. Not only we had deep conversations and lots of fun, but also I felt more relaxed and rested afterward. I started to seek what sets me into that “flow” mode outside of working hours, and my experience so far is pretty positive — it lets me recharge, reconnects me with hobbies and shuts off my brain for an hour. And I got back to cook, which is already fantastic :)</p><p>Thanks <a href="https://medium.com/u/1b15a86dc7f5">Tatu Antonovich</a> for reading drafts of this.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=36829586aa7b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[2021, Week 40 — Limit your moving parts]]></title>
            <link>https://medium.com/@agusfigueroa/2021-week-40-limit-your-moving-parts-4cd9db50676f?source=rss-4f6e9291789------2</link>
            <guid isPermaLink="false">https://medium.com/p/4cd9db50676f</guid>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[berlin]]></category>
            <category><![CDATA[expat]]></category>
            <dc:creator><![CDATA[Agustin Figueroa]]></dc:creator>
            <pubDate>Tue, 12 Oct 2021 16:27:00 GMT</pubDate>
            <atom:updated>2021-10-12T16:27:00.572Z</atom:updated>
            <content:encoded><![CDATA[<h3>2021, Week 40 — Limit your moving parts</h3><p>This week’s thought is inspired by a piece of advice I got some weeks ago when I was undergoing many changes at the same time and did not know how to even analyze the situation to decide my next steps. In college, we usually analyze marginal effects under the “ceteris paribus” assumption, but in the real world every variable can change — and so it does the marginal contribution of the one of your interest. After freaking out for a while, I saw that defining a few solid pillars would be the way to see light at the end of the tunnel.</p><p>I experienced that I can progress at something when I spend time and focus on it. But of course, Mallow’s pyramid comes into play and if any of the elements at the bottom is in doubt (e.g. where to live), then that would take all my attention and would make it really difficult to concentrate on smaller yet interesting things (e.g. get better at coding or baking). Those static elements are the stepping stones on which to build upon.</p><p>I have been in the optimization field for a while, and I carry that mindset with myself (probably forever). I see a great parallelism here: when running optimization algorithms it’s very much known that, when the problem becomes big, you can’t just “optimize” for all the variables — you pick a strategy and run it thoroughly step by step. The same logic applies here; all the aspects can be optimized, but going step by step and limiting some moving parts, will lead to finding better solutions for the aspects that sit higher on the Mallow’s pyramid. That’s why I believe that traveling that often (which I have been doing lately) might have drawbacks as it’s difficult to stick to a routine (I was my location, which is very elemental, every two weeks).</p><p>Thanks <a href="https://medium.com/u/1b15a86dc7f5">Tatu Antonovich</a> for reading drafts of this.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4cd9db50676f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[2021, Week 37 — Thinking marginally]]></title>
            <link>https://medium.com/@agusfigueroa/2021-week-37-thinking-marginally-74cd68cf225e?source=rss-4f6e9291789------2</link>
            <guid isPermaLink="false">https://medium.com/p/74cd68cf225e</guid>
            <category><![CDATA[berlin]]></category>
            <category><![CDATA[expat]]></category>
            <category><![CDATA[germany]]></category>
            <category><![CDATA[startup]]></category>
            <dc:creator><![CDATA[Agustin Figueroa]]></dc:creator>
            <pubDate>Thu, 30 Sep 2021 06:48:51 GMT</pubDate>
            <atom:updated>2021-09-30T06:48:51.835Z</atom:updated>
            <content:encoded><![CDATA[<h3>2021, Week 37 — Thinking marginally</h3><p>This week’s thought was inspired by the book <a href="https://www.effectivealtruism.org/doing-good-better/">“Doing good better”.</a> Its main topic is how to do “effective altruism”, but the most generalizable concept is shifting the focus from what’s the direct effect of your actions, to what would the outcome be if you were not doing what you are doing.</p><p>The difference is subtle but critical. My main perspective before was to evaluate the results of my own actions, and based on that define whether it was worth doing or not. If it was somehow contributing towards an overall goal (e.g. picking up papers on the street contributes to the cleanliness of the city) then I would feel that was my little but meaningful contribution to that cause. The question I ask myself now is different; in case I do not do that, would anyone else do it? Can I choose to do something that other people would not probably do? If those two answers are yes, most likely I can pick an action with a larger marginal effect — if you don’t do it, probably no one else would do.</p><p>Picking up a paper on the street is rewarding, and it makes us feel great (that’s awesome!), but it might not be the most efficient way to contribute towards that specific cleanliness goal. This concept comes particularly handy when you need to prioritize for optimizing how to spend resources. In the end, we want to choose the action that changes the outcome as a whole the most — not only considering ourselves, but also others who would be involved in the dynamic and come into play.</p><p>Thanks <a href="https://medium.com/u/1b15a86dc7f5">Tatu Antonovich</a> for reading drafts of this.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=74cd68cf225e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[2021, Week 36 — You can do anything, but you can’t do everything]]></title>
            <link>https://medium.com/@agusfigueroa/2021-week-36-you-can-do-anything-but-you-cant-do-everything-4776c46e040a?source=rss-4f6e9291789------2</link>
            <guid isPermaLink="false">https://medium.com/p/4776c46e040a</guid>
            <category><![CDATA[berlin]]></category>
            <category><![CDATA[expat]]></category>
            <category><![CDATA[startup]]></category>
            <dc:creator><![CDATA[Agustin Figueroa]]></dc:creator>
            <pubDate>Mon, 13 Sep 2021 06:49:20 GMT</pubDate>
            <atom:updated>2021-09-13T06:49:20.773Z</atom:updated>
            <content:encoded><![CDATA[<h3>2021, Week 36 — You can do anything, but you can’t do everything</h3><p>This week’s thought was inspired by David Allen’s quote <strong>“You can do anything, but you can’t do everything”.</strong> The days off I took brought me a new perspective on priorities I set a year ago, and I started to see clearly that it was time to focus on one priority and keep the other one on hold.</p><p>Building upon Naval’s tweet, I understood what it means that “returns come from compound interest”. When you switch off one priority to laser-focus on another, you are narrowing the span of options but gaining more than what you will put in — that’s the magic of compound interest.<strong> If you double your bet, your reward might triple or more!</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CIxb367tvS65K6LR" /></figure><p>Last but not least, it was key to interiorize that<strong> “what brought you here won’t take you there”. </strong>This seems very counterintuitive, but it can be modeled as a financial option — as you are ahead in time, you have more information about yourself and your surroundings. Most likely, this will help you make better decisions, so why would not you revisit them?</p><p>Revisiting priorities is never straightforward, and I believe there is no mathematical rule to state when to hold onto and when to take a step back and look. What worked for me very well on this, is to hear my inner voice and openly talk about it with people I respect. If they all see the point of your turn, you will have cut down the probability of driving straight into a complete disaster.</p><p>Thanks <a href="https://medium.com/u/1b15a86dc7f5">Tatu Antonovich</a> for reading drafts of this.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4776c46e040a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[2021, Week 35 — Discover the messages cities send]]></title>
            <link>https://medium.com/@agusfigueroa/2021-week-35-discover-the-messages-cities-send-30f57146343f?source=rss-4f6e9291789------2</link>
            <guid isPermaLink="false">https://medium.com/p/30f57146343f</guid>
            <category><![CDATA[expat]]></category>
            <category><![CDATA[berlin]]></category>
            <category><![CDATA[spain]]></category>
            <dc:creator><![CDATA[Agustin Figueroa]]></dc:creator>
            <pubDate>Tue, 07 Sep 2021 06:57:10 GMT</pubDate>
            <atom:updated>2021-09-07T06:57:10.837Z</atom:updated>
            <content:encoded><![CDATA[<h3>2021, Week 35 — Discover the messages cities send</h3><p>I took the last week off and went for some travels in the northern part of Spain. While exploring around I read Paul Graham’s essay<a href="http://www.paulgraham.com/cities.html"> “Cities and ambition”</a>, which inspired the thought of this week. My key takeaway was that cities are a collection of people who live in a delimited space. What makes people move or stay there is what the city stands for, values and rewards. That message can be more or less strong, and applies to cities, groups of cities or even regions — the more heterogeneous the sample, the more the message dilutes.</p><p>During my trip I visited two regions, and in both of them I had completely different sensations in regard to <strong>how liveable the cities were for me. </strong>Though the beauty of the place, the quality of the food and the fluency with the language have an influence, this goes far beyond. It took me a while to get it, but it ended up being that <strong>what moves the people to be locals is what the city stands for. </strong>Why do they stay? Why did they come? If applicable, why didn’t they consider leaving? Do I have something in common with the answers to those questions? The latter is a great proxy of how liveable that city would be for me.</p><p>The beauty of this game is that there’s no right/wrong. We are all moved by different beliefs, so are the places around the world. And though ranking cities in terms of economic development, nightlife, air connectivity, etc. is useful in countless cases, it might fall short in this specific one. On the other hand, exploring places with the questions above in mind will drive me to the right place where I choose to be a local — in the meantime, I am on my way!</p><p>Thanks <a href="https://medium.com/u/1b15a86dc7f5">Tatu Antonovich</a> for reading drafts of this.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=30f57146343f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>