<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[GrowthBook - Medium]]></title>
        <description><![CDATA[Discussing A/B testing, feature flags, open source, and product analytics - Medium]]></description>
        <link>https://medium.com/growth-book?source=rss----3c28b27c9524---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>GrowthBook - Medium</title>
            <link>https://medium.com/growth-book?source=rss----3c28b27c9524---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 06:12:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/growth-book" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[➡ ️We’ve Moved This Blog]]></title>
            <link>https://medium.com/growth-book/%EF%B8%8Fweve-moved-this-blog-760a5a66ea39?source=rss----3c28b27c9524---4</link>
            <guid isPermaLink="false">https://medium.com/p/760a5a66ea39</guid>
            <dc:creator><![CDATA[Graham McNicoll]]></dc:creator>
            <pubDate>Sat, 20 Jul 2024 03:51:57 GMT</pubDate>
            <atom:updated>2024-07-20T03:51:26.773Z</atom:updated>
            <content:encoded><![CDATA[<p>Visit our new blog at <a href="https://blog.growthbook.io/">blog.growthbook.io</a>. We’ve also moved all articles from this blog there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v9d5Y2JdrwpUCIain0aH_w.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=760a5a66ea39" width="1" height="1" alt=""><hr><p><a href="https://medium.com/growth-book/%EF%B8%8Fweve-moved-this-blog-760a5a66ea39">➡ ️We’ve Moved This Blog</a> was originally published in <a href="https://medium.com/growth-book">GrowthBook</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GrowthBook Version 2.9]]></title>
            <link>https://medium.com/growth-book/growthbook-version-2-9-b795d758177f?source=rss----3c28b27c9524---4</link>
            <guid isPermaLink="false">https://medium.com/p/b795d758177f</guid>
            <category><![CDATA[ab-testing]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[feature-flags]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Graham McNicoll]]></dc:creator>
            <pubDate>Wed, 03 Apr 2024 22:18:14 GMT</pubDate>
            <atom:updated>2024-04-03T22:52:24.389Z</atom:updated>
            <content:encoded><![CDATA[<h4>URL redirects, approval flows, quantile testing, and more</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W8uGj9DFXksLCrI2Vj0Qvg.png" /></figure><p>We’re proud to announce the release of GrowthBook 2.9, which includes many highly requested features, including Feature Flag Approvals, URL Redirect Testing, and Quantile Metrics. Full details are below.</p><h3>URL Redirect Testing</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XGWqjKnWw9_cthNu" /></figure><p>One of the most common use cases for A/B testing is comparing two versions of a page hosted on different URLs to see which performs better. This was possible already with feature flags, but it required writing a lot of custom code and manually handling tricky edge cases. Now, GrowthBook has built-in support for this.</p><p>Simply design a new experiment, add a URL Redirect, and start the test! All you need on your site is the latest version of our JavaScript, React, or new HTML Script Tag SDK (see below). When a user visits the targeted URL, if assigned one of the treatments, they will be redirected to the new URL immediately.</p><p>This initial release is geared toward client-side redirects in a browser, but we’re actively working on support for CDNs and Edge Workers, which we’re super excited about! URL Redirect tests require a Pro or Enterprise license. View the docs <a href="https://docs.growthbook.io/app/url-redirects">here</a>.</p><h3>Feature Flag Approvals</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Tj7-QWN1Ds-N2Slx" /></figure><p>GrowthBook now supports advanced approval flows for feature flag changes. It behaves similarly to GitHub — make a change in a new draft revision, mark it as “ready to review”, another person on your team reviews it and decides whether to approve, request changes, or just leave a comment. Once approved, you can publish your draft to make it live. There’s also a brand new “Drafts” page where you can see all of the active feature drafts and their statuses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8IS_7V_gwJElOlel" /></figure><p>In this initial release, we let you configure which environments you want to require approvals for and whether or not approvals should be dismissed when further changes are made. The Drafts tab is available to everyone, but approval flows require an Enterprise license key. Contact <a href="/service/mailto:sales@growthbook.io">sales@growthbook.io</a> or <a href="https://docs.growthbook.io/features/features/approval-flows">view the docs</a> if you’re interested in learning more. We have a lot planned here in the future, so stay tuned!</p><h3>Quantile Metrics (Median, P99, and more)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aAndO8WTUYMB4qLa" /></figure><p>We’re proud to announce that GrowthBook is the first experimentation platform to fully support Quantile metrics. You can now report on things like P99 Latency, Median purchase price, and even use quantiles for decomposition deep dives!</p><p>Quantile metrics are built on-top of Fact Tables and utilize advanced techniques to keep your SQL fast and efficient while maintaining high accuracy in the statistical results. You can read more about this (as well as a technical deep dive) on our <a href="https://docs.growthbook.io/statistics/quantile">docs</a>. Quantile metrics are available to all Pro/Enterprise customers and support all data sources except for MySQL and Mixpanel.</p><h3>Project-scoped Attributes and Environments</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kyLFzNvC1UuxZ5EX" /></figure><p>For large complex applications, projects in GrowthBook are crucial for organizing your features and experiments. For example, you might have separate projects for your front-end, back-end, and mobile app. In previous versions of GrowthBook, targeting attributes and environments were global and shared between all projects. This resulted in some weird situations, like a mobile app having a “browserVersion” attribute or your marketing site having a “staging” environment when that was only relevant for your back-end.</p><p>Now, you can restrict attributes and environments to a subset of your projects, simplifying the GrowthBook UI and reducing the chance for typos and mistakes. Plus, when combined with project-scoped roles, you now have fine-grained control over exactly who is able to manage which attributes and environments.</p><h3>New HTML Script Tag SDK</h3><p>There’s a brand new GrowthBook SDK available, perfect for all low-code websites (Webflow, Shopify, Wordpress, and more). All you have to do is add a single `&lt;script&gt;` tag to your website and you’ll get support for our Visual Editor, new URL Redirect tests (see above), and even Feature Flags! No configuration required (although there are lots of knobs and switches for those that want them). Check out the <a href="https://docs.growthbook.io/lib/script-tag">docs here</a>!</p><h3>New and Improved Webhooks with Slack/Discord Support</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*yKtDA0Tpe2PaYidl" /></figure><p>We’ve revamped our event webhooks with more powerful filtering. Want to trigger a webhook for all feature flags that change in `production` with the tag “important”? You can do that! And to make integration easier, there’s a new Formatter option to automatically render the webhook in a format that Slack or Discord understands.</p><p>Now, with only a few clicks, you can enable fine-grained notifications directly in your messaging app. Support for MS Teams as well as more event types are coming soon! Read the <a href="https://docs.growthbook.io/app/webhooks">docs here</a>.</p><h3>Other Improvements</h3><ul><li>New improved LaunchDarkly importer</li><li>Improved documentation on holdouts</li><li>50+ other bug fixes and improvements</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b795d758177f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/growth-book/growthbook-version-2-9-b795d758177f">GrowthBook Version 2.9</a> was originally published in <a href="https://medium.com/growth-book">GrowthBook</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GrowthBook Version 2.8]]></title>
            <link>https://medium.com/growth-book/growthbook-version-2-8-0f4e97ce5e5d?source=rss----3c28b27c9524---4</link>
            <guid isPermaLink="false">https://medium.com/p/0f4e97ce5e5d</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[features]]></category>
            <dc:creator><![CDATA[Graham McNicoll]]></dc:creator>
            <pubDate>Mon, 26 Feb 2024 22:40:51 GMT</pubDate>
            <atom:updated>2024-02-26T22:40:27.688Z</atom:updated>
            <content:encoded><![CDATA[<h4>Prerequisite flags, code references, versioned metrics, and more</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SEpVgKU1Z6mUR7ebVZ8VvQ.png" /></figure><p>We’re excited to announce the release of GrowthBook 2.8, with many highly requested features, including Prerequisite Flags, Code References, and a new “No Access” role. Full details are below.</p><h3>Prerequisite Flags</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D8MtnVotv7hdwU3H9yhJ9A.png" /><figcaption>Prerequisite flags UI, showing this feature flag is dependent on another flag</figcaption></figure><p>With Prerequisites, you can group together related feature flags and describe complex relationships between them. For example, you can have a bunch of features that all reference a `release-2.8` parent flag as a prerequisite. The child features will only be enabled when the parent flag evaluates to `true`.</p><p>Prerequisites can also be defined at the individual rule or experiment level. For example, only include users in your experiment who are getting assigned `b` of a separate `pricing-page-version` feature flag.</p><p>Top-level Prerequisites are available to all Pro and Enterprise customers. Rule-level and Experiment-level Prerequisites are only available to Enterprise customers. Simple cases where the prerequisite is deterministic (e.g. always true or always false) work in all SDKs with no configuration required. Advanced cases (e.g. prerequisite’s value depends on an experiment) are currently only supported in the latest JavaScript and React SDK versions. Read more about this <a href="https://docs.growthbook.io/features/prerequisites">in our docs</a>.</p><h3>Feature Code References</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w-0Czgfx8kSatrNQotf7Uw.png" /><figcaption>Example showing feature references in the code</figcaption></figure><p>Back in GrowthBook 2.6, we added stale feature flag detection. Now with Code Refs, we’re making that more actionable by showing you exactly where a feature is being used in your application’s code base.</p><p>Setup is easy — Just add a new job to your application’s CI pipeline that runs whenever code is pushed. This CI job fetches a list of feature flags from the GrowthBook REST API, scans your code base for references, and sends back the line numbers and surrounding code to GrowthBook so it can be displayed in the UI.</p><p>If you use GitHub Actions, we provide a <a href="https://github.com/marketplace/actions/growthbook-code-references">pre-built action</a> you can install. We don’t have official integrations for other CI platforms at this time, but we have published a low-level <a href="https://github.com/growthbook/gb-find-code-refs">CLI script</a> and <a href="https://hub.docker.com/r/growthbook/gb-find-code-refs">Docker image</a> you can use to integrate manually. Code References are available to all Pro and Enterprise customers.</p><h3>Official (Version Controlled) Metrics</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HiAmAwYITUKc8nSIvWCtJw.png" /></figure><p>It’s not uncommon for large organizations to have hundreds or thousands of experimentation metrics. These are often a mix of “Official” metrics (widely used and vetted by the data team) and Ad-Hoc metrics (one-off, created by a product team, etc.). GrowthBook 2.8 introduces a brand new workflow that can make this distinction clearer and lead to more trustworthy experimentation.</p><p>In a nutshell, you can now store your Official Metric definitions as code in a version control system like GitHub and changes can be automatically synced to your GrowthBook account. These Official Metrics are marked with a special badge and cannot be edited from within the GrowthBook UI. Whenever you see the Official badge, you can be confident that the definitions have gone through your version control and review process.</p><p>Check out our tutorial for <a href="https://docs.growthbook.io/integrations/github-metrics">storing Official Metrics in GitHub</a>.</p><h3>“No Access” Role</h3><p>In previous versions, the lowest role you could grant someone in your GrowthBook account was “readonly”. This meant that every user you invited to your organization could, at the very least, see every single feature flag, experiment, etc., in your account.</p><p>GrowthBook 2.8 introduces a new role called “No Access”. As the name suggests, this is even lower than “readonly” and essentially grants no permissions whatsoever. When combined with project-scoped roles, this becomes super powerful. For example, grant someone a global role of “No Access” and then override this with a more permissive role for specific projects. That user would not even be able to see the projects they don’t explicitly have access to and all of the associated feature flags, experiments, etc. would be hidden from them.</p><p>This new role is available to all Enterprise customers. You can read more about this <a href="https://docs.growthbook.io/account/user-permissions#how-does-the-no-access-role-work">in our docs.</a></p><h3>Webhooks for SDK Connections</h3><p>Every SDK Connection in GrowthBook gets a dedicated API endpoint that returns a JSON payload of all included feature flags and experiments. You can now attach Webhooks to an SDK Connection so you can be alerted any time this payload changes. For example, if you have a caching layer in front of GrowthBook, you can use a Webhook to invalidate your cache, resulting in faster feature releases to your users.</p><h3>Other Improvements</h3><ul><li>New guide on integrating GrowthBook with WordPress sites — <a href="https://docs.growthbook.io/integrations/wordpress">view it here</a></li><li>Metric lookback windows</li><li>Option to ignore zeros in percentile capping</li><li>JumpCloud SSO support</li><li>Plus, more than 50 documentation and bug fixes!</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0f4e97ce5e5d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/growth-book/growthbook-version-2-8-0f4e97ce5e5d">GrowthBook Version 2.8</a> was originally published in <a href="https://medium.com/growth-book">GrowthBook</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[New GrowthBook Version 2.7]]></title>
            <link>https://medium.com/growth-book/new-growthbook-version-2-7-ea4d0e8c2ed6?source=rss----3c28b27c9524---4</link>
            <guid isPermaLink="false">https://medium.com/p/ea4d0e8c2ed6</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[ab-testing]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[experimentation]]></category>
            <dc:creator><![CDATA[Graham McNicoll]]></dc:creator>
            <pubDate>Fri, 19 Jan 2024 08:43:14 GMT</pubDate>
            <atom:updated>2024-01-19T19:26:29.745Z</atom:updated>
            <content:encoded><![CDATA[<h3>GrowthBook Version 2.7</h3><h4>Sticky Bucketing, Reusable Targeting Conditions, Experiment Health Tab, Fact Table Optimizations, and more!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dTGa-iLOtzNMrveMeFGoVQ.png" /></figure><p>Happy New Year, everyone! We’re back at it with the release of GrowthBook 2.7. This version features sticky bucketing, reusable targeting conditions, an experiment health page, and more. Full details are below.</p><h3>Sticky Bucketing</h3><p>When running an experiment, you want to ensure users are not exposed to multiple variants (e.g. a single user seeing both A and B at different times). GrowthBook accomplishes this by using deterministic hashing, which works great most of the time. However, there are certain scenarios where this can break down. For example, if your experiment only targets German visitors, a user could flip from seeing variation B to A if they take a train to France.</p><p>Sticky Bucketing allows you to remember the first variation a user sees, so their experience will remain consistent, even if something changes that would otherwise re-assign them. This behavior is opt-in and currently is only supported in the latest versions of our Javascript and React SDKs. Read more in our <a href="https://docs.growthbook.io/app/sticky-bucketing">Sticky Bucketing Docs.</a></p><h3>Reusable Targeting Conditions</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gpNS3C4NA8uQ5jgzzkvx9g.png" /></figure><p>We’ve expanded Saved Groups to support more advanced use cases. Instead of just a list of IDs, you can now create Condition Groups with arbitrarily complex targeting rules (e.g., UK Chrome users with a Pro subscription). Just like existing Saved Groups, these can be reused across multiple features and experiments, and updating the group will immediately update everywhere that references it. You can read more about this in our completely revamped <a href="https://docs.growthbook.io/features/targeting">Targeting Docs</a>.</p><h3>Fact Table Query Optimization</h3><p>It’s common for an experiment to have multiple metrics coming from the same underlying database table — for example, <strong>Revenue per User</strong> and <strong>Orders per User</strong>, both driven by a Purchases table. In this latest release, we’re taking advantage of this relationship to drastically reduce the number of queries we need to run. For data warehouses with usage-based billing, such as BigQuery or Snowflake, this can result in large cost savings.</p><p>This optimization is only available to Enterprise customers who are using the new Fact Tables to define their metrics. Read more about this on our <a href="https://blog.growthbook.io/fact-table-query-optimization">Blog</a>.</p><h3>Experiment Health Tab</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4h2SdGw3nPdzucLp" /></figure><p>GrowthBook has always run data quality checks on your experiments to detect issues such as Sample Ratio Mismatch (SRM) and multiple exposures. In this latest release, these checks live under a new dedicated “Health” tab.</p><p>In addition, you can now pick a set of dimensions that will be checked automatically for every experiment you run. For example, if you pick a “browser” dimension, you will be able to easily detect SRM errors that only affect Safari. Read more on our <a href="https://docs.growthbook.io/app/experiment-results#health-tab">Health Tab Docs</a>.</p><h3>Safely Update Live Experiments</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/788/0*kAwgstNi73sL8glf" /></figure><p>Changing a live experiment mid-flight is much more complicated than many people realize. If you aren’t careful, it can lead to carryover bias, SRM errors, or add significant noise to your results.</p><p>The safest approach is to basically start over — begin a new experiment phase, throw away the old data, and completely re-randomize all of your users. However, on lower-traffic sites, this can be prohibitively expensive.</p><p>Now, there’s a brand new “Make Changes” button at the top of running experiments. It will guide you through the process and recommend a safe release strategy, preserving past data whenever it’s safe to do so. If you choose a different release strategy, we will give you detailed warnings outlining the risks so you can make an informed decision.</p><p>This new flow is available to everyone, but Pro and Enterprise users get access to some additional release strategies powered by Sticky Bucketing (above). Read more about this and see more examples on our <a href="https://blog.growthbook.io/changing-running-experiments-safely-and-flexibly-in-growthbook">Blog</a>.</p><h3>New Best Practices Guide</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LcSmd4ykJOHDyHQe" /></figure><p>There is a new section in our documentation that contains guides on experimentation in general and chapters specifically on how to get the most out of GrowthBook. Among other things, it covers how to organize your account with projects and tags, how to understand and interpret experiment results, and checklists to make sure your self-hosted GrowthBook deployment is secure. You can find the guide <a href="https://docs.growthbook.io/using">here</a>.</p><h3>Contextual AI Bot</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*9JFNIHVsvvetuqjg" /></figure><p>We’ve added an AI bot trained on the GrowthBook content and systems, which can provide you with detailed answers to any questions you may have. You can try it out from the documentation site by clicking on the ‘ask AI’ box on the bottom right.</p><h3>Other Improvements</h3><ul><li>Validate advanced targeting conditions before saving</li><li>Okta SCIM Improvements</li><li>More information on the compatibility of SDK versions</li><li>Display helpful query stats for BigQuery (bytes scanned, execution time, etc.)</li></ul><p>Plus many more changes and bug fixes which you can read about here: <a href="https://github.com/growthbook/growthbook/releases">https://github.com/growthbook/growthbook/releases</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ea4d0e8c2ed6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/growth-book/new-growthbook-version-2-7-ea4d0e8c2ed6">New GrowthBook Version 2.7</a> was originally published in <a href="https://medium.com/growth-book">GrowthBook</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GrowthBook Version 2.6]]></title>
            <link>https://medium.com/growth-book/growthbook-version-2-6-3d8be6712ea6?source=rss----3c28b27c9524---4</link>
            <guid isPermaLink="false">https://medium.com/p/3d8be6712ea6</guid>
            <category><![CDATA[ab-testing]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[product-management]]></category>
            <dc:creator><![CDATA[Graham McNicoll]]></dc:creator>
            <pubDate>Tue, 28 Nov 2023 22:28:15 GMT</pubDate>
            <atom:updated>2023-12-01T08:50:07.604Z</atom:updated>
            <content:encoded><![CDATA[<h4>Launch Checklists, Teams, Stale feature flag detection, and more!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-tCrPH4uW_HK2SvxJY8SXQ.png" /></figure><p>We’re back from Thanksgiving with a new version of GrowthBook! Version 2.6 contains many highly requested features and improvements and results from a lot of hard work from the team and community. Check out some of the highlights below.</p><h3>Custom Pre-Launch Checklists for Experiments</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qJSEwzP6C6witavf" /></figure><p>Top experimentation teams empower everyone at their company to design and launch their own A/B tests. Removing a centralized bottleneck is great for velocity but also tends to lower the overall quality of the experiments. With our new Customizable Pre-launch Checklists, you can now enforce your organization’s best practices around experiment design without slowing things down. There are a number of predefined options, such as requiring a hypothesis or screenshots, as well as the ability to add your own custom ones, such as creating a Jira ticket or alerting the support team. This feature is available to our Enterprise customers and you can read more about it on our blog: <a href="https://blog.growthbook.io/custom-prelaunch-checklists">https://blog.growthbook.io/custom-prelaunch-checklists</a></p><h3>Saved Groups Improvements</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OLuG19WK7gOaHkC8" /></figure><p>We’ve completely revamped Saved Groups and made them a first-class citizen for targeting feature flags and experiments to your users. In addition to Inline Groups, where a list of ids is added directly in the GrowthBook UI, there is now support for a new type — Runtime Groups, where group membership is determined within your application and passed into the GrowthBook SDK at runtime. We have much more planned for Saved Groups in the future, so stay tuned! You can read about the new improvements in our docs: <a href="https://docs.growthbook.io/features/targeting#saved-groups">https://docs.growthbook.io/features/targeting#saved-groups</a></p><h3>Teams</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cIZM1h4oRlY35ogU" /></figure><p>You can now organize your GrowthBook users into teams, each with their own set of permissions! When combined with projects, this gives Enterprises a really powerful way to manage their organization at scale. Teams are also fully integrated into our SSO and SCIM offerings so they can easily be synced from Okta, with more Identity Providers coming soon. Read more about teams in our docs: <a href="https://docs.growthbook.io/account/user-permissions#teams">https://docs.growthbook.io/account/user-permissions#teams</a></p><h3>Stale Feature Flag Detection</h3><p>Feature flags are a fantastic way to reduce risk during deployments, but they can also be a source of technical debt. It’s common for engineers to forget to clean up feature flags from the code that aren’t being used anymore. In this release, we added some basic stale feature flag detection to help solve this problem. There is a new “Stale” column on the Features page that will highlight any features that we think are good candidates to be cleaned up (along with a reason why that appears on hover). Read more about this feature and our future plans on our blog: <a href="https://blog.growthbook.io/stale-feature-flag-detection">https://blog.growthbook.io/stale-feature-flag-detection</a></p><h3>Easier Installation on Low-Code Platforms</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JZ6i8w6E2WxNtGW9" /></figure><p>For users of low-code platforms like Shopify or Webflow, you can now drop a single script tag on your site and launch your first experiment within minutes using our Visual Editor. This pre-bundled script tag handles everything for you automatically — it stores a unique anonymous cookie id for the user, sets helpful targeting attributes such as deviceType and UTM params, and sends events to Segment.io and GA4/DataLayer when available. View our dedicated guides for <a href="https://docs.growthbook.io/integrations/shopify">Shopify</a> and <a href="https://docs.growthbook.io/integrations/webflow">Webflow</a> for more info.</p><h3>Other improvements</h3><ul><li>Feature Flag Drafts Refactor</li><li>SDK Connections now support multiple projects</li><li>Visual Editor improvements</li><li>View absolute and scaled impact on experiment results</li><li>Tons of bug fixes and smaller improvements</li></ul><p>Plus many more changes and bug fixes which you can read about here: <a href="https://github.com/growthbook/growthbook/releases">https://github.com/growthbook/growthbook/releases</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3d8be6712ea6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/growth-book/growthbook-version-2-6-3d8be6712ea6">GrowthBook Version 2.6</a> was originally published in <a href="https://medium.com/growth-book">GrowthBook</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding how CUPED in GrowthBook Reduces Experiment Runtimes at the Los Angeles Times]]></title>
            <link>https://medium.com/growth-book/understanding-how-cuped-in-growthbook-reduces-experiment-runtimes-at-the-los-angeles-times-79ba7c288d71?source=rss----3c28b27c9524---4</link>
            <guid isPermaLink="false">https://medium.com/p/79ba7c288d71</guid>
            <category><![CDATA[metrics]]></category>
            <category><![CDATA[ab-testing]]></category>
            <category><![CDATA[variance]]></category>
            <category><![CDATA[experiment]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Luke Sonnet]]></dc:creator>
            <pubDate>Fri, 03 Nov 2023 16:01:13 GMT</pubDate>
            <atom:updated>2023-11-03T16:01:13.141Z</atom:updated>
            <content:encoded><![CDATA[<p>Luke Sonnet — Data Science @GrowthBook <br>Jane Carlen — Data Science @Los Angeles Times<br>Julianna Harwood — Data Science @Los Angeles Times</p><p>Variance reduction via CUPED (short for <strong>C</strong>ontrolled Experiment <strong>U</strong>sing <strong>P</strong>re-<strong>E</strong>xperiment <strong>D</strong>ata) can help your experiments reach statistical significance with a fraction of the number of users you would otherwise need, resulting in dramatically reduced experiment runtimes. In this blog post, <a href="http://growthbook.io/">GrowthBook</a>, an open core experimentation platform, and the <a href="https://latimes.com/">Los Angeles Times</a> data science team demonstrate how CUPED can reduce needed sample sizes by up to 85%.</p><p>While the ins and outs of CUPED as well its potential impact have been covered before (<a href="https://exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf">Deng et al. 2013</a>, <a href="https://projecteuclid.org/journals/annals-of-applied-statistics/volume-7/issue-1/Agnostic-notes-on-regression-adjustments-to-experimental-data--Reexamining/10.1214/12-AOAS583.full">Lin 2013</a>, <a href="https://www.kdd.org/kdd2016/papers/files/adp0945-xieA.pdf">Netflix 2016</a>, <a href="https://booking.ai/how-booking-com-increases-the-power-of-online-experiments-with-cuped-995d186fff1d">Booking.com 2018</a>, <a href="https://www.microsoft.com/en-us/research/group/experimentation-platform-exp/articles/deep-dive-into-variance-reduction/">Microsoft 2022</a>), in this blog post we focus on two important factors that can dictate how well CUPED can work for you:</p><ul><li>how the type of metric influences gains from variance reduction</li><li>how you can optimize CUPED in GrowthBook by selecting the amount of historical data to use for variance reduction</li></ul><h3>How much time can CUPED save me anyways?</h3><p>Throughout this blog post, we will use data from an experiment run in production by the Los Angeles Times’ data science team. The experiment tested different feeds for articles in a “Recommended for You” module, like the one shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/1*ZiEQo1do-01WeT1wdRHPeQ.png" /><figcaption>Figure 1: An example of the Recommended For You module, the target of the experiment</figcaption></figure><p>This experiment ran for about a month, and only registered users were eligible. While the experimenters at the LA Times had several goals in mind, in this blog post we’ll focus on three metrics:</p><ol><li>page_view<em>:</em> how many times a particular page has been viewed by a user</li><li>page_click<strong>:</strong> how many times a user has clicked something on the above page</li><li>page_ctr<strong>:</strong> a custom user-level click-through-rate measurement that is min(page_click / page_view, 1). Users with 0 page_view are also given a value of 0.</li></ol><p>Using this experiment data, we can calculate each metric’s average and variance (with and without CUPED), and compute the number of users we would need to estimate a 2% uplift (relative change) in all three metrics with 80% power. The following table shows our results with and without CUPED:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DAqdn26hMfspu8osf0FzCQ.png" /><figcaption>Table 1: Necessary sample size with and without CUPED to detect a 2% uplift at 80% power in the Los Angeles times experiment case study</figcaption></figure><p>These results show that with the default CUPED implementation in GrowthBook, the necessary number of users to detect a 2% uplift would be as little as 15% of what would be needed without CUPED! Not using CUPED for the page_view and page_click metrics would mean you need to collect over 6x as many users to detect this effect reliably.</p><p>An alternative way to view the impact of CUPED is to look at the minimum detectable effect at 80% power for the 93k user sample collected in the experiment:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wN6LipNOUEsJPrhD5dk5lw.png" /><figcaption>Table 2: Minimum detectable effects with and without CUPED for the given 93k user sample n the Los Angeles times experiment case study</figcaption></figure><p>With CUPED, the experiment can regularly detect effects that are half as small as without CUPED.</p><p>We can also see the power of CUPED in action on this experiment in the GrowthBook UI. The following screenshot shows the results for two versions of the page_click metric, one with CUPED and one without. The confidence interval is less than half as wide with CUPED, and the overall effect of the experiment becomes much clearer!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x4OOaOJ8MxUGEb8_fb7y6Q.png" /><figcaption>Figure 2: The impact of CUPED on the page_click metric as shown in the GrowthBook UI</figcaption></figure><h3>For which metrics is CUPED best?</h3><blockquote><em>CUPED has the biggest impact when there is a lot of variation among users in the behavior you are measuring, but the behavior of individual users is consistent over time</em></blockquote><p>The above results show that the necessary sample size to detect a 2% lift in page_ctr (89k) is considerably smaller than the necessary sample size for a similar lift in the other metrics (~300+k). Furthermore, the percent reduction in necessary sample size from CUPED is also much smaller in relative terms for the page_ctr metric.</p><p>This difference comes down to how page_ctr is constructed and the high variance in the page_view and page_click metrics. The page_ctr metric ranges from 0 to 1, while the other metrics range from 0 to the thousands. The following figures show the distributions of these two metrics in this experimental sample:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/633/1*-DikXyujjv6GBgcMhluKiw.png" /><figcaption>Figure 3:The distribution of user values of page_view and page_click in the experiment sample</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/403/1*3vE6ah0pexhE9BQvIdIGPw.png" /><figcaption>Figure 4: The distribution of user values of page_ctr in the experiment sample</figcaption></figure><p>The long tail of the page_view and page_click metrics means two things:</p><ol><li>A 2% increase is a relatively small effect with respect to the (large) variance of these metrics. Detecting small effects is harder and requires more users. This is why the target of 2% uplift requires more users for page_view and page_click without CUPED.</li><li>If the users out in the tail of this metric have similar behavior in the pre-experiment and during-experiment periods, then CUPED has an easier time reducing the large variance of the page_view and page_click metrics.</li></ol><p>Therefore, CUPED can be particularly effective for metrics with long tails where that long tail behavior is largely predictable. In fact, in this case, the correlation between LA Times users’ page_view metric between the pre-exposure and post-exposure periods is over <strong>0.9!</strong></p><p>This is consistent with past analysis at the LA Times. Highly engaged LA Times readers tend to stay highly engaged, while less engaged readers tend to stay less engaged. Occasionally, events and breaking news can drive up engagement for everyone, but that increase is usually proportional to readers’ baseline engagement and past behavior remains strongly predictive of future behavior.</p><p>On the other hand, the page_ctr metric is truncated. This reduces the natural variance across users and also limits the correlation between the pre- and post-exposure values. As a result, CUPED is less able to explain away in-experiment variation using pre-experiment behavior. Nonetheless, even with these limitations, CUPED reduces the necessary sample size to 73% of what is needed without CUPED.</p><h3>How can I optimize CUPED performance?</h3><p>At its core, a standard CUPED implementation reduces variance by measuring the outcome metric itself during the pre-experiment period. In this setup, CUPED performs best when the metric is correlated with itself over time (known as “auto-correlation”), as described above.</p><p>There are ways to try and fine-tune your metrics to be more stable over time, but one of the easiest ways to optimize this auto-correlation is to select the appropriate <strong>lookback window.</strong></p><p>In the following example we use 14 days (GrowthBook’s default) before an exposure to serve as the pre-exposure period. For each user, we use the 14 days before their exposure as the main covariate when reducing variance via CUPED.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*ORV8f8OxbGY4nOtSyIGLTQ.png" /><figcaption>Figure 5: The date ranges used in CUPED adjustment in GrowthBook</figcaption></figure><p>This setting is configurable in GrowthBook. At first it may be tempting to think “more data is better!” but this isn’t always the case. There are several pros and cons to increasing this window:</p><p>A longer lookback window…</p><p>✅ can ensure that metrics are more stable for a user, and more likely to be correlated with future behavior; this can lead to better performing variance reduction</p><p>✅ ensures that you get at least <strong>some</strong> data, in the case that users only visit your site every couple of weeks or so</p><p>❌ might begin to incorporate stale, noisy data; for example, a user’s behavior 3 months ago may not be indicative of their current behavior</p><p>❌ requires querying more data, which can slow analysis runtimes and increase query costs</p><p>If runtime or query costs are manageable, or not of the utmost priority, than you can simply analyze the auto-correlation in your metric over time and pick the lookback window with the highest correlation.</p><h3>Selecting the right lookback window for the LA Times experiment</h3><p>Using the experiment we analyzed above, we varied the lookback window for the three metrics and plot below the percent of the unadjusted sample size that we need with each window. As you can see, for the page_view and page_click metrics, looking back 30 days produces the best variance reduction. More days, besides requiring more data to compute, actually hurts the CUPED variance reduction as it begins to incorporate noisy data that no longer represents current user behavior.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*XlVkU6kFkZCSk3IKJurhHg.png" /><figcaption>Figure 6: The impact of lookback days on the needed sample size with CUPED when compared to not using CUPED. For example, a value of 35% would imply that with X days as your lookback window, CUPED can reduce your needed sample size to 35% of the original.</figcaption></figure><p>On the other hand, for the page_ctr metric it seems that more data is better. The calculation of the click-through rate is more complex and perhaps it takes longer for this ratio to smooth out over a user’s history.</p><p>In GrowthBook you are free to set the lookback windows at different values for each metric. For page_click and page_view, you can set the lookback window to the optimal 30 day window. For page_ctr you could go to 90 days to maximize variance reduction, but that is a lot of data to roll-up, and you may be better off choosing a value like 50 or 60 days where you receive most of the benefit while scanning much less data.</p><p>As has been well established to this point, CUPED can be incredibly valuable to accelerate your experimentation program. In this blog post, we demonstrated using an applied setting how its impact is largest for metrics that naturally have high variance and that is relatively stable overtime. We also showed how you should consider explicitly evaluating the correlation between your pre- and post-experiment windows to set the optimal number of days to include in your variance reduction method. More is not always better, and at some points there are diminishing returns to including additional historical data.</p><h3>Further reading</h3><p>If you’re interested in more of the technical details of CUPED or past discussions of its impact, check out the aforementioned write-ups:<br><a href="https://exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf">Deng et al. 2013</a>, <a href="https://projecteuclid.org/journals/annals-of-applied-statistics/volume-7/issue-1/Agnostic-notes-on-regression-adjustments-to-experimental-data--Reexamining/10.1214/12-AOAS583.full">Lin 2013</a>, <a href="https://www.kdd.org/kdd2016/papers/files/adp0945-xieA.pdf">Netflix 2016</a>, <a href="https://booking.ai/how-booking-com-increases-the-power-of-online-experiments-with-cuped-995d186fff1d">Booking.com 2018</a>, <a href="https://www.microsoft.com/en-us/research/group/experimentation-platform-exp/articles/deep-dive-into-variance-reduction/">Microsoft 2022</a></p><p>You can also read more about GrowthBook’s implementation of CUPED here: <a href="https://docs.growthbook.io/statistics/cuped">https://docs.growthbook.io/statistics/cuped</a></p><p>Lastly, feel free to check out GrowthBook, an open core experimentation platform: <a href="https://www.growthbook.io/">https://www.growthbook.io/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*Lk7WtY06uIaFhVbZ.png" /></figure><p><em>Disclosure: As users of the free, self-hosted version of GrowthBook, the Los Angeles Times data science team was interested in measuring the potential impact of CUPED, a premium feature of GrowthBook’s paid plans. GrowthBook worked with the Los Angeles Times data science team to analyze this experiment using CUPED as a way for both organizations to explore the method in practice.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=79ba7c288d71" width="1" height="1" alt=""><hr><p><a href="https://medium.com/growth-book/understanding-how-cuped-in-growthbook-reduces-experiment-runtimes-at-the-los-angeles-times-79ba7c288d71">Understanding how CUPED in GrowthBook Reduces Experiment Runtimes at the Los Angeles Times</a> was originally published in <a href="https://medium.com/growth-book">GrowthBook</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Announcing GrowthBook 2.5]]></title>
            <link>https://medium.com/growth-book/announcing-growthbook-2-5-d3e41c1ba56e?source=rss----3c28b27c9524---4</link>
            <guid isPermaLink="false">https://medium.com/p/d3e41c1ba56e</guid>
            <category><![CDATA[product-development]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[ab-testing]]></category>
            <category><![CDATA[features]]></category>
            <dc:creator><![CDATA[Graham McNicoll]]></dc:creator>
            <pubDate>Mon, 16 Oct 2023 10:47:00 GMT</pubDate>
            <atom:updated>2023-10-16T10:47:00.675Z</atom:updated>
            <content:encoded><![CDATA[<h4>Metric Fact Tables, SCIM Support, AI Content Suggestions, and More</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PC8lEZO7oStHy8HD" /></figure><p>GrowthBook 2.5 may be our biggest release to date! This packed release includes metric fact table support, SCIM integration for Okta, remote evaluations for Javascript/React SDKs, data pipelining mode for Snowflake and Bigquery, AI copy suggestions in the visual editor, and more. See the full list below.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FXUq37I5ShT4%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DXUq37I5ShT4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FXUq37I5ShT4%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/dfbf39091914a18e3ba1f0131ab19927/href">https://medium.com/media/dfbf39091914a18e3ba1f0131ab19927/href</a></iframe><p>This release reflects a ton of hard work from our team and the community and we could not be more excited to hear what you think. Our primary focus is building the best feature flagging and A/B testing platform that allows companies to adopt and scale an experimentation-driven software development process.</p><h3>Metric Fact Tables</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ce2xquybQHX91XkK" /></figure><p>Fact Tables are a brand new way to create Metrics in GrowthBook. Every analytics event usually results in several very similar metrics. For example, when an order is placed, you may want to know how many people ordered, how much they spent, what percent of orders used a coupon code, and more. With Fact Tables, you define the SQL for getting order data once and can quickly create tons of metrics on top of this without writing any code.</p><p>We have a lot planned for Fact Tables in the near future, including some massive SQL performance/cost improvements and the ability to easily add dimension slices for metrics. Stay tuned!</p><h3>SDK Remote Evaluation Mode</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b3qzi3e-jBroMK_QbxiENg.png" /></figure><p>When we first built the client-side GrowthBook SDKs, we took a radically different approach from other feature flagging tools — all evaluations happened locally on a user’s device. This had huge performance benefits (extremely cacheable) and much better data privacy (no user PII sent to 3rd parties). There is a downside, though — your users can see all the business logic and experiments that you’re using to assign feature flag values.</p><p>With this release, we’re adding a new Remote Eval option to our JavaScript and React SDKs where feature evaluation happens on a back-end server and your users only see their own personalized result. The most exciting part is that we found a way to preserve many of the benefits of local evaluation! Evaluation happens entirely on your infrastructure, so you can still have amazing performance and privacy. We’ll be adding support to mobile SDKs soon and adding more deployment options.</p><h3>SCIM Support for Okta</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RSY5DSnX5AeQY9rs" /></figure><p>SCIM, or System for Cross-domain Identity Management, is an open standard that allows for the automation of user provisioning. Any of our Enterprise customers using Okta for SSO can now add and remove users from GrowthBook directly from within the Okta UI. Support for other identity providers is coming soon.</p><h3>Data Pipeline Mode</h3><p>We added a new option for BigQuery and Snowflake to enable Data Pipeline Mode. In this mode, GrowthBook will write some intermediate tables back to your data warehouse while analyzing experiment results. This can drastically reduce the query cost, especially for teams that run a lot of experiments, each one with tens or hundreds of metrics. We’re adding support for other data warehouses soon.</p><h3>AI Copy Suggestions</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tvoK-l15YI-byCh4" /></figure><p>Enhance your written content using GPT-powered AI. Transform human-readable text into any desired emotion effortlessly. Our visual editor just got better with the inclusion of AI powered copy suggestions. Simply click on any text element, and then have the AI suggest alternative copy. Give it a try, you might be surprised with some of the improvements it suggests!</p><h3>Simulations and User Archetypes</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tEfBykjkpf5VZcxYkXm-pA.gif" /></figure><p>When setting up feature flag values, it is helpful to know that you’ve set up the rules correctly for the users you’re targeting. With Simulations, you can now easily see how rules will be applied to users by setting the attributes. We will also show debug information about why each rule is used or skipped. And, as many teams have specific sets of users they often target features too, you can now save sets of attributes as an ‘Archetype’ to quickly see what values they will get.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vErFovJ78h4RxyYcN7b3mg.png" /></figure><p>Feature flag rule testing is free for everyone, while Archetypes are available to our enterprise customers.</p><h3>API Endpoints for Feature Flags</h3><p>We’ve added REST endpoints that allow you to create, edit, and toggle feature flags via our API. We’re excited about all of the use cases this unlocks and can’t wait to see what the community builds! You can read more about these endpoints in our API docs: <a href="https://docs.growthbook.io/api#tag/features">https://docs.growthbook.io/api#tag/features</a></p><h3>Multi-organization Deployments</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*x008WzcpQ8DNNcr_" /></figure><p>GrowthBook has Projects, which let you group together related feature flags and experiments. Companies often use this to give each team or product its own space. Some larger Enterprises need an additional layer to represent their complex business structure, which is why we now offer self-hosted Multi-Organization deployments. Organizations in GrowthBook share a common SSO identity provider, but are otherwise completely isolated from each other. Users in GrowthBook can belong to one or more organizations and can easily toggle between them in the top nav. Along with this change comes a dedicated Super Admin page and REST API endpoints to manage your deployment.</p><h3>Other Changes</h3><ul><li>Improved experiment velocity graphs, with the ability to segment by status, results, or projects.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UBUnrB2xQEtkMASQNWo-sw.png" /></figure><ul><li>Experiment search improvements</li><li>Snowflake query tagging for easier cost accounting</li><li>Stats engine selection for experiments</li></ul><p>Plus, there are many more changes and bug fixes which you can read about here: <a href="https://github.com/growthbook/growthbook/releases">https://github.com/growthbook/growthbook/releases</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d3e41c1ba56e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/growth-book/announcing-growthbook-2-5-d3e41c1ba56e">Announcing GrowthBook 2.5</a> was originally published in <a href="https://medium.com/growth-book">GrowthBook</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GrowthBook Version 2.4]]></title>
            <link>https://medium.com/growth-book/growthbook-version-2-4-c1b235e189df?source=rss----3c28b27c9524---4</link>
            <guid isPermaLink="false">https://medium.com/p/c1b235e189df</guid>
            <category><![CDATA[ab-testing]]></category>
            <category><![CDATA[feature-flags]]></category>
            <category><![CDATA[datadog]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Graham McNicoll]]></dc:creator>
            <pubDate>Thu, 14 Sep 2023 10:31:39 GMT</pubDate>
            <atom:updated>2023-09-14T10:31:38.768Z</atom:updated>
            <content:encoded><![CDATA[<h4>Experiment Page Redesign, Built-in Sample Data, Datadog integration guide, and more</h4><p>We’re excited to announce the latest release of GrowthBook! Version 2.4 includes significant UI improvements, more powerful feature flag experiments, built-in sample data, a Datadog integration guide, and much more. Here are the highlights:</p><h3>Redesigned Experiment Page V2</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zAE6Oa0uOYAmZ9iM" /></figure><p>We continued listening to your feedback and iterating on the experiment page. This release contains a ton of changes, including:</p><ul><li>A new tabbed layout to quickly switch between overview info and the results</li><li>Vertical stacking of results when there are 3+ variations (no more horizontal scrolling!)</li><li>Clear CTAs at the top of the page to start an experiment, ramp up traffic, and make a decision</li></ul><h3>Feature Flags + Experiments = Better Together</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Paxik_TyST3NsQL4" /></figure><p>We’ve made massive changes behind the scenes to integrate Feature Flags and Experiments better. You can now have multiple Features controlled by a single Experiment, more control over the Experiment lifecycle, and combo experiments that use both Feature Flags and our Visual Editor. This background work will enable much more in the future, so stay tuned!</p><h3>Datadog Integration Guide</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6DG5i4sPRGg3QY3I" /></figure><p>By integrating GrowthBook and Datadog, you can set up custom workflows to automatically toggle Feature Flags based on changes in your Datadog metrics. For example, roll back a release if error rates increase. You can find the guide in our docs, <a href="https://docs.growthbook.io/integrations/datadog">here</a>.</p><h3>Improved Onboarding &amp; Built-in Sample Data</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Y56G1fgDyiJN3x85" /></figure><p>It’s now much quicker and easier to set up a new GrowthBook account from scratch. If you aren’t ready to connect GrowthBook to your application and data yet, there’s a new sample dataset built-in, including metrics, experiments, and feature flags (with an easy cleanup button when you’re done). And, once you’re ready to fully integrate GrowthBook, we streamlined the database connection process and SDK integration.</p><h3>Additional Features and Improvements</h3><ul><li>BigQuery performance and cost improvements</li><li>Full Handlebars support for SQL template variables</li><li>Self-hosted Docker image now supports multiple platforms (x86 + arm)</li><li>New documentation organization</li><li>New organization setting to specify a default data source</li><li>Realtime streaming support in the Java SDK</li></ul><p>Plus, there are many more changes and bug fixes which you can read about here: <a href="https://github.com/growthbook/growthbook/releases">https://github.com/growthbook/growthbook/releases</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c1b235e189df" width="1" height="1" alt=""><hr><p><a href="https://medium.com/growth-book/growthbook-version-2-4-c1b235e189df">GrowthBook Version 2.4</a> was originally published in <a href="https://medium.com/growth-book">GrowthBook</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GrowthBook Version 2.3]]></title>
            <link>https://medium.com/growth-book/growthbook-version-2-3-76c81795d993?source=rss----3c28b27c9524---4</link>
            <guid isPermaLink="false">https://medium.com/p/76c81795d993</guid>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[feature-flags]]></category>
            <category><![CDATA[ab-testing]]></category>
            <dc:creator><![CDATA[Graham McNicoll]]></dc:creator>
            <pubDate>Mon, 31 Jul 2023 04:07:23 GMT</pubDate>
            <atom:updated>2023-08-02T05:50:08.466Z</atom:updated>
            <content:encoded><![CDATA[<h4>New Experiment Page Design, LaunchDarkly Importer, Automatic Metric Discovery, Improved CDN, and more</h4><p>We are thrilled to announce the launch of our latest GrowthBook version, packed with a host of new features and enhancements! This version includes a redesigned experiment page, automatic metric creation, a new CDN, OpenTelemetry support, and more. Here are some of the highlights in this update:</p><h3>Redesigned Experiment Page and Left Nav</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*massYAk7t5h1GOij" /></figure><p>We listened to your feedback and improved the design of the experiment results page, as well as adjusted the structure of the left navigation. The top of the experiment page now gives all the context you need to understand exactly what was tested, including any linked feature flags. This section can also collapse to focus on the results. Check it out, and let us know what you think! We have some more big changes planned for the experiment results section in the next release, so stay tuned!</p><h3>Automatic Metric Discovery with Segment, Rudderstack, or GA4</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_apQQa8AoUmU78zv" /></figure><p>For users of these popular event tracking tools, we are now able to auto-detect and automatically create metrics for your tracked events, including all of the SQL. This feature can be accessed when adding a new data source or by viewing the details of an existing data source. Soon, we will be adding support for additional event trackers, such as Amplitude. Read more about this feature on our engineering blog <a href="https://blog.growthbook.io/new-feature-save-time-with-automatic-metric-generation">here</a>.</p><h3>New Cloud CDN with Real Time Purging and Streaming</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SSpUAkTxyVnjEfYp" /></figure><p>We migrated our CDN on GrowthBook Cloud to Fastly, bringing faster response times and greater reliability. This also allowed us to enable instant purges and real time streaming for everyone. When a feature is published, it is now available globally in under one second, and updates are streamed down to supported SDKs automatically (JavaScript/React only right now, more will be coming soon).</p><p>For self-hosted users, we are working on documentation for configuring your own CDN to get many of these same benefits.</p><h3>Percentile Capping (Winsorization)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*I_c5lmoijX0qPXBD" /></figure><p>Since the beginning, GrowthBook has supported capping metric values to remove outliers and reduce variance in experiments (also known as Winsorization). Previously, it required custom analysis outside GrowthBook to determine the best capping value to use, but with GrowthBook’s new support of percentiles, this process is now much simpler and less error prone.</p><h3>Observability for Self-Hosted GrowthBook Instances (OpenTelemetry)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sru977w2ABVZ3EhU" /></figure><p>For our self-hosted users, we have integrated OpenTelemetry into GrowthBook. With this integration, you can easily monitor the health and performance of your GrowthBook instances in production. Learn how to configure this with our <a href="https://docs.growthbook.io/self-host/env#observability-opentelemetry">instructions in the docs</a>.</p><h3>Import from LaunchDarkly</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EWbgnDeAhKWeZcob46RgTQ.png" /></figure><p>Migration from LaunchDarkly to GrowthBook is now easier than ever! Easily transfer features, projects, and environments from LaunchDarkly directly from within GrowthBook from the new “Import Your Data” page in the Settings menu.</p><h3>Additional Features and Improvements</h3><ul><li>Optional confirmation step for feature kill switches</li><li>New REST API endpoints for metrics and experiments</li><li>SQL performance improvements (up to 40% in some cases!)</li><li>Improved experiment presentations</li></ul><p>Plus many more changes and bug fixes which you can read about here: <a href="https://github.com/growthbook/growthbook/releases">https://github.com/growthbook/growthbook/releases</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=76c81795d993" width="1" height="1" alt=""><hr><p><a href="https://medium.com/growth-book/growthbook-version-2-3-76c81795d993">GrowthBook Version 2.3</a> was originally published in <a href="https://medium.com/growth-book">GrowthBook</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GrowthBook Version 2.2]]></title>
            <link>https://medium.com/growth-book/growthbook-version-2-2-495a8a494be1?source=rss----3c28b27c9524---4</link>
            <guid isPermaLink="false">https://medium.com/p/495a8a494be1</guid>
            <category><![CDATA[ab-testing]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[product-development]]></category>
            <category><![CDATA[experimentation]]></category>
            <dc:creator><![CDATA[Graham McNicoll]]></dc:creator>
            <pubDate>Sat, 10 Jun 2023 04:44:54 GMT</pubDate>
            <atom:updated>2023-06-11T23:01:54.307Z</atom:updated>
            <content:encoded><![CDATA[<h4>Visual editor improvements, secure feature values, JSON validation and more</h4><p>We are pleased to announce the release of version 2.2. This version includes improvements to the Visual Editor, secure feature values, JSON validation, semantic version targeting support and more. Find out more below.</p><h3>Visual Editor — JS injection and CSS property editing</h3><figure><img alt="Showing an example of the visual editor editing javascript and CSS rules" src="https://cdn-images-1.medium.com/max/900/0*zctwpFMiS9ym_yWz" /><figcaption>JS editor and CSS attribute editor in the improved visual editor</figcaption></figure><p>This release expands the types of A/B tests you can create with our Visual Editor. You can now run custom javascript per variation to make complex changes to the page. Plus, you can now edit CSS properties (font size, color, etc.) for an element directly without needing to manipulate class names.</p><h3>JSON Schema Validation</h3><figure><img alt="Showing that JSON Schemas that can be added to features, with an example of the value validation error" src="https://cdn-images-1.medium.com/max/1024/0*DhjBLKxjuWQ6uZ3F" /><figcaption>JSON Schemas can be added to validate values</figcaption></figure><p>One of the really powerful ways to use GrowthBook is as a remote configuration tool. You can pass down entire JSON objects to your application and make changes without a new deploy. However, typing JSON into a text area can be very error prone. With version 2.2 you can now attach a JSON Schema to a feature flag to validate every value before saving. This feature is available to Enterprise customers.</p><h3>Secure Targeting Attributes</h3><figure><img alt="Attribute add/edit box showing new secure string type" src="https://cdn-images-1.medium.com/max/1024/0*g6-qhEbhetmtrbIF" /><figcaption>Select secure strings to hash values sent to SDKs</figcaption></figure><p>There are times when you want to target feature flags to specific users, but don’t want their IDs or emails exposed in the JSON payload when using our client-side SDKs. You can now mark any attribute as a “secure string” which will apply SHA256 hashing to the values in targeting conditions before sending the JSON payload to our SDKs. For additional protection, we even allow specifying a unique salt for your organization. You must enable this feature in your SDK Connections for it to take effect. This feature is available for Pro users.</p><h3>REST API Permissions</h3><p>As we add more REST endpoints to programmatically interact with your GrowthBook data, there’s been a growing need for more fine-grained permission controls for API tokens. This release adds two new types of tokens to accomplish this.</p><p>First, any user can now create their own <strong>Personal Access Tokens</strong>, which will assume their current role when used. If the user’s role changes in the future (or if they are removed from your organization), their access tokens will be updated immediately to reflect the changes. Personal Access Tokens are accessible from the user drop-down menu.</p><p>Second, admins can now create <strong>Read-only Tokens</strong> on the API Key settings page. These are great for scripts that don’t need write access, such as one that syncs experiment results into your data warehouse. You can now confidently build on top of GrowthBook without the risk of accidentally deleting or modifying something in your account.</p><h3>Custom <strong>Currencies</strong></h3><p>By popular request, we’ve added the ability to adjust the currency shown within GrowthBook for revenue metrics. You can select any global currency from the drop down on the General Settings page.</p><h3>FullStory Integration</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/0*avG0FW21FFVEOqSn" /></figure><p>Users of FullStory that have set up a <a href="https://help.fullstory.com/hc/en-us/articles/6295300682903-Data-Destinations">data destination</a> to BigQuery or Snowflake can now use that data for metrics and experiment results from within GrowthBook. This event source is in beta and may require some small adjustments.</p><h3>MongoDB 6.0 Support</h3><p>We updated our internal drivers for MongoDB so you can now use GrowthBook with the latest MongoDB 6.0.</p><p>As part of this upgrade, we did have to drop support for super old MongoDB versions (version 3.4 and lower). In some rare cases, you also may have to adjust the query string options in your MongoDB Connection URL. See <a href="https://github.com/growthbook/growthbook/releases/tag/v2.2.0">here</a> for more details.</p><h3>Other Features and Improvements</h3><ul><li>Version string comparison operators in client-side SDKs (semantic versioning)</li><li>Lots of UI and SDK bug fixes</li><li>Tons of new commands in the GrowthBook CLI</li></ul><p>Plus many more changes and bug fixes which you can read about here: <a href="https://github.com/growthbook/growthbook/releases">https://github.com/growthbook/growthbook/releases</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=495a8a494be1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/growth-book/growthbook-version-2-2-495a8a494be1">GrowthBook Version 2.2</a> was originally published in <a href="https://medium.com/growth-book">GrowthBook</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>