<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Axel Delmas on Medium]]></title>
        <description><![CDATA[Stories by Axel Delmas on Medium]]></description>
        <link>https://medium.com/@axel_57048?source=rss-e6c9a630869f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*GkWr4FN5mQDanqFr0wpJAQ.png</url>
            <title>Stories by Axel Delmas on Medium</title>
            <link>https://medium.com/@axel_57048?source=rss-e6c9a630869f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 07:12:04 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@axel_57048/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to be data-driven when you aren’t Netflix (or even if you are) — part 2: AB Testing]]></title>
            <link>https://medium.com/lumen-engineering-blog/how-to-be-data-driven-when-you-arent-netflix-or-even-if-you-are-part-2-ab-testing-e57ecc46102d?source=rss-e6c9a630869f------2</link>
            <guid isPermaLink="false">https://medium.com/p/e57ecc46102d</guid>
            <category><![CDATA[ab-testing]]></category>
            <category><![CDATA[streaming-video]]></category>
            <category><![CDATA[video-delivery]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[video-workflow]]></category>
            <dc:creator><![CDATA[Axel Delmas]]></dc:creator>
            <pubDate>Mon, 25 Jun 2018 16:08:52 GMT</pubDate>
            <atom:updated>2018-06-25T16:18:57.987Z</atom:updated>
            <content:encoded><![CDATA[<p>In the <a href="https://medium.com/streamroot-developers-blog/how-to-be-data-driven-when-you-arent-netflix-or-even-if-you-are-part-1-b1d3e1ec9758">previous article of this series</a>, we covered the first steps of our journey towards becoming more data driven. We’ve explained why time series are simply not enough because of the noise introduced by environmental factors. In this article we’ll explain how we got over this by setting up an AB test workflow.</p><p>AB tests don’t remove noise from your dataset, but they do ensure that both populations, A and B, are subject to the same environmental variations, by testing two different configurations over the same time period. When your viewers are randomly allocated to each population, and with enough payloads in each dataset, the noise will average out; you can be reasonably sure that if you’re seeing a major difference in your metrics between the two populations, it’s due to the change you’re measuring, rather than an environmental change. Basically, it provides a reference point while allowing you to filter signals from noise, and therefore take better informed decisions.</p><p>Let’s go over a simple example to illustrate how AB testing eliminates background noise. Our customers’ video rebuffering rate tends to decrease as the number of concurrent users rises; as more users connect to a stream, the more local sources become available to the viewer, bringing the content closer to the individual user’s device. Let’s say we want to test a configuration change that is supposed to even further decrease rebuffering rates. If we try the new configuration on all users, it would be hard to pinpoint the new configuration’s exact effect on rebuffering, especially in cases where the number of concurrent users is volatile. And that’s without talking about other environmental factors that could affect rebuffering, such as geographical location and internet connection speed. Dividing the test pool into two groups randomly — group A with the old configuration, and group B with the new one — and comparing rebuffering results from the same time period means that we can safely assume that the difference in rebuffering rate that we witness is due to our configuration change. With enough data this would allow us to filter out the environmental noise, as any condition that might affect rebuffering would be equally present in both populations.</p><h4><em>Putting it all together: getting rid of noise</em></h4><p>When we started leveraging AB testing, we stopped looking at time series altogether to determine which population was behaving better. Although time series for both populations are subject to the same environmental noise during the time of the AB test — unless the change you’re testing has a tremendous impact — the time series will keep intersect, making it rather hard to determine if one of the population is “on top”. Let’s take a look at a time series showing the percentage of P2P offload during an AB test. Each point in the time series represents the average value over a 5-minute interval:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fgiKz98_hR_B-K06" /></figure><p>To overcome this representation issue, we started comparing a single aggregate value for each population:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nWYg3kzbmArM8YRdqoadEw.png" /></figure><p>We thereby remove the time variable from the equation. When running an AB test for long enough, the aggregate value will start converging to a more stable value and successfully average out the environmental noise.</p><p>Let’s take the same data set, but this time let’s plot the cumulative average value of the entire data set over time:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2UzyD_M_Uhr2al-A" /></figure><p>This clearly shows that when we don’t have enough data points at the beginning of an AB test, the result is still very noisy. That’s pretty normal since we split the two populations randomly; an impactful environmental factor might still be over-represented in one of the two populations at this point. But as we let our AB test run and gather more and more data points for both populations, the environmental factors become more uniformly represented within each population, eventually converging into stable, legible, noise-free results.</p><p>So now that we’ve established that it’s important to gather enough data points during an AB test, you might ask yourself how much data is enough? Well, that’s the tricky point. It really depends on how noisy your data is: how much the metric you’re looking at is affected by different environmental factors, and to what extent these factors are variable in the test population.</p><p>Let’s take an example: let’s say we’re looking at the rebuffering ratio for viewers in Scandinavia. Since most internet connections in this region enjoy plenty of bandwidth, and users usually have similar conditions, we expect very low noise levels in our data and therefore require less of it. When looking at rebuffering ratio for a given stream for viewers across the whole world, it is a different story. Bandwidth conditions will be very different from country to country, creating a pretty noisy data set, and therefore require that we run my AB test for a longer period of time and gather more data points in order to have relevant information for our AB test.</p><p>So how can we determine the quantity of data points we required for trusting AB test results? The simplest way is to run an AA test and see when the value for both populations becomes consistently identical: the two time series should be almost superimposed starting from the moment you gathered enough data.</p><p>Another option is to use statistical analysis tools on your data set in order to determine a level of confidence. We’ll cover those in the next article of our series.</p><h4><em>AB Testing at Streamroot</em></h4><p>We implemented two different levels of AB testing in our workflow: <em>Release AB Testing</em>, which allows us to compare two different versions of our library; and <em>Configuration Injection AB Testing</em>, which enables us to inject parameters to fine-tune our algorithms and to toggle features on and off.</p><p>We perform <em>Release AB Testing </em>by using a reverse proxy: whenever a user requests our client-side library in the beginning of a video session, the request first hits a reverse proxy that can be configured to return different files according to different rules. We use <em>Release AB Testing</em> whenever we roll out a new release of our product. You can configure the reverse proxy to randomly return files A and B with a 50–50 distribution to do AB testing, or with a 95–5 distribution to do “canary testing”, i.e. providing the new version to a small fraction of users, and then ramping up those numbers to distribute the new version gradually as you collect more data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/788/0*Lby0DL8q8Kn7dG5v" /></figure><p>We perform Configuration Injection to test small changes in configuration parameters, as well as to toggle specific features on and off while rolling out a new version of our product. In order to perform C<em>onfiguration Injection AB Testing </em>our library loads a tiny configuration from our servers. In the same fashion as for <em>Release AB Testing</em>, our servers can be configured to return different configuration files according to any distribution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/0*zk4NP3OFMeEAa1z_" /></figure><p>This configuration file contains different values for parameters that will change how our algorithm behaves. By AB testing different values for these parameters we can find those that behave best and fine tune our algorithms. This configuration file also allows us to toggle specific features on and off so we can compare two different flavors of an algorithm.</p><p>These two mechanisms have truly empowered us in our journey to make our technology even more efficient. This process has enabled something we consider crucial when rolling out changes in a complex technology, where real world results can fall far from expectations and from what you can observe in a lab environment: the ability to test changes atomically.</p><p>When we prepare a new release, we use both <em>Release AB Testing</em> and <em>Configuration Injection AB Testing</em>. In every new release, we make sure every new feature is togglable, and deactivated by default. That way, the new release with its default configuration is identical to the previous one. We then proceed to <em>Release AB testing</em> to test both releases side by side and make sure all our metrics are identical. This way we ensure that there is not any regression hidden in our build. Once the new release is validated, we gradually deploy it for 100% of the viewers. Once our new version is 100% deployed, we use <em>Configuration Injection</em> to AB test the new version features, one by one, activating toggles one at a time to verify that each new feature generates the expected results compared to populations where the feature hasn’t been turned on yet. If a change doesn’t produce the improvements we expect, even after some parameter tweaking, we disable it and ended to figure out why it did not work as expected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/0*BPsj2bpts0UhUEyU" /></figure><h4><em>AB Testing: An effort well worth making</em></h4><p>This approach may look cumbersome, but quality is paramount to us. We believe this is the most robust method to roll out changes for our customers. We have a rigorous engineering process including code reviews, automated testing and manual testing, but software being software, it is possible that bugs may find a way to slip through the cracks. This holds true even in mission critical software you’d think would be tested in and out, and thus bug proof. Like <a href="https://www.theregister.co.uk/2013/09/27/30_years_on_the_day_a_computer_glitch_nearly_caused_world_war_iii/">this glitch</a> that almost started World War III some 35 years ago.</p><p>And if you’re still thinking this is overkill, here’s another thing to consider: without this process, every regression means a complete rollback that can stall new releases and product enhancements and improvements. With this mechanism in place, we minimize the risk of a bug in the first place, and in the event of an issue in the one of the new togglable features, we just toggle it back off and fix it in the next release, saving time, money, and working far more efficiently.</p><p>Finally, testing changes atomically has another benefit: imagine we released several features in a single release, without a toggle mechanism. Let’s say each of the first three of these features mildly improves QoS, whereas the fourth new feature degrades it, just about cancelling the effect of the first three improvements. Looking at the stats, we might think that none of these features changed anything, when in fact they averaged out. We might decide not to keep any of them, as they don’t seem to improve QoS, actually losing 3 improvements in the process. We might decide to keep all of them, with the goal of tweaking them in the next release, letting a regression slip into our product without being aware of it. Either way is far from ideal..</p><p>AB testing is a powerful, indispensable tool that allows us to be better informed and make the right data-driven decisions. In this article, we discussed mostly how to test different product versions and configurations. This brings us to the next, equally important question — what data do we test? To reach the right conclusion, we have to take into consideration the right data, make sure that we have enough of it, and make the proper data splits. We’ll try to answer all those questions in part 3 of this series, so stay tuned!</p><p>At Streamroot, we are always on the lookout for new tech talent to join our team. Check out open positions on our <a href="https://www.streamroot.io/careers/">career page</a> or send us your CV directly to contact@streamroot.io.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e57ecc46102d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lumen-engineering-blog/how-to-be-data-driven-when-you-arent-netflix-or-even-if-you-are-part-2-ab-testing-e57ecc46102d">How to be data-driven when you aren’t Netflix (or even if you are) — part 2: AB Testing</a> was originally published in <a href="https://medium.com/lumen-engineering-blog">Lumen Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to be data-driven when you aren’t Netflix (or even if you are) — Part 1]]></title>
            <link>https://medium.com/lumen-engineering-blog/how-to-be-data-driven-when-you-arent-netflix-or-even-if-you-are-part-1-b1d3e1ec9758?source=rss-e6c9a630869f------2</link>
            <guid isPermaLink="false">https://medium.com/p/b1d3e1ec9758</guid>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[Axel Delmas]]></dc:creator>
            <pubDate>Thu, 05 Apr 2018 13:50:18 GMT</pubDate>
            <atom:updated>2018-04-05T13:51:13.242Z</atom:updated>
            <content:encoded><![CDATA[<p>Data analytics has always played a key role at Streamroot. From day one, our core goal was to improve quality of service for end viewers, as well as offloading as much bandwidth as possible from servers without compromising on QoS. That’s why we set out to implement a comprehensive analytics module built-in to our solution right from the start, at the prototype phase.</p><p>For those of you who are unfamiliar with our solution, Streamroot is a leading provider of innovative OTT delivery technologies for media groups, content publishers and enterprise customers. Our Distributed Network Architecture solution — Streamroot DNA™— prevents congestion and server saturation by offloading significant traffic off CDNs, offering broadcasters improved quality of service, greater audience reach and infinite delivery capacity at a reduced delivery cost.</p><h4>The crucial role of data analytics</h4><p>Our solution includes quite a few different algorithms, many of them distributed, and we realized quite early that modifications could have quite a different outcome out there than in our office environment. Real-life conditions can be tricky to reproduce, and there are infinite possibilities when there are millions of devices implementing your distributed technology. We realized pretty early that relying on intuition only was like trying to land a rocket on the moon with a walkie-talkie and a swiss army knife.</p><p>When it comes to improving our technology, we have a very iterative process. As appealing as the idea of crushing your enemy with a single blow may seem, improving a complex software is more like standing toe-to-toe against an army of small issues, advancing by baby steps. You don’t want to compromise all those hard won improvements by making a single bad decision and go back to where you were three months before. The only way to truly move forward is to validate each step along the way using comprehensive data analysis. You would probably like to do that too, when you make decisions and optimizations in your video workflow, like tweaking your ABR logic or stream packaging parameters to improve quality of service.</p><p>And quality of service is crucial for most video broadcasters. Whether you’re a SVOD platform that seeks to acquire and retain subscribers, or an ad-driven streaming service that is looking to increase viewer engagement, QoS is central in building revenue. Bad viewing experience will make your users leave. And they might never come back.</p><p>Building tools that allow you to be more data-driven and make informed decisions about QoS can be intimidating for broadcasters that don’t always have enough people or resources to invest, but it will pay off in the long run. And the good news is, it is possible to start slow and build your own solution incrementally. <strong>You don’t need to have a huge team of data scientists or set up a full fledged AB testing program to start reaping the benefits of analytics.</strong></p><p>In this series of articles, I’ll cover our own experience building our in-house data solutions, and hopefully help you move forward with your own data-driven decision-making process.</p><h3>The start of our quest — set up your data pipeline</h3><p>The first step in our data odyssey was to build our data pipeline. Here is how it looked when we first started:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cPIA81pyiaRLh3lMegm8Xw.jpeg" /></figure><p>Streamroot’s client-side library runs a small module that collects all sorts of statistics, from P2P offload and QoS metrics, down to more granular metrics related to some specific P2P algorithms. It runs some pre-aggregations when applicable, then sends data payloads to our backend.</p><p>We wanted to be able to scale as we grow our business, so we set up an architecture that leverages a message broker. These data payloads are received by an HTTP endpoint that forwards them directly to our message broker (<a href="https://kafka.apache.org/">Kafka</a>). This technology has been designed for a very high write rate, ingesting messages as fast as they arrive so no data is lost. It is perfect for our use case since we can suddenly have millions of viewers arriving at once, as for example at the beginning of a high-profile live event like the <em>FIFA World Cup</em>, which we’ll be powering for some of our tier-one customers this summer.</p><p>Once the data payloads have been secured in a message queue, they will be consumed by different services. We have an array of such “consumers,” which perform tasks such as enriching the data payload with metadata or running aggregations, then outputting the resulting data points in a database. Since we needed time series for our customer dashboard, we chose <a href="https://www.influxdata.com/">InfluxDB</a> as a time series database. We’ve later introduced other technologies better suited to our constant improvement needs, but I’ll come back to that later in our series.</p><p>This was a first step; it provided us with basic insights on the performance of our solution, but but it had quite a few limitations.</p><h3>Why a time series is simply not enough</h3><p>When we first started with this data pipeline, we only had time series to represent our metrics. Whenever we pushed a new version or a configuration change, the only way we could verify its expected effect on traffic was to look at the time series, before and after the time of the change. This approach has a serious limitation: video data have huge variations due to environmental factors. We might see global changes in our P2P offload; however, it is almost impossible to tell with a great certainty that these effects are due to the configuration change, or the new version we implemented.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sGQBOsbJw7YRtOF0CvngFQ.jpeg" /></figure><p>A good example of external factors that may obscure results while testing in a global environment is the state of the network infrastructure. Network conditions are highly variable: a couple of edge servers in your CDN might fail, or maybe there’s an exceptional traffic peak from a geographical region where CDN presence is more limited. All these will make their marks on your graphs, but they don’t necessarily mean your solution has gotten any better or worse due to the change you made in its configuration, or the new version you distributed. Some of those variations might be important to note in order to improve your CDN configuration, but for our purposes they’re mostly environmental noise.</p><p>With time series, we were able to have some level of understanding on how our solution was performing, but we could not validate whether the changes we were implementing were moving the needle in the right direction. We had to find a way to remove noise and environmental factors. This is when we started working on our AB testing workflow.</p><p>Part 2 of this series or articles will be dedicated to our AB testing process, so stay tuned for my next article here on the Streamroot Tech Blog.</p><h3>Learn more about our data analytics at NAB</h3><p>I’ll be talking about our data-driven decision making process next week at NAB. Drop by to say hi if you’re there:</p><p><strong><em>Wednesday, April 11 at 11:00 am, North Hall Meeting Rooms — N255</em></strong>.</p><p>Also, I’ll be happy to see you and catch up at our <strong>booth SU9114</strong> during the week. Hope to see you at NAB!</p><p>As usual, we are always looking for new tech talent to join our team, check out open positions on our <a href="https://www.streamroot.io/careers/">career page</a> or send us your CV directly to <a href="/service/mailto:contact@streamroot.io">contact@streamroot.io</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b1d3e1ec9758" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lumen-engineering-blog/how-to-be-data-driven-when-you-arent-netflix-or-even-if-you-are-part-1-b1d3e1ec9758">How to be data-driven when you aren’t Netflix (or even if you are) — Part 1</a> was originally published in <a href="https://medium.com/lumen-engineering-blog">Lumen Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>