<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Frédéric Gaudet on Medium]]></title>
        <description><![CDATA[Stories by Frédéric Gaudet on Medium]]></description>
        <link>https://medium.com/@fred_24339?source=rss-ef3d4900f29c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*BXBlArU7fkqg4wWxWl35sQ.jpeg</url>
            <title>Stories by Frédéric Gaudet on Medium</title>
            <link>https://medium.com/@fred_24339?source=rss-ef3d4900f29c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 11:11:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@fred_24339/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Warmup your pods using Istio]]></title>
            <link>https://medium.com/blablacar/warmup-your-pods-using-istio-5249ec68f0e9?source=rss-ef3d4900f29c------2</link>
            <guid isPermaLink="false">https://medium.com/p/5249ec68f0e9</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[istio]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[tech]]></category>
            <dc:creator><![CDATA[Frédéric Gaudet]]></dc:creator>
            <pubDate>Mon, 03 Feb 2025 15:36:51 GMT</pubDate>
            <atom:updated>2025-02-03T15:36:51.799Z</atom:updated>
            <content:encoded><![CDATA[<h3>Warmup Your Pods Using Istio</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RWiwnjkpM6k2tJYxt5dkeQ.png" /></figure><p>At BlaBlaCar we continuously improve our platform to provide the best user experience. In this article we’ll share how we leveraged the new Istio load balancing warmup feature to decrease pods latency during rollout phases.</p><h3>Problem</h3><p>First let’s dig into the problem we faced. We are using Java micro-services, which typically need a warmup phase to achieve optimal performance.</p><p>When a JVM-based application starts, the first requests it receives are generally significantly slower than the average response time. Indeed, during the first moments of the life of an app, the JVM starts by taking its bytecode and running it in the interpreter, which requires lots of CPU resources and is slower to execute. The JIT compiler will then kick in and try to optimize the bytecode, compiling hot code paths into native — highly optimized — code. This is what a typical application startup will look like in terms of speed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/746/0*ofx3uei-navvJTva" /><figcaption><em>Speed of the application through the different phases — Source : </em><a href="https://docs.azul.com/prime/analyzing-tuning-warmup"><em>https://docs.azul.com/prime/analyzing-tuning-warmup</em></a></figcaption></figure><p>It usually takes more than 10K invocations before the optimization process stabilizes.</p><p>Running non-optimized code while consuming lots of CPU to optimize it leads to poor performance, resulting in timeouts and user-facing errors.</p><p>This situation could be partially mitigated by increasing the CPU shares allocated to the application, but this is costly since these extra shares will only be useful at startup: it becomes a waste 99% of the time.</p><p>The problem of poor performance at startup is becoming one of the top priorities for the JVM community and many projects try to address it: Leyden, GraalVM, CraC among the most known initiatives, but also tools like <a href="https://github.com/google/kube-startup-cpu-boost">Kube startup CPU boost</a>. We evaluated most of them but none worked for us, either because they are not ready for prime time yet or because they are not adapted to our current technology choices.</p><p>We do have a warmup solution that works pretty well, which we talk about in <a href="https://medium.com/blablacar/warm-up-the-relationship-between-java-and-kubernetes-7fc5741f9a23">this blog post</a>, but it requires extra maintenance and adds its own complexity. Also, this solution doesn’t cover all use cases as we don’t want to interfere with the production data (so it typically only covers GET requests).</p><p>We therefore kept trying to find other low-maintenance solutions to, if not eliminate the warmup issues completely, at least reduce their impact.</p><h3>Warmup Config in Istio</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dMDBhfEmrd6TaTM-" /></figure><p>Since we use Istio as a service mesh in our Kubernetes cluster, we also had a look into the “slow start mode” in order to solve this issue.</p><p>It’s a very interesting pattern as it ramps up the traffic during a custom duration ensuring a smooth startup. The goal is obviously to avoid giving 100% of the traffic to a new READY pod but instead send a smaller amount during a short time period.</p><p>It sounded great initially, but after some tests in our context it wasn’t: we discovered a significant drawback as it comes with a hardcoded minimum of 10% traffic which was already way too much for us.</p><p>Since Istio relies on envoy proxy which supports a configurable load start, we’ve made a <a href="https://github.com/istio/istio/pull/53472">contribution</a> in Istio to update the destinationRule API. The slow start mode has been replaced by the <a href="https://istio.io/latest/docs/reference/config/networking/destination-rule/#WarmupConfiguration">warmup configuration</a> to add two new parameters along with the existing duration:</p><ul><li><strong>minimumPercent</strong>: which configures the minimum percentage of traffic for a fresh instance</li><li><strong>aggression</strong>: this parameter controls the speed of traffic increase over the warmup duration. Defaults to 1.0, so that endpoints would get a linearly increasing amount of traffic. When increasing the value for this parameter, the speed of traffic ramp-up increases non-linearly.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/515/0*QYrAx2lS7NEWxLas" /><figcaption>Source : <a href="https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/load_balancing/slow_start">https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/load_balancing/slow_start</a></figcaption></figure><p>It’s worth mentioning that this is most effective when few new endpoints come up like scale events in Kubernetes. When all the endpoints are relatively new like new deployment, this is not very effective as all endpoints end up getting the same amount of requests. Finally, this is only supported for ROUND_ROBIN and LEAST_REQUEST load balancers.</p><h3>Testing</h3><p>After the Istio 1.24 release, we tested this new feature in a simple way: using HTTP requests load testing with <a href="https://fortio.org/">fortio</a> pods to check the behavior of the warmup during pods’ lifecycle: eviction, rollout and scale up. The goal is only to test the contribution we’ve made but not to test if it solves the java latency issue described earlier.</p><h3>Testing Setup</h3><p>During those tests we used fortio client and fortio server deployments. A load test is sent from the client to the server at 1000 QPS with 10 threads.</p><h3>Pods Configuration</h3><p>✅ 1 Client Pod with 1 core CPU request</p><p>✅ Minimum of 2 fortio server pods with 100m CPU Request</p><p>✅ Fortio server has an <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/">HPA</a> CPU target at 70%</p><h3>Warmup Configuration</h3><pre>apiVersion: networking.istio.io/v1<br>kind: DestinationRule<br>metadata:<br>  name: xxx<br>spec:<br>  host: fortio-server.infra.svc.cluster.local<br>  trafficPolicy:<br>    loadBalancer:<br>      warmup:<br>        minimumPercent: 2.0<br>        duration: 2m<br>        aggression: 1.0</pre><h3>Deletion of One Pod</h3><p>We first experimented with the deletion of one pod, which is immediately recreated thanks to the deployment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*j_woeBnlRmzHQwG-" /></figure><p>We can observe that existing pods take the extra load while the new pod warms up. Ultimately, after 2 minutes it has the same load weight and accepts an equal share of the traffic load.</p><h3>Brutal Scale Up</h3><p>Then we did another experimentation with 2 pods running which receive no traffic. Then we brutally increased the load 😀. Each vertical red bar indicates a pod creation event.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Uj7RCL6frk6oidmg" /></figure><p>The 2 existing pods take the load until the HPA triggers new pods creation to reach 8 replicas as seen below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ec_qhDCcDC89N1fg" /></figure><h3>Rolling Update</h3><p>The key factors here are the <strong>maxSurge</strong> and <strong>maxUnavailable</strong> settings applied in our deployment. If we don’t use them, all pods will be rolled out at the same moment which makes the load weight very inaccurate. We then don’t enjoy the benefit of the incremental load.</p><p>Demonstration with this deployment setting:</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: fortio-server<br>spec:<br>  strategy:<br>    type: RollingUpdate<br>  template:<br>    metadata:<br>[...]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*l-ZpoqqPrH2CWQD2" /><figcaption>No incremental load if all pods are roll out at the same moment</figcaption></figure><p>If we apply for example a strategy of <strong>maxSurge = 34%</strong> and <strong>maxUnavailable = 0</strong>, then the rollout is much more progressive allowing the calculated load share to be distributed more accurately. This is to be adapted to each environment but this is the setting we generally use in production at BlaBlaCar.</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: fortio-server<br>spec:<br>  strategy:<br>    rollingUpdate:<br>      maxSurge: 34%<br>      maxUnavailable: 0<br>    type: RollingUpdate<br>  template:<br>    metadata:<br>[...]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Pe-IstS8N0YbauHH" /></figure><h3>Rollout in Production</h3><p>After testing the warmup feature in a staging context, we deployed it in production and observed the results on one of our most sensitive micro-service.</p><p>Here is a comparison without (on your left hand side) and with (on your right hand side) the Istio warmup activated with the following settings:</p><pre>apiVersion: networking.istio.io/v1<br>kind: DestinationRule<br>metadata:<br>  name: xxx<br>  namespace: yyy<br>spec:<br>  host: xxx.yyy.svc.cluster.local<br>  trafficPolicy:<br>    loadBalancer:<br>      warmup:<br>        aggression: 1<br>        duration: 5m<br>        minimumPercent: 0.1</pre><p>This comparison is done during a restart of the service with the rollout strategy mentioned above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bDSaawhfIfzE89gakQ18DQ.png" /><figcaption>The loading shape is very different</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mf9N_gGj0h_Eod01tUrAug.png" /><figcaption>Less latency spike with the warmup activated</figcaption></figure><p>Without the Istio warmup activated, we suffer from multiple latency spikes during the rollout of the service, the consequence being a degrading experience for clients calling our service.</p><p>However, once the Istio warmup has been activated, we can observe as expected that the traffic is progressively being distributed to each pod, minimizing latency spikes. Of course it does not resolve 100% of the latency spikes we encountered, the very first requests routed to the cold pods will still be slow but overall all the following requests are processed faster.</p><h3>Service Level Objective Impacts</h3><p>We waited a few more days to see if the impact we saw at a first glance was confirmed on our latency SLIs, here is a comparison of some of our latency SLI and the resulting SLO impact on this micro-service.</p><p>First endpoint:</p><p>The SLI threshold is 500ms of request latency with percentile p99, under this threshold it’s what we consider as a bad event.</p><p>The SLO Target is 99.80% of good event over one month.</p><p><strong>Before: 99.08% — After: 99.97%</strong></p><p>We can observe some impressive results on this endpoint that was affected by the startup issue of the JVM: once we activated the Istio warmup, most of the spike disappeared and the SLI is now reaching the target SLO that we have set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*COjXvHMCl3wsErD3" /></figure><p>Second endpoint:</p><p>The SLI threshold is 700ms of request latency with percentile p99</p><p>The SLO target is 99.80% of good events over one month</p><p><strong>Before: 98.60% — After: 99.91%</strong></p><p>For this second endpoint, we observe mostly the same results, although it still occasionally experiences latency spikes. However, these spikes are minor compared to before and we are now meeting the target SLO.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*z9KZzev-MlOECNUe" /></figure><h3>Conclusion</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FLWBXP5uq-b8wPLE" /></figure><p>The new Istio warmup configuration ensures services in the mesh are fully initialized and reliable before handling most of live traffic. The setup is really easy and only requires a couple of settings. Sometimes easy solutions have great impacts. 😉</p><p>This results in better system stability, accurate observability and smoother traffic management, ultimately improving the overall resilience and user experience of the micro-services environment.</p><p>We are now applying this setup for all our services, even those in PHP!</p><p><em>I would like to warmly thanks </em><a href="https://medium.com/u/f930abce0bb4"><em>Adrien Loison</em></a><em> and </em><a href="https://medium.com/u/f28043bf8586"><em>Bastien Lelarge</em></a><em> for respectively helping me with the detailed Java section and the production results.</em></p><p><em>I also thanks (in no special order) Erwann Robin, </em><a href="https://medium.com/u/fcbc73b17f0a"><em>Aurelien Beltrame</em></a><em>, David Ratel, Samuel Masué, @tony.demol and Benoit Puertolas for their insights in reviewing this article.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5249ec68f0e9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/blablacar/warmup-your-pods-using-istio-5249ec68f0e9">Warmup your pods using Istio</a> was originally published in <a href="https://medium.com/blablacar">BlaBlaCar</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SLOs at BlaBlaCar]]></title>
            <link>https://medium.com/blablacar/slos-at-blablacar-59a126622c3e?source=rss-ef3d4900f29c------2</link>
            <guid isPermaLink="false">https://medium.com/p/59a126622c3e</guid>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[metrics]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[startup]]></category>
            <dc:creator><![CDATA[Frédéric Gaudet]]></dc:creator>
            <pubDate>Mon, 07 Jun 2021 10:45:10 GMT</pubDate>
            <atom:updated>2021-06-07T14:22:38.758Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<p><em>Discover how we set up SLOs at BlaBlaCar and the steps we followed from design all the way to technical implementation.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*37l0kML_DnHoi7HF4CQCUA.png" /></figure><p>BlaBlaCar operates a web-based application serving over 90 million members and aims to become the go-to marketplace for shared road mobility.</p><p>Being able to monitor the platform from a quality standpoint is key to support our user experience and enlighten business decisions. At BlaBlaCar, the SRE team is a cross-functional team designed to provide tooling and expertise to empower service teams on the “you built it, you run it” path. As part of its mission, the SRE team is responsible for the observability stack and monitoring frameworks.</p><p>For years, the engineering teams simply tracked infrastructure and application metrics, raising alerts when errors occurred… but moving from an on-premise monolithic architecture to a microservice architecture <a href="https://medium.com/blablacar/blablacar-a-journey-to-the-cloud-7c6af2701196">hosted on the cloud</a> the context changed, requesting a new pattern.</p><p>Having a service-oriented architecture, we needed to define for each of the distributed services a commitment towards the platform. By doing that, we wanted to help the service teams to balance delivery velocity versus reliability: that’s exactly what <a href="https://sre.google/sre-book/service-level-objectives/">Service Level Objectives</a> (SLOs) are intended for, by defining a threshold that describes the expected “quality” of a service.</p><p>To evaluate how SLOs could improve our reliability and how to implement them at BlaBlaCar, the SRE team bootstrapped a working group. I’ll describe through this article how we set up SLOs at BlaBlaCar, step by step from initial discussions to the technical implementation.</p><p>Before starting, let’s begin with some definitions :</p><ul><li><em>Service Level Indicator</em> (SLI) is what you actually measure. Example: availability, latency.</li><li><em>Service Level Objective </em>(SLO) is what you have internally set as a target. It’s defined from a SLI, with a target over a period of time. Example: “99.9%” availability over 1 month (the so-called « three 9s »).</li><li><em>Service Level Agreement</em> (SLA) is an agreement that includes penalties if the objectives are not met. We don’t use them at BlaBlaCar.</li><li><em>Error Budget</em> provides a clear objective metric that determines how unreliable a service is allowed to be within a single SLO window.</li></ul><p>BlaBlaCar’s service teams mainly consume HTTP backend services, so we started with two easy SLOs: HTTP availability and latency. The initial objective was to select a few services, start to implement SLOs, build dashboards and let the system run for a couple of months…. So how did we get started?</p><h3>It started with a chat</h3><p>Those two questions were the starting point :</p><ul><li>Who are your users? Is your service being consumed by another service consumer or an end-user?</li><li>What are your users facing? What are their expectations about this particular service?</li></ul><p>The key here is to talk with product managers, users, and developers to understand how the service is built and consumed. Thanks to this discussion, you will be able to identify the service boundaries and select a couple of SLIs, to start with. Don’t hesitate to write a literal sentence explaining what the SLI is supposed to measure. It helps to check if the SLI actually means something to the users, which is key.</p><p>There is no need to be exhaustive at this early stage: start small and iterate over time to improve the initial coverage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S2yXP27nHhzf94gucjzSWQ.png" /></figure><h3>We implemented fast</h3><p>We have identified a couple of SLIs, but how are we going to measure them?</p><p>Speaking about HTTP SLIs, we had several options: either capture the metrics from the load balancer, from the service mesh, or from the application itself. Since the SLOs are going to be tracked at a company level, we needed to define a consistent way to measure HTTP requests across all the services. In this context, application metrics are not coherent enough leaving us with two options:</p><ul><li><strong>The load balancers.</strong> They are a great option but in our context, they only provide logs and we didn’t want to manage a whole pipeline of log processing to create metrics.</li><li><strong>The service mesh, Istio.</strong> It provides substantial and consistent observability metrics, it is already deployed and supported by the Foundations team… let’s keep it simple and use it!</li></ul><p>Using Istio metrics we built a few PrometheusRules with the help of <a href="https://jsonnet.org/">jsonnet</a> to calculate our SLIs along with a bunch of useful labels to allow us to graph and combine our SLOs.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/146c7bb7e3e1c1c4dd03b1a064c55354/href">https://medium.com/media/146c7bb7e3e1c1c4dd03b1a064c55354/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9beb7055ebfbfa3acb2a1c659fd416b8/href">https://medium.com/media/9beb7055ebfbfa3acb2a1c659fd416b8/href</a></iframe><p>The above rule is evaluated each minute and computes a time series that indicates whether the service is available or not.</p><p>Depending on the service, you will need to carefully select the traffic. 4xx, 3xx HTTP codes can all be right or wrong traffic.</p><p>From a user perspective, 4xx codes are errors, but is it an error the provider should be held responsible for? If for instance, a request is not authorized due to a token expiration: 4xx is a normal response that can’t be considered a failure.</p><p>On the 3xx codes side: those queries are usually very fast, capturing them could lead to misrepresenting the service user latency.</p><p>Last but not least, it is sometimes useful to combine real user traffic with a probe …or even to only rely on a probe to capture metrics.</p><p>For example, after implementing the HTTP SLIs/SLOs, we decided to work on MariaDB: there was some friction about the database response time that needed to be clarified. To avoid any subjective feelings and only rely on the facts and figures, the <a href="https://medium.com/blablacar/organizing-the-migration-of-a-hundred-databases-to-the-cloud-f3b162eb4a9">Database Reliability Engineering Team</a> wrote a probe to track the queries&#39; response time. Thanks to this probe, SLI is representing the response time for a known request. Having an unchanged complexity on the input side (queries), we are able to define a smart objective for the measured output (response time).</p><p>Having a meaningful indicator helped a lot: if query performance is not good while the SLI’s is showing expected metrics… then the query complexity is to be reviewed.</p><p>At this stage we had meaningful indicators, clearly defined with all stakeholders, and we were able to measure them. That’s a good start but there is still some work!</p><h3>We struggled to define SLOs</h3><p>Coming back to our two HTTP SLOs (latency, availability ): when we started the working group we honestly had no real clue about which targets shall be set and how we would measure them. There are two classic options: volumes-based or time window-based. Picking one has a lot of implications and needs to be considered carefully.</p><p>The first option would be to define a “success ratio”. Such SLO would basically be the target percentage of requests matching the success criteria.</p><p>Doing that, we would end up with SLO statement like these:</p><blockquote>We want to have 99% of good <em>(eg &lt;5ms response time)</em> requests over the month</blockquote><blockquote>We have 1% of failed requests per month in our monthly error budget. The error budget could be down to 500 or up to 3 million requests per month depending on the service traffic.</blockquote><p>Another option is to use an “incident ratio”.</p><p>Representing the month as a multiple of small-time buckets (one minute for instance) we would evaluate the success ratio for each bucket and consider how many buckets have a success ratio above the objective.</p><p>In this case, we would end up with SLO statement like these:</p><blockquote>99% of the time the service must have 90% of successful (eg : <em>&lt;5ms response time</em>) requests or above</blockquote><blockquote>We have 7h downtime per month in our monthly budget</blockquote><p>Each method has its own advantages and drawbacks. With the success ratio method, the SLO score varies in real-time and is directly correlated with the user activity. Having a huge spike of well-handled requests yesterday will minimize the impact of an outage during a low activity period today. That’s perfectly fair. It’s however harder to define a proper error budget policy and thus make decisions easily understandable by all stakeholders.</p><p>Using windows based SLOs, the main drawback is the non-proportionality towards end-users impact. Using our previous example, a time bucket with a 100% failure rate weighs as much as one with 15%. Similarly, the weight is not proportional to the number of affected users: a downtime at night has the exact same weight as downtime during a peak period. As a consequence, such SLOs could be noisy for low request workloads.</p><p>After a lot of discussions, we decided to adopt time-based SLOs, as they are more meaningful for our users and stakeholders.</p><p>We leverage the drawbacks positively: this method is more challenging for BlaBlaCar’s engineering teams than the volume one. Agreeing on a statement as <em>99% of the time the service must have 90% of successful requests or above</em> means that, if it doesn’t have at least a 90% success rate, we consider it as down for one minute, consuming the error budget. This is a positive exigence towards our system reliability and our end users&#39; experience. However, to avoid noisy alerts (and ensure smooth on-call turns) we introduced a minimum number of queries per seconds for each time bucket, in order to make it eligible for SLO evaluation. As a consequence, if an outage occurs during the night with only a very few bad requests it doesn’t violate our SLO and wakes up an engineer.</p><p>This approach combines the best of the two worlds and fits our needs.</p><p>Finally, regarding the availability and latency thresholds, we have decided to keep it simple. We set SLOs consistently with the ratios observed on the platform. We needed to get used to this new framework without adding any pressure on the service teams. From the beginning, we decided to set up a control loop to adjust the SLOs in cooperation and transparency.</p><p>At this stage, we have measured indicators and defined objectives, that we agreed to review regularly. But stopping here would not have brought any evolution to day to day management of the platform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LbQWD7GPnhQLo_pI5qvjiA.png" /></figure><h3>We set alerts meaningful for service users</h3><p>Nowadays, engineering teams are running hundreds or even thousands of servers, building and using software designed to manage server (or even network) failures. As a consequence, it does not make sense to wake up the on-call folks as soon as a pod (or a server) crashes but move to a new paradigm: you should wake up on-call engineers when the user experience is downgraded and alert on <em>symptoms</em> (when your error budget is burning) rather than on a <em>cause </em>that might have no consequence.</p><p>The<a href="https://sre.google/workbook/alerting-on-slos/"> Google SRE’s book</a> introduces several alerting techniques which come along with their pros and cons. At BlaBlaCar, we adopted the multi-burn rate and multi-window alerting because we found it more accurate and efficient to avoid useless alerts while still preserving the user experience.</p><p>Actually… Multi-burn rate and multi-window alerting is simpler to setup than advocating for its usage 😅 . I have to admit that I practiced a lot before feeling very comfortable with the figures and being able to explain it fluently. To help the service teams, we explicitly wrote in front of every alerting graph how the alerting is processed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lJ8HxG1iJHvJl6x3EUx1oA.png" /><figcaption>fig3: Alerting panels</figcaption></figure><p>However, even if the initial presentation was difficult, quickly a team made my day by switching from their legacy alerting to the new SLOs based one !</p><p>To sum up, remember that a SLO alerting system is not intended to help you to find an outage’s root cause. It is designed to alert you about a user (end-user, internal service…) having a poor experience using your service, requesting further investigation (logs, CPU &amp; memory consumption, pod rollout …).</p><h3>Spread the culture … and go further!</h3><p>As we added a lot of new SLOs, enriched with labels, we are able to build a nice status map of every SLO we manage.</p><p>That’s easy to do with the <a href="https://grafana.com/grafana/plugins/flant-statusmap-panel/">Flant Grafana plugin</a> and give a fancy result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JQDCxwvvAhtvA8s2DPVB5w.png" /><figcaption>fig4: SLO status map</figcaption></figure><p>This kind of dashboard spreads the SLO’s culture all over your company, brings questions, gives insight to everyone about what is important for our users.</p><p>Setting up SLOs was a nice step, but we want them to be actionable, otherwise they are just like any other technical KPIs. As a consequence, the SRE team is currently implementing a policy template to help businesses, product owners and developers having relevant conversations when SLOs are not met.</p><p>Breaking a SLO can happen. SLOs are not designed to be blameful nor to do finger-pointing. Using them shall help to take a step back and analyze from facts and figures the user experience. Thanks to them, you might find out what could be done to improve this experience and have enriched discussion when talking about backlog prioritization.</p><p>The policy itself is finally quite simple, it is divided into two main parts:</p><ul><li>An agreement between the dev team, business team and product team on when to slow down the features releasing versus reliability work. Depending on your organization, it could be evaluated on a monthly or quarterly basis.</li><li>A SLO control loop to ensure the objectives are still relevant: are the SLIs still representative? Are the SLOs ambitious enough?</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tvwM2s4AXFr6EadzJqmqKg.png" /></figure><h3>Never-ending story</h3><p>SLO is a useful toolbox, which you will always need to adapt, and can’t be applied as a magic recipe: it must be adjusted according to your context, your business, your organization, your technologies, your culture… All of these have an impact on the implementation, and as your business, organization and technologies will evolve, the SLOs will have to evolve frequently as well.</p><p>Setting up SLOs was an awesome experience. From a SRE perspective, it allows me to sharpen the global vision of our architecture; and it made a lot of sense to help business and engineering teams to rely on facts and figures while addressing the user-perceived quality.</p><p>It takes time to spread the SLO culture at every level of the company. But it is worth it.</p><p><em>Special thanks (in no special order) to Jean Baptiste Favre, </em><a href="https://medium.com/u/6f95002a98e1"><em>Maxime Fouilleul</em></a><em>, </em><a href="https://medium.com/u/9ba009cde4e"><em>Nicolas Salvy</em></a><em> and </em><a href="https://medium.com/u/3195f59feced"><em>Blake Faulds</em></a><em> for their sharp and kind comments in reviewing this article.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=59a126622c3e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/blablacar/slos-at-blablacar-59a126622c3e">SLOs at BlaBlaCar</a> was originally published in <a href="https://medium.com/blablacar">BlaBlaCar</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>