<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Xabier Larrakoetxea on Medium]]></title>
        <description><![CDATA[Stories by Xabier Larrakoetxea on Medium]]></description>
        <link>https://medium.com/@slok?source=rss-13dde3ada38b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*FhIOOJAjQLUBKa8m5ICxeg.jpeg</url>
            <title>Stories by Xabier Larrakoetxea on Medium</title>
            <link>https://medium.com/@slok?source=rss-13dde3ada38b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 04:47:07 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@slok/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Community-based SLIs to boost SLO reliability and adoption]]></title>
            <link>https://itnext.io/community-based-slis-to-boost-slo-reliability-and-adoption-d849c8b921eb?source=rss-13dde3ada38b------2</link>
            <guid isPermaLink="false">https://medium.com/p/d849c8b921eb</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[plugins]]></category>
            <category><![CDATA[service-level-objectives]]></category>
            <category><![CDATA[slo]]></category>
            <category><![CDATA[observability]]></category>
            <dc:creator><![CDATA[Xabier Larrakoetxea]]></dc:creator>
            <pubDate>Mon, 30 Aug 2021 09:08:46 GMT</pubDate>
            <atom:updated>2021-08-30T09:37:08.588Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Iv3saNgv0rdJg-m6lYed_w.png" /><figcaption>Common SLI plugin-based SLOs</figcaption></figure><p>Some months ago, I published an article called <a href="https://itnext.io/slos-should-be-easy-say-hi-to-sloth-9c8a225df0d4"><em>SLOs should be easy, say hi to Sloth</em></a><em>.</em> This article described the release of <a href="https://github.com/slok/sloth">Sloth</a>, an application/process to create SLOs for <a href="https://prometheus.io/">Prometheus</a> in a very easy way (If you haven’t read it already I recommend you read that first).</p><p>Sloth was born because of my concerns around the complexity and error-prone processes that exist when developing/creating SLOs. This involves many parts (Spec UX, CI, deploy…). On this process, Sloth tried solving them, including the one that we are going to talk about today, the source of the SLO, the service level indicator, or <strong>SLI</strong>.</p><p>What’s an SLI? The <a href="https://sre.google/sre-book/table-of-contents/">SRE book</a> describes it like this:</p><blockquote>An SLI is a service level <strong>indicator</strong> — a carefully defined quantitative measure of some aspect of the level of service that is provided.</blockquote><blockquote>Most services consider <strong>request latency</strong> — how long it takes to return a response to a request — as a key SLI. Other common SLIs include the <strong>error rate</strong>, often expressed as a fraction of all requests received, and <strong>system throughput</strong>, typically measured in requests per second. The measurements are often aggregated: i.e., raw data is collected over a measurement window and then turned into a rate, average, or percentile.</blockquote><h3>Then the problems started…</h3><p>When you create an SLO, normally, the friction/problem isn’t always deciding the target/objective, first, you need to know what and how to measure the SLIs:</p><ul><li>What do I need to measure using X technology to have SLOs on this service?</li><li>How can I query the service metrics to obtain a real SLI?</li><li><em>ERR_TOO_MANY_METRICS</em>, what are the good ones for this SLO?</li><li>These Prometheus queries are complex, I don’t fully understand what I’m doing?</li><li>Is this a valid SLI?</li><li>Can this be measured using different time windows?</li><li>…</li></ul><p>It&#39;s very likely that you end asking yourself these kinds of questions while you are creating SLOs.</p><p><strong>This shouldn’t be like that, it should be straightforward and easy.</strong></p><p>We already tackled this problem in other areas.</p><ul><li>I search for an already existing library when I want to develop something.</li><li>I search for a Grafana dashboard when I want to graph/visualize metrics.</li><li>I search for existing applications when I want to do something.</li><li>I search for existing Kubernetes manifests when I want to deploy something in Kubernetes.</li></ul><p>In the end, we want reusability to be productive, learn, avoid errors… So why can’t we have the same for the SLIs? a big community and index of common OSS SLIs?</p><p>Let&#39;s introduce SLI plugins!</p><h3>SLI plugins</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RZOQiMltTQUxlbP7iZfvFg.jpeg" /><figcaption><a href="https://www.pexels.com/es-es/foto/espacio-pared-cable-cargando-4097207/">Plug from Pexels</a></figcaption></figure><p>When creating Sloth some of my SLI related concerns were:</p><ul><li>Don’t write the same 1000 times and be able to reuse already created SLIs.</li><li>Remove query complexity from the spec.</li><li>Decrease errors and increase reliability on SLIs.</li><li>Standardize SLI on OSS technologies (e.g Kubernetes, Traefik…).</li><li>Allow technology/app devs to share SLIs along with the code (e.g: the same as they do with Kubernetes manifests).</li><li>Centralize SLIs on the company and abstract complexity for teams.</li></ul><p>As a solution to these, I thought about how to have easy and pluggable SLIs and I created the concept of SLI plugins inside Sloth.</p><p><strong>An SLI plugin is a simple and small code file written in Go that will receive some options and will return a Prometheus query.</strong></p><p>Very simple and easy. Yet, very powerful.</p><p>Sloth spec supports this as an optional SLI type (along with event and raw types). A person could use these pluggable SLIs instead of setting complex metric queries:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/766df84e924ede063982f406388cddd6/href">https://medium.com/media/766df84e924ede063982f406388cddd6/href</a></iframe><p>As it can be seen in this example we didn’t need to think about queries, we just know that we are using Traefik v1, so we used the <a href="https://github.com/slok/sloth-common-sli-plugins/tree/main/plugins/traefik/v1/availability">Traefik v1 plugin</a> and passed some options.</p><h3>The common plugin index</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*_a8xR8vrdBDIdmSty9T3QQ.png" /></figure><p>Apart from Sloth, There is a repository called <a href="https://github.com/slok/sloth-common-sli-plugins"><em>sloth-common-sli-plugins</em></a> that is growing every day with new plugins. These plugins are ready to be used by anyone using SLOs with sloth.</p><p>Currently, there are multiple plugin types (availability, latency…) for multiple technologies:</p><ul><li><a href="https://kubernetes.io/">Kubernetes</a></li><li><a href="https://coredns.io/">CoreDNS</a></li><li><a href="https://traefik.io/">Traefik</a></li><li>HTTP go libraries</li><li><a href="https://prometheus.io/">Prometheus</a></li><li>…</li></ul><p>Using these plugins is very easy, it’s a 2 step thing:</p><ul><li>Download the plugins (e.g: Clone the repo).</li><li>Run Sloth, pointing to the repo path using <em>“sli-plugins-path”</em> flag.</li></ul><p>Sloth will discover the plugins and load them so they can be used in the specs.</p><pre>git clone https://github.com/slok/sloth-common-sli-plugins.git<br><br>sloth generate --<em>sli-plugins-path</em> ./sloth-common-sli-plugins -i ${MY_SLO_MANIFEST}</pre><h3>Contributing and developing an SLI plugin</h3><p>So, imagine that you like what you have just read and you want to create your own plugins. Even better, you want to contribute with new plugins to the community!</p><p>Let&#39;s deep dive into SLI plugins architecture and creation!</p><h4>Architecture</h4><p>The decision of using Go for the plugins was key to have good shareable plugins. Plugins should be easy to develop, test, and simple by default. However, at the same time, flexible enough so we don’t end generating accidental complexity full of workarounds.</p><p>Go is simple, easy to understand, and has a very powerful standard library to be able to make decisions based on options, mutate strings…</p><p>Sloth’s SLI plugins are loaded using <a href="https://github.com/traefik/yaegi">Yaegi</a> and developed using a set of conventions and rules established by Sloth. Check all of them <a href="https://github.com/slok/sloth#sli-plugins">here</a>. TL;DR:</p><ul><li><strong>Only use the standard library</strong> (no external dependencies).</li><li><strong>One single file per plugin, called “<em>plugin.go”</em></strong><em> </em>(portable).</li><li><strong>No reuse between plugins</strong> (plugin independence).</li><li><strong>Implement a method called “<em>SLIPlugin</em>”</strong> (convention).</li></ul><h4>Example</h4><p>In the end, the plugins are very simple, they must return a Prometheus query with:</p><ul><li>The SLI error ratio that we want to measure, <em>“bad events/valid events”</em></li><li>The ability to measure this in different time windows.</li></ul><p>So, imagine that we have these metrics for our availability plugin:</p><ul><li><em>my_event_errors_total{app=””,kind=””}</em>: Counter with the errors events.</li><li><em>my_events_total{app=””,kind=””}</em>: Counter with all events.</li></ul><p>And our plugin must accept as options:</p><ul><li>“app” (Required): Selects the app to measure its events.</li><li>“kind” (Optional): By default all, but the user can select a specific type of event.</li></ul><p>Let&#39;s see the plugin code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/071fd81419675bdb94e5099e8e4f24ba/href">https://medium.com/media/071fd81419675bdb94e5099e8e4f24ba/href</a></iframe><p>In the above code, we see:</p><ul><li><strong>SLIPluginVersion</strong>: Sloth will use this to know what type of plugin should load.</li><li><strong>SLIPluginID</strong>: Sloth will register this plugin with this ID. This is the ID used by the specs to use this plugin.</li><li><strong>SLIPlugin</strong>: The plugin logic that will be executed.</li></ul><p>The plugin logic receives metadata (SLO name…), labels (User-defined labels on the SLO spec), and plugin options. In our case, we only need options to get the data from the user SLO spec:</p><ul><li>If the user doesn’t have “app”, we return an error.</li><li>If the user has “kind”, we create the filtering label “kind”.</li><li>Return 0 error ratio if there are no metrics available (avoids NaN metrics).</li></ul><p>Then we render a template with these options to generate our query. A nice thing about using Go plugins is that we can have unit tests easily to check that our plugin is returning what we expect:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2b6019bfcb4d5da663712144eb99f12d/href">https://medium.com/media/2b6019bfcb4d5da663712144eb99f12d/href</a></iframe><p>To use it, would be as simple as pointing to the path where the “plugin.go” file is and referencing the plugin in the spec:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/61be880f77816c5e5229729081dea087/href">https://medium.com/media/61be880f77816c5e5229729081dea087/href</a></iframe><p><strong>For more examples, check </strong><a href="https://github.com/slok/sloth-common-sli-plugins/"><strong>sloth-common-sli-plugins</strong></a><strong>.</strong></p><p>To contribute with a new plugin, check <a href="https://github.com/slok/sloth-common-sli-plugins#contributing">contributing</a> section on this same repository.</p><h3>Future</h3><p>The future is to continue adding more and more common SLI plugins to the index, some examples that are planned:</p><ul><li><a href="https://traefik.io/">Traefik V2</a></li><li><a href="https://www.envoyproxy.io/">Envoy</a></li><li><a href="https://www.nginx.com/">Nginx</a></li><li><a href="https://argoproj.github.io/">Argo (cd and workflows)</a></li><li><a href="https://fluxcd.io/">Flux</a></li><li><a href="https://www.mysql.com/">MySQL</a></li><li><a href="https://www.postgresql.org/">Postgres</a></li><li><a href="https://redis.io/">Redis</a></li><li><a href="https://kubernetes.io/">Kubernetes</a></li><li><a href="https://grpc.io/">GRPC</a></li><li><a href="https://etcd.io/">Etcd</a></li></ul><h3>Conclusion</h3><p>At this point I think that we already know what are the benefits of sharing SLI plugins as a community:</p><ul><li>Maintained by people that know and use the metrics/SLIs every day.</li><li>Solving an error on one SLI solves in all the users of that SLI plugin.</li><li>Solves SLI staleness: Updates on new metrics/changes are centralized.</li><li>Optimization on query executions.</li><li>Reliability of metric queries by removing corner cases.</li><li>Discoverability of SLI by users.</li><li>Easy, secure, and fast delivery of SLOs.</li><li>As learning examples to develop personal plugins or SLOs.</li><li>…</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*Ef4Old1tRPNqjgKnFe1vhA.gif" /></figure><p><strong>If you like what you see, be part and help us building this </strong><a href="https://github.com/slok/sloth-common-sli-plugins/"><strong>shared-common SLI plugin index</strong></a><strong>!</strong></p><p>Thanks for reading and happy <em>SLOing</em>!</p><p>Bye!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d849c8b921eb" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/community-based-slis-to-boost-slo-reliability-and-adoption-d849c8b921eb">Community-based SLIs to boost SLO reliability and adoption</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Clean and simple hot-reloading on uninterrupted go applications]]></title>
            <link>https://itnext.io/clean-and-simple-hot-reloading-on-uninterrupted-go-applications-5974230ab4c5?source=rss-13dde3ada38b------2</link>
            <guid isPermaLink="false">https://medium.com/p/5974230ab4c5</guid>
            <category><![CDATA[livereload]]></category>
            <category><![CDATA[configuration]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[hot-reload]]></category>
            <dc:creator><![CDATA[Xabier Larrakoetxea]]></dc:creator>
            <pubDate>Mon, 19 Jul 2021 05:37:52 GMT</pubDate>
            <atom:updated>2021-07-19T11:57:28.156Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UQLTbGRjkOQhHft2UfpINQ.jpeg" /><figcaption><a href="https://www.pexels.com/photo/gray-scale-photo-of-gears-159298/">Gears</a></figcaption></figure><p>In this article, we will learn how to hot-reload any application component without stopping a running Go app, in an easy, well-structured, and clean way, using <a href="https://github.com/slok/reload"><strong>slok/reload</strong></a> library. But let&#39;s start from the beginning…</p><h3>Introduction</h3><p>Often, people confuse the term hot-reload of an app (watch some files, kill the process and run again) with hot-reloading application components, where <strong>the main process isn’t killed at all and we recreate/reconfigure some components of the application on the fly, this, while the app continues running without interruption</strong>.</p><p>This is important for some apps because we need to reduce the interruption time when it is not required, and this can be avoided by using hot/live reload.</p><p>Some of the most used apps need this, e.g:</p><ul><li><a href="https://prometheus.io/">Prometheus</a></li><li><a href="https://www.nginx.com/">Nginx</a></li><li><a href="https://www.postgresql.org/">PostgreSQL</a></li><li><a href="https://www.envoyproxy.io/">Envoy</a></li></ul><p>Normally, this reload process can be triggered by using an HTTP endpoint (e.g: POST request on “<em>/-/reload”</em> path) or using <a href="https://en.wikipedia.org/wiki/SIGHUP">SIGHUP</a> OS signal.</p><h3>oklog/run</h3><p>Some years ago I came across <a href="https://github.com/oklog/run"><strong>oklog/run</strong></a>, a very small go lib, yet very powerful and easy to use.</p><p><em>oklog/run</em> lets you control the different main application execution entry points lifecycles. For example, when you start an application you would:</p><ul><li>Start an HTTP server for a REST API in one goroutine.</li><li>Start an HTTP server for the metrics in another goroutine.</li><li>Start a long-running goroutine that passes some checks at regular intervals.</li><li>A channel that listens to OS signals, and when received it stops the application.</li><li>…</li></ul><p>You would use oklog/<em>run </em>to define these execution entry points with a run and a stop function. Oklog/r<em>un, </em>will manage the start and graceful shutdown (on all of the components) when one of the components ends with or without error. e.g:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c42ecdc81a1e47a5e35375ab7040e5c3/href">https://medium.com/media/c42ecdc81a1e47a5e35375ab7040e5c3/href</a></iframe><p>With this, you gain a clean and reliable start/stop of the application because all your entry points will have a clean way of declaring what to do at the start (e.g run a server) and what to do at the stop (e.g close server and drain connections). Plus, the app forces you to manage every running entrypoint in a single way and in one place. So simple and clean… awesome.</p><h3>slok/reload</h3><p>Let&#39;s deep dive into the hot-reload process… that&#39;s why we are here right?</p><p>Sometimes when we create an application, some of the components (services, functions…) need to hot-reload at any given point, this reload logic normally is inside the component itself, however, things aren’t so simple: multiple components need to be reloaded at the same time, or we need a reload order, also, there are reloading errors, reloads are triggered from the same sources, or maybe from different sources…</p><p>In a few words, is not an easy thing and starts getting messy easily. <strong>We need a clean and well-structured way of declaring these hot-reload actions.</strong></p><p>Inspired by <a href="https://github.com/oklog/run">oklog/run</a> and its simplicity, I started thinking of how to manage a clean reload process in a similar way…</p><p><strong>Say hi to </strong><a href="https://github.com/slok/reload"><strong>slok/reload</strong></a><strong>!</strong></p><p>I had some more requirements apart from the ones that <em>oklog/run</em> already has and solves:</p><ul><li>A single entry point (e.g: <em>func main()</em>) that manages all the hot-reload processes.</li><li>Easy to use and configure with a small API and library.</li><li>Manage app component reloads in a single way.</li><li>Multiple reload components triggered by multiple triggers.</li><li>Be able to reload different components in order.</li><li>Reliable reload when multiple reload triggers happen at the same time.</li><li>Flexible enough for any kind of reload process.</li></ul><h4>API</h4><p>Let&#39;s start by showing the library&#39;s small API.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/1*LQaudDbhS2dAUyecnvBXmg.png" /><figcaption><a href="https://github.com/slok/reload">slok/reload</a> API</figcaption></figure><p>The API is based on 3 main components:</p><ul><li><a href="https://pkg.go.dev/github.com/slok/reload#Notifier">Notifier</a>: Triggers the reload process (e.g: HTTP handler, OS signal…) when it ends the execution.</li><li><a href="https://pkg.go.dev/github.com/slok/reload#Reloader">Reloader</a>: Executes the component reload process (e.g: Config loader, client factory…).</li><li><a href="https://pkg.go.dev/github.com/slok/reload#Manager">Manager</a>: Manages the reload lifecycle using the notifiers and the reloaders. It will run forever until a notifier or reloader returns an error or the context ends.</li></ul><p>The API comes with some handy helpers so the users don’t need to declare new types to create Notifiers and Reloaders (e.g: Create a notifier from a function or a channel).</p><h4>How does it work</h4><p>In the following diagram, we see what is running inside the <a href="https://pkg.go.dev/github.com/slok/reload#Manager">Manager</a>.</p><p>We have 5 notifiers that have been executed and are waiting to end at some time to trigger the reload process.</p><p>We have 5 reloaders (divided into 3 groups) waiting to be executed when the notifiers trigger the reload process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/579/1*QToyCvooXM5xnUjWcmyHTw.png" /><figcaption><a href="https://github.com/slok/reload">slok/reload</a> high-level architecture</figcaption></figure><ul><li>1: One of the notifiers will end its execution and will trigger the reload process.</li><li>2: The first reloader group with priority 10 will execute the single reloader that it has.</li><li>3: The second reloader with priority 99 will execute the two reloaders concurrently.</li><li>4: The third reload group with priority 500 will execute the two reloaders concurrently and everything will end.</li></ul><p>The manager will ensure that while a reloader group is being executed, no other notifiers can trigger that reload group reloading process.</p><p>When one of the notifier&#39;s execution ends (to trigger the reload process), it immediately will start again so that notifier is ready for a new reloading trigger.</p><h3>Practical example</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6-eUYK_FZT-_7IybFZYExA.jpeg" /><figcaption><a href="https://www.pexels.com/photo/lighthouse-snail-staircase-53554/">Lighthouse snail stair</a></figcaption></figure><p>We are going to create a simple application to show how we can set a reliable, simple, and clean application startup and teardown having hot/live-reload features, by using:</p><ul><li><strong>oklog/run</strong>: To manage the running entry points lifecycle.</li><li><strong>slok/reload</strong>: To manage the hot-reload lifecycle.</li></ul><p><strong>TL;DR: </strong><a href="https://github.com/slok/reload/tree/main/_examples/simpleapp"><strong>Full example here</strong></a></p><h4>Requirements</h4><p>We have an app that loads a JSON configuration file. This configuration will be used to create two domain services:</p><ul><li>Printer: Prints a message with what we have in the config.</li><li>Curler: Makes a request to an URL with what we have in the config.</li></ul><p>The app should be able to hot-reload the configuration and afterward these two services.</p><p>The hot-reload should be triggered, in multiple ways:</p><ul><li>SIGHUP (on demand).</li><li>HTTP API handler (on demand).</li><li>File system file change (on file change event).</li></ul><p>When the app starts, it should run 4 entry points:</p><ul><li>An OS signal handler: Handlers stop signals and hot-reload trigger.</li><li>An HTTP server: Handles domain services execution and also hot-reload trigger.</li><li>A file watcher: Triggers hot-reload on file changes.</li><li>A hot-reload manager: Handles all the hot-reload process execution.</li></ul><h4>Configuration loader</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4c0056070183673e8da0cc2153a6f9a4/href">https://medium.com/media/4c0056070183673e8da0cc2153a6f9a4/href</a></iframe><p>In the above code, we see the configuration loader. This is the service that knows how to load the configuration from a file, and any component of the code can use it to get the configuration using a public method.</p><ul><li>Uses an RW mutex to lock the read/writes configuration value inside the loader.</li><li>Has a <em>Reload</em> method that re-reads the config file and loads it.</li><li>Has a <em>Get</em> method that returns the loaded configuration at that moment.</li></ul><h4>Domain services</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/668f39852dd2832755e58f9d61c6f452/href">https://medium.com/media/668f39852dd2832755e58f9d61c6f452/href</a></iframe><p>In the above code, we can see the <em>Printer</em> service, the most important parts are these:</p><ul><li>Use an RW mutex to lock the read/writes value access, that we will be used to reload and print.</li><li>Has a <em>Reload </em>public method that can reload the configuration (using the configuration loader) inside the service and sets the required service configuration message so the service can print it.</li></ul><p>For the <em>Curler</em> service, we have a similar thing.</p><h4>Reloaders</h4><p>Let&#39;s discuss the reloaders: <em>Config loader, Curler, Printer</em>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/24d46be1695f590da90365c1668744be/href">https://medium.com/media/24d46be1695f590da90365c1668744be/href</a></iframe><p>We just started analyzing the application main code, in the first part of the above code, the domain services are created. Then we create the reload manager.</p><p>The interesting part happens when we start using <em>slok/reload</em>. We are declaring 3 reload actions in a specific order in a very clean and easy way:</p><ul><li>First, 0 priority reloaders, the configuration service (we warn on a config loading failure, but no error).</li><li>Second, 100 priority reloaders, reload the curler and printer service (when this reloads the config loader has already been reloaded, so, they will pick the new configuration).</li></ul><h4>Signal reload Notifier</h4><p>Finally the last part, the notifiers. Let’s start with the OS signal notifier.</p><p>We want to listen to OS signals. And when there is a SIGHUP, we want to trigger the reload process through the <em>slok/reload.Manager</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/510ce4e5dd12bf34baa0f64c41ca0521/href">https://medium.com/media/510ce4e5dd12bf34baa0f64c41ca0521/href</a></iframe><p>In the above code, we see that we create multiple channels:</p><ul><li>sigC: To receive the OS signals.</li><li>exitC: To stop explicitly the listening of the OS signals.</li><li>reloadC: To trigger the reload process (This channel is used as a notifier using <a href="https://pkg.go.dev/github.com/slok/reload?utm_source=godoc#NotifierChan"><em>reload.NotifierChan</em></a><em> </em>helper).</li></ul><p>So, the logic mainly is, wait on <em>sigC </em>for the OS signals <em>forever</em>, when we receive a signal of the kind SIGHUP, use the <em>reloadC</em> to trigger the reloading process. Anything else, just exit.</p><h4>HTTP reload Notifier</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/710193b16263413a26a5c3b6c209148d/href">https://medium.com/media/710193b16263413a26a5c3b6c209148d/href</a></iframe><p>In the above code, we declare the notifier for the HTTP service, as with the SIGHUP notifier, we do this using the <a href="https://pkg.go.dev/github.com/slok/reload?utm_source=godoc#NotifierChan"><em>NotifierChan</em></a> helper that converts a <em>chan string </em>into a Notifier.</p><p>The logic is small, we register our hot-reload HTTP handler, when the HTTP server receives a request on “<em>/-/reload</em>” path, we will trigger the reload process using this <em>chan string</em> channel.</p><h4>Config file watcher Notifier</h4><p>Lastly, we have the notifier that watches the configuration file for changes and triggers a reload</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/71f134168b0bf826099a69492822686d/href">https://medium.com/media/71f134168b0bf826099a69492822686d/href</a></iframe><p>To watch the file we are using <a href="https://github.com/fsnotify/fsnotify">fsnotify</a> library. We create a Watcher that automatically starts watching all the specified files, then we add the path to the config file to be watched.</p><p>The watcher has a channel for the events called <em>Events. </em>We will use this to get event changes on files and trigger the reload.</p><p>Unlike the other two notifiers, this one doesn’t use the <em>NotifierChan</em> helper and uses the <a href="https://pkg.go.dev/github.com/slok/reload?utm_source=godoc#NotifierFunc"><em>NotifierFunc</em></a> helper. This helper transforms a function into a notifier. This function will be executed by the reload Manager and when it ends the execution it will trigger the reload process.</p><p>In this case, this function will wait by listening to the events on the File watcher channels (event and error), and depending on what it receives it will trigger a reload, or tell the reload manager that there was an error.</p><h3>Conclusion</h3><p>Controlling and orchestrating a reliable app startup is not an easy thing. If we add the hot-reload requirement, things start getting very messy and complex. However, <strong>we have seen how to design and implement a simple, well-structured, and clean, run/hot-reload process for any kind of Go application.</strong></p><p>Also, <a href="https://github.com/slok/reload">slok/reload</a> library <em>forces </em>you somehow to set all the reloading components in a single way and mostly in the same place (app’s main entry points), making the code clear for you and the ones that will come in the future.</p><p>As a bonus, we’ve seen <a href="https://github.com/oklog/run">oklog/run</a>, an amazing and simple piece of software to define your application start (that plays very nicely with <a href="https://github.com/slok/reload">slok/reload</a>).</p><p>Thanks for reading and if you have comments, suggestions, issues, or opinions, please left a comment or open an issue <a href="https://github.com/slok/reload/issues">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/220/1*8DYd6Hlf4wXOPcNEtPf2Rw.gif" /><figcaption>Bye!</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5974230ab4c5" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/clean-and-simple-hot-reloading-on-uninterrupted-go-applications-5974230ab4c5">Clean and simple hot-reloading on uninterrupted go applications</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SLOs should be easy, say hi to Sloth]]></title>
            <link>https://itnext.io/slos-should-be-easy-say-hi-to-sloth-9c8a225df0d4?source=rss-13dde3ada38b------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c8a225df0d4</guid>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[service-level-objectives]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[slo]]></category>
            <category><![CDATA[prometheus]]></category>
            <dc:creator><![CDATA[Xabier Larrakoetxea]]></dc:creator>
            <pubDate>Sun, 06 Jun 2021 10:16:27 GMT</pubDate>
            <atom:updated>2021-06-30T21:16:05.696Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4tYtJVfDD0DpK-jPqksHtg.png" /><figcaption>Kubernetes apiserver latency SLO with Sloth</figcaption></figure><p>As in other areas, in the technology world, every year there are some buzz words that are being said more than others. Some examples:</p><ul><li>2017: Cryptocurrency and blockchain</li><li>2018: Observability and tracing</li><li>2019: Service mesh</li><li>2020: Gitops</li></ul><p>And this year, the fancy word is SLO.</p><p>I’m sure you have been hearing <em>service level objectives</em> lately (not only because of the <a href="http://sloconf.com/">SLOConf</a>). SLOs aren’t new, and most likely that if you are reading this, you are already familiar with SLI, SLO, SLA, error budgets… lingo.</p><p>Apart from the buzzword, SLOs are awesome, and if you are not using them at this moment, explore them if you can, because I’m sure that you will get value from them.</p><p><em>Now I’ll talk about how I end up implementing </em><a href="https://github.com/slok/sloth"><em>Sloth</em></a><em>, so you can skip directly to </em><a href="https://slok.medium.com/slos-should-be-easy-say-hi-to-sloth-9c8a225df0d4#4b95"><em>Sloth section</em></a><em> if you want.</em></p><h3>A long long time ago…</h3><p>I don’t know when exactly SLOs were created, however, I was introduced to these when I started being fascinated by the SRE culture, I read them for the first time in <a href="https://sre.google/sre-book/service-level-objectives/">chapter 4 of the SRE book</a>, sometime later continued on the <a href="https://sre.google/workbook/part-I-foundations/">SRE workbook</a>.</p><p>For me, SLOs are not (only) a tool/framework, but, a way of thinking and applying observability (<em>Ding! 2018</em>), a language… in a word, <em>culture</em>. However, every time that I tried explaining to friends, family, coworkers… in multiple companies, happens the same:</p><blockquote>Everyone understands SLOs, the terms, purpose… but lots of them struggle to apply them, understanding the process, or putting them in a practical way.</blockquote><h3>(2018–2019) Service level operator</h3><p>It was 2018, I was working in <a href="https://www.spotahome.com/es">Spotahome</a>, I was a platform <em>engineer/SRE/someone that writes code and does ops. W</em>e wanted to start applying SLOs in the company, there wasn’t too much out there and we wanted to make it easy, so people didn’t need to think in all of the existing SLO terms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5BUo9SV9idWAKP85w8L3Eg.png" /></figure><p>We created a Kubernetes operator that had the SLOs of a service defined as a <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/">CRD</a>. We named it <a href="https://github.com/spotahome/service-level-operator"><em>Service Level Operator</em></a><em>. </em>The CRD was simple: give me a query for total events (e.g requests), give me a query for the error events (e.g requests 5xx), and the Operator will get a ratio (0–1) calculated at regular intervals and give it to a Prometheus using some metric conventions.</p><p>The operator worked well, we had very simple dashboards that gave us insights quickly, however, weren’t period window metrics, alerts, SLO metadata information… we lack a lot of things about SLOs, but as a first step was very nice.</p><h3>(2019–2020) Asadito</h3><p>At the end of 2019, I moved to a new company called <a href="https://cabify.com/">Cabify</a>. Some teams in the company were already applying SLOs and implementing them. However, there were a lot of error-prone manual tasks when creating these SLO Prometheus rules. Even, we started creating Prometheus unit tests to be sure about these manually created Prometheus rules! <strong>it was pure </strong><a href="https://sre.google/sre-book/eliminating-toil/"><strong>toil</strong></a><strong>.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/1*kf77EiYL2T9t4Z_BpCgK9Q.png" /></figure><p>I thought about that, and how we could do something to improve the process, and started a new project inside the company. <em>Asadito,</em> a CLI that generates all the Prometheus rules required to create the SLI recording rules and SLO-based alerts (<a href="https://sre.google/workbook/alerting-on-slos/">multi-window multi-burn</a>). The main idea was simplicity and flexibility to improve SLO adoption and that moment SLOs usage.</p><p>In other words, I took some of the experience I had creating the service level operator and evolve it to adapt to this company’s needs.</p><p>Multiple teams started using this tool and suddenly we had standardized SLOs around the company (only the teams that wanted to use it), you could discover all different teams’ SLOs, we had a generic SLO Grafana dashboard, a repository with easy to understand SLO specs…</p><p>A very good evolution of what I did back in 2018 with the service level operator, and worked successfully.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/656/1*mUPRuhfQ07OLkTX01ozbeg.jpeg" /><figcaption>Success!</figcaption></figure><h3>(2021) Sloth</h3><p>I’m no longer in any of those companies. <strong>It&#39;s 2021 and I continue thinking that people have difficulties implementing SLOs, and shouldn’t be like this.</strong> Also, I think that I contributed in a positive way in this area in these past companies so…</p><p>After the experience of the <em>service level operator</em> and <em>Asadito</em>, I wanted something similar and to be available for everyone, and like everything that I develop in my free time, OSS.</p><p><strong>Say hi to </strong><a href="https://github.com/slok/sloth"><strong><em>Sloth</em></strong></a><strong>!</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/267/1*fzNfHeZalcxwCCpJCJU71w.png" /></figure><blockquote>Sloth generates SLOs easily for Prometheus based on a spec/manifest that scales. Is easy to understand and maintain.</blockquote><p>Is it a CLI? is it a Kubernetes controller/operator?. It’s both, It adapts to you so you can choose how you want to use it.</p><p>Sloth spec tries removing all the difficult configuration parts that make the people struggle when applying SLOs in a practical way: strange error budget formulas, different time windows, different forms of visualization graphs…</p><p>Sloth focuses on simplicity, has safe defaults and the purpose is to be able to work for 90–95% of the people (that’s marketing!). I’m sorry if you are in the 5–10%, anyway, is most likely that you have already created your own tool or custom SLO rules.</p><p>Let’s see an SLO example and what would Sloth offer you.</p><h4>Spec</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2ed8a856442b9648fa79eda5ceff06d8/href">https://medium.com/media/2ed8a856442b9648fa79eda5ceff06d8/href</a></iframe><p>As it can be seen on the spec, we define multiple SLOs for a service, the idea behind this is that every service should have its own clean and simple SLO manifest. this has 2 purposes:</p><ul><li>Having SLOs per service.</li><li>Documented service SLOs.</li></ul><p>Each SLO has an ID, the objective/target for the SLO (e.g <em>99.9%</em>), an SLI, in this example is an SLI <em>events</em> type (there is also <em>raw </em>SLI type), and finally, the optional alerting block to enable/disable the automatic <a href="https://sre.google/workbook/alerting-on-slos/">multiwindow-multiburn</a> alerts generation.</p><p>The SLI Prometheus queries, require a templated variable in the query, <em>{{.window}}, </em>SLOth will expand this on the generated rules to the required SLOs windows (e.g 5m, 1h, 3d…).</p><h4>Validation</h4><p>When Sloth generates the SLO rules, it validates the spec, including SLI queries, options… so the feedback loop of an incorrect SLO is the fastest as possible and you don’t end with a wrong SLO or spending 3h creating a simple SLO.</p><h4>Usage (CLI)</h4><p>If you use it as a regular CLI, for examples for <em>gitops</em> (<em>Ding! 2020</em>):</p><pre>sloth generate -i ./myservice-slo.yml</pre><h4>Usage (Kubernetes controller/operator)</h4><p>If Sloth is running as a controller, you could submit a similar manifest (it’s a <a href="https://github.com/slok/sloth/tree/main/pkg/kubernetes/api/sloth/v1">CRD</a>) automatically with <em>Kubectl</em>, Sloth controller will generate the rules, and store them as Prometheus rules for <a href="https://prometheus-operator.dev/">prometheus-operator</a>.</p><p>You can run the controller with:</p><pre>sloth kubernetes-controller</pre><p>Get The SLOs created in the k8s cluster and its state:</p><pre>kubectl get slos --all-namespaces</pre><h4>Prometheus rules</h4><p>The Prometheus rules that Sloth generates are standardized. This will make that the discoverability of all SLOs easy, don’t lack information, and have a uniform SLO system in place.</p><p>The rules can be categorized into 3 areas:</p><ul><li>SLI recording rules.</li><li>SLO metadata recording rules.</li><li>Multiwindow-multiburn alert rules.</li></ul><p>You can get all the metrics that Sloth has generated using this Prometheus query:</p><pre>count({sloth_id!=””}) by (__name__)</pre><h4>Dashboard</h4><p>As was said before, all the SLOs share the same uniform implementation, so we can create a generic <a href="https://grafana.com/grafana/dashboards/14348">Grafana dashboard</a> to show all SLOs status:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B4TtLfijv0b4s00xxxSnDg.png" /><figcaption>My home WiFi experience SLO on the Sloth dashboard.</figcaption></figure><p>As you already know, Sloth removes SLOs’ configuration tricky parts and assumes safe defaults, like month (30d) time windows. This way the dashboard comes with an error budget burndown chart to see how is the monthly budget doing and make decisions based on it (I’m thinking setting also a 7d/week. Use the comments or GH issues if you think this would be interesting).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2ohX1yqIOfbQKDVka2-buw.png" /><figcaption>My home wifi experience SLO error budget month burn down chart.</figcaption></figure><h4>Alerts</h4><p>We talked about <a href="https://sre.google/workbook/alerting-on-slos/">multiwindow-mutiburn</a> alerts, Sloth implements the standard that Google describes, they are good enough for most cases as it tracks slow and fast error budget-burning based alerts. Sloth will create two types of alerts (each of these can be enabled or disabled):</p><ul><li><strong>Page (critical) alerts</strong>: Pay attention right now</li><li><strong>Ticket (warning) alerts</strong>: Something is not correct, don’t need to worry now, but take an eye when you have some time.</li></ul><p>You don’t need to think about how to set alerts, as with other things, Sloth will set safe default time windows to alert in these different cases.</p><h3>Future</h3><p>And what’s the future of Sloth?</p><h4>OpenSLO</h4><p>Lately, <a href="http://openslo.com/">OpenSLO</a> has been published as an SLO standardized spec. It’s funny because it has been created at the same time that Sloth was being developed, we didn’t know each other.</p><p>At this moment, I’m trying to adapt Sloth spec to an OpenSLO spec. I don’t think that will be 100% compatible (e.g OpenSLO doesn’t have anything about alerts), however, if we end supporting OpenSLO on Sloth, Sloths’ specs will be always maintained and be a first-class citizen due to its simplicity and full-featured spec.</p><p><strong>Update (2021/06/30): OpenSLO support available</strong></p><h3>Xabier Larrakoetxea on Twitter: &quot;🦥Sloth v0.5.0 has been released!This release comes with a very interesting thing... 🥁@openslo_ support! 🎉Also Hot-reload of 🔌 file plugins on #Kubernetes controller/operatorhttps://t.co/aLUQI0C8jx#Observability #slo #openslo pic.twitter.com/F2jn2RaPgA / Twitter&quot;</h3><p>🦥Sloth v0.5.0 has been released!This release comes with a very interesting thing... 🥁@openslo_ support! 🎉Also Hot-reload of 🔌 file plugins on #Kubernetes controller/operatorhttps://t.co/aLUQI0C8jx#Observability #slo #openslo pic.twitter.com/F2jn2RaPgA</p><h4>SLI plugins</h4><p>I’m very excited about this feature, soon will be available, stay tuned in the following week…</p><p><strong>Update(2021/06/10): SLI plugins available with the </strong><a href="https://github.com/slok/sloth/releases/latest"><strong>latest Sloth release</strong></a></p><h3>Xabier Larrakoetxea on Twitter: &quot;New 🦥 Sloth release! I&#39;m very excited about this new feature:🔌SLI plugins!SLI plugins are external Go files that users can load on Sloth start and reference them on the #SLO to get SLIs from these!Let&#39;s explore this new feature! 👇🧵#observability #kubernetes / Twitter&quot;</h3><p>New 🦥 Sloth release! I&#39;m very excited about this new feature:🔌SLI plugins!SLI plugins are external Go files that users can load on Sloth start and reference them on the #SLO to get SLIs from these!Let&#39;s explore this new feature! 👇🧵#observability #kubernetes</p><p>These are simple Go files that anyone can develop outside Sloth. Sloth will load on start (passing some flag to read the from the FS). From that moment any SLO can reference these SLI plugins on the SLI spec part instead of writing a Prometheus query. You can imagine the power of this extensibility, some examples:</p><ul><li>Share common SLIs on a repository at the company level, this could be maintained by a team or a group of experienced people.</li><li>Community-based SLI repositories (common ones, frameworks, libs, apps).</li><li>Complex Prometheus SLI queries made them easy.</li><li>Avoid query repetition on multiple services of a team or company…</li><li>Use the SLI plugins to learn and as examples to create your own.</li><li>Validation and safety of queries on the SLIs.</li></ul><p>Sloth will have a common-SLI repository that people can contribute to, and also use to set up SLOs even easier than before.</p><h3>Conclusion</h3><p>I covered very quickly what Sloth can do for you, on <a href="https://github.com/slok/sloth">Sloth documentation</a> you can read more in-depth all the features.</p><p>In a few words, I like to say that Sloth:</p><blockquote>Makes applying SLOs simple and easy.</blockquote><p>If you can, give it a try with some services, you have <a href="https://github.com/slok/sloth/releases/latest">prebuilt binaries</a>, just download, create a simple spec and generate your Prometheus rules, in less than 5m you can have SLOs for this service up and running, :)</p><p>If you can give me feedback, opinions, or if are already using it for you or your company… I would appreciate it a lot! this kind of feedback motivates me a lot. You can use the comments below or GH issues :)</p><p>Thanks for reading and happy <em>SLOing</em>!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/356/1*s8Qt2UzpkKAXAA4ojGB7qw.gif" /><figcaption>Bye!</figcaption></figure><p>Oh, I forgot…</p><p>Blockchain! <em>(Ding! 2017)</em>, Service mesh!<em> (Ding! 2019).</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c8a225df0d4" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/slos-should-be-easy-say-hi-to-sloth-9c8a225df0d4">SLOs should be easy, say hi to Sloth</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Gontroller: a Go library to create reliable feedback loop controllers]]></title>
            <link>https://medium.com/spotahome-product/gontroller-a-go-library-to-create-reliable-feedback-loop-controllers-832d4a9522ea?source=rss-13dde3ada38b------2</link>
            <guid isPermaLink="false">https://medium.com/p/832d4a9522ea</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[controller]]></category>
            <category><![CDATA[feedback]]></category>
            <category><![CDATA[control-theory]]></category>
            <category><![CDATA[golang]]></category>
            <dc:creator><![CDATA[Xabier Larrakoetxea]]></dc:creator>
            <pubDate>Fri, 06 Sep 2019 09:18:09 GMT</pubDate>
            <atom:updated>2019-09-06T09:51:25.363Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/357/1*lk1I0eKq6J4XLKxMbzqvwg.png" /></figure><p>Say hello to <a href="https://github.com/spotahome/gontroller">Gontroller</a>, a Go library to create controllers (<strong>without the need of Kubernetes resources</strong>).</p><p>If you are reading this and you’ve been aware of the <a href="https://kubernetes.io/es/">Kubernetes</a> and <a href="https://www.cncf.io/">CNCF</a> ecosystem you may be wondering, <em>another library to create Kubernetes controllers?!</em></p><p>Not at all… Don’t worry, you can keep reading independently if you are familiar with Kubernetes ecosystem or not.</p><p><a href="https://github.com/spotahome/gontroller">Gontroller</a> has nothing to do with Kubernetes, well that’s not 100% true… it implements the <strong>feedback loop/control controller</strong> pattern used by Kubernetes controllers/operators in a similar way.</p><p>Before explaining what does this mean, what <a href="https://github.com/spotahome/gontroller">Gontroller</a> can solve and explaining how it can be used, let’s start explaining a bit of <a href="https://en.wikipedia.org/wiki/Control_theory">control-theory</a> and feedback loops.</p><h3>Control theory</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c4GS98wio630SbM8lQLuYA.jpeg" /><figcaption><a href="https://pixabay.com/photos/feedback-review-gut-bad-3677258/">https://pixabay.com/photos/feedback-review-gut-bad-3677258/</a></figcaption></figure><p>So what is this control theory thing, from <a href="http://shop.oreilly.com/product/0636920028970.do"><em>Feedback Control for Computer Systems</em></a> book, Feedback means:</p><blockquote>Feedback works by constantly comparing the actual behavior of a sys‐<br>tem to its desired behavior. If the actual behavior differs from the de‐<br>sired one, a corrective action is applied to counteract the deviation and<br>drive the system back to its target. This process is repeated constantly,<br>as long as the system is running.</blockquote><p>We could say that a controller is an application that tries to meet the desired state of a resource applying actions (<em>forever</em>).</p><p>For example:</p><ul><li>A thermostat: sets the heater temperature of a room based on the room’s temperature.</li><li>An autoscaler: runs and deletes instances based on the traffic.</li><li>A cache system: increases and decreases the size based on the hits and the system is running on.</li><li>An application concurrency system: sets the available concurrent limits based on the traffic.</li><li>An application that maintains the TLS certificates up to date: renews them before they expire based on their expiration date.</li><li>A Loadbalancer: maintains a healthy set of endpoints for services based on health checks.</li><li>Distributed file storage: Replicates the data in different regions based on the state of data (e.g: S3).</li></ul><p>These systems can be of different types and combinations (open and closed feedback loops, linear, <a href="https://en.wikipedia.org/wiki/PID_controller">proportional, integral, derivative</a>…).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*2Pyk2nCbtapFrnqpApl_CA.png" /><figcaption>Closed-loop controller diagram (<a href="https://en.wikipedia.org/wiki/Control_theory">Wikipedia</a>)</figcaption></figure><p>If you are using Kubernetes, is likely you are using this kind of systems every day, they are also called controllers or operators. For example:</p><ul><li><a href="https://github.com/kubernetes/kubernetes/tree/master/pkg/controller/podautoscaler">Horizontal pod autoscaller</a>: Scales up and down the number of replicas (pods) based on the CPU usage.</li><li><a href="https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/replicaset">Replicaset controller</a>: Maintains the desired number of replicas of a service using pods (if you want 3 pods, it will ensure 3 pods exist all the time).</li><li><a href="https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/">Ingress controller</a>: Ensures the desired services have access based on hosts and are being routed correctly to the desired services. Depending on the controller could be an ELB, ALB, GCLB, Nginx, HAProxy, Envoy…</li></ul><p>As you see, this pattern is applied to lots of things and we use it every day.</p><h3>Creating controllers</h3><p>Depending on the context, a <strong>reliable</strong> controller needs to meet a number of basic features, sometimes all of them, others, only some:</p><h4>Consistent</h4><p>A controller needs to maintain the desired state all the time, that’s why a controller needs to be in a constant loop checking and trying to meet the desired state of a resource by taking actions.</p><h4>Real-time</h4><p>Some controllers need to maintain the desired state as fast as possible, in this case apart from the constant loop they need to take actions on real-time events or changes.</p><h4>Level triggering</h4><p><a href="https://hackernoon.com/level-triggering-and-reconciliation-in-kubernetes-1f17fe30333d">Explained here.</a></p><h4>Scalable</h4><p>A controller would scale overtime having to process more and more resources, that’s why it will need to act on multiple different resources at the same time, this means that concurrency is required, for example:</p><ul><li>Multiple AWS autoscaling groups (hundreds of AWS autoscaling groups from different clusters).</li><li>Multiple TLS certificates (e.g. thousands of automatic hosts creation have their own certificates).</li><li>Multiple groups of instances (e.g. Kubernetes deployments).</li></ul><h4>Safe</h4><p>Errors exist and will happen, delays exist, scale and concurrency increase the problems… that’s why a safe controller should:</p><ul><li>Process the same resource only once at the same time (although there are different resources being processed at the same time).</li><li>Handle the resource based on the latest state of the resource (not a stale state, delays exist…).</li><li>Retry in case of error.</li></ul><p><a href="https://github.com/spotahome/gontroller">Gontroller</a> should have all these features and at the same time be flexible to be able to implement all these different controller kinds.</p><h3>Gontroller</h3><p>At <a href="https://www.spotahome.com/">Spotahome</a> we use and develop a lot of Kubernetes controllers for multiple things, they are awesome and maintain the system in a correct state, this gives us reliability and resilience.</p><h4>Motivation</h4><p>The main problem is that we wanted to create controllers for another kind of resources that weren’t based on Kubernetes (e.g. Github repositories), but we love how the Kubernetes controllers have been designed, so we developed <a href="https://github.com/spotahome/gontroller">Gontroller</a> with the same idea except some small changes to make it more flexible and easier to use.</p><h4>Features</h4><ul><li>Easy and fast usage.</li><li>Automatic retries on error.</li><li>Ensure only one worker is handling the same resource (based on ID) at the same time.</li><li>Concurrent handling of different resources.</li><li>A resource will be only processed once if it’s not been handled yet (although it’s been received multiple times before it has been handled).</li><li>Handle all resources at regular intervals (reconciliation loop) and updated objects in real-time.</li><li>Observability as a first-class citizen: Metrics with <a href="https://prometheus.io/">Prometheus</a> implementation.</li><li>Extensible, all is based on behavior (go interfaces) and not concrete types.</li><li>Easy to test, business logic not coupled with infrastructure code (controller implementation/library).</li><li><em>If you are used to Kubernetes: No Kubernetes dependencies on the code.</em></li></ul><h4>How does it work</h4><p>To implement a controller using <a href="https://github.com/spotahome/gontroller">Gontroller</a> the user needs to implement 3 Go interfaces.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*hIjzIR5YvkN0YGmTwTDnFg.png" /><figcaption>High-level architecture of <a href="https://github.com/spotahome/gontroller">Gontroller</a></figcaption></figure><h4>ListerWatcher</h4><p>This is the first component, is responsible for knowing what resources will handle the controller, this is done by retrieving the IDs of the resources the controller will handle in the future. e.g:</p><ul><li>TLS certificates.</li><li>AWS Autoscaling groups.</li><li>Github repositories.</li><li>Users.</li><li>Services.</li></ul><p>This component is composed of 2 methods that need to be implemented:</p><pre>type ListerWatcher interface {<br>    List(ctx context.Context) (ids []string, err error)<br>    Watch(ctx context.Context) (&lt;-chan Event, error)<br>}</pre><ul><li><strong>List</strong>: This will list all the resource ids, this list will be used to process each of the resources at regular intervals, also called reconciliation loop (e.g. I want to check/handle all TLS certificate expiration every 30m).</li><li><strong>Watch</strong>: This will watch events on the resources (delete, edit, create), unlike the List that will be used on a constant loop, these events will be processed in real-time (e.g. I don’t want to wait for the next reconciliation loop for this new created TLS certificate).</li></ul><h4>Storage</h4><pre>type Storage interface {<br>    Get(ctx context.Context, id <em>string</em>) (interface{}, <em>error</em>)<br>}</pre><p>The storage component is the component that transforms the ListerWatcher obtained ids into the resource state/data. e.g:</p><ul><li>The Autoscaling group current instances and the cluster free size.</li><li>The number of replicas of a service.</li><li>The metadata and files of a Git repository.</li><li>The location of a package on the truck and road.</li><li>The expiration date of a TLS certificate.</li></ul><p>Once the storage gets the resource state based on the id, the controller will call the handler with this data.</p><h4>Handler</h4><p>The handler is where most of the controller logic lives, is the one that takes the actions to meet the expected state of the resource. The Handler needs to implement 2 methods.</p><p>When handling a resource, the controller will call one of these two methods:</p><pre>type Handler interface {<br>    Add(ctx context.Context, obj interface{}) error<br>    Delete(ctx context.Context, id string) error<br>}</pre><ul><li><strong>Add</strong>: This method will be called when the object exists (Events of add or edit), here we will take actions based on the received data (previously obtained by the Storage)</li><li><strong>Delete</strong>: This method will be called when the object does not exist (received a delete event by a Watch), this method only receives the id and is mostly used for cleanup actions.</li></ul><p>For a full example of how you can implement a controller using <a href="https://github.com/spotahome/gontroller">Gontroller</a>, check the <a href="https://github.com/spotahome/gontroller/tree/master/examples">examples</a>.</p><p>If you want to know the internal implementation, you can <a href="https://github.com/spotahome/gontroller#internal-architecture">check the repository</a>.</p><h3>Use case examples</h3><h4>Project description in a repository file on Github</h4><p>What we want is to have a file (e.g. spotahome.yaml) that describes the project information in a repository (name, team owner, cluster…). This file should be downloaded and parsed so everything about the project can be initialized and registered.</p><ul><li><strong>ListerWatcher(List)</strong>: Will list all the Github repositories from an organization and return the name as the unique id (e.g. spotahome/gontroller)</li><li><strong>ListerWatcher(Watch)</strong>: Will subscribe to Github webhooks and when there is a push event, it will check that is in themaster branch and there is a modification in spotahome.yaml file.</li><li><strong>Storage</strong>: Will get the spotahome.yaml data file from the repo and get the current project initialization state in the cluster, teams… This will be returned in a state object.</li><li><strong>Handler</strong>: Will use the received state object with the expected data and the current state, and create, modify or ignore depending on what we want and what is present.</li></ul><p>We described one approach but it could be designed in another way, for examples, the storage only return the spotahome.yaml data and let the handler get the current state, compare and take actions.</p><p>With this controller, we would have in sync every project in our clusters, chat rooms like slack, monitoring systems, AWS roles...</p><p><em>Note:</em> This needs a garbage collection job in the background in case we miss a deletion event or the controller wasn’t running at the moment of the deletion.</p><h4>Directory replication</h4><p>What we want is to have a directory replicated in the FS.</p><ul><li><strong>ListerWatcher(List)</strong>: Will list all files in a directory, in case of being a directory, it will get the subdirectory files also.</li><li><strong>ListerWatcher(Watch)</strong>: Will subscribe to the Kernel notifications on the different files like edit, create and deletion (e.g. using <a href="https://github.com/fsnotify/fsnotify">fsnotify</a>).</li><li><strong>Storage</strong>: Will get the file hash for verification and check if the file exists in the replicated directory, if it is there it will get also the verification hash. Then will return a state struct with the expected state and the current state.</li></ul><pre>type State struct {<br>    OriginalPath <em>string<br>    </em>originalHash <em>string<br>    </em>ReplicaPath <em>string<br>    </em>ReplicaHash <em>string<br></em>}</pre><ul><li><strong>Handler</strong>: In case of being an add/edit event it will check if the file has not been replicated yet or the hashes of both files are different and will replicate the file. In case of deletion and the file exists it will delete the file.</li></ul><p>Like the previous approach, the Storage could get only the information of the original file and let the handler get the hashes and take actions, that depends on how you would like to design your application but is flexible enough to give you the option to select whatever you want.</p><p>With this controller, we could have a directory replicated, like a Dropbox or Google Drive application.</p><p><em>Optimization example</em>: This can be optimized to use a cache for the ListerWatcher and Storage, so the list of all the files is only done the first time (<a href="https://golang.org/pkg/sync/#Once">sync.Once</a>), and then use the notifications (like the watcher) on files to update that cached state and return that on the List calls (this is the method used by Kubernetes).</p><p><em>Optimization example2</em>: When we handle a file that is a directory, create a new replication controller for that directory and when deleted everything stops the controller (kind of recursion but with controller runs).</p><p><em>Note:</em> This needs a garbage collection job in the background in case we miss a deletion event or the controller wasn’t running at the moment of the deletion.</p><h3>Conclusion</h3><p>Hope you like the library and if you have use cases using it that you want to share we would love to hear them!</p><p>You can reach us on <a href="https://github.com/spotahome/gontroller">Github</a> to give us feedback about features, errors and pain points, or contributing by sending PRs.</p><p>Thanks for reading!</p><p><em>P.S. Now you don’t have an excuse to write reliable and clean controllers if you are not using Kubernetes resources :)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=832d4a9522ea" width="1" height="1" alt=""><hr><p><a href="https://medium.com/spotahome-product/gontroller-a-go-library-to-create-reliable-feedback-loop-controllers-832d4a9522ea">Gontroller: a Go library to create reliable feedback loop controllers</a> was originally published in <a href="https://medium.com/spotahome-product">Spotahome Product</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Goresilience: Adaptive resilience]]></title>
            <link>https://slok.medium.com/goresilience-adaptive-resilience-9500db789609?source=rss-13dde3ada38b------2</link>
            <guid isPermaLink="false">https://medium.com/p/9500db789609</guid>
            <category><![CDATA[resilience]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[adaptive]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Xabier Larrakoetxea]]></dc:creator>
            <pubDate>Sat, 02 Mar 2019 08:54:01 GMT</pubDate>
            <atom:updated>2019-03-02T08:54:01.531Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VFqUbXMD60RewT8hKumRPg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/RC1_2Q4oM9k?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Cécile Brasseur</a> on <a href="https://unsplash.com/search/photos/chameleon?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>In December <a href="https://github.com/slok/goresilience">Goresilience</a> was <a href="https://medium.com/@slok/goresilience-a-go-library-to-improve-applications-resiliency-14d229aee385">released</a>. For the ones that don’t know what it is: Goresilience is a Go library that implements resilience patterns.</p><p>Today I released the <strong>0.2 </strong>version of the library and it comes with a huge improvement, <strong>Adaptive runners.</strong></p><p>Adaptive resilience comes from <strong>adaptive control </strong>theory.</p><blockquote><strong>Adaptive control</strong> is the control method used by a controller which must adapt to a controlled system with parameters which vary, or are initially uncertain</blockquote><p>In our case, it means that the Goresilience runners will adapt automatically to the environment they are running in by getting information in real time.</p><h3>Little background</h3><p>To understand how adaptive runners are designed and how to use them, first, you will need to understand Goresilience library design, check <a href="https://medium.com/@slok/goresilience-a-go-library-to-improve-applications-resiliency-14d229aee385">this</a> post to read about its design.</p><h3>Design</h3><p>These new adaptive runners are based on concurrency limiters, this is not new, Goresilience has been inspired by <a href="https://github.com/Netflix/concurrency-limits">Netflix Concurrency-limits</a> library (Java implementation).</p><p><a href="https://github.com/slok/goresilience#concurrency-limit"><strong>concurrencylimit</strong></a><strong> </strong>runner will limit the concurrency based on the feedback of the executions results (latency and errors).</p><p>In this <a href="https://medium.com/@NetflixTechBlog/performance-under-load-3e6fa9a60581">Netflix post</a> it’s explained perfectly how these concurrency limits work, so, if you don’t know what is adaptive concurrency, <a href="https://medium.com/@NetflixTechBlog/performance-under-load-3e6fa9a60581">this</a> posts explains it in a fast and very simple way.</p><p>But to keep it short… the execution of funcs/jobs will be measured by latency and the final result of the execution, this result will be processed and depending on the algorithm it will configure a new concurrency limit. We could reduce to the simplest way and think that is like a <em>dynamic bulkhead</em> that adapts to the environment.</p><h4>Components</h4><p>concurrencylimit runner is based in 3 components (all of them configurable).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/782/1*-RrX9KYV8PwZ1vt7CRigzQ.png" /><figcaption>Concurrencylimit Runner architecture design</figcaption></figure><ul><li><strong>Executor</strong>: The executor knows how to prioritize the work ( goresilience.Func) it’s receiving and executes.</li><li><strong>Limiter</strong>: The limiter will calculate a limit for the execution workers the executor will configure based on the different algorithm implementations.</li><li><strong>Execution result policy</strong>: is a simple function that will receive the result of the work executed by the executor component and based on it, it will translate this result to a result (success, fail or ignore) that the Limiter understands and uses to calculate the concurrency limit. Maybe you are wondering why do we need this component and we don’t take the execution result directly, for example, sometimes an error on the execution is a “ignore” (and not a “failure”) result for the limiter because we want to base the calculation of limits only using latency.</li></ul><h3>Implementations</h3><p>At this moment concurrencylimit has different implementations of the three different components, by combining them you can get different results that will adapt to different kind of apps, implementations, and contexts.</p><h4>Limiters</h4><p>The limiters are mostly algorithms based on TCP congestion existing algorithms. The will come more on the next months (at least the ones that Netflix recommends like <a href="https://en.wikipedia.org/wiki/TCP_Vegas"><em>Vegas</em></a>)</p><ul><li><strong>Static</strong>: This limiter does not change, you select the number of workers and it will return always the same.</li><li><strong>AIMD</strong>: This limiter is based on <a href="https://en.wikipedia.org/wiki/Additive_increase/multiplicative_decrease">AIMD</a> algorithm (and <a href="https://github.com/Netflix/concurrency-limits/blob/d541bf037e7c620ea9f7718d3c8badf4b3daecd6/concurrency-limits-core/src/main/java/com/netflix/concurrency/limits/limit/AIMDLimit.java">here</a>). It will increase the number of workers in additive mode until it detects congestion (errors and/or latency), at that moment it will decrease by a multiplicative factor (can be configured).</li></ul><h4>Executors</h4><ul><li><strong>FIFO</strong>: First in first out queue execution priority.</li><li><strong>LIFO</strong>: <a href="https://es.wikipedia.org/wiki/Last_in,_first_out">Last in first out</a> queue execution priority.</li><li><strong>Adaptive LIFO + CoDel</strong>: This is based on Facebook’s <a href="https://queue.acm.org/detail.cfm?id=2839461">Fail at scale</a> article. This executor has two modes of working, the regular one (FIFO + regular queue timeout that is 100ms by default) and the congestion mode (LIFO + aggressive queue timeout 5ms). The executor changes from one to another mode when it detects the execution congestion.</li></ul><h3>Examples</h3><p>Netflix concurrency-limit style:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/65646cf4abc2c0be557fb0172e8da268/href">https://medium.com/media/65646cf4abc2c0be557fb0172e8da268/href</a></iframe><p>Facebook AdaptiveLIFO+CoDel style:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/05065ac91fc5a57db46a2f98454183ff/href">https://medium.com/media/05065ac91fc5a57db46a2f98454183ff/href</a></iframe><p>These are regular goresilience Runners. That’s the aim of this library, to be idiomatic and use it always in the same, clean and simple way.</p><p>For example, we could use them as an HTTP middleware that controls the concurrent requests handled by the server and return a controlled failure when is congested, in this example is a 429 status code, but could be default data, cached data, empty data...</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/130321aa7f0289cdb9baa08b97018417/href">https://medium.com/media/130321aa7f0289cdb9baa08b97018417/href</a></iframe><h3>Observability</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/901/1*veBwPtGsIA3uE-0k7RXvnA.png" /><figcaption>Concurreny limit calculated by an AIMD algorithm</figcaption></figure><p>The concurrencylimit <em>Runners </em>come with metrics like every runner kind:</p><ul><li><strong>Inflight executions</strong> executing concurrently by the executor.</li><li>The <strong>calculated limits</strong> calculated by the limiter.</li><li>The <strong>results translation</strong> made by the result policy.</li></ul><h3>Conclusion</h3><p>The adaptive Runners are very flexible and like all the other runners from the library, are extensible and can be combined, they aren’t different from the others.</p><p>Hope you like this new version, grab it on the <a href="https://github.com/slok/goresilience">repository</a>, and as always, any feedback is more than welcome.</p><p>Thank you!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9500db789609" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A tale of evolutionary applications, iterations, and performance (part 3)]]></title>
            <link>https://medium.com/spotahome-product/a-tale-of-evolutionary-applications-iterations-and-performance-part-3-55b459884462?source=rss-13dde3ada38b------2</link>
            <guid isPermaLink="false">https://medium.com/p/55b459884462</guid>
            <category><![CDATA[iteration]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[operators]]></category>
            <category><![CDATA[latency]]></category>
            <dc:creator><![CDATA[Xabier Larrakoetxea]]></dc:creator>
            <pubDate>Thu, 14 Feb 2019 12:25:49 GMT</pubDate>
            <atom:updated>2019-02-16T10:54:05.972Z</atom:updated>
            <content:encoded><![CDATA[<p>This post is the 3rd part of a series of posts, on <a href="https://product.spotahome.com/a-tale-of-evolutionary-applications-iterations-and-performance-part-2-e6afdff56a36">part 2</a> we saw the bottlenecks we encountered after the first usage and iterations on the application. Let’s continue with the story…</p><h4>4th age: Frontend degradation</h4><p>Since we saw the latency spikes and we applied the concurrency, the frontend started to have a performance degradation, not at usage level but at resources usage by the browsers.</p><p>This was because the <em>CI operator</em> returned all the builds information of a project without pagination and without ordering. To tackle this problem they started to order and get the latest 150 builds per project (discarding the rest of received builds).</p><p>After the concurrency change, we knew that the problem was the number of operations being made to get all the builds from a project at once.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WHQUijMiwladdHFJ956oXA.png" /><figcaption>Redis CPU usage</figcaption></figure><p>At this point it was the moment of iterating again in the performance and return paginated results, we would gain:</p><ul><li>Frontend simpler code</li><li>Frontend performance improvement</li><li>Less JSON unmarshaling (less CPU usage == less throttling)</li><li>Less time hitting Redis (less IO, less CPU == less latency)</li></ul><p>To start paginating results through the API we needed to change the Redis <em>set</em> data structure to a <em>sorted set</em> data structure (use the build creation timestamp as the score of the sorted set).</p><p>Knowing that we were listing only the latest builds, we could make this migration in three stages so we don’t have downtime:</p><ul><li>Deploy the new data structure (sorted set) writes along with the current data structure (set) writes and read as always from the current data structure (set). This will populate the new structure with the latest data.</li><li>After some time, deploy the change to return the results from the new data structure (sorted set) with default pagination of 100–150 page size.</li><li>Check everything works, then migrate rest of the data (if required) and clean old data structure (set) data.</li></ul><p>We implemented this in two days of one person and set in production, and these are the results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*24nIBLbMGBp5Rs1enPqZzg.png" /><figcaption>Latency after pagination release (zoom in)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tMtHdj_x10_Ve9ldw9CLWg.png" /><figcaption>Latency after pagination release (zoom out)</figcaption></figure><p>At this moment our app is stable and has a max latency of 260ms (p99)</p><p>The usage resources of the app have been also reduced a lot:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7r40DLY-7unpdh0HnsXOLQ.png" /><figcaption>CPU and memory usage after pagination release.</figcaption></figure><p>There is room for improvement like using caching but we don’t need it at this moment.</p><h4>Conclusion</h4><p>Wow, that was a huge post (altogether)!</p><p>The conclusion here can have two different interpretations</p><p>The first one:</p><ul><li><em>You had insane latencies!</em></li><li><em>You could have added pagination from day one</em></li><li><em>You could have added concurrency from day one</em></li></ul><p>Or the second one, let me ask some questions and answer them:</p><ul><li><em>Did the users were affected?</em> no, or at least they didn’t yell about it :).</li><li><em>Did we interrupt the service?</em> no, that we are aware of (using alerts and people feedback).</li><li><em>Did we have a valid solution to the problem from day one?</em> yes, a very simple one and not the final one on this post.</li><li><em>Did we invest our time efficiently?</em> yes, or at least we tried, we have lots of different apps and infrastructure to maintain and divide the cycles among all of them (apart from the incidents, meetings, mentoring…)</li><li><em>Is our code simple, maintainable and flexible to changes?</em> yes, we could add changes easily and fast and new developers understand the code without problems.</li><li><em>Would we end at the same place from the clients/users point (results, latency…) if we had developed everything at first? </em>maybe, we don’t know but is very likely.</li><li><em>Did we develop things that were not necessary?</em> no, from our point of view we developed the things that we required at that moment and nothing more.</li><li><em>Did we overengineered or prematurely optimized?</em> no, mainly due to the previous question answer.</li><li><em>Did we optimize things that were not required?</em> not at a first sight, but some of them could, but is likely that are minor things (for example, more worker quantity on the operator).</li></ul><p>With both interpretations or approaches we would end at the same place, but what are the differences between both?</p><p>The second interpretation gives us the ability to give value from day one and invest time only on the required parts with good designs and approaches, <strong>just because we have more information than before, nothing more</strong>. For example, without information, the performance optimizations could happen on wrong places adding complexity, corner cases and not having a big impact in performance, but yes on maintainability of the app (in a bad way).</p><p>The time invested operating and developing the application has been low in our opinion (~10–15 days in total of 1 person in &gt;4 months) and also, in the way of achieving the final result, as a <strong>bonus</strong>, we learned things like CPU behavior in Kubernetes, improved debugging, monitoring and profiling skills.</p><p>The last thing that is very important to have in mind is that <strong>every application is different and the context matters</strong>, we could manage these latencies due how the application was designed on the frontend, because the nature of the data we are managing is not required to be near real-time, and finally the rate of request on this service is not too high.</p><p><strong>Make things as simple as possible but not simpler.</strong></p><p>Thank you.</p><p><em>P.S.</em> <em>We did not talk about </em><a href="https://landing.google.com/sre/sre-book/chapters/service-level-objectives/"><em>SLI and SLOs</em></a><em> on this post but do you think we have violated our SLOs or burned all the error budget? :)</em></p><p><em>P.S. Thanks to </em><strong><em>frontend-core team </em></strong><em>for the awesome frontend app that we use every day.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=55b459884462" width="1" height="1" alt=""><hr><p><a href="https://medium.com/spotahome-product/a-tale-of-evolutionary-applications-iterations-and-performance-part-3-55b459884462">A tale of evolutionary applications, iterations, and performance (part 3)</a> was originally published in <a href="https://medium.com/spotahome-product">Spotahome Product</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A tale of evolutionary applications, iterations, and performance (part 2)]]></title>
            <link>https://medium.com/spotahome-product/a-tale-of-evolutionary-applications-iterations-and-performance-part-2-e6afdff56a36?source=rss-13dde3ada38b------2</link>
            <guid isPermaLink="false">https://medium.com/p/e6afdff56a36</guid>
            <category><![CDATA[iteration]]></category>
            <category><![CDATA[operators]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[latency]]></category>
            <dc:creator><![CDATA[Xabier Larrakoetxea]]></dc:creator>
            <pubDate>Thu, 14 Feb 2019 12:24:12 GMT</pubDate>
            <atom:updated>2019-02-18T11:52:13.733Z</atom:updated>
            <content:encoded><![CDATA[<p>This post is the 2nd part of a series of posts, on <a href="https://product.spotahome.com/a-tale-of-evolutionary-applications-iterations-and-performance-8053370c795f">part 1</a> we saw the problem and the first solution we delivered. Let’s continue with the story…</p><h4>2nd Age: 1st bottleneck</h4><p>After 2 weeks and more and more builds were stored on the service we saw that the latency was growing a bit. The users were not affected but we wanted to reduce the latency because we wanted to reduce technical debt.</p><p>So we started improving the performance of the application, but where do we start?</p><p>This research process had a duration of ~2 months using spare time/cycles. In total were around three days of one person.</p><p>We only knew that the latency started when we changed the storage type from in-memory to Redis. We wanted to start small, our possible bottlenecks at first sight were:</p><ul><li>Database</li><li>Application resources (CPU, Memory, IO)</li></ul><p>Our first guess was the Redis. After checking the stats and metrics of the Redis, we concluded, the Redis was perfect, very small latency and resource consumption, <strong>the database wasn’t the bottleneck</strong>.</p><p>Then we started with the application resource usage.</p><p>These were the tests made to isolate the bottlenecks (some of them have been omitted because they were dead ends and not interesting):</p><ul><li>Increate instance number on the cluster: It didn’t increase/reduce latency (Redis discarded once again because ops per second increased).</li><li>Redis connection pool on app increased: performance increased but not much, so we also discarded.</li><li>Run application from localhost to the production database in RO mode: better performance &lt; 400ms p99.</li></ul><p>At this point based on the latest result, we concluded that was very likely that the environment where the app was running was the main problem and not the Redis implementation. We discarded the network problem because we had more applications that transfer more data in the same conditions. So… everything points to the CPU.</p><ul><li>Increased <a href="https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/">CPU requests</a> and limits: The performance improved! but… the CPU real usage didn’t increase.</li></ul><p>At this point, we were on something.</p><p>We had some guess on a hot path where concurrency was being used aggressively without limits to insert project builds on the database.</p><p>We made different combinations increasing and decrease CPU requests/limits, operator workers and limiting concurrency.</p><p>For example one of the combinations changes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bZx_b65BPfEfMpDaT4_NuQ.png" /><figcaption>from 2 workers to 4 workers and 200m CPU request</figcaption></figure><p>But it was strange because although the application’s resources were increased and the performance was improved, the CPU usage was not being increased.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pFB2iTd8Y-yFp9fr2f2Tqg.png" /><figcaption>CPU usage</figcaption></figure><p>At this point we knew that the CPU was the problem, we did some profiling on the application with <a href="https://golang.org/pkg/net/http/pprof/">pprof</a> and check what was consuming the most CPU.</p><p>Example to profiling 30s of CPU usage:</p><pre>go tool pprof <a href="http://localhost:6060/debug/pprof/profile?seconds=30">http://localhost:6060/debug/pprof/profile?seconds=3</a>0</pre><p>And we used <a href="https://www.speedscope.app/">Speedscope</a> to load the samples in <a href="http://www.brendangregg.com/flamegraphs.html">flamegraphs</a> (for us is more visual than google/<a href="https://github.com/google/pprof">pprof</a> one).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vzjo73hsCPTQEuL6q7KEfQ.png" /><figcaption>Flamegraph of the CPU usage (ordered on the left the most CPU consuming)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*swbphblF11UFftGf07i9HA.png" /><figcaption>List of instructions consuming CPU (descending order)</figcaption></figure><p>Like we see on the pictures the most time is spent on JSON unmarshaling (23%). So we changed the JSON implementation, instead of using Go’s standard library implementation we tried with <a href="https://github.com/json-iterator/go">json-iterator/go</a> library. We created a branch and deployed that branch with this small change, and the results were almost the same (very minimal improvements) so this wasn’t the bottleneck we were looking for.</p><p>The CPU profiling although didn’t point the bottleneck, give us the hint that the problem was out of the application. So… we checked how the CPU was working in Kubernetes… we started checking if the CPU was being <a href="https://engineering.linkedin.com/blog/2016/11/application-pauses-when-running-jvm-inside-linux-control-groups">throttled</a> by <a href="https://kubernetes.io/blog/2018/07/24/feature-highlight-cpu-manager/">the kubernetes CPU manager</a> although the CPU was not being consumed (strange behavior but we checked).</p><p>Make a small Prometheus query on our metrics:</p><pre>max(<br>    rate(container_cpu_cfs_throttled_seconds_total{<br>      pod_name=~&quot;^ci-operator.\*&quot;, <br>      container_name=&quot;app&quot;}[5m]<br>    )<br>)</pre><p><strong>The query showed us that the <em>CI operator </em>was being throttled (between 1s and 3s!).</strong></p><p><strong>We found the current bottleneck. </strong>On the next image, you can see the heatmap of the different CPU throttled seconds on all Kubernetes cluster operators, it’s clear where the throttling of the <em>CI operator</em> is.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SQk4LAFgEyhzNw3xumrAAw.png" /><figcaption>Heat maps of all operators CPU throttling</figcaption></figure><p>After researching a bit about the unnecessary throttling that was being applied by the CPU manager to the <em>CI operator, </em>we read (<a href="https://github.com/kubernetes/kubernetes/issues/51135">here</a>, <a href="https://github.com/kubernetes/kubernetes/issues/67577">here</a> and <a href="https://github.com/kubernetes/kubernetes/pull/63437">here</a>) that is a common problem/bug on Kubernetes and increasing the CPU limits or disabling them it would make a real and safer throttling.</p><p>We increased the limits a lot (300m to 1200m):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kbnpKIMMZvvWaB1jNS0qVA.png" /><figcaption>The operations on the storage rate kind and its latency (with Redis and CPU 1200m)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z2HIyT4fhqEnIhvUHKadvQ.png" /><figcaption>The latency of the API (with Redis and CPI 1200m)</figcaption></figure><p><strong>As it can be seen on the images there was a huge improvement… (from 2.5s to 900ms).</strong></p><p>Good enough, we stopped here until next performance improvement iteration.</p><h4>3rd Age: 2nd bottleneck</h4><p>And here we are at this point (1st of February of 2019) the latency started to increase during this time and although there was no impact on the user experience the latency measures were insane for one of the handlers ~7s getting project builds.</p><p>Knowing the problematic endpoint and how the performance has been degrading on this endpoint after weeks we knew that the problem was how the implementation of Redis was retrieving the builds of a project, instead of doing iteratively, we changed to a concurrent way using the <a href="https://docs.microsoft.com/en-us/azure/architecture/patterns/bulkhead">bulkhead</a> pattern and <a href="https://github.com/slok/goresilience">goresilience</a> library implementation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uBUbPG2HRd0_iHM0qEF3Ww.png" /><figcaption>Latency before and after the concurrency change.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*79Tr4zb6TyfaQVAER3bScA.png" /><figcaption>P99 latency (by handler) before and after the concurrency change.</figcaption></figure><p><strong>This change reduced the latency of the slowest handler (7s spikes to 2.5s) but increased the latency of the others (100ms to 300ms)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9AiXK4VHATe3bJZCqXhhSQ.png" /><figcaption>P99 latency (by handler without project builds one) before and after the concurrency change.</figcaption></figure><p>This seemed that after setting concurrency the CPU usage increased and the throttling started again (remember our first bottleneck).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LqHNzNboJ9UxtvTXUmu1qg.png" /><figcaption>CI operator CPU throttling before and after adding concurrency</figcaption></figure><p>So we increased the limit from 1200m to 3000m and 8000m and the latencies went down more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y2mDUufdErOUHYdL2ecPyg.png" /><figcaption>Latency before and after the concurrency and CPU change.</figcaption></figure><p>At this moment we had the service at the place we wanted using one day of work of one person. For now, we were good, let’s continue iterating when needed.</p><p>The story continues on the <a href="https://product.spotahome.com/a-tale-of-evolutionary-applications-iterations-and-performance-part-3-55b459884462">3rd part</a>.</p><p>Thank you!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e6afdff56a36" width="1" height="1" alt=""><hr><p><a href="https://medium.com/spotahome-product/a-tale-of-evolutionary-applications-iterations-and-performance-part-2-e6afdff56a36">A tale of evolutionary applications, iterations, and performance (part 2)</a> was originally published in <a href="https://medium.com/spotahome-product">Spotahome Product</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A tale of evolutionary applications, iterations and performance]]></title>
            <link>https://medium.com/spotahome-product/a-tale-of-evolutionary-applications-iterations-and-performance-8053370c795f?source=rss-13dde3ada38b------2</link>
            <guid isPermaLink="false">https://medium.com/p/8053370c795f</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[operators]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[latency]]></category>
            <category><![CDATA[iteration]]></category>
            <dc:creator><![CDATA[Xabier Larrakoetxea]]></dc:creator>
            <pubDate>Thu, 14 Feb 2019 12:23:50 GMT</pubDate>
            <atom:updated>2019-02-18T11:50:45.688Z</atom:updated>
            <content:encoded><![CDATA[<h3>A tale of evolutionary applications, iterations, and performance (part 1)</h3><p>This is a story of an application called <strong><em>CI operator</em></strong>, that has been with us for 4 months, this application has been developed in the<strong> infrastructure team</strong>. In these four months, the application has evolved in many aspects and in this post, we will tell how it was created, how evolved, what decisions were taken and why those decisions were applied.</p><p>We will split this story into three posts, in part 1 we will see the initial design and creation of the application, on <a href="https://product.spotahome.com/a-tale-of-evolutionary-applications-iterations-and-performance-part-2-e6afdff56a36">part 2</a> we will see the bottlenecks we encountered and on <a href="https://product.spotahome.com/a-tale-of-evolutionary-applications-iterations-and-performance-part-3-55b459884462">part 3</a> we will see the current state and some conclusions of this trip.</p><p>First of all, we need to be on the same page and have some information about the context.</p><p>In Spotahome almost all of our projects use <a href="https://brigade.sh/">Brigade</a> as the pipeline/CI/CD system. Brigade is awesome because it gives us the ability to have dynamic pipelines(made in JS) with native <a href="https://kubernetes.io/">Kubernetes</a> runtime which we are already using intensively for most of our platform.</p><p>Engineering department(&gt;80 people) has been migrating all their projects from <a href="https://concourse-ci.org/">Concourse</a> to Brigade this past 2018 and to use it correctly the <strong>frontend-core team</strong> developed a web interface based on <a href="https://reactjs.org/">React</a>, <a href="https://samnewman.io/patterns/architectural/bff/">BFF pattern</a>, and <a href="https://graphql.org/">GraphQL</a> that calls the Brigade HTTP API to get the information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/919/1*7pVmkVlmiFKCOd8EN9wIyQ.png" /><figcaption>The application for deploying and checking the builds in Spotahome</figcaption></figure><h3>The problem</h3><p>Like almost every application, the application is the solution to one or more problems, and this is not different, the <em>CI operator</em> solves a problem with Brigade: <strong>The information about the builds and logs the Brigade API returns is ephemeral</strong>, it gets this information from Kubernetes itself so, when a <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod/">pod</a> of a build has gone, it’s logs and information (exit code, id…) will disappear. If we are talking about old builds this is not a problem, but our clusters autoscale constantly and its common to lost pods information in a small amount of time (in extreme cases, the build finishes and in few seconds the information has gone).</p><p>To tackle this problem we created the <strong>CI operator</strong>, this application is a Kubernetes operator that stores information of all the Brigade installations on the cluster (we have one Brigade installation per team on the cluster, and we have ~11 teams), the operator will expose that information to the app that the <strong>frontend-core team</strong> developed.</p><p>The application has a simple design, its based in 2 components:</p><ul><li><strong>The storer (operator)</strong>: The operator itself that is getting all the information from the cluster (builds, logs…) and storing in a database.</li><li><strong>The HTTP API</strong>: An HTTP REST API exposed to the clients to get that information from the database and avoid having to share a database which is an antipattern.</li></ul><p>In this picture, you can visualize better the solution:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/775/1*nXQd0sMgJlakqtSbgTW6mA.png" /><figcaption>The architecture of clients and CI operator</figcaption></figure><p>There are more details in the process of the solution but is not required for this post.</p><h3>Team premises</h3><p>In the Infrastructure team, we try to follow some premises that can help to understand the processes we followed with <em>CI operator.</em></p><ul><li>Think and design before writing a line of code.</li><li>Try following best practices while developing without dogmatisms (testing, documentation, PRs, SOLID…).</li><li>Each line of code is technical debt (the best code is no code).</li><li>Design and implement things as simple as possible.</li><li>Tomorrow you will have more information than today (postpone decisions until is necessary).</li><li>Ship fast and iterate every day.</li><li>Observability is a requirement, not a feature.</li><li>Avoid overengineering and premature optimization.</li><li>Automate repetitive tasks and reduce <a href="https://landing.google.com/sre/sre-book/chapters/eliminating-toil/">toil</a> every day.</li><li>Test in production and rollback fast.</li><li>Complexity will kill us.</li></ul><p>In other words, we love simplicity and “boring” things.</p><h3>History of the CI operator</h3><p>Let’s start with the history of our application and why we took those decisions. I’ll split it in ages.</p><h4>Age 0: Initial release</h4><p>On October of 2018, we arose the Brigade ephemeral information problem and we started thinking a solution.</p><p>The solution as previously was described it was the CI operator, a Kubernetes operator with an API that will store all the data and is written in <a href="https://golang.org/">Go</a>.</p><p>This was the definition of our storage (the simples one we could imagine):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e5f815df88e05b9069e7e0d80a6a9c59/href">https://medium.com/media/e5f815df88e05b9069e7e0d80a6a9c59/href</a></iframe><p>Our first implementation was based <strong>in memory storage and without any listing options</strong>. Why did we make this decision?</p><ul><li>We didn’t have information about the usage of the data to decide what kind of storage backend is a good candidate (K/V, SQL, document, graph…).</li><li>Ship fast.</li><li>Solve partially the current problem (be better today than yesterday).</li><li>Check how our initial store design, model (business logic) and API was working to validate the process.</li></ul><p>We try following <a href="https://en.wikipedia.org/wiki/SOLID">SOLID</a> principles when we develop, this gives us the ability to postpone decisions easily, solve the immediate problem, create maintainable code and have the door open to fast improvements.</p><p>We created a simple implementation with the memory storage based on maps.</p><p>The 10th of October of 2018, we deployed this first implementation and the results were very good. There was only one drawback, <strong>the data also was ephemeral</strong>, we knew that, but it was less ephemeral than the Brigade one. People were happier and it was better than before in just 3 days of work (including design) of 1.5 people.</p><p>Our applications try to ship with metrics from day one and this is no exception, based on the metrics that we recorded we had this data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-7lBBvtXMpFyx7wqaWD29g.png" /><figcaption>The operations on the storage rate kind and its latency (In memory)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XPL1lFFAxREbtCN6RHLXnQ.png" /><figcaption>The latency of the API (In memory)</figcaption></figure><p>The results seemed good.</p><h4>1st Age: persistent storage</h4><p>After a week of seeing the application working, we validated that the design at that moment was more than enough, the HTTP API was more than enough and the requirement meets from the users were more than enough. Except… that we wanted to persist the data.</p><p>We only needed to design the storage part, all the other parts of the application were working good in production. Based on the collected data and the type of HTTP API we designed, we made the decision to go with a <a href="https://redis.io/">Redis</a> because:</p><ul><li>Easy to use and maintain.</li><li>Flexible and lots of structures.</li><li>Fast.</li><li>Mature and tested in many environments and by many companies (us included).</li><li>Possible use of expiration and remove old data easily in the future when we knew the growth rate.</li><li>If we lost everything if something very bad happens is not a problem.</li></ul><p>In one day, one person and ~300 LoC it was ready and shipped by the end of the day. We took the simplest approach we had for the Redis, use sets to store the builds of the projects and use simple keys to store the builds data serialized in JSON and the logs in raw.</p><p>And we shipped the 24th of October…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p2Iz4HBIc-eULyx8hdBrBg.png" /><figcaption>The operations on the storage rate kind and its latency (with Redis)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YL534O_nB98PnDIMUFZz3g.png" /><figcaption>The latency of the API (with Redis)</figcaption></figure><p>We expected a latency increase, but we didn’t understand why the latency was so high (2s p99).</p><p>Luckily the web UI that uses data from this application was working ok and the users didn’t have a bad experience using the service, so, although we knew that it wasn’t an ideal latency it satisfied our needs:</p><ul><li>Users experience not affected.</li><li>Users use it and is a valid solution for them.</li></ul><p>You are wondering why with this latency the users were not affected?</p><p>The web that the users use, uses continuous polling and updates the data after the first load dynamically, giving the user the false sensation of real-time updates, eventually everything will be updated correctly, the user doesn’t need millisecond real-time updates for this kind of applications (what’s the difference of showing the build has started at T0 or T0 + 4s? for us, nothing), so the huge latency increase was not affecting the user experience.</p><p>For now, we were good, let’s continue iterating when needed.</p><p>The story continues on the <a href="https://product.spotahome.com/a-tale-of-evolutionary-applications-iterations-and-performance-part-2-e6afdff56a36">2nd part</a>.</p><p>Thank you!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8053370c795f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/spotahome-product/a-tale-of-evolutionary-applications-iterations-and-performance-8053370c795f">A tale of evolutionary applications, iterations and performance</a> was originally published in <a href="https://medium.com/spotahome-product">Spotahome Product</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Goresilience a Go library to improve applications resiliency]]></title>
            <link>https://slok.medium.com/goresilience-a-go-library-to-improve-applications-resiliency-14d229aee385?source=rss-13dde3ada38b------2</link>
            <guid isPermaLink="false">https://medium.com/p/14d229aee385</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[resilience]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Xabier Larrakoetxea]]></dc:creator>
            <pubDate>Mon, 17 Dec 2018 16:24:31 GMT</pubDate>
            <atom:updated>2019-03-02T09:09:50.401Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_Uo1X9Y14LOCyYuL--_HwA.jpeg" /></figure><p>When we start developing an application lot’s of people forget about one of the most important facts of our programs. <strong>Our application will be in production and nothing is guaranteed there</strong>.</p><p>We know that is true but we forget about it. Could be as easy as asking ourselves some of these questions…</p><ul><li>What if the network isn’t working?</li><li>It’s one of my dependency services responding? or worse, is responding slowly?</li><li>Can I become DoSed by my clients? (internal clients?!)</li><li>Can I be collapsed by my own concurrency with my current assigned resources?</li></ul><p>I could continue… but you know what I mean.</p><p>Why does this happen to us? Well, maybe the answer is that on the happy path these things don’t happen, but this happens, especially in high scale environments.</p><p>That’s why companies like Netflix that work at this huge scale develop things like <a href="https://github.com/Netflix/Hystrix">Hystrix</a>.</p><blockquote>Hystrix is a latency and fault tolerance library designed to isolate points of access to remote systems, services and 3rd party libraries, stop cascading failure and enable resilience in complex distributed systems where failure is inevitable.</blockquote><p>With these libraries, they can control these kinds of problems and <a href="https://github.com/slok/goresilience"><strong>Goresilience</strong></a><strong> is one of those libraries.</strong></p><h3>Goresilience</h3><p><a href="https://github.com/slok/goresilience">Goresilence</a> is a library to improve the resilience of Go applications in an easy and flexible way. It can be extended and tries to be Go idiomatic.</p><p><a href="https://github.com/slok/goresilience">Goresilience</a> is inspired by <a href="https://github.com/Netflix/Hystrix">Hystrix</a> and similar libraries (<a href="https://github.com/resilience4j/resilience4j">resilience4j</a>, <a href="https://github.com/Netflix/concurrency-limits">concurrency-limits</a>…) but the main goals of <a href="https://github.com/slok/goresilience">goresilience</a> (apart from being a resilience library) are:</p><ul><li>Flexibility: Easy to extend by anyone.</li><li>Testable: Easy to test.</li><li>Go idiomatic: Embrace Go language “standards”.</li><li>Observability: Give the user the ability to expose metrics about what’s happening inside the application execution(white box monitoring).</li><li>Easy to use: Clean design and safety defaults.</li></ul><h3>Design</h3><p>The library architecture is simple, and if you are already a Gopher it will be easy to understand how it works because it follows the same ideas of the http.Handler.</p><p>The core of the library is based on one component, the goresilence.Runner, this is the executor.</p><p>The Runner is an interface and all the resiliency components in <a href="https://github.com/slok/goresilience">Goresilience</a> implement it. With this interface, the library uses the power of the <a href="https://refactoring.guru/design-patterns/decorator">decorator pattern</a> and decouples all the logic of the different resiliency components. In Go is common to have seen this pattern with middlewares of the http.Handler.</p><p>Let’s explain how the library works with a small example.</p><h4>Example</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d9f1e6dbb748d4c749b24cfa11998dbf/href">https://medium.com/media/d9f1e6dbb748d4c749b24cfa11998dbf/href</a></iframe><p>As seen on the snippet, we have a service that knows how to grab music information from our users.</p><p>But this is not very resilient, what if it&#39;s getting slow and the dependencies get stuck for more than 1 Second while we are waiting to their response? we don’t want to make our users wait for everything because the user favorite genres list is slow. Let’s fix that with a <a href="https://github.com/slok/goresilience">goresilience</a> timeout.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/71faeb89e59af48e25798afcb68c74f2/href">https://medium.com/media/71faeb89e59af48e25798afcb68c74f2/href</a></iframe><p>On the first line we create our runner using <a href="https://github.com/slok/goresilience">goresilience</a> with the default configuration, using that runner we can execute any function. Next we set a fallback for the results and then execute using the runner.</p><h4>Middlewares</h4><p>As you see is clean and easy to follow. But the best part of the library design comes with the middlewares.</p><p>The Runners can be used as middlewares in a chain of middlewares so you could chain the different resiliency componentes in a execution chain.</p><p>Following the past example.. imagine that we want to customize the timeout and also add retries with exponential backoff.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/376b14c975f61b5244387c42fdb6f96b/href">https://medium.com/media/376b14c975f61b5244387c42fdb6f96b/href</a></iframe><p>We just change the runner code. Now we create a Chain of runner middlewares. This runner execution would be represented with something like this: (retry(timeout(Func))).</p><p>And like this chain, you can create the flow that you want with any of number and type of Runners (including yours! it just needs to implement interface goresilence.Runner).</p><p>As you see in the example every Runner from the library comes with 2 factory methods: Newand NewMiddleware. Newis used in standalone mode and NewMiddlewareis used in a chain of runners.</p><p>As a final example imagine you want to create a more `Hystrix-like` runner chain, you could:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/917472de7bd077457e19dd602475b938/href">https://medium.com/media/917472de7bd077457e19dd602475b938/href</a></iframe><h3>Runners available</h3><p>At the moment of this writing the library comes with this (for now only reactive and static, but I want to add adaptive ones).</p><ul><li>Timeout: Timeouts the execution based on the duration of it.</li><li>Retry: Retries with exponential backoff and Jitter if the execution fails.</li><li>Bulkhead: Controls the concurrency of the execution.</li><li>Circuitbreaker: <a href="https://martinfowler.com/bliki/CircuitBreaker.html">Circuitbreaker pattern</a> based on the metrics of the execution.</li><li>Chaos (failure injection): Injects errors and latency on demand.</li></ul><h3>Why this library?</h3><p>Maybe you are wondering why another circuitbreaker library?</p><p>Well… this is not a circuit breaker library, is a toolkit to increase the resilience of your applications although it implements the circuit breaker pattern.</p><p>When looking out there I feel the lack of a toolkit that could adapt to my execution flows without using multiple libraries (one for retries, one for circuit breakers…) and createing glue code to wire everything, messing up my application with code out of my business logic.</p><p>Apart from that, I like the idea of creating chains of middlewares like the http.Handler style but for execution flows.</p><p>So… I implemented…</p><p>The way the library is designed, it can be extended and customized the reliability for each particular case, the responsibility of each resilience component is separated.</p><h3>Metrics</h3><p>All the Runner know how to measure their actions using a metrics recorder, at this moment only Prometheus recorder is implemented. Following the previous example, we could do this to have Prometheus metrics.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d1f16475cf0989711d6ba3e120f78f88/href">https://medium.com/media/d1f16475cf0989711d6ba3e120f78f88/href</a></iframe><p>Like you see is another Runner!</p><h3>Conclusion</h3><p>Hope you like the library and if you like it, start using it!</p><p>If you want you can go to <a href="https://github.com/slok/goresilience">Github</a> to give feedback about errors and pain points or if you like to contribute to improving the current Runners (or new ones), they will be welcome!</p><p>Thanks for reading.</p><p><strong>P.S.</strong> A <a href="https://medium.com/@slok/goresilience-adaptive-resilience-9500db789609">second article</a> has been published based on the next version of the library that implements adaptive resilience using concurrency limits</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=14d229aee385" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tracing meets Kubernetes controllers and operators]]></title>
            <link>https://medium.com/spotahome-product/tracing-meets-kubernetes-controllers-and-operators-2f1efe594e52?source=rss-13dde3ada38b------2</link>
            <guid isPermaLink="false">https://medium.com/p/2f1efe594e52</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[operators]]></category>
            <category><![CDATA[kooper]]></category>
            <category><![CDATA[controller]]></category>
            <category><![CDATA[opentracing]]></category>
            <dc:creator><![CDATA[Xabier Larrakoetxea]]></dc:creator>
            <pubDate>Wed, 23 May 2018 14:44:32 GMT</pubDate>
            <atom:updated>2018-05-23T14:44:32.021Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*s9x-GhMTBVs5ZQTExBWhuQ.png" /></figure><p>Some months ago we open sourced our library to create controllers and operators along the way Kooper has fixed bugs and add new features like Prometheus metrics or leader election. Today Kooper has add a new amazing feature to increase the observability in our controllers and operators… <strong>tracing!.</strong></p><h3>Tracing and Opentracing</h3><p>Like metrics or logs,traces add another way to get insights and debug how our distributed systems is working. To explain what tracing and traces are let’s use the explanation given by <a href="http://opentracing.io/">Opentracing</a>.</p><blockquote>At the highest level, a trace tells the story of a transaction or workflow as it propagates through a (potentially distributed) system. In OpenTracing, a trace is a directed acyclic graph (DAG) of “spans”: named, timed operations representing a contiguous segment of work in that trace.</blockquote><p>And what is <a href="http://opentracing.io/">Opentracing</a>?</p><blockquote>Vendor-neutral APIs and instrumentation for distributed tracing</blockquote><p>For more information about tracing and <a href="http://opentracing.io/">Opentracing</a> go <a href="http://opentracing.io/documentation/">here</a>.</p><h3>Kooper and tracing</h3><p>Tracing has been added using <a href="http://opentracing.io/">Opentracing</a> standard APIs, this means that the controllers implemented using Kooper can add tracing easily with this tracing standard and its <a href="https://github.com/opentracing/opentracing-go/">go client lib</a>.</p><p>Now controller handlers that use Kooper will receive a <a href="https://golang.org/pkg/context/#Context"><em>context.Context</em></a><em> </em>instance that has the parent span set, this way the the controller event handling code can continue it’s trace.</p><p>Kooper controller will create the root span when it starts handling the object adding important information like the object key itself so the trace can be propagated around the system and identify the trace to a particular object processing. It will also add a child span with the event kind (add or delete).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rCO1D8nxjOWnj8RXeW2S4A.png" /><figcaption>Controller object processing trace with error</figcaption></figure><p>The trace started by the controller has more information like the controller configuration, the number of processed times (retries made on a particular object processing after error), the final state of the handling of the object (failed or not)…</p><p>The spans created by the controller injects some logs during the handling of the object events.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G-Gxuw8BRkAaSHMWCBs2dw.png" /></figure><h3>Extending tracing on controllers</h3><p>Kooper started trace gives valuable insights, but the real value of tracing is when the <em>handlers </em>implementation of different controllers and operators continue the trace creating more spans from this parent spans.</p><p>To use it, the <a href="https://golang.org/pkg/context/#Context"><em>context.Context</em></a> object received from the handlers can be used to get the parent span to create child spans on our custom controllers. For example:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9e1d1307a5feec6c3ad5ade35d0ce2dd/href">https://medium.com/media/9e1d1307a5feec6c3ad5ade35d0ce2dd/href</a></iframe><p>A working <a href="https://github.com/spotahome/kooper/tree/master/examples/traced-controller">traced example</a> can be found on Kooper repository and also <a href="https://github.com/spotahome/kooper/blob/master/docs/tracing.md#example">how to</a> run it with a development instance of <a href="http://www.jaegertracing.io/">Jaeger</a>.</p><p>Here is a video of jaeger UI with the traces created by the <a href="https://github.com/spotahome/kooper/tree/master/examples/traced-controller">controller traced example.</a></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FAL3oftDV5ck%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DAL3oftDV5ck&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FAL3oftDV5ck%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/441178e60afea625635e097246f6bd49/href">https://medium.com/media/441178e60afea625635e097246f6bd49/href</a></iframe><p>As we always say, any doubts, feature request or Pull requests are very welcome on <a href="https://github.com/spotahome/kooper">Kooper Github project page</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2f1efe594e52" width="1" height="1" alt=""><hr><p><a href="https://medium.com/spotahome-product/tracing-meets-kubernetes-controllers-and-operators-2f1efe594e52">Tracing meets Kubernetes controllers and operators</a> was originally published in <a href="https://medium.com/spotahome-product">Spotahome Product</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>