<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Michael de Jong on Medium]]></title>
        <description><![CDATA[Stories by Michael de Jong on Medium]]></description>
        <link>https://medium.com/@michaeldejong?source=rss-36807c7a223d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*_MDYZ0cO2bQP0j8Z</url>
            <title>Stories by Michael de Jong on Medium</title>
            <link>https://medium.com/@michaeldejong?source=rss-36807c7a223d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 09:24:27 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@michaeldejong/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[There’s a lot less to container orchestration than you think]]></title>
            <link>https://medium.com/magnet-me-engineering/theres-a-lot-less-to-container-orchestration-than-you-think-ea3f45845422?source=rss-36807c7a223d------2</link>
            <guid isPermaLink="false">https://medium.com/p/ea3f45845422</guid>
            <category><![CDATA[continuous-deployment]]></category>
            <category><![CDATA[orchestration]]></category>
            <category><![CDATA[chatops]]></category>
            <category><![CDATA[scalability]]></category>
            <category><![CDATA[container-orchestration]]></category>
            <dc:creator><![CDATA[Michael de Jong]]></dc:creator>
            <pubDate>Mon, 13 Mar 2023 12:14:09 GMT</pubDate>
            <atom:updated>2023-03-13T12:14:09.344Z</atom:updated>
            <content:encoded><![CDATA[<h4>A story on how NOT moving to Kubernetes paid off for us…</h4><p>We recently celebrated Magnet.me’s 10 year anniversary. Meeting up with some of my former Software Engineering colleagues, made me reminisce about where we came from as a team, working with the tech and resources available to us back then. As I was doing that, I concluded that 2016 was a very interesting year for us backend engineers. Back then our product probably consisted of no more than 10 unique services. Life was a lot simpler.</p><p>In fact, only a few months prior, we first started using Docker in production as means to simplify how these services were run. Around that same time <a href="https://medium.com/magnet-me-engineering/how-we-built-our-service-mesh-with-consul-rxjava-zipkin-e00d5c2b4f7e">we rolled out Consul to facilitate our services to talk to each other without having to know on which server or IP/port each service was running</a>. But which service ran on which server(s), was ultimately still statically defined in our Puppet configuration and deployed by Puppet periodically as before.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Np8DJuP2dVBQ7KsMR4yHbA.jpeg" /><figcaption>View of the Erasmus bridge, a short walk from our office.</figcaption></figure><p>Back then it felt like literally hundreds of startups popped-up with a mission to orchestrate, deploy, and scale services across docker clusters. We just didn’t know which (or if any) of the many new Docker orchestrators were going to stick around long-term. Kubernetes was gaining popularity but felt really complex (even back then), Docker Swarm was really interesting — it was already part of the docker ecosystem — but didn’t look and feel ready for production environments just yet. Other products like Dokku, Flynn, Rancher, and others were on our radar as well, but had yet to prove themselves, or didn’t exactly do what we wanted. We wanted in on the container orchestration adoption — primarily to simplify our own deployments — but we really didn’t want to bet on software that might not be around anymore in a year or two.</p><p>I had to double-check, but it was all the way back in March 2016 when we launched an internal service called <strong>infra</strong>. Infra was a bit special because it started out as a bit of an experiment. The thought was along the lines of…</p><blockquote>If we don’t want to pick an orchestrator just yet, why don’t we just build the simplest version we can think of ourselves, and use that until we migrate to whichever orchestrator “wins” in the end.</blockquote><p>And so we did. <strong><em>But the interesting bit is that we never actually migrated to a different orchestrator afterwards...</em></strong> and for no other reason than that what we built — despite being very simplistic — worked so well for us that we never had the need to migrate away. Especially considering that the orchestrator that arguable “won” the industry’s favor turned out to be Kubernetes — infamous for its complexity.</p><h3>So, how does infra work?</h3><p>Our services run on around 15–20 servers serving purely as hosts for docker containers. Most of our services run anywhere from 2 to 5 instances somewhere on these host machines. Which service runs where exactly is still dictated by Puppet, but once a service is running Puppet’s job is done. From there on infra takes over.</p><ol><li>When we merge a change into our Git master branch, our CI-pipeline compiles, and tests the code, before finally building container images for the services affected by the change. These containers are labeled with the build number to describe the “version” of the container. These container images are then pushed to our internal Docker registry.</li><li>The Docker registry fires a webhook event to the infra service, letting infra know a new image is available. Infra checks if the new image has the now highest known version, and if so initiates a deployment for that service. (As a backup infra also periodically checks for updates, but webhooks make this near instant).</li><li>Infra then starts a thread in a special Slack channel where it regularly posts about the progress of the deployment as it progresses.</li><li>Then for each server running outdated containers for the affected service, infra pulls the new container image from the registry in parallel.</li><li>Once that’s done it will redeploy each instance on every affected server sequentially: making a copy of the old container’s configuration, gracefully terminating the old container, starting a new container with the new image and the old configuration, and waiting for all health checks in Consul to report healthy statuses once again.</li><li>Once all instances have been redeployed, infra notifies everyone in the Slack thread, that the deployment has completed. A full deployment of all instances of a service generally takes 2–3 minutes.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/1*KPhwEl8wCXOlsiCms75mNQ.png" /></figure><p>The upside to copying existing configuration from older containers is that we don’t need to configure anything in infra, “it just works” (tm). Need to change the config? Just update the config in Puppet.</p><h3>But wait there’s more…</h3><p>Now this already works perfectly fine in the background, but as a software engineer you’d not only want to know what infra is doing, but also be able to affect its operation. To do just that we’ve implemented a couple different Slack commands which infra listens for, and will execute. Here’s just a couple useful ones we’ve implemented:</p><ul><li><strong>/ops deploy [service name | all]</strong> will redeploy a specific service or all services. You can optionally specify the version you would like to deploy.</li><li><strong>/ops hold [service name | all] </strong>will prevent a specified or all services from being redeployed.</li><li><strong>/ops release [service name | all]</strong> will allow a specified or all services to be redeployed again.</li><li><strong>/ops queue</strong> to see which services are scheduled to be redeployed.</li><li><strong>/ops abort [service name]</strong> aborts the redeployment of the specified service immediately. Great for when a deployment gets stuck, or you run into an “oops” situation.</li></ul><h3>Monitoring</h3><p>But that’s not the only thing. We also added support for monitoring memory usage, garbage collection, and CPU usage, through Telegraf. This allows us to determine if a service instance is misbehaving, and if so simply terminate that particular docker container (graceful if possible), and simply start a brand new docker container with the same configuration. This too, is obviously reported back to the developers through Slack, so that they can investigate if this service requires some attention.</p><h3>But what about scaling?</h3><p>Now you might be thinking, “what about (auto-)scaling”. What if you need more instances of a particular service, what if you need more servers, what if there’s a big influx in users all of a sudden, etc? Well we don’t really have a need for such scalability. Most companies probably don’t actually. In fact it’s a lot cheaper for us to run a pool of Virtual Private Servers 24/7, than to elastically spinning machines up and down based on our workloads using Amazon AWS or Google Cloud.</p><p>That doesn’t mean we never scale things up or down though. When a service could benefit from having another instance run in production — typically because we want to have them handle more HTTP requests, or process tasks more quickly — we simply add one line to our Puppet config, telling Puppet where we want to run another instance, and commit that change with Git. Our puppetmaster automatically picks up on that change, and typically within minutes a new service instance will come online automatically. Likewise, if the current pool of servers is at or near capacity, we can simply provision a new server using cloud-init. Within a couple of minutes we’ll have a new server, and it will startup services we’ve assigned to it in the mean time. You probably don’t need auto-scaling!</p><h3>The takeaway</h3><p>If there’s anything you should take away from this story, it’s that sometimes, simple is (unexpectedly) good enough. Simple stuff is easy to maintain, run, and explain to others, giving you and your colleagues more time to work on the more important things, <a href="https://medium.com/magnet-me-engineering/building-a-new-mobile-magnet-me-35e90b307a4">like building a new mobile app</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ea3f45845422" width="1" height="1" alt=""><hr><p><a href="https://medium.com/magnet-me-engineering/theres-a-lot-less-to-container-orchestration-than-you-think-ea3f45845422">There’s a lot less to container orchestration than you think</a> was originally published in <a href="https://medium.com/magnet-me-engineering">Magnet.me Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How & Why we standardized our APIs]]></title>
            <link>https://medium.com/magnet-me-engineering/how-why-we-standardized-our-apis-3ca22f530af8?source=rss-36807c7a223d------2</link>
            <guid isPermaLink="false">https://medium.com/p/3ca22f530af8</guid>
            <category><![CDATA[api-management]]></category>
            <category><![CDATA[api-development]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[rest]]></category>
            <category><![CDATA[rest-api]]></category>
            <dc:creator><![CDATA[Michael de Jong]]></dc:creator>
            <pubDate>Mon, 04 Nov 2019 08:19:02 GMT</pubDate>
            <atom:updated>2019-11-06T16:01:35.350Z</atom:updated>
            <content:encoded><![CDATA[<p>At <a href="https://magnet.me/">Magnet.me</a> we’re currently running 40+ self-built services in production. As you’d expect most of these expose some sort of internal API to manage and manipulate the data they’re responsible for. And while microservices give great freedom to backend developers in choosing languages, libraries, and approaches, they should also take great care to design consistent APIs so they are easy to use, and appear consistent and coherent to its users. To achieve this at Magnet.me we follow a couple simple guidelines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*byCYvbINl7cWJM6b4rmHEQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@chuttersnap?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">chuttersnap</a> on <a href="https://unsplash.com/s/photos/container?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>Naming scheme</h3><p>One important thing to get right early on is to settle on a naming scheme for URLs. This way URLs become predictable, and for the API consumer it becomes less relevant to know with which service and naming scheme it’s dealing with. It’s not important which naming scheme you end up choosing, but it <em>IS</em> important to apply that scheme consistently throughout your services. In our case:</p><ul><li>We settled on kebab-casing URLs.</li><li>We use plurals to describe entities.</li><li>When referring to sub-entities we stick to the following pattern:<br>/entity/{id}/sub-entity/{id}</li></ul><p>Here are some examples of URLs we have in our services:</p><pre>GET /job-seekers/1<br>GET /organizations/2/opportunities<br>GET /conversations/3/messages/4</pre><p>Unfortunately not everything fits into the RESTful paradigm. For us the most common case where we have to deviate from it, is when we want to perform an action on an entity. Communicating this in a RESTful approach isn’t always possible or clear to the consumer. Some examples of these are how we perform searches, issue admin commands, or fetch statistics or aggregated summaries on specific entities. For all of these we prepend the name of the action with an underscore at the end of the URL. That way it’s clear to the API consumer that they’re performing an action on a specific entity, or on all entities.</p><p>Here are some example URLs which follow this pattern:</p><pre>POST /job-seekers/_search<br>POST /job-seekers/1/_reindex<br>GET /conversations/2/_summary</pre><h3>GET versus POST for searching</h3><p>You could argue that searching for specific entities, like users or conversations, should be done with a GET operation on the top-level entity path, where filters are provided through query parameters:</p><pre>GET /job-seekers?name=Michael<br>GET /conversations?query=Michael</pre><p>And that would make for a fine RESTful API approach. However, if you’re unlucky, and have to support a large number of options or filters for your search, your representation of that search can quickly exceed the size limitation of URLs. To deal with this, we expose more powerful searches on special action endpoints, exactly like Elasticsearch’s HTTP API does. With this approach we POST the search request as a JSON payload to a URL that ends in /_search .</p><pre>POST /job-seekers/_search <br>{ &quot;name&quot;: &quot;Michael&quot; }</pre><pre>POST /conversations/_search <br>{ &quot;query&quot;: &quot;Michael&quot; }</pre><p>You can still expose limited search capabilities on the top-level entity URL using GET in addition to the more powerful POST endpoint if you so desire. But since the POST endpoint tends to make the GET endpoint irrelevant, and having two search endpoints for the same entity can be confusing for an API consumer, we usually don’t expose both endpoints to the consumer.</p><h3>Search results</h3><p>Another little gem we borrowed from Elasticsearch’s API design, is exposing totals in search results. Whenever you’re performing a search through a GET or POST endpoint, we always return a JSON object with the following structure:</p><pre>{<br>  &quot;data&quot;: [<br>    { ... },<br>    { ... }<br>  ],<br>  <strong>&quot;totalSize&quot;: 192</strong><br>}</pre><p>This is really useful because in a lot of cases you don’t want to retrieve every matching entity with a search endpoint, but knowing how many entities matched is already enough. It’s also extremely helpful when it comes to displaying data in the UI. You might have retrieved 10 entities, but knowing there are 192 in total also means, you’ll have 20 pages worth of results. This means you can easily and dynamically update pagination every time you perform a search. It also eliminates the need for a /_count endpoint, since you can achieve the same result with the /_search endpoint.</p><h3>Every entity is addressable</h3><p>We’ve also made a habit of making every entity we expose over our API to be addressable. This means that besides exposing an id field for every entity, we also expose a url field describing the location of that entity within our backend services. We do this because you can’t expect an API consumer to just know where every entity lives — neither in which service, nor on what URL. Here’s what that looks like in our API:</p><pre>{<br>  &quot;id&quot;: 1,<br>  <strong>&quot;url&quot;: &quot;http://rest-api.services.magnet.me/users/1&quot;</strong>,<br>  &quot;firstName&quot;: &quot;Michael&quot;,<br>  ...<br>}</pre><p>In addition to this, if one entity refers to another we don’t just specify the id , we also specify the url field in a reference object. This clearly communicates to the API consumer where they can find the entity to which we are referring:</p><pre>{<br>  &quot;id&quot;: 1,<br>  &quot;url&quot;: &quot;http://messaging.services.magnet.me/conversations/1&quot;<br>  <strong>&quot;startedBy&quot;: {<br>    &quot;id&quot;: 1,<br>    &quot;url&quot;: &quot;http://rest-api.services.magnet.me/users/1&quot;<br>  }</strong>,<br>  ...<br>}</pre><h3>Limiting output</h3><p>Many years ago, if you requested information on a specific user from our API, that API would return a very fat model containing all the info we had on that user. But as an API consumer, you’re not always interested in all of that information. As an example, in case of our “user” entities we very often only want to fetch the name and avatar of a user to display those in the UI, but we need nothing else. If our API returned a very fat model for every such request it would put tremendous strain on our network and backend services, not to mention the user’s network connection and device.</p><p>To solve this we borrowed another technique from Elasticsearch. Both allow you to specify which fields you’re interested in, and which ones can be omitted from the response. So if you’re interested only in the name and avatar of a user, as a consumer you can specify that using query parameters, and the API will only return that data:</p><pre>GET /users/1?include=firstName,lastName,avatar.templateUrl</pre><pre>{<br>  &quot;firstName&quot;: &quot;Michael&quot;,<br>  &quot;lastName&quot;: &quot;de Jong&quot;,<br>  &quot;avatar&quot;: {<br>    &quot;templateUrl&quot;: &quot;http://images.magnet.me/x_{width}_{height}.png&quot;<br>  }<br>}</pre><h4>But wait there’s more!</h4><p>Not long after we introduced this method of limiting what data is returned over the API, we extended the use of the include parameter to also retrieve and return data for referenced entities (even when that data is managed and served by another service). We call this mechanism <em>API Resolving</em>, but you don’t need to remember that. For example, let’s say that you want to show the 3 most recent conversations in the UI. You might send a request like this to the messaging service:</p><pre>POST /conversations/_search <br>{<br>  &quot;limit&quot;: 3,<br>  &quot;sortField&quot;: &quot;timestamp&quot;,<br>  &quot;sortOrder&quot;: &quot;DESC&quot;<br>}</pre><p>Which would result in a pretty slow response with very fat representations of the top 3 conversations for that user:</p><pre>{<br>  &quot;data&quot;: [<br>    {<br>      &quot;id&quot;: 1,<br>      &quot;url&quot;: &quot;http://messaging.services.magnet.me/conversations/1&quot;,<br>      &quot;startedBy&quot;: {<br>        &quot;id&quot;: 1,<br>        &quot;url&quot;: &quot;http://rest-api.services.magnet.me/users/1&quot;<br>      },<br>      ...lots of data...<br>    },<br>    { ...lots of data... },<br>    { ...lots of data... }<br>  ],<br>  &quot;totalSize&quot;: 21<br>}</pre><p>But now let’s say you only want to display the names and avatars of the users that started these conversations. Normally you’d first have to do the request above, and then send off 3 additional requests to find the names and avatars of each conversation starter that the first request yielded. This is not ideal, since the network distance between the browser and our load-balancer becomes a deciding factor in how long this is going to take. This is out of our control, and if the user has a slow or unreliable connection it could take very long for this data to load.</p><p>Instead our APIs also allow you to specify which fields you desire to load from entities that are referred to in the response. For instance, the startedBy field in the previous response is a reference field, referring to a user entity which resides in another service. But if you add startedBy.firstName to the include query parameter, our backend will actually fetch that data for you before returning the result of your request to you. This allows you to do a lot of work in one single request:</p><pre>POST /conversations/_search?include=startedBy.firstName, startedBy.lastName,startedBy.avatar.templateUrl<br>{<br>  &quot;limit&quot;: 3,<br>  &quot;sortField&quot;: &quot;timestamp&quot;,<br>  &quot;sortOrder&quot;: &quot;DESC&quot;<br>}</pre><p>Which would return the following response:</p><pre>{<br>  &quot;data&quot;: [<br>    {<br>      &quot;startedBy&quot;: {<br>        &quot;firstName&quot;: &quot;Michael&quot;,<br>        &quot;lastName&quot;: &quot;de Jong&quot;,<br>        &quot;avatar&quot;: {<br>          &quot;templateUrl&quot;: &quot;http://...&quot;<br>        }<br>      }<br>    },<br>    { ... },<br>    { ... }<br>  ],<br>  &quot;totalSize&quot;: 21<br>}</pre><p>This is a pretty neat feature, but is definitely not the first thing that should be on your list of todos when standardizing your API. It just shows you what sort of features are possible when you start standardizing the APIs of your services.</p><h3>Conclusion</h3><p>I hope I’ve helped you see the value of implementing some of these rules, and how following a set of rules or guidelines can make (micro)services feel a little less like the wild wild west — where everybody just does what they want, and a little more like an organized and understandable structure where everybody can easily add value to. Hopefully we’ve even given you some ideas for setting some guidelines of your own when designing your own APIs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3ca22f530af8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/magnet-me-engineering/how-why-we-standardized-our-apis-3ca22f530af8">How &amp; Why we standardized our APIs</a> was originally published in <a href="https://medium.com/magnet-me-engineering">Magnet.me Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How we built our service mesh with Consul, RxJava & Zipkin]]></title>
            <link>https://medium.com/magnet-me-engineering/how-we-built-our-service-mesh-with-consul-rxjava-zipkin-e00d5c2b4f7e?source=rss-36807c7a223d------2</link>
            <guid isPermaLink="false">https://medium.com/p/e00d5c2b4f7e</guid>
            <category><![CDATA[zipkin]]></category>
            <category><![CDATA[service-mesh]]></category>
            <category><![CDATA[service-discovery]]></category>
            <category><![CDATA[rxjava]]></category>
            <category><![CDATA[consul]]></category>
            <dc:creator><![CDATA[Michael de Jong]]></dc:creator>
            <pubDate>Fri, 25 Oct 2019 09:01:01 GMT</pubDate>
            <atom:updated>2019-11-03T22:03:09.778Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1wEBbkhrTwwTRsJPOu_c4A.jpeg" /></figure><h3>Let’s go back to 2015</h3><p>Back in 2015 the microservices hype was really kicking off. Back then Magnet.me consisted of 3 separate services:</p><ul><li>rest-api : our monolith backend which exposed a RESTful API</li><li>web-frontier : our user-facing service serving up an AngularJS app</li><li>oauth : our authentication and authorisation backend</li></ul><p>The concept of microservices promised us more flexibility: scaling up individual components of our backend, the ability to try out different programming languages, libraries, and approaches, while also limiting the blast radius of outages in case of failures. But it would also mean a far more complex infrastructure, having to run, monitor, and automate general management of a multitude of services, as well as way of getting services to talk to each other somehow. After a short period of prototyping we ultimately settled on the approach covered in this post.</p><h3>Consul</h3><p>In 2015 we opted to use <a href="https://consul.io/">Consul</a> to manage <em>Service Discovery,</em> and handle the <a href="https://medium.com/magnet-me-engineering/configuration-management-with-consul-a36cfa1252ea">configuration</a> for our (micro)services in one centralized place. Back then there weren’t that many options for <em>Service Discovery</em>, and seemingly every platform and tool had a different approach. It’s worth noting that this was before Kubernetes, Docker swarm, etc really took off, and it really wasn’t clear which one was going to “win” in this space. We also didn’t really want to buy into a specific tool, and potentially end up depending on a tool that was losing community support in the long-term. Instead what we wanted was to pick the best approach for us, and find the best tool to support that approach. For us this was Consul.</p><p>Consul makes <em>Service Discovery </em>very easy. It even exposes a DNS service to your cluster which your own services can use to find each other. In this setup you’d simply query that DNS service to get a location (IP address + port) for the service you’re trying to contact. However, for reasons that will become more apparent later on, we decided to go a slightly different route.</p><h3>What were our options?</h3><p>Back then there were basically 3 options we could pick from: Relying on DNS to locate other services, using a load-balancer with dynamic routing to direct traffic to a specific instance, or using a sidecar container which magically handles the network for you (typically these came along with a container orchestration tool).</p><p>It’s important to note, that each service either registers itself somewhere on start-up, or that it’s automatically registered by the container orchestration tool when the service is started.</p><h4>Relying on DNS</h4><p>With this option, you’re using a custom DNS service to keep an actively updated view of where every instance of every service is running. Services can either lookup addresses with a DNS client, or simply configure their runtime to use the DNS service as the default nameserver. In that case if the front-end service wanted to retrieve information about a specific user from the backend, the front-end service would simply ask therest-api service by sending a HTTP request to http://rest-api.local.consul/users/1 or something similar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Lxu47rDS1_W1wsmDlsFJRQ.jpeg" /></figure><h4>Using a load-balancer</h4><p>One alternative is using a centralized load-balancer of which every service knows the location, to redirect requests to one of the instances of the targeted service. The configuration of the load-balancer is actively updated as services get registered and deregistered from the cluster. Using this approach you’d simply send a HTTP request like <a href="http://rest-api.local.consul/users/1">http://rest-api.services.magnet.me/users/1</a> and the load-balancer would send it to an instance of rest-api.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZgLrS7VyNbehxq1utqJZSA.jpeg" /></figure><h4>Using a the sidecar-pattern to handle networking</h4><p>This one felt a bit like black magic, and leverages container isolation. If your service runs in a container, with this pattern your service is intentionally kept isolated from the rest of the network. Instead it can only talk to one other container called <em>the sidecar</em>. This container receives HTTP requests from your service, and then directs them through the cluster, relaying back responses it receives.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8sGAoFrnvqXx99o3GyCQ2A.jpeg" /></figure><h4>Yeah, that’s no bueno…</h4><p>We didn’t really like any of these options. Primarily because all of these options try to abstract away all the details and all of the control from your service. Your service has no control over which specific instance of the service gets its request, only the guarantee that <em>some</em> instance will receive it. Where that instance is located, or what the state of that instance is, is entirely hidden from the service, while there’s no real reason to do so.</p><p>At the same time we were inspired by Netflix’s approach to microservices using <a href="https://github.com/Netflix/Hystrix">Hysterix</a>. Hysterix is a library that allows services to services to talk to each other in a distributed environment with fault-tolerance built in. Requests get automatically retried on failures, and if downstream services get overloaded or become unresponsive, circuit-breakers prevent requests from being sent to the affected services, allowing them to recover.</p><p>We really liked this approach but it also takes a considered approach to communicating this way. Not only do you need to define what your service should do if a request cannot be executed within a timely fashion, but also how it can find and talk to several other instances autonomously.</p><h4>Our approach</h4><p>Our services register themselves with a local Consul agent when they start up, and automatically deregister themselves when they terminate. In addition to this a service exposes its health over a HTTP endpoint, which it instructs Consul to check periodically. To set this up, we use our own open-sourced library called <a href="https://github.com/Magnet.me/consultant">Consultant</a>.</p><p>When we want to make a request to another service we do the following:</p><ol><li>We use Consultant to ask Consul to give us a list of instances that the local Consul agent is aware of.</li><li>Since Consul keeps track of the health of a service for us, we ask it filter out any unhealthy or unresponsive instances, and sort the list of instances by network distance (from closest by to furthest away).</li><li>We then randomly select and remove one instance from the list (heavily biased to closer by instances). This acts as a bit of a round-robin effect, ensuring every instance gets a similar workload, while still avoiding services which would take longer to talk to.</li><li>We then send off our request to that instance. And if that request is successful, that’s basically the end of the story.</li><li>However, if we receive a 503 Service Unavailable, we’ll know that this instance is no longer accessible or capable of handling requests. We then select another instance from that same list (excluding instances we already tried), and resend the request to that instance. It’s important to note that this is only possible because we have the ability to locate all these instances, and we have control over which instance we contact. If we didn’t, there would be a significant chance we’d be sending requests to instances we already tried sending this request to before, but failed .</li><li>We repeat this last step until either the request succeeds, a pre-specified timeout is exceeded, or if we tried all of the listed instances. In these cases, it’s up to the service performing the request to determine what happens, and in most cases the error propagated back to the upstream service which made the request to our service, which in turn may decide to retry the request by itself.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0oCGlJUsTxsUmnfOE7Bp5Q.jpeg" /><figcaption>Example of how our setup would recover from a downstream request that fails, and is subsequently retried on another instance.</figcaption></figure><p>To achieve all of this we wrote our own wrapper around async-http-client which uses Consultant to talk to Consul, and uses RxJava internally for sending off the request and retrying the request if it couldn’t send it to the selected instance of the targeted service. This allows us to perform HTTP requests in a familiar way, while performing all of the service locating, and retrying on a lower level. We already tend to write tiny clients for most of our microservices in our most predominantly used programming language in the backend, so this makes it really easy to call any services from almost any other service.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/31eb3199cedd5140365cd647bcedd227/href">https://medium.com/media/31eb3199cedd5140365cd647bcedd227/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/15616de9846aca878d9c6a1fd8fdc524/href">https://medium.com/media/15616de9846aca878d9c6a1fd8fdc524/href</a></iframe><h3>But we can do better…</h3><p>This only retries requests if they failed with a 503 Service Unavailable status code. What if we want to retry on an arbitrary error code, or an exception being thrown because of some entity could not be found, or networking issues throwing IOExceptions? For this we implemented an ExponentialBackoff class which will reattempt whatever Single or Observable stream it is attached to, when a problem occurs. On which errors it retries, as well as how often and how fast can all be configured.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d02d21be42555e2fdc92d29a42a41cea/href">https://medium.com/media/d02d21be42555e2fdc92d29a42a41cea/href</a></iframe><p>In a distributed environment that means that you’ve drastically improved your chances of successfully executing a request, even if the network is having an off day. But this brings us to another problem: big fan-outs with retried requests.</p><h3>Observability</h3><p>When you take the microservices route, you’ll quickly end up running a big variety of services — all preferably with multiple instances. But when you grow the number of services and instances over time, you’ll eventually end up with a situation where a user request will arrive in your brokering service — sometimes also called a <a href="https://samnewman.io/patterns/architectural/bff/">BFF</a> — , which then needs to contact a variety of services, sometimes even in a nested fashion. We’ll discuss the details on how we do this at Magnet.me in another post. But as you can imagine it’s important that you build some kind of logging mechanism for internal requests belonging to the same user request. If not you’ll never know why a certain request is timing out, or returns a generic 500 Server Error .</p><p>To deal with this in our stack, we chose <a href="https://zipkin.io/">Zipkin</a> a couple of years ago. This means that when a user request enters our infrastructure, it’s assigned a unique trace ID. This trace ID is passed along through every internal request to downstream services. Every internal client, and internal service, registers their outgoing and incoming requests with that trace ID to Zipkin. This allows Zipkin to know which internal requests belong to which user request, and produces an easy to read interactive graph, allowing you to easily pinpoint any issues for any given user request.</p><p>Although you can search for a specific trace in Zipkin’s UI fairly easy, we also expose the trace ID in a HTTP header when answering the user request. That way it’s easier to look that specific trace up, if you come across an anomaly in the logs, or in your own browser.</p><p>We have since also started tracing operations on our Elasticsearch cluster as well as operations on Redis instances. This can give a very detailed view of what actually happens when a user request is processed. In turn this may also lead you to the understanding that some fan-outs have simply gotten too big, and may need some optimization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cJFwWY9zjeP0YQm-ygUk1g.png" /><figcaption>Example of a Zipkin trace showing how the user request fanned out to multiple internal requests between our microservices. It’s also showing which requests/operations were slow relatively speaking.</figcaption></figure><h3>Conclusion</h3><p>We hope this gave you a nice peek into some of the core principles that shaped our stack and infrastructure over the years. Although the bulk of this work was done in 2015 and 2016, much of it is still very relevant, and we see no reason to change anything.</p><blockquote>If we had to build Magnet.me from scratch today, we might not pick the exact same tools, but the principles we followed are still sound and we’d probably follow those again.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e00d5c2b4f7e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/magnet-me-engineering/how-we-built-our-service-mesh-with-consul-rxjava-zipkin-e00d5c2b4f7e">How we built our service mesh with Consul, RxJava &amp; Zipkin</a> was originally published in <a href="https://medium.com/magnet-me-engineering">Magnet.me Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Configuration management with Consul]]></title>
            <link>https://medium.com/magnet-me-engineering/configuration-management-with-consul-a36cfa1252ea?source=rss-36807c7a223d------2</link>
            <guid isPermaLink="false">https://medium.com/p/a36cfa1252ea</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[consul]]></category>
            <category><![CDATA[configuration-management]]></category>
            <dc:creator><![CDATA[Michael de Jong]]></dc:creator>
            <pubDate>Sat, 12 Oct 2019 20:41:02 GMT</pubDate>
            <atom:updated>2019-10-13T18:49:30.776Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NC22-mZewMcc7nFaMQBipg.png" /></figure><p>Microservices are <strong><em>hot</em></strong>, and there are many challenges you’ll face when adopting this way of building software. In this post we’ll tackle the challenge of “configuration”. In particular how to manage the configuration of many different services located in a cluster where services are running in different datacenters, on a multitude of host machines or containers, and potentially different roles. Using <a href="http://github.com/Magnetme/consultant">Consultant</a> — <em>an open-sourced Magnet.me project</em> — we’ll show you how to load your service’s configuration with zero effort and stay subscribed to any change in near real-time.</p><h3>Consul</h3><p>We’ve been using <a href="https://consul.io/">Consul</a> for years now at Magnet.me to manage service discovery, routing inter-service requests, and configuration for our (micro)services. For those that don’t know; Consul is a service developed by <a href="https://hashicorp.com/">HashiCorp</a> which you can run in your cluster. Consul uses <a href="https://serfdom.io/">Serf</a> to gossip between multiple running instances of Consul to share the location and state of services running in the cluster. In addition to this, Consul also provides a key/value store which allows you to store configuration and other meta-data for the operation of your services. Consul is also datacenter aware, meaning that the you can not only search for the locations of a particular service in the entire cluster, but also limit that search to a particular datacenter to avoid sending requests across the globe if it can be avoided. We’ll save the service discovery awesome-ness for another post. Let’s focus on configuration!</p><h3>Configuration</h3><p>Configuration is an integral part of many services. If your service needs to talk to a database, it’s likely you’ll want to store the username, password, database name, and perhaps even the location of the database server if you’re not using service discovery. These settings are typically static, and don’t change while the service is running. There are also settings which are more dynamic and can change while a service is running, like settings regarding the number of workers for a certain task, or a service-based feature flag. All these settings are typically stored inside one or more configuration files which the service reads on startup. If the service supports dynamic configurations, typically the service watches these configuration files for changes, and reloads the configuration from disk when a change is detected.</p><p>The problem is managing all this configuration for every instance of the service in the entire cluster. For instance, what do you need to do if you want to alter the config of all service instances in a particular datacenter, running on a particular host, or with a particular role? With microservices this can be a bit of challenge as it’s likely the service instances might be running just about anywhere in your cluster. The solution to this problem is to make your configuration available from a service such as Consul, and make your services look their own configuration up depending on their location and role in the cluster.</p><h3>Integrating Consultant with a service</h3><p>Assuming you’ve already got Consul set up for your cluster, let’s look at how we can integrate Consultant into your JVM-based services, and let them retrieve their own configuration from Consul. Consultant was designed to be extremely simple to use, and to integrate with existing services. Below you’ll see how we can set up Consultant for a billing service, and retrieve a Properties object from it:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/727bd5c5f36940a6a20ba3c201197814/href">https://medium.com/media/727bd5c5f36940a6a20ba3c201197814/href</a></iframe><p>It’s highly recommended to provide environment variables specifying the datacenter, host, and role of this particular service instance. In case of our billing service we’d want to specify the following environment variables:</p><pre>SERVICE_DC=eu-central <br>SERVICE_HOST=docker12.internal.magnet.me</pre><p>This way this instance of the billing service will know it’s running in our EU datacenter, and that it’s running on a specific host machine.</p><p>Consultant will instantly retrieve the configuration applicable for the billing service (including configuration that’s specific to the eu-central datacenter, and the docker12.internal.magnet.me host machine).</p><h3>Subscribing to changes</h3><p>But that’s not all Consultant does. Should we change any of settings in Consul’s key/value store, Consultant is immediately notified of this, and will update the Properties object. Should you wish to validate any configuration changes, then you can do that too:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cc1b266cd08531ea8efce5ec4c31a304/href">https://medium.com/media/cc1b266cd08531ea8efce5ec4c31a304/href</a></iframe><p>This will ensure that every time Consultant is notified of changes to the configuration, it will first validate the entire configuration before exposing it to your service. Should the validation fail, the Properties object will not be updated. This means that your service will remain unaware of these changes in its configuration until they change again and pass the validation.</p><p>Should you wish to be notified of when the service’s configuration is successfully changed you can also subscribe to these changes:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/124a145181cbbb0e80c706b32c6a1e6a/href">https://medium.com/media/124a145181cbbb0e80c706b32c6a1e6a/href</a></iframe><p>Or if you’d like to be notified of a particular setting being changed:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/31f6d8b4c00974e78213f1a6443924c4/href">https://medium.com/media/31f6d8b4c00974e78213f1a6443924c4/href</a></iframe><p>These listeners will be called every time Consultant is able to retrieve a configuration from Consul’s key/value store which passes the validation — <em>if you’ve specified any validation checks.</em></p><h3>Storing the config in Consul’s key/value store</h3><p>Now that we can have our services retrieve their configuration from Consul’s key/value store, it’s time to see how we can store our settings in there. The easiest way to do this, is to manually add the configuration through Consul’s web UI. Simply by adding a new entry with a key matching the pattern: config/&lt;service_name&gt;/&lt;setting_name&gt; you can define a setting with that particular name, for that particular service.</p><p>If you wish to define a setting which overrides the default value for a particular datacenter, host, or instance you can do so by defining a selector block. In that case the key should follow one of these patterns:</p><pre>config/&lt;service_name&gt;/[dc=&lt;datacenter_name&gt;].&lt;setting_name&gt; config/&lt;service_name&gt;/[host=&lt;host_name&gt;].&lt;setting_name&gt; config/&lt;service_name&gt;/[instance=&lt;instance_name/role&gt;].&lt;setting_name&gt;</pre><p>Note that you can even mix and match these to define even more specific groups such as:</p><pre>config/billing/[dc=eu-central,host=docker12.internal.magnet.me].&lt;setting_name&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jy-i-xQhzDGRVezV4_GTvw.png" /><figcaption>Consul’s web UI showing some of the configuration of our OAuth service</figcaption></figure><h3>Summary</h3><p>Using Consul and Consultant, Magnet.me is able to have each service retrieve its own configuration from Consul. By subscribing to changes in that configuration, we are able to alter the behaviour of our services at runtime, tweak the performance of our services at runtime, and even run the same configuration in development environments as we do in production environments. We hope you’ve enjoyed this post, and that you’ll give Consul and Consultant a try! Consultant is available from <a href="http://search.maven.org/#search%7Cga%7C1%7Cme.magnet.consultant">Maven’s Central Repository</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a36cfa1252ea" width="1" height="1" alt=""><hr><p><a href="https://medium.com/magnet-me-engineering/configuration-management-with-consul-a36cfa1252ea">Configuration management with Consul</a> was originally published in <a href="https://medium.com/magnet-me-engineering">Magnet.me Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>