<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Rastko Vukasinovic on Medium]]></title>
        <description><![CDATA[Stories by Rastko Vukasinovic on Medium]]></description>
        <link>https://medium.com/@metaphorical?source=rss-d0d24ba520a9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Yfini-XE4BRxC0Fie4k8Jw.jpeg</url>
            <title>Stories by Rastko Vukasinovic on Medium</title>
            <link>https://medium.com/@metaphorical?source=rss-d0d24ba520a9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 07:07:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@metaphorical/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Dynamic configuration management and circuit breakers in Kubernetes]]></title>
            <link>https://medium.com/adidoescode/dynamic-configuration-management-and-circuit-breakers-in-kubernetes-e75ccc832b5b?source=rss-d0d24ba520a9------2</link>
            <guid isPermaLink="false">https://medium.com/p/e75ccc832b5b</guid>
            <dc:creator><![CDATA[Rastko Vukasinovic]]></dc:creator>
            <pubDate>Sat, 08 May 2021 22:38:28 GMT</pubDate>
            <atom:updated>2021-05-08T22:38:28.205Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><em>Building scalable dynamic configuration control and circuit breaker system for microservices deployed to Kubernetes.</em></blockquote><h3>Capability to scale — inability to control</h3><p>Cloud-Native comes with high scale, portability, standardization, and modularity. For that, it is being treated almost like a silver bullet, which proves wrong more often than you would think.</p><p>Even something of the cloud-native pedigree at the level of cloud-native microservice deployed to Kubernetes comes with a tradeoff.</p><h3>Statelessness</h3><p>One of the core characteristics of cloud-native microservices is statelessness, it is one of the core architectural concepts that allows it to scale horizontally to any level. If it does not contain any dynamic internal state, traffic can be load balanced to any instance, even globally between datacenters. This makes it really easy to horizontally scale. It is what made container orchestration at scale possible on this level.</p><h3>The tradeoff</h3><p>It also makes us jump the hoops if we want to dynamically manage the state of our service. The state of our service is minimal, and, same as the service itself, distributed to a multitude of small encapsulated identical instances, statelessness in service of immutability. If you then say that we are talking about microservice deployed to more than one data center globally, we got the full scale of the problem.</p><h3>Use Cases</h3><p>There are two core use cases for dynamic state management in container orchestration (Kubernetes):</p><ul><li>Dynamic configuration management</li><li>Circuit breaker</li></ul><h3>General config control</h3><p>Controlling the configuration on the application level, or maybe some other structural level like sub-application, market, Kubernetes cluster (as in the environment) or Kubernetes namespace is the core use case here. It covers things like:</p><ul><li>Feature flags</li><li>Integration point configuration (URLs, api keys, etc)</li><li>Encryption keys</li><li>etc</li></ul><p>Full spec would probably include the dashboard which allows overview and control of system configuration. Ideally, it should be flexible in format to a certain extent.</p><h3>Circuit breaker</h3><p>A tool that controls the flag switching something on and off in your system based on external factors is called Circuit Breaker. The external factor is usually a high error rate or some other type of service quality deterioration. Additionally, flags can control some other type of system functionality regulation.</p><p>The circuit breaker is configuration control in the sense that you are controlling a flag. It is obviously simpler in terms of config format.</p><p>On the other hand, the application needs to be programmatic and its interface should allow control through some kind of trigger, most usually system alert of some sort (Grafana or Prometheus alert triggering a webhook?).</p><h3>The problem</h3><p>Our system is a microservice deployed to Kubernetes in multiple clusters, which makes it</p><ul><li>Containerized</li><li>Stateless — means deployed with a certain state and unchanged by request passing through it</li><li>Disposable — can live for 2 seconds or 2 days (rarely even more) and be replaced (or not) by the new one, quite usually identical</li><li>Highly distributed</li></ul><h3>Part 1 — distribution of state</h3><p>Our service is not a box that would connect to the database and then just pull the configuration — it is a multitude of smaller boxes, potentially existing in different Kubernetes clusters or different Kubernetes namespaces that all need to be in the same state always.</p><p>Having different pods in a different state would mean that the app behaves differently depending on where the request goes. That ultimately means an uneven experience for the consumers.</p><p>Session stickiness, or keeping any additional state on ingress level is a bad and really rare practice. And, realistically, we want the change to be near real-time, not wait for the database read cycle.</p><p>So, in summary — <strong>we need highly available, low latency, and scalable solution(s) that will allow us to, push the state change to all of our pods globally.</strong></p><h3>Part 2 — control of the state</h3><p>Based on use cases we need the state controllable in both manual and programmatic manner.</p><p><strong>This implies both an API and UI that allow centralized control of the system configurations at scale.</strong></p><h3>Building the solution — case study</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/951/0*7YX7BIE7FVuNkR1O.jpg" /></figure><p>This is our microservices, deployed on 4 cloud-based Kubernetes clusters across the world.</p><p>All the pods are identical, they are deployed from the same container that is hosted in a container repo somewhere in the world (most probably in the distributed repository that allows it to be downloaded from three closest locations)</p><p>Each cluster identifies itself to the pods through environmental variable which is then used to switch between deployed config sets (like integrated service endpoints, api keys, some other specificities).</p><blockquote><strong><em>Clarification 1:</em></strong><em> We can not solve this problem by using the env vars — to change the env var you got to restart the pod, this is the reality of Kubernetes at the moment.</em></blockquote><p>We are looking for a way to live distribute the state of the pods on the level of the higher structural entity in K8s.</p><p><strong>So, as the first part of our solution, we want to be able to change something in the namespace and it reflects on all the pods of the same type deployed into that namespace. Then need to have this scale across the clusters.</strong></p><h3>Changing the state</h3><p>Looking at the problem, we see that inside the cluster storage part the solution would be some kind of storage that allows us to observe it for changes and receive the change as part of the change event.</p><blockquote><em>Now that we know the direction we want to go, I will just skip the research step for the sake of the length of this article and jump straight to what I use.</em></blockquote><h3>The Key-Value store</h3><p>Since I am a big fan and consumer of many <a href="https://www.hashicorp.com/">Hashicorp</a> products, <a href="https://www.consul.io/">Consul</a> is already in my architecture repository — this time we are using it from KV aspect.</p><p>Consul is a cloud and Kubernetes native, scalable, and multipurpose solution that, amongst <a href="https://www.consul.io/docs/intro">other things</a>, has a key-value store that allows you to control a flexible set of values and watch for changes on them. Dynamic configuration and feature flagging are listed as primary (ideas) use cases for Consul KV Store on its website.</p><p><strong>So, the plan is to use Consul deployed in our namespace directly to read values.</strong></p><p>It is generally easy to connect to Console from your microservice and watch for changes. Most of the major programming languages used for backend development have <a href="https://www.npmjs.com/package/consul">libraries to handle this</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/0*vEzaNHackGAVwbc7.jpg" /></figure><p><strong>In the picture </strong>— We have Consul deployed inside our k8s namespace(s) and pods in our deployment are connected to Consul (from the microservice code).</p><p>Now, things left for us to solve are:</p><ul><li>How do we add and change stuff in the consul?</li><li>How do we distribute changes globally?</li><li>How can we make changes to config programmatically (remember circuit breaker case)?</li></ul><blockquote><strong><em>Clarification 2:</em></strong><em> Consul </em><strong><em>does</em></strong><em> have a web interface and HTTP API which we can use, but it is rather a foundation than a solution for our case. The web interface can be used to change the configuration in one k8s namespace in one k8s cluster, and HTTP API allows the same thing through the API.</em></blockquote><h3>Dynamic (re)configuration</h3><p>Planning how to control the content stored in <strong><em>Consul KV</em></strong> requires considering one more aspect of the problem — the level of distribution.</p><p>We have multiple (in our case 4) clusters with k8s namespaces in which our microservice (and consul) is deployed. We also want to plan for the growth across this dimension (more k8s clusters, more k8s namespaces).</p><p>We solve this by having a central, single source of truth, a database to store configurations. The idea is to utilize Consul API, build a microservice with an API for it, and build a UI to use this API.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/934/0*-CVp8jhb1ixIftVi.jpg" /></figure><p><strong>Right?</strong></p><p><strong>Wrong!</strong> In this case, API is redundant since we could have just used Consul UI. But, it is also not scalable — the only way to scale this is to deploy another one like that for every cluster and change it manually as you would anyway.</p><h3>Final design — Scaling configuration</h3><p>In order to scale configuration distribution, we need to build up the architecture a bit. We want to make our configuration control able ot scale across multiple k8s clusters and multiple k8s namespaces.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Lq_dWcb7jz7hKmpd.jpg" /></figure><p>Above you see the final design for the scalable dynamic configuration control system.</p><h3>Single source of truth — UI + API + Storage</h3><p>Here we introduced the UI-API-Storage system that is the capability center for configuration management and the place where all the configuration is distributed to clusters. This is a component that is providing storage for any new namespace Consul to fetch configuration from. Generally, it does not matter where it is deployed, it will probably be in one of the existing clusters.</p><blockquote><strong><em>Clarification 2.1:</em></strong><em> Yes, theoretically, we could have chosen single central storage that every pod around the world just listens to, even the S3 bucket would do in a certain way of application. This would, however, compromise the availability of the storage to newly created/restarted pods and with that — integrity of the system. Highly available, high performance, in-cluster key-value store as storage proxy ensures the availability of configuration and better resilience of the system using it.</em></blockquote><p>Additionally API gives us the flexibility of being able to do programmatic changes through it. This will be handy for the circuit breaker extension when we get to that.</p><p>Do not worry about the geographical proximity of this component, we are making sure that latency does not make too much difference, and we are building it in a way that will gracefully handle networking issues.</p><p><strong>UI </strong>— not too much to say here, set of forms and potentially different dashboards utilizing the API component in order to make configuration easier to manage.</p><p><strong>API</strong> will carry most of the functionality, accepting configuration CRUD and passing it to the storage in a proper way. API should also be the one handling Auth part — <strong>this system is essential security-wise.</strong></p><p><strong>Storage</strong> — it can be anything, really. We are not expecting it t grow too big, data structures are not too complex (the most complex case would be feature flags), so there is no need to prescribe any particular storage. Wherever life takes you cloud-wise. <strong>One good to have thing is the possibility to subscribe to events on objects or similar, this is super handy for this use case.</strong></p><h3>Config service — in cluster/namespace reconciliation</h3><p>Depending on if you opted for the storage that is broadcasting change events (and if you want to use that functionality) this can be a microservice or a cron job.</p><p><strong>Microservice</strong> would be subscribed to events coming from the storage, and every time it receives a change event on the storage it would pull the configuration object from the central storage and deploy it to the consul.</p><blockquote><strong><em>Clarification 3:</em></strong><em> In some cases, the change event contains an object that represents what changed, so theoretically you can use this to deploy the config diff. However, this approach goes against the resilience of the system in the sense that we are assuming that all the change events have reached all the clusters all the time. Assume that nothing will reach everything all the time.</em></blockquote><p><strong>Cron job</strong> would be set to a low interval (in seconds, minutes in some cases) and would just pull configuration object from storage and deploy it to Consul.</p><h3>The resilience of the config management</h3><p>The first precondition — nothing should be originally empty — everything should be equipped with defaults.</p><p>For the resilience considerations, let’s take a look from the perspective of one of the services we are configuring:</p><ul><li>Service has its own configuration and will change the configuration on consul connection and every time it observes a change in consul. If it receives nothing from the consul, or it does not connect to the consul it will have its original (default) configuration.</li><li>Consul has a default configuration or the configuration from the last time it was updated by the config service.</li><li>Config service will always pull the full config from the central storage and store it in Consul, if there is no connection nothing happens.</li><li>Central service should be done the same as everything else, with cloud-native best practices in mind, and according to internal or some good external architectural guidelines — for example, <a href="https://aws.amazon.com/architecture/well-architected/?wa-lens-whitepapers.sort-by=item.additionalFields.sortDate&amp;wa-lens-whitepapers.sort-order=desc">AWS well-architected framework</a> or <a href="https://cloud.google.com/architecture/framework">google cloud architecture framework</a> or similar.</li></ul><blockquote><strong><em>Clarification 4:</em></strong><em> No partial updates of the configuration — with configuration management we are talking about transferring a really small amount of data (if that changes, optimize accordingly), so I fear more data corruption than long download times.</em></blockquote><p><strong>The most important conclusion </strong>— configuration can be outdated but we need to make sure that we always have some configuration and that it is not corrupted<strong>.</strong></p><h3>Implementing the circuit breaker</h3><p>Now that we came to a robust and scalable config control solution that involves us building some software ourselves, let’s use that to our advantage and build a nice Circuit breaker.</p><h3>Use Case</h3><p>We want to implement Circuit breaker capability on top of the newly designed Config Control solution.</p><ul><li>It should have the ability to manage multiple binary (true/false) flags for circuit breaker control (<strong>check</strong>, it is already baked into the config management solution)</li><li>It has to have the ability to programmatically trigger the config change (receive an alert)</li></ul><h3>Receiving an alert</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dm_R7Vx224U3bJll.jpg" /></figure><p>The amazing thing about having few components that we have built here is the flexibility we get with it.</p><p>On the picture above you can see that architecturally, what we did is, we added just one more API consumer.</p><p>I am using <a href="https://prometheus.io/">Prometheus</a> as a monitoring and alerting tool of choice because I have used it in practice for the same thing, but most of the modern monitoring tools have the same functionality.</p><p>The functionality you need is the usage of <a href="https://prometheus.io/docs/alerting/latest/configuration/#webhook_config">generic/custom webhooks</a>. This would allow you to develop a logic on one HTTP API (not necessarily part of your REST API) that reacts to the triggers from the Prometheus backend and switches the config.</p><h3>On and off — a bit about implementation</h3><p><strong>When you receive an alert</strong> from the monitoring system (or alert manager), you would program the system to, based on parameters passed change the state of the system. State change is usually turning something off in order to recover from the failure happening, switch to some other integration, degrade the service to maintain the performance, etc.</p><p><strong>Recovering the system to its original state,</strong> however, is not that straightforward. You will, in most cases, be able to configure the system to send you the signal when the alert state was resolved. In the case of Prometheus, you can set <strong>send_resolved</strong> to “ <em>true</em> “ in the webhook config and it will do exactly that. My article on monitoring microservice data flows amongst other things covers the area of alerting thresholds.</p><p>This makes it easy to just switch back, but in most cases, you will need a human hand in confirming that everything is, indeed back to normal. The reality is — <strong>you had a kind of outage and it was caused by something, maybe you…</strong> There is a whole lot to do around detecting frequent circuit breaker triggers and reviewing service health.</p><p><strong>It is an operationally complex matter and should be handled on a case-to-case basis.</strong></p><h3>Security</h3><p>With all the flexibility that comes with building your own enterprise-grade software solution comes a need for enterprise-grade security.</p><p>Not too much advice I can give here beyond the usual.</p><p><strong>Auth </strong>— Make sure you got API-level authentication and authorization on the whole thing. For the scalability of a solution, you want to have the flexibility of multiple roles/user types. So, you should be able to segment the authorization — the best way is RBAC.</p><p><strong>Take care of the cloud security of your app</strong> — I write about this, other people write about this better than me, also Google/Microsoft/AWS/IBM writes about this on their own solutions side. Not in this article.</p><p><strong>Software side security</strong> — do what you need to do for your app to be secure. Please?</p><h3>The modularity of the solution — how to use it for your particular case</h3><p>You can use this solution partly or in full, depending on your system’s needs, you can pick a part that you need.</p><ul><li><strong>If you just want to manually configure a Kubernetes deployment in one cluster</strong>, you can just use Consul. It is the core of cluster data management and it has its own UI.</li><li><strong>If you want to only do a circuit breaker</strong>, you need an alert receiver that talks to Consul and apply config changing logic based on the received alert. You do need to build a microservice, but you do not need storage and a centralized system with the UI. You can just have an alert receiver in every cluster and send alerts to the multiple places f needed.</li><li><strong>If you want the mobile app to control this,</strong> it is really about just building a mobile app since you already got REST API, but please think twice… talking about a waste of time.</li></ul><h3>Summary</h3><p>This is one of the ways to build a dynamic configuration control system and a circuit breaker for Kubernetes deployed microservices.</p><p>The solutions based on this or similar/older versions of this design are running in production for a few years now. During this time they have been tested on the high scale and business-critical systems.</p><p>I consider this approach one of the best examples of what handling the scale should always be about.</p><p>If you decide to use this solution or a part of it, I would like to hear about the results.</p><p>Thank you!</p><p><em>Originally published at </em><a href="https://rastko.tech/devops/sre/circuit-breaker/automation/blueprints/cloud/kubernetes/architecture/2021/04/27/kubernetes-cloud-native-circuit-breaker.html"><em>https://rastko.tech</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e75ccc832b5b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/adidoescode/dynamic-configuration-management-and-circuit-breakers-in-kubernetes-e75ccc832b5b">Dynamic configuration management and circuit breakers in Kubernetes</a> was originally published in <a href="https://medium.com/adidoescode">adidoescode</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The 7 (and a half) tools to assist you on your DevOps journey]]></title>
            <link>https://medium.com/adidoescode/the-7-and-a-half-tools-to-assist-you-on-your-devops-journey-b45759f7410a?source=rss-d0d24ba520a9------2</link>
            <guid isPermaLink="false">https://medium.com/p/b45759f7410a</guid>
            <dc:creator><![CDATA[Rastko Vukasinovic]]></dc:creator>
            <pubDate>Sat, 08 May 2021 22:34:31 GMT</pubDate>
            <atom:updated>2022-01-14T13:51:48.081Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><em>Of course that having almost no automation, monitoring only 2–3 crucial http calls, having no real way to save yourself quickly if something goes terribly wrong, and releasing every 3 weeks while yelling “Leeroy Jeeenkins” (to name few of my top favorites) has to stop some time.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aE_AqTY2b2P_cjef.jpg" /></figure><p>DevOps journey of any company or a team starts whenever you deployed anything anywhere. Of course the emergence of any need to understand the term, not to mention really adopt some random principles comes at a later time.</p><p>You do, of course, have a choice of when it happens, and I will not caution you that there will be a bloody price to pay if you do it too late, but I will offer you some tools to take with you when you decide to embark on this journey.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*HRaJDyvmTRBPAaUT.jpg" /></figure><p>Two important things before getting further into reading:</p><ol><li>These are engineering tools, this list is a list of concepts that you can use to develop tools, or choose tools for certain tasks in your team. Another side of the whole thing is organizational and that aspect is not the subject of this article.</li><li>DevOps journey never ends, it is an infinite loop of continuous improvement and software delivery until our software’s end of life takes us somewhere else.</li></ol><h3>1 — the ChatOps</h3><p>The ChatOps is the meta-tool — the automation to glue together the things needed for you to operate your CICD easier. It gets you to the next level of DevOps for your team.</p><p>ChatOps is about the chat and about operations (obviously), it is also known as <strong>conversation-driven development.</strong> It is something usually built by the teams themselves because it is highly dependent on the tooling you use.</p><p>Here is quickly a concept of how it works:</p><ul><li>For your team communication tool of choice, for example, Slack or Ms Teams, you build an integration to your CICD task runner/build system, for example, Jenkins (not Leeroy in this case).</li><li>Integration assumes the special channel to which you will push notification cards for any successful or failed builds.</li><li>Fail build notification cards got the URL to stack trace of the build or some other page that shows why it failed</li><li>For successful builds on branches, card would have an URL to visit its dev environment</li><li>For successful build on the main branch, it would have:</li><li>An URL to its dev env</li><li>Button to trigger promotion to staging</li><li>A successful promotion to staging would have a button to trigger the automatic part of testing, and would potentially trigger other means of communication to set things in motion so we can quickly prepare to release.</li><li>Testing ending in positive result would give you the button to push and go to prod</li></ul><p>Voila — the conversation-driven… automation for something… development… CICD :).</p><p>It is great because you can control your CICD from the place in which you are having day-to-day work-related conversations, which greatly reduces the context switching and helps teams keep the focus on the task at hand.</p><p>I initiated building and polishing ChatOps for almost every project I have worked on for years now, and I can highly recommend it as a focus and scalability tool for your team.</p><h3>2 — Feature flags</h3><p>Another meta-concept that meant a whole DevOps world to me, is the concept of feature flags.</p><p><strong>Feature flags are configuration-controlled logical encapsulations of features,</strong> which allow us to decouple code deployment/release and feature release.</p><p>Imagine developing a feature for your application and code is done and tested so you have to hold off with merging it to the main branch because the production content or database or something else is not ready, or because the business wants to release it on a certain day. This is a costly wait because in time for the release you got to go through a new round of source code alignment, a new round of testing etc.</p><p>What the feature flag concept brings to the table is the idea that, if you have a situation similar to above, you can actually put the logical (if) block around the feature enabling code, and control the entry logic with a flag in the config (naively simplifying it for the sake of an article length).</p><p>Having this, you can test before the release, disable the flag in the production configuration, and deploy to prod. You still have everything in the code, it is tested and ready so you can enable it when external conditions are met.</p><p><strong>You can control the config in whatever way you generally do it </strong>— some teams need a code release to make this type of config change (it is a lot better to release just a config change than the full code change), some teams use databases or third-party config control systems… whichever you use, the concept is the same.</p><blockquote><strong><em>Dimensions of feature flags</em></strong><em> — feature flag does not have to be just true or false, it can also depend on the environment or other factors. For example, basic implementation of feature flags would need an env dimension, so you can say “it is enabled in dev, but not in staging and prod”, in multiple market e-commerce, it would probably be useful to have configuration like </em><strong><em>“enable in dev for all markets, in staging for given set of markets, and in prod for another set of markets”.</em></strong><em> Unfortunately, every application is different and the environment dimension is the only rule of thumb I can offer.</em></blockquote><h3>3 — Circuit breaker</h3><p>A circuit breaker is the simplest concept but hardest to implement well. It is what we call a self-healing mechanism, and represents the pinnacle of reliability engineering.</p><p>Assuming you have a feature that is an occasional source of instability, or integration that is so important for your flow (value stream), that you want to have a backup for it.</p><p>Circuit breaker would be the concept that would use system metrics to evaluate the state of the feature/integration and react when it becomes problematic — by shutting the thing down or activating the backup integration.</p><p><strong>Example:</strong> microservice in which one of the integrations used to enrich the data, occasionally starts performing super slow. In order to maintain the performance at the proper level, we want it shut down and have a bit of data richness degradation whenever this happens. Our circuit breaker system would trigger the operation of turning off the integration and degrading the data when the response time of the service passes a certain threshold for X amount of time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/726/0*RejN4OdlhJY02L-q.png" /></figure><h3>4 — Canary/shadow-prod releases</h3><p>Kubernetes gives this ability <a href="https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#canary">out of the box</a> — you would be using certain Kubernetes annotations to manage two separate deployments under the same host and manage the traffic to one or the other.</p><p>There are two usual uses of this tool:</p><ul><li>having portion of your real consumers traffic going into your new release and rest still going to your current/old release, until you confirm all is working correctly and then just shifting 100% of traffic to a news release.</li><li>Having the code released to production as a secondary (shadow) production deployment and testing it there internally.</li></ul><p>Of course, having the old version already deployed and having the ability to send the traffic there in case of emergency is a possible option for some failure scenarios. Which gets me to the next tool.</p><h3>5 — Rollback</h3><blockquote><strong><em>Disclaimer</em></strong><em>- my recommendation is to always follow the roll-forward strategy, which means that you are always fixing the problem by deploying a new change. Of course, that “always” means </em><strong><em>as much as possible.</em></strong></blockquote><p>In cases that it will take time to fix a big production bug, or your SRE is only one available and there are no feature flags to switch off, it is always handy to have a <strong>last resort panic button.</strong> Goes nicely with the “do not panic” sign.</p><p>Rollback is usually another CICD job deploying (current version of the container — 1) to production, effectively neutralizing complete last release to production.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/0*XgJNjwXzzzyM5xis.png" /></figure><h3>6 — The seeds</h3><p>DevOps is an equal part smooth development-prod-operations cycle and good developer experience. Technical enablement is a huge thing in the DevOps savvy environment.</p><p>Nothing speaks development experience better than smooth development project bootstrap, and a good step in this direction are what we call “seeds”.</p><p>Concept is simple, if you want to drive something like:</p><ul><li>Microservice adoption</li><li>Cloud adoption</li><li>Kubernetes adoption</li><li>All of the above or you just want to boost DevOps culture in your engineering</li></ul><p>One of the things you can do is develop seeds for the projects so engineering teams can just jump almost straight into writing business logic.</p><p>Types of seeds you can have:</p><ul><li><strong>Microservice code seeds</strong> — bootstrap repo containing initial microservice project setup including standard libraries import, initiation of the app, standard logging and code instrumentation, tests, container file, etc. For any programming languages you use in your ecosystem.</li><li><strong>CICD seed</strong> — build and deployment setup with standardized containerization and all of the setups for uploading containers, cluster deployments, infrastructure as a code, quality gates, different env setups, etc. You use these together with code seed.</li><li><strong>Cloud bootstrap</strong> — if you are in a business where you often prep a new cloud account in the standard way (certain compliances, monitoring, budgets, etc.) initial seed can be infrastructure as a code setup for all of these.</li></ul><p>This is not a final list, you can have whatever you want actually, these are just the most usual cases I encountered.</p><p>Seeds are often:</p><ul><li>Repos you can clone/fork</li><li>Scaffolding scripts pulling from different repos and assembling new ones based on your project needs</li></ul><p>In practice, these are amazing to drive any standardization and transformation in the engineering space, if done right of course.</p><h3>6.5 — Libraries</h3><p>Libraries are about standards — the standard way you do http calls, the standard way to log stuff, instrument the code in a standard way, with proper logging, secure handling of headers, parameters sanitization… blah blah.</p><p>These are integrated into seeds but are important to mention — if you develop libraries that are giving your app, and new apps that decide to integrate it a fully standardized way of working on certain topics, you are a star of your team’s DevOps journey.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/891/0*QkjZH3HnIMU2US0A.png" /></figure><h3>7 — DevOps checklist</h3><p>Over time, as you build your toolset and set up the process, the list of essential things that are defining the progress along the DevOps path will crystalize.</p><p>This list should be written down and refined as a checklist to start any new project.</p><p>As a next step, I would propose attacking measurements to each point in the checklist and annually (or more often) running it back as a sort of maturity assessment.</p><p>As an example of a quite good, and mature tool of this kind (and completely open-source tool you can use), I will leave here <a href="https://github.com/adidas/adidas-devops-maturity-framework">the adidas DevOps maturity framework</a>.</p><blockquote><strong><em>Yes, I did say this is the list of tech tools and I smuggled a governance tool in. If it is worth anything — it is an amazing tool.</em></strong></blockquote><h3>Conclusion — the affordable governance</h3><p>Smoothness in creating DevOps culture and standardization around it is all about achieving a certain level of self-service or rather making it easier to follow the rules and standards than doing anything else.</p><p>Tools should break the communication and boring work boundaries and standards should be guidance into a better state. It should be a no-brainer that standards are followed — governance should not be a burden on the way of achieving the goal, especially when driving DevOps adoption and lifecycle it is quite easy to make it affordable.</p><p><em>Originally published at </em><a href="https://rastko.tech/devops/automation/cicd/blueprints/cloud/kubernetes/architecture/sre/2021/04/10/tools-concepts-that-make-devops-flow.html"><em>https://rastko.tech</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b45759f7410a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/adidoescode/the-7-and-a-half-tools-to-assist-you-on-your-devops-journey-b45759f7410a">The 7 (and a half) tools to assist you on your DevOps journey</a> was originally published in <a href="https://medium.com/adidoescode">adidoescode</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cloud-Native CICD]]></title>
            <link>https://medium.com/adidoescode/cloud-native-cicd-c703cad711f9?source=rss-d0d24ba520a9------2</link>
            <guid isPermaLink="false">https://medium.com/p/c703cad711f9</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[cicd]]></category>
            <dc:creator><![CDATA[Rastko Vukasinovic]]></dc:creator>
            <pubDate>Sat, 08 May 2021 22:27:15 GMT</pubDate>
            <atom:updated>2021-05-08T22:27:15.772Z</atom:updated>
            <content:encoded><![CDATA[<p>The software delivery process and technology, popularly known as CICD (Continuous Integration + Continuous Delivery), is a structure supporting your software (development) lifecycle.</p><blockquote><strong><em>CICD</em></strong><em> is the one of most overloaded terms in modern software development, my theory is that at the moment something gets into a job description as a skill, it takes life of its own and is hard to determine the generally applicable definition. In this article, and ones I wrote before, I am referring to CICD as an SDLC supporting structure, as a process and set of technical solutions, not as a skill or, service, or magic dust.</em></blockquote><p>Delivering software is hard, delivering it frequently should be easier, or at least make it easier over time.</p><p>Setting your process, project, and system for stable and frequent delivery is a complex and often puzzling endeavor.</p><p>Through this article, I want to give you guidelines to help you get to smooth delivery of your cloud-native software.</p><p>I will try to give you an insight into one full end-to-end development and delivery setup — a blueprint of a full SDLC production line, as I like to call it.</p><h3>Prologue — stay confident, deliver frequently</h3><p>In the majority of cases, the main measurement of anything we do in software development is business value.</p><p>For a long time in software delivery history, we have delivered “full products”, something that we learned from production in the other industries. We reasoned that the whole thing needs to be assembled, tested, and delivered exactly as pre-defined. This resulted in big value releases, the prod deployment events that were perceived as life-changing events every time it happens.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*szEJO0AjIhIv-eol.jpg" /></figure><p>The problem with big releases is their big value. You got just one job to do and that delivers value, so delivering a big chunk of value is going to make your stakeholders happy at some time in the future (if the market does not change by the time we are done), and is going to make you and your stakeholders anxious about tons of things.</p><p>The pain is — a big chunk of business value in software means many changes, lots of new things, lots of software to test and to confirm. And if it goes in the wrong direction, as it most certainly will, one wrong change can take down the whole release.</p><blockquote><em>This sounds like a hyperbolic story in modern times, but believe me that some of the biggest players in the industry, including SaaS giants, have some gigantic releases. Imagine </em><strong><em>quarterly</em></strong><em> (as in four times a year) releases on some important SaaS platform that causes billions of euros in damage from some undiscovered bug, only to roll back and cause more damage to companies integrating some other new features that came with the release which had to be rolled back. Amazing stuff.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/278/0*jOdUfzTFmuk1Mkvh.png" /></figure><p><em>source: xkcd</em></p><p>So, the solution is to, as most of the readers will know already, set the pace of your SDLC to a lot shorter cycles. With this you will get:</p><ul><li>Shorter lead times to delivery of a single feature.</li><li>Less value in a single release, reducing the anxiety and increasing the confidence. You do not get too anxious about releasing one feature, and then maybe rolling it back quickly or it causing small financial damage.</li><li>Easier to test.</li><li>Easier to locate the problem.</li><li>More frequent feedback loops.</li><li>You exercise the whole CICD system more often so the system itself is more polished — iterations add finesse.</li></ul><p>You will have to get rid of:</p><ul><li>Releasing fortnightly/monthly/quarterly/yearly… even worse.</li><li>Long discussions about preparing everyone for moon landing, sorry, I meant production release.</li><li>Long reviews, testing sessions, and security checks.</li><li>Slow and outdated CICD system, lots of manual work (which is allowed if everything is slow).</li></ul><p>This is a part of setting up your organization to work in a full DevOps model in some nice agile or lean process setup.</p><blockquote><strong><em>Important note:</em></strong><em> I am mostly focusing on the architecture and technical solutions, so talking about team, work management process methodologies, release cycles and other generally organizational decisions is out of the scope of this article. I will keep focusing on the technical capabilities you want to provide and connect in order to let your software delivery fly.</em></blockquote><h3>Path to Cloud-Native SDLC</h3><p>Cloud-native principles are having a major influence on our thought process while creating our full SDLC supporting structure. This is how I like to present them in short form (good for conversation or whiteboarding).</p><p><strong>Four central ideas of Cloud-Native:</strong></p><ul><li>A service-based architecture.</li><li>API-first.</li><li>Containerization.</li><li>DevOps.</li></ul><p><strong>Eight elements of Cloud-Native path:</strong></p><ul><li>DevOps evolution.</li><li>Modularity/reusability.</li><li>Statelessness.</li><li>The right tool for the right task.</li><li>Self-service tooling and infra.</li><li>Automate everything.</li><li>Continuous delivery and advanced deployment methods</li><li>Continuously optimize and improve your ecosystem.</li></ul><p>I am bringing this in to reflect on what is driving us on the path we are taking towards fully optimized business value delivery. This means through the delivery process and good dev experience.</p><blockquote><strong><em>Service-based architecture</em></strong><em> — not microservices, not CQRS, not magic, just services working to deliver capabilities, some big, some small, some micro — all delivering business value and allowing us to have decently short and as much as possible separated release cycles for all.</em></blockquote><p>So, where do we start? What is making our SDLC tick?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6SanIdrYKbIx6RU3.jpg" /></figure><h3>The flow of delivery — in theory</h3><p>Every production line has its tempo, speed, and throughput (intentionally staying away from official theories and terminology):</p><ul><li><strong>Tempo</strong> sets when and how often production or processing of separate elements starts.</li><li><strong>Speed</strong> sets lead time from start to finished product.</li><li><strong>Throughput</strong> is the capacity for parallel production lines.</li></ul><h3>Setting the tempo — Source code management</h3><p>Getting into the core of the topic — SCM (most usually git) has all the capabilities we are looking for in the module that manages the tempo of our SDLC. We can trigger events on:</p><ul><li>Branch created/pushed to the repo</li><li>Before branch is pushed (with a hook)</li><li>PR created</li><li>PR merged</li><li>Merged to certain branch (master)</li><li>Creation of certain branch (release/xx)</li><li>Basically, any other event related to operations done with source code.</li></ul><h3>Need for speed — quality and compute</h3><p>Throughout the development of your system, there are going to be many automation and series of automation (AKA pipelines) that get triggered on different SDLC events. They are mostly:</p><p>Most of these are essential for the quality and delivery of your software, they will be there every time you so much as touch the code, not to mention deploy it. Their level of presence makes them a major influence on lead times within the delivery cycle. Keeping automation execution times low is essential for efficient software development and delivery.</p><h3>Capacity and throughput — 500 drills theory</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/349/0*7appFhC46jnChkTA.png" /></figure><p><em>source: xkcd</em></p><p>Looking at the path to cloud-native, few items directly impact how we look at the throughput of our delivery system:</p><ul><li>modularity/reusability</li><li>Self-service tooling and infra.</li><li>Automate everything.</li><li>Continuous delivery and advanced deployment methods</li></ul><p>In a secondary manner, statelessness and the right tool for the right task as part of solution space, are also bringing some solutions into space.</p><p>I like to set it up in this way: <strong>Creating building blocks for self-service tooling, automation, and infrastructure for every team to build their own continuous delivery and custom deployment methods.</strong></p><blockquote><em>Of course, self-service means that there is a team around with almost a sole purpose to hand-hold everyone through the process of self-servicing, assembling, and configuring stuff.</em></blockquote><p>In simple terms — we want it built with capabilities and at the level of flexibility that allows us to spawn many on-demand pipelines and development environments.</p><h3>Measure and optimize</h3><p>Every part of the CICD system has to be well monitored and analyzed. Measurements around the system should be used for determining the fitness of the technology solutions and identifying the next improvement area.</p><h3>Delivery flow in practice</h3><p>So, beyond the fancy theory (and all the comedy), <strong>the technical embodiment of SDLC starts with the quick delivery review and breakdown of needs</strong>.</p><p>So, let’s set the stage:</p><ul><li>Code is in git, using the “GitHub-flow” model, basically branches for features that merge to the main branch where they are considered prod ready</li><li>Is deployed to Kubernetes</li><li>Needs 3 types of environments — dev, staging/preview (pre-prod) and production</li><li>SCM flow is setting the expectation that every feature branch has a separate environment</li><li>Has dependencies deployed to the cloud</li><li>We want to always be able to release</li></ul><h3>Source code flow — code comes in</h3><p>As already described, we want SCM to be the main driver behind our CICD setting — setting the rhythm.</p><p>We would utilize the hooks from whatever source code repository we are using to trigger certain actions in the system.</p><p>This means that development flow until actual merge to the main branch will fully be driven by the events on the source control:</p><ul><li>New branches created and pushed</li><li>Pull requests created</li><li>Pull requests merged</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WDyuJ4VKHQlt3Mc9.jpg" /></figure><h3>The containerization</h3><p>Containerization implies another repository/registry system — the one to keep all the containers we will be building in. Ideally one with some additional features like security and integrity scans and maybe some other fancy stuff.</p><blockquote><strong><em>Apart from what is in the container</em></strong><em> — you might need more than just a place for your app code to live, there are of course different data persistence layer components, networking, storage, etc… If these are all in the cloud for you, </em><strong><em>please make sure you got as much as possible infrastructure as a code (ideally have EVERYTHING AS A CODE POLICY in the dev team)</em></strong><em>. Avoiding any manual change is always the best, it also makes it optimal for automation.</em></blockquote><h3>The development environment — no judgment place to work</h3><p>When it comes to the early phases of SDLC, you want it free-flowing. There is no point in scrutinizing any aspect of work in the starting phase, not just of the system, but of any feature. <strong>If every feature is a built-in separate code branch, that is.</strong></p><blockquote><strong><em>To make sure this is followed, also make sure to block any pushes to the main branch (whatever you call it).</em></strong></blockquote><p>So, we basically want every new branch created on the repo to spawn its own environment in K8s, <strong>no questions asked</strong>.</p><p>On the other hand, you will probably never need separate databases and other supporting structures between different dev environments — it is good to keep things simple on this side.</p><p><strong>For this you need:</strong></p><ul><li>Build pipeline ( no quality gates used apart from the one that is actually if the build succeeded or not)</li><li>Build pipeline will output the containerized application, so you need to structure your usage of a container registry</li><li>Deployment enabling a multitude of environments.</li></ul><blockquote><strong><em>Multitude </em></strong><em>— Imagine 3 people on the project, each having some features they work on and some test/PoC, totaling 3 branches per person — you would ultimately have at least 9 builds, and if they are successful, 9 separate environments under one ingress in your dev Kubernetes cluster. Depending on your application’s API structure this might be a source of complexities as well.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bNhSH8SdXokhDdRk.jpg" /></figure><h3>Mainline — the pull request (PR)</h3><p>Getting the code further through the delivery process should depend the most on what happens next in the source code story.</p><p>After the successful development of a new feature, it is time to merge it to the main branch and start preparations for production release.</p><p>This is where pull request is created and code should go through all imaginable checks before ultimately being merged and built <strong>one last time</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iX_OuQ4Az2gXSmeY.jpg" /></figure><p>So, when PR is created, the CICD system starts a sequence of operations, mostly related to code functional and non-functional quality, and tests — <strong>ofc we are writing tests</strong>.</p><p>These are known as <strong>quality gates.</strong></p><p><strong>&gt; The Quality Gates</strong> are all outputting go/no-go status and it is used fr both the PR merge and the final build of the code version.</p><p>Without all green, the very button to merge the PR or any pipeline to merge the code should stay locked.</p><p>A not complete, but hopefully illustrative list of quality gates:</p><ul><li>Unit tests</li><li>Integration tests</li><li>Static security scan</li><li>Code quality scan</li><li>Peer review</li></ul><h3>Merged! Now what?</h3><h4>The Last Container!</h4><blockquote><em>You should be able to always consider the code merged to the main branch production-ready, if not, this is the first thing you should fix</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hq77Zp4Y1qLuh1lN.jpg" /></figure><h3>Release the Kraken!!!</h3><p>Now, in building your CICD architecture comes the time to deliver the app version to the new environment.</p><p>The next environment can be production or staging, which also determines what you are doing on the main branch dev build. If you have no need of a particular staging environment, you can also run final quality gates (functional, acceptance, API contracts…) on the dev environment of the main branch build.</p><blockquote><em>In some situations, you can have even more separated environments — I worked on software that needed special security checks and compliance review environments to continuously run production code and certain replication of prod data…</em></blockquote><p><strong>In conclusion — beyond deploying the main branch build to the dev environment, you are only promoting it further. it is about performing the same action on the same, already built, container, but in a different environment and running post-deployment tests.</strong> The only difference might be the environment variables on the deployed container that you use to control the configuration.</p><blockquote><em>If you are maintaining any kind of consumer-facing application (API or UI used by internal or external consumers), you will mostly have a staging (pre-production) environment for a few purposes like:</em></blockquote><ul><li>Testing the app prior to going to prod</li><li>Testing content prior to going to prod</li><li>Having next version available for clients’ integration purpose</li></ul><p>After all the tests are done, you are ready to open the gates.</p><blockquote><strong><em>Promote</em></strong><em> — actual terminology for the release in this cloud-native approach is quite often “promote to staging” and “promote to production”, for the reason that you are taking the container built from the main source control branch and deploying it to dev (as a branch-build), then taking the same container and deploying it to staging and then to prod, which is basically promoting it to the next level.</em></blockquote><p>So, this is it, go back to the part where it says “code comes in” and start again.</p><p>Here is the holistic view of what you’ve just read about:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FuuTHGI4hNN2pWNX.jpg" /></figure><p><em>Originally published at </em><a href="https://rastko.tech/cicd/blueprints/cloud/kubernetes/architecture/2021/04/05/cicd-cloud-native-software-delivery.html"><em>https://rastko.tech</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c703cad711f9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/adidoescode/cloud-native-cicd-c703cad711f9">Cloud-Native CICD</a> was originally published in <a href="https://medium.com/adidoescode">adidoescode</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The two scales of DevOps]]></title>
            <link>https://faun.pub/the-two-scales-of-devops-4513d7a4c41a?source=rss-d0d24ba520a9------2</link>
            <guid isPermaLink="false">https://medium.com/p/4513d7a4c41a</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[cloud-native]]></category>
            <dc:creator><![CDATA[Rastko Vukasinovic]]></dc:creator>
            <pubDate>Thu, 05 Nov 2020 21:55:25 GMT</pubDate>
            <atom:updated>2020-11-16T15:53:01.567Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><strong><em>Whatever happens always happens on time.</em></strong><em> <br> </em>Zen quote about software delivery</blockquote><p>This article is a continuation of the discussion about NFRs of being <a href="https://rastko.tech/blueprints/cloud/cloud-native/kubernetes/infrastructure/architecture/solutions/2020/10/22/native-to-cloud.html">Native to the Cloud</a></p><h3>The scale of things to come</h3><p>The set of complexities that you have to manage in your system, from its inception to the end of days, is immense. It represents a summary of its architecture, the way it is built, a structure that creates and delivers it, and the structure that is being serviced by it.</p><p><strong>The architecture</strong> of the system defines the end-to-end structure of the system, the layout of components, and their correlation. It defines events and data flows and makes the footprint in the infrastructure. This is the part that gets deployed to Kubernetes and other connected systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SWTTEg6ncHjpmxM-.jpg" /></figure><p><strong>In the picture </strong>— Microservice architecture with event-driven components and two UIs, which would be considered medium complexity projects at the start. The central piece is handled by Kubernetes while there are few mounted components from cloud solutions and CDN to handle static content (UI) and caching.</p><p>Further on, I would like you to understand that your system will, in most cases, have to handle the fluid nature of software architecture: you have to be ready for ever-changing software design (emergent design). And there we get to the actual problem space.</p><p>Effects of emergent design manifest on all aspects of your software’s development, delivery, and lifecycle. This is continuously changing NFR problem space, which consequentially requires you to continuously adapt your solution. The actual problem we are solving is represented by twofold complexity you are bound to manage till the end of the day. These two parts of the problem are best analyzed and tackled together, through one thought process.</p><p>Over a period of many years I have spent working on all the aspects of software, I ended up calling this <strong>“the two scales of devops”.</strong> Of course, scales are complexity and volume measurements of particular software aspects:</p><ul><li><strong>The development and delivery</strong> — the scale of development.</li><li><strong>The production part</strong> — the scale of life.</li></ul><p>This <strong>problem of two scales</strong>, as I would usually call it, has to be solved day in and day out, which means that, in our case, it becomes a <strong>paradigm — we want to discuss and collect some established patterns for solving its usual components</strong>.</p><p>The scale of development is about providing a good development experience at any pace, about development tooling, about flow efficiency, security, and ease of onboarding. You measure the quality of the solution in this area through build times, lead times to production, testing capabilities, release confidence, employee satisfaction (eNPS?), security, and process transparency metrics.</p><p>At this side of the equation, structure and method of delivery is the central point of the solution, the focus is mostly not on tools (apart from Kubernetes and few chosen specifics just for this analysis to even be possible), but on a way to utilize set of tools you chose to best manage any development process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*muDSBU_UhbhVy7TO.jpg" /></figure><p><strong>In the picture </strong>— Developing and delivering a system with certainty and high quality is quite a complex set of tasks and problems on its own.</p><p>Creating fast and resilient tooling to deliver your application smoothly, and allow flexibility to set up a good process around it is the key. It is about technologically enabling your engineering team and setting the whole team up for success.</p><ul><li>Deployment frequency — relying on the quality and flexibility of the process and technical solutions with a focus to make changes smaller and more frequent.</li></ul><blockquote><em>Good further read on this topic is </em><a href="https://itrevolution.com/book/accelerate/"><em>Accelerate</em></a><em>, the book which explains the approaches around building annual State of DevOps report.</em></blockquote><h3>The second scale</h3><p>This side of your system takes the software over after it is delivered to production, it is the area where the long term stability of the app is defined. It is about a long term perspective of running it in production.</p><p>The scale of life is focused on operation, stability (resilience), security, and capacity planning. You measure the quality in this area through SLOs, API contract tests, quality of service, and incident response metrix (MttX).</p><blockquote><em>This scale is about serving the end-user with a high-quality product, it is about the business value we are delivering and how to serve it in the best way.</em></blockquote><p>Deeper down it is all about operational tooling, telemetry, good capacity planning, and other more specific SRE approaches.</p><h3>The DevOps and SRE</h3><blockquote><strong><em>KPIs and signals</em></strong><em> — KPIs (key performance indicators) of your service can be observed through two lenses:</em></blockquote><ul><li>Operational — how good capabilities and people are working together, which is measured through Minimum time to Detect, Diagnose and Resolve a problem (MttX where X can be D for detect, R for resolve and many more based on what you want to measure). Signals to measure here are partly communicational — how automated the system is and how close are the teams involved, and partly in the following category.</li><li>Application — Quality on Service level, which is measured through service level objectives (how we want it to perform), indicators (what system metrics we use to determine this performance), and agreements /contract that we make with our clients in regards to how our app should perform (SLX). Signals we observe and combine in these cases are also known as <strong>four golden signals — error rate, latency, traffic, and saturation.</strong></li></ul><p>In order to efficiently handle the problem space we just defined, the team has to be able and empowered to fully own everything they are building. The team should also own the process, and sometimes the team itself. Depending on the size of the scope, the team, and the company, it might not be the same team owning the whole thing end to end, it might be a couple of teams working together in certain, hopefully, efficient collaboration.</p><p>This approach of high ownership is better known as DevOps, the concept aiming to blur the boundaries between software development, delivery, and operation, bringing it closer together, with the goal to make software development and delivery a lot more efficient and high quality.</p><blockquote><strong><em>SRE </em></strong><em>— site reliability engineering is a set of software engineering practices focused on infrastructure reliability and operations. It is born from the direct application of DevOps methodologies on live systems with a heavy focus on automation and observability. SRE focused teams usually split their focus roughly 50–50 between software engineering (building tools, features, or automation), and operations/support tasks (like incident handling, on-call, and so on).</em></blockquote><blockquote><em>Good further read on this topic and many other topics that I write about can be found in the </em><a href="https://landing.google.com/sre/sre-book/toc/"><em>SRE book</em></a></blockquote><h3>To tame the scales</h3><p>The goal of delivering high-quality software as soon as possible represents both scales. It puts focus on the smooth delivery of your software and making sure it is reliable (high quality). <strong>Taking DevOps concepts and more closely applying them to live systems belongs to the site reliability engineering (the SRE)</strong>.</p><p>Systems that are native to the cloud are able to adopt DevOps approaches faster than others through building tools and capabilities (like on-demand infrastructure for engineers, fast and safe releases, security-first design…).</p><p>If built correctly, these systems often have the right tool for the given task, deliver continuously with low to no risk, and are generally more open-ended architecture wise, fully manifesting DevOps culture.</p><p><strong>Looking from the practical standpoint, and especially in Cloud Native systems, DevOps and SRE practically mean the same thing. SRE pays less attention to how software is being delivered (first scale) and focuses heavily on applying the DevOps methodologies on live systems (second scale).</strong></p><p>It all ultimately means that your team is able to pick up the software at its inception, develop it, efficiently deliver it to production, and afterward also operate at a high level of reliability while continuing to deliver new business value at a steady pace throughout the whole lifecycle of the system.</p><blockquote><strong><em>Keeping the system available and QoS high in many cases brings more business value than a new feature you would release. If you can’t have both, stick to quality.</em></strong></blockquote><h3>DevOps and SRE on Kubernetes</h3><p>Kubernetes’ architecture and the area of technical solution it covers with its core functionality is already a big incentive to plan for a full DevOps approach — it would actually be harder to not do DevOps. This also goes for a big part of the Cloud space, which makes Cloud-Native concepts a big DevOps driver.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uDSrwh7zCP1eNwpR.jpg" /></figure><p><strong>In the picture:</strong> SRE uses DevOps to dive deeper into the live system’s reliability and observability</p><p>Having a broad spectrum of tools, giving us solution components for most of the problems we are facing is sort of a blessing, but it is really putting a lot on your development team. Owning the code, building, deploying, and operating is good for the quality and speed of delivery of any software, but building all the tools and capabilities needed to be successful in that might be too hard for your team.</p><blockquote><strong><em>Cognitive load</em></strong><em> — Cognitive load in general represents amount of intellectual effort you have to invest in a certain topic in order to either perform tasks, learn how to perform a task, or form long term knowledge about it. Quite often it is all of the mentioned in some short sequence or in parallel. Level of cognitive load is in direct correlation with (oh, what a surprise) intellectual performance. </em><strong><em>Take care of cognitive load of your team</em></strong><em>.</em></blockquote><blockquote><em>Fun and quite high quality read on the topic (and many more) — </em><a href="https://teamtopologies.com/book"><em>Team Topologies</em></a></blockquote><h3>About technical enablement</h3><p>We have to acknowledge that, although we can easily agree that tooling was always a factor in failing to do the “full DevOps,” it is cognitive load that plays a big part in it, if not even a major part. It turns out that, even if it all fits together pretty well, and it is a practically proven technique to fix the velocity of your team, the team can’t always keep pace with developing new features on your software and solutions around all tooling and non-functional requirements.</p><blockquote><strong><em>Integrating all parts of the software lifecycle into one seamless flow of delivery that radiates feedback and assures smooth operation is not magic, it is rather really hard work.</em></strong></blockquote><p>Tradeoffs are too often really costly in the long run. That is why you have to identify the point during your growth when it becomes crucial to have a team (or a person) dedicated to enabling development teams to do full DevOps.</p><p>The ultimate result might look like one of those amazing yoga flows, you do not understand how someone gets there but you see it is totally realistic to be able to do it. To have this kind of ownership over your system, all the things you had to do before, in separate knowledge silos, you have to do now as well — in a team of t-shaped expert engineers.</p><p>For this, we need to identify and provide tools and capabilities that team or teams need to do this kind of work. We have to understand what is coming and prepare accordingly, but since we actually do not want to know and we want well-tamed chaos (weird way to spell a-g-i-l-e) to rule our lives, we need to have a solid preparation. It would, actually, be amazing if we would get some kind of solution management capability, a project foundation that can contain chaos. How lovely.</p><p>I am scratching under the surface of chaos starting from the next article.</p><p>For now, remember — <strong>this is not a finite game, it is the practice, and we are in it for a journey</strong>.</p><p>❤ DevOps.</p><p><em>Originally published at </em><a href="https://rastko.tech/blueprints/cloud/cloud-native/devops/sre/kubernetes/infrastructure/architecture/solutions/2020/11/02/two-scales-of-devops.html"><em>https://rastko.tech</em></a><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/0*Piks8Tu6xUYpF4DU" /></figure><p>👋 <a href="https://faun.dev/join"><strong>Join FAUN today and receive similar stories each week in your inbox!</strong></a><strong> </strong>️ <strong>Get your weekly dose of the must-read tech stories, news, and tutorials.</strong></p><p><strong>Follow us on </strong><a href="https://twitter.com/joinfaun"><strong>Twitter</strong></a><strong> </strong>🐦<strong> and </strong><a href="https://www.facebook.com/faun.dev/"><strong>Facebook</strong></a><strong> </strong>👥<strong> and </strong><a href="https://instagram.com/fauncommunity/"><strong>Instagram</strong></a><strong> </strong>📷 <strong>and join our </strong><a href="https://www.facebook.com/groups/364904580892967/"><strong>Facebook</strong></a><strong> and </strong><a href="https://www.linkedin.com/company/faundev"><strong>Linkedin</strong></a><strong> Groups </strong>💬</p><figure><a href="https://www.faun.dev/join?utm_source=medium.com/faun&amp;utm_medium=medium&amp;utm_campaign=faunmediumbanner"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_cT0_laE4iPcqW1qrbstAg.gif" /></a></figure><h4>If this post was helpful, please click the clap 👏 button below a few times to show your support for the author! ⬇</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4513d7a4c41a" width="1" height="1" alt=""><hr><p><a href="https://faun.pub/the-two-scales-of-devops-4513d7a4c41a">The two scales of DevOps</a> was originally published in <a href="https://faun.pub/">FAUN.dev() 🐾</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Native to the cloud]]></title>
            <link>https://medium.com/adidoescode/native-to-the-cloud-149643d57690?source=rss-d0d24ba520a9------2</link>
            <guid isPermaLink="false">https://medium.com/p/149643d57690</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[cloud]]></category>
            <dc:creator><![CDATA[Rastko Vukasinovic]]></dc:creator>
            <pubDate>Thu, 05 Nov 2020 21:51:29 GMT</pubDate>
            <atom:updated>2021-06-25T08:22:57.963Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jAjMK0hK9SzuEWlU7lgcKw.jpeg" /></figure><p>Cloud’s biggest breakthrough wasn’t actually being a cloud. Although it might seem that putting computing in a metaphorical cloud that you could rent through the interface made a whole difference; it made it resilient, and made managing all the aspects of infrastructure and resilience easy, making its guarantee of availability something that new generations of engineers and some of us ancient folk take for granted. Cloud did a good job of convincing its consumers that actual hardware machines do not exist.</p><p><strong>The biggest thing, in my opinion, is that cloud technology expansion changed web software architecture by shifting the attention of architects and engineers towards more important stuff, like better architecture, faster delivery, etc.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aVUCVgEwujAdbry_.jpg" /></figure><h3>Native to the cloud</h3><p>The rise of cloud computing has made it easier to do lots of things that were originally super hard for us. More importantly, it made a promise that our lives as software engineers and architects, and especially as system administrators are going to be easier in so many ways. We suddenly had the new world to venture to.</p><p>It is also a whole new world to adapt to.</p><p>The first reflex, of course, was to just move everything from “raw iron” to the cloud… a big wave of migration projects happened — the new turf got explored along the way, not beforehand. New projects were also mostly created in the same way as before — as we would do it on servers we owned and ran ourselves for 24h a day… It was the only thing we knew how to do.</p><p>At the same time, we realized that, since we can now get resources on-demand and actually plan how we deploy and connect services, we can start exploring some architectural concepts that were hard to implement to full potential before the cloud. We started exploring microservices in-depth, working on event-driven architectures, explored scaling principles, etc.</p><p>Since all the cloud tools provided some carefree resilience, and we already worked on figuring out how to do DevOps in every scenario, we quickly promoted “move fast and break things” into the main mantra, and … we moved fast. Things getting broken along the way did not really bother us because that was what the likes of Facebook, Amazon, and Google were saying was a way forward.</p><p>Somewhere down that path, we started understanding technology a lot better conceptually, so it became clear that there is a lot more to it than just repacking what we have to this new environment.</p><p><strong>Enter Cloud-Native</strong>, the collection of concepts that brought us lots of flexibility and a better way forward for our architecture. The actual mantra about breaking stuff was never fully revised, altho I believe initial promoters moved on from it.</p><blockquote><strong><em>Cloud-Native</em></strong><em> as the overarching concept was created in order to make delivering software more adjusted to a new state of the infrastructure and tooling world where it promises to give us better release pace and quality and easily manageable systems and tools through the deploying our software as a containerized components, highly reliable infrastructure, and vendor-agnostic tooling.</em></blockquote><blockquote><em>Altho this is, and always will be, only partially true, through few iterations of open source and closed projects we managed to create an infrastructure abstraction system, often also referred to as a new operating system, that is bringing a certain level of vendor agnosticism but cosmically high level of infrastructure virtualization and management — </em><strong><em>Kubernetes</em></strong><em>.</em></blockquote><h3>Kubernetes as the foundation</h3><p>At the current moment, Kubernetes epitomizes everything that Cloud Native is, it provides the core tooling to realize fully cloud-native architectures. The struggle is — it is not simple… it is everything but simple. How could anything that gives you such a broad toolset ever be simple?</p><p>Moreover, the case of Kubernetes actually goes to the whole cloud-native space. What actually is quite a simple and straight forward concept that teaches us how to think in the space of flexible and available compute has easily created the most complex integral software engineering concept out there. What grew out of cloud-native teachings is the only set of approaches that can guide the creation of the set of tools for every little non-functional piece you might need for your application.</p><p>Building blocks available (non-exclusive list):</p><ul><li>Kubernetes and all of its ecosystem</li><li>All the cloud infrastructure</li><li>Many observability and monitoring tools and concepts</li><li>Many databases</li><li>Many different CICD approaches and tools</li><li>And counting</li></ul><h3>Cloud-native problem space</h3><blockquote><em>So, what is this all about…</em></blockquote><p>Building software in cloud-native space brings forward a whole new set of challenges We are now taking a lot more emergent design approach and stepping into a new set of requirements in regards to fast delivery, scalability, resilience, developer experience etc.</p><p><strong>I wanted to share some thoughts, approaches, and analyses about achieving high quality throughout your software development lifecycle from the side of the non-functional requirements of the system. It means that, this time, we will not be looking at the system’s functionalities but at its scalability, resilience, security, speed, and capacity to deliver new features to production, etc.</strong> <strong>My idea is to do this through the series of articles that will live under a special category in this blog called <em>Blueprints</em> to which this article will serve as an intro.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vQfNSdn4mCIEXNqW.jpg" /></figure><p>So, this article covers the intro to developing software that is native to the cloud, and the next article will start unwrapping DevOps capability as such using the problem space of two scales of DevOps</p><p>While the series will cover a lot more, it will be <strong>focused on using Cloud-Native concepts and will mostly be Kubernetes-centric</strong> in order to reduce the number of variables and make it at least a bit more realistic to connect all the dots together at the end.</p><blockquote><em>Just kidding, there is no end. Software development projects are never done. While there are consumers, of course. #neverdone</em></blockquote><h3>Developing software</h3><blockquote><strong><em>Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.</em></strong><em> (from </em><a href="https://agilemanifesto.org/"><em>Manifesto for agile software development</em></a><em>)</em></blockquote><p>Developing any new system is a lot about tailoring things to fit; the same as human interaction, it is about the way things interface, about the way it all fits together end to end. You want to make it perfectly serve the purpose. Software architecture has to be clean; the general solution has to fit the purpose well; it has to be scalable, resilient, secure, and easy to change. You want to deliver it whenever needed or whenever a change is made… You want to make it work. Probably a few years down the line you want 20–30–50 engineers to actively contribute to your product. At this point, you probably want the magic coffee machine, as well as the flying carpet.</p><p>The solution, usually well known in the global cognitive machine of software development circles is the following:</p><blockquote><em>As soon as any piece of software is delivered everyone wants it in production… Of course, it is Agile so we do it, no matter if it is 100% ready — we can always rollback or fix forward, as soon as that happens maintenance and operations processes start and that is also your life now. As the first thing is done, further development continues, and usually a few other features in parallel, all of them being designed as new briefs are arriving. The plan is to continuously integrate new changes into the live system, but with the operation of the existing system in mind.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gf0E_6r6douAafHH.jpg" /></figure><p>It is chaos, nothing is ever 100% done, planning is easy — easy to put in, easy to cancel, easy to change (which we always do), you always need an architect, and QA and security won’t get off your back about the next release. It is chaos, but it is our chaos and we love it.</p><p>But, how do you plan for chaos? How do you set chaos up for success?</p><h3>Inception of container orchestration — enter Kubernetes</h3><p>As the cloud picked up the pace and we understood that now we got more time to focus on other things, like those great architectural concepts that would never work for us because of the practicalities of big on-prem machines. You can now rent a server in quite a flexible capacity range, so we can just decide what sizes we need for our smaller components, we got Docker to containerize it so it would be quite easy to just go.</p><p>Realizing that deploying containerized applications was now a central thing for cloud usage, different ways of managing container deployments started appearing, we tried most of them, and then again any would work — but none covered even a part of the solution that was needed. With every step of growth, a new problem was discovered:</p><ul><li>How do we manage multiple deployments?</li><li>How can we manage resources in a more flexible way?</li><li>How can we scale based on internal system metrics?</li><li>How do we mount a drive?</li><li>How do we secure a system and properly manage communication between different services and then have them communicate to the outside world through a unified interface?</li><li>Secret management?</li><li>Environmental variables?</li><li>How can we manage it through centralized manifests?</li><li>How to seamlessly deploy, rollout, and rollback?</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pProAMEgY6Uz_X1Q.jpg" /></figure><p>This is a small fraction of the actual list, but illustrative enough, I hope. This called out for a robust open-ended solution with many core capabilities and a completely open way of extending the solution range, which ended up in Google brewing Kubernetes and the global community taking it over and polishing it. Along the way, there were more than a few vendor-specific and open source solutions to the same (or subset of) problems, but Kubernetes is now established as <em>the solution</em> in the space. As already mentioned, going forward, in this particular article series I want to use Kubernetes as the underlying infrastructure for any of the technical solutions proposed.</p><p>Kubernetes is bringing a good solid structure that is quite beneficial for any microservices project. More importantly, as you look at many challenges that your software development life cycle is bringing, Kubernetes provides capabilities and means to develop capabilities that your team needs in order to fully own their work and make the project you are working on go fast.</p><p>As you go into a new project, no matter if it is a fresh microservice architecture, monolith to microservices migration, or an upgrade to the existing system you will want to onboard the team quickly, help them set up, and start writing code as soon as possible, deploy quickly in stages, test in a flexible way, deliver to production, easily plan capacity, make it secure, etc. In a project of any magnitude, Kubernetes can be a central part to achieving the above and more, while helping provide a great experience for people who are building the software and operating it.</p><p>If you got one or many different containerized components that you need to deploy and scale, manage different versions of, and provide various testing and performance environments. If your risk factors are quick growth of a team and number of features, security, and capacity (or any of these). If you are looking for a good system to manage the growth of the influx of code and new people to the project with ease, the same ease you are managing the capacity planning, Kubernetes might be the solution for your problems. Actually, I believe that Kubernetes is the best tool to base your medium to high complexity system on.</p><p>The solution, however, might not be that easy to define and finalize, and this is where careful planning and analysis with experience can help a lot. I want to help you plan well and build the base in order to be able to move fast. I am not a big fan of breaking things, but you can choose to have that as well, it is purely depending on your preference and I believe we will get to talk about that as well further down the road.</p><h3>What is next</h3><p>For now, let’s identify two main groups of concerns as we are planning the development of our new system:</p><ul><li>Scale and complexity of the system (the product that we are building)</li><li>The capability to own the development, delivery, and system itself end to end</li></ul><p>The first group depends a lot on the technology you chose, but it does not change the way you approach a problem from a cloud-native perspective, for this group of concerns we need to have a good set of practices to understand how to protect your new application’s architecture to cloud-native centric system. There are not a lot of moving pieces when doing this, there might be lots of small and big complexities, but understanding the concepts is pretty straight forward.</p><p>The problems from the second group, however, would take a significant part of any discussion and will be a dominant content in the core matter of this series.</p><p>In order to not get too deep into the topic of the next article, I will stop here, quickly summarize and soon follow up with a post about two scales of DevOps.</p><p>Summary:</p><ul><li>Cloud gave us the ability to think about important stuff</li><li>The real advancements occurred as the cloud concepts matured with the emergence of cloud-native</li><li>This is an introduction to the Blueprints series</li><li>Blueprint series addresses non-functional requirements in the microservice architecture and cloud-native space in the hope to help you by making your software delivery smoother and your quality of service better.</li><li>Kubernetes is taken as the foundational infrastructure component since it, along with its ecosystem, epitomizes cloud-native.</li><li>I believe that Kubernetes is the best tool to base your medium to high complexity system on.</li><li>Don’t break stuff whatever others say.</li></ul><p><em>Originally published at </em><a href="https://rastko.tech/blueprints/cloud/cloud-native/kubernetes/infrastructure/architecture/solutions/2020/10/22/native-to-cloud.html"><em>https://rastko.tech</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=149643d57690" width="1" height="1" alt=""><hr><p><a href="https://medium.com/adidoescode/native-to-the-cloud-149643d57690">Native to the cloud</a> was originally published in <a href="https://medium.com/adidoescode">adidoescode</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monitoring data flows in a microservice system]]></title>
            <link>https://medium.com/adidoescode/monitoring-data-flows-in-a-microservice-system-5ea04532db3f?source=rss-d0d24ba520a9------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ea04532db3f</guid>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[alerting]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[monitoring]]></category>
            <dc:creator><![CDATA[Rastko Vukasinovic]]></dc:creator>
            <pubDate>Thu, 05 Nov 2020 21:50:09 GMT</pubDate>
            <atom:updated>2020-11-05T21:50:09.989Z</atom:updated>
            <content:encoded><![CDATA[<p>To provide the highest possible quality of service, you have to keep a close eye on data flows in your app. This is something that, in most cases, requires you to set up the metrics of your code well and set up good monitoring on these metrics.</p><blockquote><strong><em>Metrics</em></strong><em> are the fastest observability data. They have the quickest output, can be consumed directly (since they are in most of the cases already data series easy to plot), require the smallest throughput, and have the smallest retention footprint. You can find a more detailed analysis of observability planning in my article </em><a href="https://rastko.tech/sre/observability/monitoring/logging/analytics/microservices/kubernetes/basics/2020/08/29/planning-observability-monitoring-analytics.html"><em>Planning observability of your system</em></a></blockquote><p>In microservice architectures (or any service-oriented architecture), it is essential to understand how to use the metrics of distributed integration points for the early identification of problems. This is the topic I am covering in this article.</p><p>Let’s first set the stage for a discussion.</p><h3>The architecture of the system</h3><p>We are talking about data flows in microservice architecture, so the first thing we want to introduce is the microservices that are doing stuff together.</p><p>I decided to go with something that you regularly see nowadays — your friendly neighborhood microservice-based system that still did not outgrow and eat itself, but might as well be on a similar trajectory.</p><p><strong>The System has:</strong></p><ul><li><strong>Frontend API microservice</strong> that should have been an API gateway, but the architects could not agree where some of the logic goes, so they just left it in (“It is a proper place for it anyways.”).</li><li><strong>Integration Aggregators (A.I.) microservices</strong> because some logic did move, but also stayed (if you know what I mean).</li><li><strong>Subsystem microservices</strong> — a microservice specialized for a domain aggregate (the way we see it). Eric Evans is proud of us (the way we see it).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/0*D6Hip9U9gI_Ejm_Z.jpg" /></figure><blockquote><em>Any similarity to a real-life system is accidental. I would never embarrass anyone by showing their work this way.</em></blockquote><p>Anyways, we got it all in there, arguably even more than we would need for this topic. So, let’s start.</p><p>The case I decided to cover is a case of a regular data flow that has multiple calls within it but synchronously ends after everything is processed. This is one of the last cases that are representable in a straight forward series of specialized diagrams. For more async variations, the approach is mostly the same but there are nuances, and further discussions can be derived (you can expect this in some of my future writings).</p><h3>The flow</h3><p>Consider the following sequence diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-a0NRjpAfj2CQKjI.jpg" /></figure><p>This is your regular backend flow, usually involving a few calls to additional services to get the requested data, aggregate, and prepare and send back to the user. Everything, or most of it done in sequence, and emitting few callbacks to keep the UX proper.</p><p>The diagram is also showing two regular subflows:</p><ul><li>When there is an erroneous response to one of the calls.</li><li>Timeout flow, when the call is taking too long to finish (to simplify — frontend call is taking too long).</li></ul><h3>Observing the flow</h3><p>Let’s put this diagram under the observability lens and see what we get.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*k2D-ag4kJj6uviU3.jpg" /></figure><blockquote><em>The first thing to note here is that this flow touches almost every part of the system, which is not unusual for imaginary examples at all.</em></blockquote><p>We want to expose metrics about the flow for two reasons:</p><ul><li>Performance measurement</li><li>Flow exactness confirmation (making sure flow is right)</li></ul><p>For this kind of measurement we want to employ two types of metrics:</p><ul><li><em>Counters</em> — displayed as a square/rhomboid shapes</li><li><em>Timers</em> — displayed as arrow bands</li></ul><blockquote><em>Here I am starting to use </em><strong><em>Prometheus terminology</em></strong><em> when talking about metrics, just because Prometheus is my tool of choice for metrics handling and because it is so awesome. It does not make text more difficult to understand by any means, but you will notice some tech language particularities, for example, that metrics get exposed.</em></blockquote><p>In the sequence diagram above you can see my recommendation for measurements and measurement placements for any flow of this type. You should expose noted metrics while you are building this kind of software, so you can always confirm that the live system is performing up the spec and quality standard.</p><h4>What to measure</h4><p>For the best observable flow we want to expose:</p><ul><li>Counter for every initiated and completed call — labeled to be relatable to the part of the flow</li><li>Counter for every caught error — labeled to see where it is happening. It is also important to at least have a distinction between 5xx and 4xx calls, but it can depend on the type of systems that are doing the communication. Sometimes some specific errors need to be tracked separately.</li><li>Timer for every successful call. If it is erroneous we do not actually care how much time it took, but you need to keep in mind that it might be important in your particular case.</li></ul><blockquote><strong><em>Tracking the status codes or other important request specifics </em></strong><em>— to maximize flexibility it is really important to always have exact status codes, the most usual and recommended way is to have it in labels (for example </em>statuscode=”200|401|404|500&quot;<em>). It is mostly quite easy to group and remove granularity using proper queries, but if you do not have enough data it is not possible to generate proper granularity after events are done and measured.</em></blockquote><h4>Where to measure</h4><p>Since we are not measuring processing times or data throughput, but pure data flow metrics, we should focus on:</p><h4>Why to measure</h4><p>One word — <strong>Quality.</strong> The main and overarching topic of this is quality of service provided to the consumer(QoS). In order to provide good and reliable service, you need to know how your system performs. To know how your system performs you should observe the data you exposed.</p><p>And what should you observe?</p><p><strong>Performance</strong></p><p>The performance of your API endpoint is a no-brainer in this case — that is the longest arrow in the successful flow (the arrow closest to the frontend API).</p><p><strong>Plotting:</strong> We can more or less plot that over time to keep an eye on any changes in performance. To do this we can combine over time plots of different essential metrics. Usually a graph of the total performance of the flow goes nicely on the same dashboard with smaller graphs of performance of separate segments, and in parallel with the call counter (top-left rhomboid).</p><p>When the alarm goes off you want to know how much the total performance is off, is there any particularly influential segment in the flow and is the performance degradation related to the number of requests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wO3GmezBwxMdo1y3.jpg" /></figure><p>In the example above you see the application performance plot over 12 hour period. At some point performance degradation occurred due to a sudden spike in traffic (number of requests). Observe the total response time and how sub service 2 and sub service 3 both influenced the response time degradation, each in it’s own way.</p><p>The yellow line on the main performance plot represents the start of the performance breach (response over 300ms). The red line is an alert that was triggered after a breach going on for some time ( a threshold you set — a few minutes for example). The green line represents a moment when the system is considered to be back to normal</p><p><strong>Alerting:</strong> In case of performance monitoring there are two approaches that I recommend to set up alerting, and it is not uncommon to see both set up for some flows:</p><ol><li>Based on performance budget (for example — respond within 300ms).</li><li>Based on system performance history (for example — if responses are 20% slower then yesterday at the same time).</li></ol><blockquote><em>You do not get to decide how well you are doing</em></blockquote><blockquote><em>Make sure to know that this way of measuring, which is proper, dynamic and quality-driven, comes with a caveat. The same way service#1 that is calling service#2 needs to qualify the performance of service#2, your users are qualifying your application’s performance, and you are there to make sure that it is the best performance you can achieve.</em></blockquote><blockquote><em>This is why it is important to involve all segments of the (eco)system in the general performance measurement.</em></blockquote><p><strong>Error rates</strong></p><p>Basically, all the counters on error responses are important for monitoring error rates. This measurement is important for the quality of service, but if you are doing things properly these metrics will be important to identify any technical flaws as well.</p><p><strong>Plotting:</strong> Errors are characterized by status codes between 400 and 599. The same as anything else, you want to track these over time. There are many cases when you would want to actually know the difference between 4xx and 5xx errors so here is a general rule of thumb for choosing one or the other:</p><ol><li>Observe them <strong>separately</strong> if the subsystem whose errors we are observing has specific 400 behaviors apart from 404 (for example authentication/authorization…) so we can track any potential problem on this side.</li><li><strong>Together</strong> if your system does not have special 4xx behaviors and anything in this rage would represent system error.</li><li>In many cases you can exclude 404 errors from erroneous behavior monitoring — on the web it is generally quite usual for people to miss a page address.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9H4FLmZPLaE8BwtZ.jpg" /></figure><p>Above we can see the error rate of the endpoint going up because of the high error rate on subservice#1.</p><p>We can observe two things from this plot (monitoring dashboard screenshot):</p><ul><li>The high error rate is caused by subservice#1</li><li>The error rate on the outside was ramping up slowly although the problem on the subservice was obviously immediate in nature. This is usually due to a good resilient architecture choice — some responses are cached and errors were only starting to be visible after the cache has expired, so the system as a whole was less impacted.</li></ul><p><strong>Alerting:</strong> Error rate alerts should be loud and strict. These are usually based on SLOs which means that our alert trigger is if our API breaches the SLO for x period of time. This should be implemented on the end of the whole flow, but also on every segment.</p><blockquote><em>All errors are made equal, but some are more equal than others.</em></blockquote><blockquote><em>You got to know which errors are costing you the most, not just in terms of money but also in terms of UX and the general appeal of your app. Some examples:</em></blockquote><ul><li>Is your system a private application and do your users have to be logged in and authenticated? 4xx errors will mean a lot to you, because in behind-the-login apps authentication/authorization errors (401, 403, 405… depending how deep you go in your API design) and many 404s are indicators of unwanted behavior of either app, or users (or both).</li><li>If it is one of the systems you are integrating with (as in our subsystems), and you start receiving 404, that is a cause for immediate alert and attention by your team — something changed on the provider side.</li><li>Does your system have public search functionality? Many 404s are ok for you, especially on search and detail pages because of bots and all other different crawling automations that will just go and try random stuff. Of course, this has many other implications to SEO and such, but this article does not care about that.</li><li>5xx errors are never ok.</li></ul><p><strong>Make sure not to waste time on staring at monitoring dashboards — set up alerts and go do something fun.</strong></p><h3>Summary</h3><ul><li>Code instrumentation is a part of the software development process.</li><li>You need to consider and design the observability of your system with as much attention to detail as when you created the architecture itself.</li><li>The proper perspective to monitor the performance of the service from, is the perspective of the service consumer. Monitoring the internal performance of the service does not account for integration and networking.</li><li>ERRORS BAD.</li></ul><p><em>Originally published at </em><a href="https://rastko.tech/microservices/sre/monitoring/alerting/observability/programming/architecture/solutions/2020/09/12/monitoring-microservice-data-flows.html"><em>https://rastko.tech</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ea04532db3f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/adidoescode/monitoring-data-flows-in-a-microservice-system-5ea04532db3f">Monitoring data flows in a microservice system</a> was originally published in <a href="https://medium.com/adidoescode">adidoescode</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Planning observability of your system]]></title>
            <link>https://medium.com/adidoescode/planning-observability-of-your-system-96b6ad13f03?source=rss-d0d24ba520a9------2</link>
            <guid isPermaLink="false">https://medium.com/p/96b6ad13f03</guid>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[logging]]></category>
            <category><![CDATA[microservices]]></category>
            <dc:creator><![CDATA[Rastko Vukasinovic]]></dc:creator>
            <pubDate>Thu, 05 Nov 2020 21:42:59 GMT</pubDate>
            <atom:updated>2021-06-29T07:38:03.753Z</atom:updated>
            <content:encoded><![CDATA[<p>This is the guide through the observability space to assist you with planning the observability of your next project.</p><blockquote><strong><em>Observability</em></strong><em> is the ability of your system to expose data about its performance and behavior so we can further monitor, analyze, and alert on certain trends or changes. Making your system observable requires certain effort in your project.</em></blockquote><h3>Why should you care</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XjnS2Qb6LCn0rSxn.jpg" /></figure><p>Observability of your system is the <strong>top non-functional requirement</strong>. In my opinion, not even application performance tops observability. It is absolutely necessary that you know what is happening in your system at any given time and <strong>in level of detail necessary to be able to keep everything running smoothly by adequately reacting to change in the system</strong>.</p><blockquote><em>Of course, there is a possibility that you do not share this opinion, in which case you would, probably, rather develop your product, do a bit of testing, UAT, and push it out. Probably because it requires a certain amount of work and is considered non-essential as you start. If Google Analytics is all you need, I am not here to argue that, refer back here when you change your mind. If, however, you want to dive into this topic and start planning, I want to share some insights that should help you pin down stuff that is most important for you and prioritize.</em></blockquote><p>There are quite a few aspects of observability in service-oriented architecture, and in this article, I will be focusing on discussing every aspect with a little bit of depth in order to help you better understand, focus and plan developing your observability. To help you plan and have a broader conversation with all the people involved in planning your project, I will provide some insight into the most popular stacks as well and give insights into the most important defining factors.</p><p>Your system being observable can mean multiple things on as many levels, but generally, it means that the system exposes data that we can consume to:</p><ul><li>Monitor system in real-time.</li><li>Alert or perform some automated task based on defined data state, or rather a change of state.</li><li>Analyze and predict behaviors, requirements etc…</li><li>Do a detailed analysis of the incident or any distinct event in the system.</li><li>Any combination of the above.</li></ul><p>Since it is all about using proper tools, let’s spend some time on quick discussion about what we need and what tool/type of observability we choose.</p><h3>Characterizing capabilities</h3><p>When we talk observability of the system, the way we are planning it really depends on what we want to achieve. Ultimately, we would love to achieve everything, but some ability to focus and plan would be nice. Let us take a look at some factors to take into account when prioritizing observability.</p><p><strong>Main types of data:</strong></p><ul><li><strong>Logs</strong> — Old, but gold. The first that ever existed has all the data about what is going on, but slow, hard to process, and needs huge system throughput to bring usage close to real-time. You can still go to the file that was generated last and grep it for a particular phrase or just look for the time when the error happened, which will forever be my favorite thing I ever learned how to do (apart from making pizza from scratch).</li><li><strong>Metrics</strong> — Quick, flexible, made for quick availability and low throughput requirement, easy and long retention. Not too much data per unit of information, real-time trend analysis and reaction.</li><li><strong>Traces</strong> — Everything you would like to know about the call you made, seems like magic especially in a well-instrumented distributed system. Sometimes slower than logs, but allows for deeper analytics.</li></ul><p><strong>Performance qualities:</strong></p><ul><li><strong>Usability</strong> — Do metrics allow a good way to monitor and analyze them, with flexibility in data architecture to give enough versatility.</li><li><strong>Speed/throughput requirements</strong> — how much metrics system can take before receiving some performance penalty. Also, how fast are the metrics available for consumption/use (both manual and automated).</li></ul><p><strong>Way we plan to consume data:</strong></p><ul><li><strong>Monitoring</strong> — real-time plotting and observation of system state.</li><li><strong>Alerting and automatic triggers</strong> — reacting to events by broadcasting notification or triggering events.</li><li><strong>Analytics</strong> — analyzing collected data to gather useful learnings about the system and/or perform <strong>root cause analysis (RCA)</strong> of the problem in the past (usual case) or ongoing problem (rarer since Analytics is mostly about applying complex algorithms to big data sets to draw conclusions, and that takes time).</li></ul><p>We can quickly draft the following table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6SG03pz-c2RfSIw8cLhcBw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*swN2vSiQjqJrFb2I.jpg" /></figure><h3>Creating observability</h3><p>As you are building a system, no matter if it is a huge monolithic application, or a multiservice-something, planning observability you want to achieve will get you places. For that, you’ll have to know what kind of tools you need and what additional infrastructure will it take… of course, but there is also a choice of where to start. It is less obvious, but deciding to do everything might be a costly decision depending on how much time you got to deliver the first version of your system.</p><blockquote><strong><em>DevOps</em></strong><em> — in this moment it is worth mentioning that for most of the stuff that I write, I assume your goal is full DevOps, and that your team is building everything and running it in all the environments. If you are getting something pre-built, you have an army of engineers or reasonably long timelines there might be a way to get it all in time.</em></blockquote><h3>Logging</h3><p>Generally, you do not need to do anything in order for your system to start gathering logs, logs will be available to you anywhere from the default log directory for the OS of choice to Kubernetes logs. What you will need is everything else — literally.</p><p>Going from just a stack of call logs inside a gazillion line file on the disc to a searchable log database that can also be used for a certain level of alerting and many different applications is the matter of planning. I usually recommend two well-known stacks, that differ in the log collection segment:</p><ul><li><a href="https://www.elastic.co/elastic-stack"><strong>ELK</strong></a> — Elastic, Logstash, Kibana which is more standard one</li><li><a href="https://www.digitalocean.com/community/tutorials/how-to-set-up-an-elasticsearch-fluentd-and-kibana-efk-logging-stack-on-kubernetes"><strong>EFK</strong></a>- Elastic, Fluentd, Kibana which is a more flexible one in terms of data flow management and further design scalability. It is also recommended for Kubernetes.</li></ul><p>I guess everyone loves 3-letter abbreviations so nobody bothered to add F at the start of both since it became a standard. F for <a href="https://www.elastic.co/beats/filebeat">Filebeat</a>.</p><p>Basically, you need to setup EXK and Filebeat, Filebeat will pick up the logs as they arrive to any of the mentioned standard log outputs and send them to a log collector, that gets stored into the Elastic database so logs will be indexed and made searchable and visualizable through Kibana.</p><p>Obviously, the more activity you got in your system, the more logs there is, more data throughput becomes crucial for your logs to get to you — Filebeat needs to send logs to Elastic that then needs to index them (on the way there is a throughput of whatever you use). I have seen cases of hundreds of terabytes of logs a day with the need of month to 3 months retention… It becomes a whole separate system to maintain.</p><p>The good thing is that, when overwhelmed, these setups are quite good in the degrading performance but keeping data, so if you do not run out of physical hardware and everything just halts, your logs may be late to Kibana because elastic can’t index, or they might be stuck in some queue, but they will not be lost. That is, at least, something.</p><blockquote><strong><em>In the wild</em></strong></blockquote><blockquote><em>Stack itself is relatively easy to deploy with lots of resources available online, from tutorials to docker-composed sets, </em><a href="https://www.terraform.io/"><em>Terraform</em></a><em> submodules and the </em><a href="https://helm.sh/"><em>Helm</em></a><em> charts to do it for you. The first hard thing to do is set everything right the first time from the delivery of logs perspective. The second hard thing to do is not to kill your logging stack with too much or too big logs.</em></blockquote><blockquote><em>Good luck.</em></blockquote><h3>Metrics</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-JKci7283MaOLm-f.jpg" /></figure><p>On the metrics side of things are a lot different than on the logs side of things — although there is a general majority choice of a tool, and thus an approach to it, there are actually two different approaches based on a way metrics are exposed/collected:</p><p>Based on two stand out representatives of two groups, I will provide a quick comparison.</p><h4>Pull — Prometheus</h4><p><a href="https://prometheus.io/">Prometheus</a> is my current metrics collection stack of choice, developed by <a href="https://soundcloud.com/">SoundCloud</a> and fully promoted to open source in 2015, although it was available for longer.</p><p>Pull means that the metrics collector is scraping services that are being monitored by it. In turn, the system requires that any service that needs to be monitored by Prometheus exposes metrics. Data exposition format allows “condensation” of the metrics that are exposed, so they basically can accumulate until scraped for both variable scraping intervals, but also the longer periods in case of incidents with collector.</p><blockquote><em>Metrics are kept in the service’s memory until the collector pulls (scrapes) the metrics.</em></blockquote><p>The architecture of the metrics collection and way Prometheus is built gives us two important architectural advantages:</p><ul><li><strong>Resilience</strong> — If Prometheus collector dies for some time, metrics are not lost because they are kept in the service’s memory until scraped</li><li><strong>Federation</strong> — the same way the service exposes metrics to be collected by Prometheus, Prometheus itself can expose aggregated metrics to be collected by other Prometheus system, allowing for metrics collection scaling by creating a tree hierarchy collection structure known as Federation in the Prometheus ecosystem</li></ul><p>Prometheus has a full set of needed components within its ecosystem — it collects metrics in form of time-series and stores them, it also has a alert manager to handle alerts and push gateway to support sort lived services (jobs, container orchestration, serverless…).</p><p>There is a certain complexity to using pull metrics that you have to take into consideration — since it scrapes services for metrics you would need service discovery to always know what to scrape, especially in container orchestration space.</p><blockquote><strong><em>In the wild</em></strong></blockquote><blockquote><em>Prometheus is a multi-component system that is relatively easy to deploy tanks to the community, but it has lots of quirks for you to handle. It will be super useful, and I recommend it wholeheartedly, but be ready to do some maintenance on it as it is a fast data collection and parsing system having many resilience points which often cause stuff like memory leaks and hard drive space overload.</em></blockquote><blockquote><em>Every time you want to plot any real-life scenario data, you are making mid to high complexity query to Prometheus, which can overload it, especially in high-resolution intensive plots we can make Prometheus less efficient for its main purpose. Luckily, you can record (sort of index/cache) certain queries (rules) and make the whole thing more performant.</em></blockquote><blockquote><em>Prometheus can be used as a source directly in </em><a href="https://grafana.com/"><em>Grafana</em></a><em>, which is the best open source monitoring+analytics frontend dashboard tool out there.</em></blockquote><h4>Push — StatsD</h4><p>The first monitoring stack I could seriously call that was <a href="https://github.com/statsd/statsd">StatsD</a>, developed by <a href="https://www.etsy.com/">Etsy</a>.</p><p>Push means that your system is pushing the signals to StatsD all the time.</p><p>This simplifies lots of architectural requirements since service discovery is not needed, the service itself needs to know where the StatsD server (also called daemon) is, and that is that it can receive signals from many sources.</p><blockquote><em>I personally think that price in a bit of complexity on the Prometheus side is worth it.</em></blockquote><p>Looking at the stack itself, where Prometheus has everything in one place, StatsD has its daemon and clients (for every language), but everything else needs a custom thing to plugin:</p><ul><li>To read ad transform data you need something like <a href="https://graphiteapp.org/">Graphite</a>, that you can later also use as a source for Grafana.</li><li>For alerting you need another module.</li></ul><blockquote><strong><em>In the wild</em></strong></blockquote><blockquote><em>Fact that the daemon gets hit for every event in your service and that your service needs to make a request every time the event you are measuring is done leads to new requirements on the software engineering side if the number of request to your app grows. You need to sample every metric if you do not want most of your resource consumption to go into metrics. There is this really good anecdote from Etsy that they actually had alerts on metrics sampling, so Ops guys would come for you if you did not sample or you did not sample enough.</em></blockquote><blockquote><em>The best way to visualize in this case is also Grafana, but in this case you can not use StatsD as a data source directly, you need Graphite.</em></blockquote><blockquote><strong><em>Sampling</em></strong><em> — the technique of reducing number of metrics in a way that you are only sending samples of data (that is why it is also known as downsampling). What you do is send one in 100 metrics generated (one in every 100 requests gets measured).</em></blockquote><blockquote><em>Sampling helps in both push and pull case, but it is more useful in push case since it reduces resource consumption in the collection and storage space needed for retention. In Pull case, it just saves you disk space and some resources for more complex queries.</em></blockquote><h3>Tracing</h3><p>As the data boom hit the industry it did not miss the observability of stuff we are building. Now we can expose every small detail about the long term behavior of our app in the form of a huge amount of data that tracks any request and to end, even in distributed systems.</p><p>The main difference between the two large-volume observabilities is that logs in their essence represent a collection of individual events and traces are designed to represent set of connected events and their holistic picture for further analysis. The base of tracing observability is code instrumentation and there are many ways it is done (including connecting log events using request ids), but the currently accepted global standard is OpenTracing API (now part of <a href="https://opentelemetry.io/">OpenTelemetry</a>).</p><p>Two tools to mention are <a href="https://zipkin.io/">Zipkin</a> and <a href="https://www.jaegertracing.io/">Jaeger</a>, developed by <a href="https://twitter.com/">Twitter</a> and <a href="https://www.uber.com/">Uber</a> respectively, both using OpenTracing API and both supporting every programming language in almost every form (framework) out there.</p><p><strong>Zipkin</strong> is simpler to deploy a more conventional client-server collection system that is oriented towards less complex, non-container-orchestrated systems. <strong>Jaeger</strong> is more complex, distributed tracing oriented, and preferably deployed to Kubernetes.</p><blockquote><strong><em>In the wild</em></strong></blockquote><blockquote><em>Tracing tools logically need more complex setups in their backends with fast databases, map/reduce systems, and special scaling needs in order to provide us with the best trace grouping and analysis capability. These are always complex systems of many small and big components to deploy and maintain across the space.</em></blockquote><blockquote><em>In return, we get the ability to see requests travel through our system, and the capability to identify the exact place where any of the erroneous ones failed and which logs the failure produced etc…</em></blockquote><blockquote><em>Data inflation!</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nu7UJfbJSEDvWueg.jpg" /></figure><h3>Summary</h3><ul><li>Logs have all the data, but you need a delivery system, database, indexing, and search/dashboard tool in order to be able to use them beyond merely skimming the text in a file on a disk somewhere</li><li>Metrics do not have too much data but they provide an essential way of measuring your system’s performance and reacting to it in real-time</li><li>Tracing is giving us deep insight into the system’s behavior, and holistic analysis of every request. Price is a complex system to gather, store, analyze, and store to be searched and visualized.</li><li>Prometheus is the best</li><li>Use Grafana.</li></ul><h3>P. S.</h3><p>I wrote this as a separate article since I realized that intro to another observability article I have been writing can actually be a planning and tools insight guide for a broader audience — aiming for all specialties like engineers, architects, tech POs, QA engineers… I hope it worked :D</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/0*gYjLwV8M9oL9ByE_.jpg" /></figure><p><em>Originally published at </em><a href="https://rastko.tech/sre/observability/monitoring/logging/analytics/microservices/kubernetes/basics/2020/08/29/planning-observability-monitoring-analytics.html"><em>https://rastko.tech</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=96b6ad13f03" width="1" height="1" alt=""><hr><p><a href="https://medium.com/adidoescode/planning-observability-of-your-system-96b6ad13f03">Planning observability of your system</a> was originally published in <a href="https://medium.com/adidoescode">adidoescode</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Internal and external connectivity in Kubernetes space]]></title>
            <link>https://medium.com/adidoescode/internal-and-external-connectivity-in-kubernetes-space-a25cba822089?source=rss-d0d24ba520a9------2</link>
            <guid isPermaLink="false">https://medium.com/p/a25cba822089</guid>
            <category><![CDATA[nginx-ingress]]></category>
            <category><![CDATA[ckad]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[services]]></category>
            <dc:creator><![CDATA[Rastko Vukasinovic]]></dc:creator>
            <pubDate>Sat, 13 Jul 2019 22:41:36 GMT</pubDate>
            <atom:updated>2021-08-03T09:01:31.175Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>Services and networking — from ClusterIP to Ingress</strong></h4><p>As you are making your way through all the stages of your app’s development and you (inevitably) get to consider using Kubernetes, it is time to understand how your app components connect to each other and to the outside world when deployed to Kubernetes.</p><blockquote>Knowledge you will get from this article also covers “services &amp; networking” part of <a href="https://www.cncf.io/certification/ckad/">CKAD exam</a>, which currently takes 13% of the certification exam curriculum.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/633/1*yvSj52Q7s3V70LaT59QoCA@2x.jpeg" /></figure><h3><strong>Services</strong></h3><p><strong>Kubernetes services</strong> provide networking between different components within the cluster and with the outside world (open internet, other applications, networks, etc)</p><p>There are different kinds of services, and here we’ll cover some:</p><ul><li><strong>NodePort </strong>— service that exposes Pod through a port on the node</li><li><strong>ClusterIP </strong>— service that creates virtual IP within the cluster to enable different services within the cluster to talk to each other.</li><li><strong>LoadBalancer </strong>— creates (provisions) load balancer to a set of servers in the Kubernetes cluster.</li></ul><h4><strong>NodePort</strong></h4><p>NodePort service maps (exposes) port on the Pod to a port on the Node. There are actually 3 ports involved in the process:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/1*T5VLoopP7Eqzx6awLGK7Zw@2x.jpeg" /><figcaption>NodePort service exposes deployment (set of pods) to the outside of the k8s node</figcaption></figure><ul><li><strong>targetPort</strong> — port on the Pod (where your app listens). This is an optional parameter, if not present, Port is taken</li><li><strong>Port</strong> — port on the service itself (usually same as pod port)</li><li><strong>NodePort</strong> — port on the node that is used to access web server externally. By standard, NodePort can be in the range between <em>30000</em> and <em>32767</em>. This is an optional parameter, if not present, a random available port in a valid range will be assigned.</li></ul><h4><strong>Creating the NodePort service</strong></h4><pre>apiVersion: v1<br>kind: Service<br>metadata:<br> name: k8s-nodeport-service<br>spec:<br> type: NodePort<br> ports:<br> - targetPort: 6379<br>   port: 6379<br>   NodePort: 30666<br>selector:<br> db: redis</pre><h4><strong>Linking pods to a service</strong></h4><p><strong>Selectors</strong> are the way to refer (link) service to a certain set of pods.</p><p>A set of pods gets selected based on the selector (in almost all cases, pods from the same deployment), <strong>service starts sending traffic to all of them in a random manner effectively</strong> <strong>acting as the load balancer</strong>.</p><blockquote>If mentioned pods are distributed across the nodes, service will be created across the nodes to be able to link all the pods. In case of multi node service, service exposes same port on all nodes.</blockquote><h4><strong>ClusterIP</strong></h4><p>In the case of an application consisting of multiple tiers deployed to different sets of pods, a way to establish communication between different tiers inside the cluster is necessary.</p><p>For example, we have:</p><ul><li>5 pods of API number 1</li><li>2 pods of API number 2</li><li>1 pod of redis</li><li>10 pods of frontend app</li></ul><p>Each of above mentioned 18 pods have their own distinct IP addresses, but making communication that way would be:</p><ul><li>Unstable since pods can die and be recreated with new IP any time</li><li>Inefficient since we would have to load-balance within the integration part of each app</li></ul><p><strong>ClusterIP service provides us with unified interfaces to access each group of pods — it provides a group of pods with internal name/IP.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/384/1*cERBKWAopNkgN5nH2XArJg@2x.jpeg" /><figcaption>ClusterIP service provides inside the cluster addressing of different deployments</figcaption></figure><pre>apiVersion: v1<br>kind: Service<br>metadata:<br> name: api-1<br>spec:<br> type: ClusterIP<br>ports:<br>- targetPort: 9090<br>  port: 9090<br>selector:<br> app: api-1<br> type: backend</pre><blockquote><strong>ClusterIP</strong> is default type of service, so if service type is not specified, k8s assumes ClusterIP.</blockquote><p>When this service gets created, other applications within the cluster can access the service through service IP or service name.</p><h4><strong>LoadBalancer</strong></h4><p>In short, a LoadBalancer type of service is provisioning external load balancer in cloud space — depending on provider support.</p><p>The deployed load balancer will act as NodePort, but will have more advanced load balancing features and will also act as if you got additional proxy in front of NodePort in order to get new IP and some standard web port mapping (30666 &gt; 80). As you see, it’s features position it as the main way to expose service directly to the outside world.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*NcCpQOB5yFdb7rx-DkWdgw@2x.jpeg" /><figcaption>Load balancer services handling external access to pods in kubernetes</figcaption></figure><p>The main downside of this approach is that any service you expose needs it’s own load balancer, which can, after a while, have a significant impact on complexity and price.</p><p>Let’s briefly review the possibilities:</p><pre>apiVersion: v1<br>kind: Service<br>metadata:  <br> name: lb1<br>spec:  <br> externalTrafficPolicy: Local  <br> type: LoadBalancer  <br> ports:  <br> - port: 80    <br>   targetPort: 80    <br>   protocol: TCP    <br>   name: http  <br> - port: 443    <br>   targetPort: 443    <br>   protocol: TCP    <br>   name: https  <br> selector:    <br>   app: lb1</pre><p>Above creates external load balancer and provisions all the networking setups needed for it to load balance traffic to <strong>nodes</strong>.</p><blockquote><strong>Note from k8s docs: </strong>With the new functionality, the external traffic will not be equally load balanced across pods, but rather equally balanced at the node level (because GCE/AWS and other external LB implementations do not have the ability for specifying the weight per node, they balance equally across all target nodes, disregarding the number of pods on each node).</blockquote><p>If you want to add AWS ELB as an external load balancer, you need to add the following annotations to load balancer service metadata:</p><pre>annotations:    <br> service.beta.kubernetes.io/aws-load-balancer-backend-protocol:&quot;tcp&quot;    <br> service.beta.kubernetes.io/aws-load-balancer-proxy-protocol:&quot;*&quot;</pre><h3><strong>Ingress</strong></h3><h4><strong>Real life cluster setup</strong></h4><p>When getting into space where we are managing more than one web server with multiple different sets of pods, above mentioned services turn out to be quite complex to manage in most of the real life cases.</p><p>Let’s review the example we had before — 2 APIs, redis and frontend, and imagine that APIs have more consumers than just frontend service so they need to be exposed to open internet.</p><p>Requirements are as following:</p><ul><li>frontend lives on <a href="http://www.example.com/"><strong>www.example.com</strong></a></li><li>API 1 is search api at <a href="http://www.example.com/api/search"><strong>www.example.com/api/search</strong></a></li><li>API 2 is general (everything else) api that lives on <strong>www.example.com/api</strong></li></ul><p>Setup needed using the above services:</p><ul><li><strong>ClusterIP</strong> service to make components easily accessible to each other within the cluster.</li><li><strong>NodePort</strong> service to expose some of the services outside of node, or maybe</li><li><strong>LoadBalacer</strong> service if in the cloud, or</li><li><strong>proxy server</strong> like nginx, to connect and route everything properly (30xxx ports to port 80, different services to paths on the proxy etc)</li><li><strong>Deciding on where to do SSL implementation and maintaining it across</strong></li></ul><h4><strong>So</strong></h4><p>ClusterIP is necessary, we know it has to be there — it is the only one handling internal networking, so it is as simple as it can be.</p><p><strong>External traffic</strong> however is different story, we have to set up at least one service per component plus one or multiple supplementary services (load balancers and proxies) in order to achieve requirements.</p><blockquote>Number of configs / definitions to be maintained skyrockets, entropy rises, infrastructure setup drowns in complexity…</blockquote><h4><strong>Solution</strong></h4><p>Kubernetes cluster has <strong>ingress</strong> as a solution to the above complexity. Ingress is essentially a layer 7 load balancer.</p><blockquote><strong>Layer 7</strong> load balancer is name for type of load balancer that covers layers 5,6 and 7 of networking, which are <strong>session</strong>, <strong>presentation</strong> and <strong>application</strong></blockquote><p>Ingress can provide load balancing, SSL termination, and name-based virtual hosting.</p><p>It covers HTTP, HTTPS.</p><blockquote>For anything other then HTTP and HTTPS service will have to be published differently through special ingress setup or via a NodePort or LoadBalancer, but that is now a single place, one time configuration.</blockquote><h4><strong>Ingress setup</strong></h4><p>In order to set up ingress, we need two components:</p><ul><li><strong>Ingress controller</strong> — a component that manages ingress based on provided rules</li><li><strong>Ingress resources</strong> — Ingress HTTP rules</li></ul><h4><strong>Ingress controller</strong></h4><p>There are few options you can choose from, among them nginx, GCE (google cloud) and Istio. Only two are officially supported by k8s for now — nginx and GCE.</p><p>We are going to go with <strong>nginx</strong> as the ingress controller solution. For this we, of course, need new deployment.</p><pre>apiVersion: extensions/v1beta1<br>kind: Deployment<br>metadata:<br> name: nginx-ingress-controller<br>spec:<br> replicas: 1<br> selector:<br>  matchLabels:<br>   name: nginx-ingress<br> template:<br>  metadata:<br>   labels:<br>    name: nginx-ingress<br>  spec:<br>   containers:<br>   - name: nginx-ingress-controller<br>     image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller<br>     args:<br>     - /nginx-ingress-controller<br>     - configMap=$(POD_NAMESPACE)/ingress-config<br>     env:<br>     - name: POD_NAME<br>       valueFrom:<br>         fieldRef:<br>          fieldPath: metadata.name<br>     - name: POD_NAMESPACE<br>       valueFrom:<br>        fieldRef:<br>         fieldPath: metadata.namespace<br>     ports:<br>     - name: http<br>       containerPort: 80<br>     - name: https<br>       containerPort: 443</pre><p>Deploy ConfigMap in order to control ingress parameters easier:</p><pre>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: nginx-configuration</pre><p>Now, with basic deployment in place and ConfigMap to make it easier for us to control parameters of the ingress, we need to set up the service to expose ingress to open internet (or some other smaller network).</p><p>For this we setup node port service with proxy/load balancer on top (bare-metal /on-prem example) or load balancer service (Cloud example).</p><p>In both mentioned cases, there is a need for Layer 4 and Layer 7 load balancer:</p><ul><li>NodePort and possibly custom load balancer on top as L4 and Ingress as L7.</li><li>LoadBalancer as L4 and Ingress as L7.</li></ul><blockquote><strong>Layer 4 load balancer</strong> — Directing traffic from network layer based on IP addresses or TCP ports, also referred to as transport layer load balancer.</blockquote><p>NodePort for ingress yaml, to illustrate the above:</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br> name: nginx-ingress<br>spec:<br> type: NodePort<br> ports:<br> -targetPort: 80<br>  port: 80<br>  protocol: TCP<br>  name: http<br> -targetPort: 433<br> port: 433<br> protocol: TCP<br> name: https<br>selector:<br> name: nginx-ingress</pre><p>This NodePort service gets deployed to each node containing ingress deployment, and then load balancer distributes traffic between nodes</p><p>What separates ingress controller from regular proxy or load balancer is additional underlying functionality that monitors cluster for ingress resources and adjusts nginx accordingly. In order for ingress controller to be able to do this, service account with right permissions is needed.</p><pre>apiVersion: v1<br>kind: ServiceAccount<br>matadata:<br> name: nginx-ingress-serviceaccount</pre><blockquote>Above service account needs specific permissions on cluster and namespace in order for ingress to operate correctly, for particularities of permission setup on <a href="https://kubernetes.io/docs/reference/access-authn-authz/rbac/">RBAC</a> enabled cluster look at <a href="https://kubernetes.github.io/ingress-nginx/deploy/rbac/">this document in nginx ingress official docs</a>.</blockquote><p>When we have all the permissions set up, we are ready to start working on our application ingress setup.</p><h4><strong>Ingress resources</strong></h4><p>Ingress resources configuration lets you fine-tune incoming traffic (or fine-route).</p><p>Let’s first take a simple API example. Assuming that we have just one set of pods deployed and exposed through service named <em>simple-api-service </em>on port 8080, we can create <em>simple-api-ingress.yaml.</em></p><pre>apiVersion: extensions/v1beta1<br>kind: Ingress<br>metadata:<br> name: simple-api-ingress<br>spec:<br> backend:<br>  serviceName: simple-api-service<br>  servicePort: 8080</pre><p>When we <strong>kubectl create -f simple-api-ingress.yaml</strong> we setup an ingress that routes all incoming traffic to simple-api-service.</p><h4><strong>Rules</strong></h4><p>Rules are providing configuration to route incoming data based on certain conditions. For example, routing traffic to different services within the cluster based on a subdomain or a path.</p><p>Let us now get to the initial example:</p><ul><li>frontend lives on <strong>www.example.com</strong> and everything not<strong> /api</strong></li><li>api 1 is search api at <strong>www.example.com/api/search</strong></li><li>api 2 is general (everything else) api that lives on <strong>www.example.com/api</strong></li></ul><p>Since everything is on the same domain, we can handle it all through one rule:</p><pre>apiVersion: extensions/v1beta1<br>kind: Ingress<br>metadata:<br> name: proper-api-ingress<br>spec:<br> rules:<br> -http:<br>   paths:<br>   -path: /api/search<br>    backend:<br>     serviceName: search-api-service<br>     servicePort: 8081<br>   -path: /api<br>    backend:<br>     serviceName: api-service<br>     servicePort: 8080<br>   -path: /<br>    backend:<br>     serviceName: frontend-service<br>     servicePort: 8082</pre><p>There is also a <strong>default</strong> backend that is used to serve default pages (like 404s), and it can be deployed separately. In this case, we will not need it since the frontend will cover 404s.</p><p>You can read more at <a href="https://kubernetes.io/docs/concepts/services-networking/ingress/">https://kubernetes.io/docs/concepts/services-networking/ingress/</a></p><figure><img alt="Illustration of multi service kubernetes deployment being connected internally through ClusterIP and externally via ingress." src="https://cdn-images-1.medium.com/max/1024/1*kwzSoapGCmltYvOKW45-Yw@2x.jpeg" /><figcaption>Managing ingress and internal traffic in kubernetes</figcaption></figure><h4><strong>Bonus — More rules, subdomains, and routing</strong></h4><p>And, what if we changed the example to:</p><ul><li>frontend lives on <strong>app.example.com</strong></li><li>api 1 is search api at <strong>api.example.com/search</strong></li><li>api 2 is general (everything else) api that lives on <strong>api.example.com</strong></li></ul><p>It is also possible, with the introduction of a new structure in the rule definition:</p><pre>apiVersion: extensions/v1beta1<br>kind: Ingress<br>metadata:<br> name: proper-api-ingress<br>spec:<br> rules:<br> -host: api.example.com<br>  http:<br>   paths:<br>   -path: /search<br>    backend:<br>     serviceName: search-api-service<br>     servicePort: 8081<br>   -path: /<br>    backend:<br>     serviceName: api-service<br>     servicePort: 8080<br> -host: app.example.com<br>  http:<br>   paths:<br>   -path: /<br>    backend:<br>     serviceName: frontend-service<br>     servicePort: 8082</pre><blockquote><strong>Note (out of scope):</strong> You can notice from the last illustration that there are multiple ingress pods, which implies that ingress can scale, and it can. Ingress can be scaled like any other deployment, you can also have it auto scale based on internal or external metrics (external, like number of requests handled is probably the best choice).</blockquote><blockquote><strong>Note 2 (out of scope):</strong> Ingress can, in some cases, be deployed as <a href="https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/">DaemonSet</a>, to assure scale and distribution across the nodes.</blockquote><h4><strong>Wrap</strong></h4><p>This was a first pass through the structure and usage of k8s services and networking capabilities that we need in order to structure communication inside and outside of the cluster.</p><p>I, as always, tried to provide to the point and battle-tested guide to reality… What is written above should give you enough knowledge to deploy ingress and setup a basic set of rules to route traffic to your app and give you context for further fine tuning of your setup.</p><blockquote><strong>Important piece of advice:</strong> Make sure to keep all the setups as a code in files, in your repo — infrastructure as a code is essential part of making your application reliable.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a25cba822089" width="1" height="1" alt=""><hr><p><a href="https://medium.com/adidoescode/internal-and-external-connectivity-in-kubernetes-space-a25cba822089">Internal and external connectivity in Kubernetes space</a> was originally published in <a href="https://medium.com/adidoescode">adidoescode</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Local kubernetes setup with minikube on Mac OS X]]></title>
            <link>https://metaphorical.medium.com/local-kubernetes-setup-with-minikube-on-mac-os-x-eeeb1cbdc0b?source=rss-d0d24ba520a9------2</link>
            <guid isPermaLink="false">https://medium.com/p/eeeb1cbdc0b</guid>
            <category><![CDATA[helm]]></category>
            <category><![CDATA[minikube]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[k8s]]></category>
            <dc:creator><![CDATA[Rastko Vukasinovic]]></dc:creator>
            <pubDate>Wed, 02 Jan 2019 16:55:18 GMT</pubDate>
            <atom:updated>2019-07-19T20:38:08.077Z</atom:updated>
            <content:encoded><![CDATA[<h3>Local Kubernetes setup with minikube on Mac OS X</h3><h4>Kubernetes, container registry, Helm…</h4><p><a href="https://kubernetes.io/docs/setup/minikube/">Minikube</a> is ideal tool to setup <a href="https://kubernetes.io/">kubernetes</a> (k8s from now on) locally to test and experiment with your deployments.</p><p>In this guide I will try to help you get it up and running on your local machine, drop some tips on where and how particular stuff should be done and also make it helm capable (I assume when you use k8s that at some point you will want to learn about and use Helm, etcd, istio etc).</p><blockquote>This is your local k8s environment scaffolding guide.</blockquote><h3>Minikube installation</h3><p>Minikube works with virtual machine, and for this it can use various options <a href="https://hackernoon.com/tagged/depending">depending</a> on your preference and operating <a href="https://hackernoon.com/tagged/system">system</a>. My preference in this case is Oracle’s <a href="https://www.virtualbox.org/wiki/Downloads">VirtualBox</a>.</p><p>You can use <strong>brew</strong> to install everything:</p><pre>$ brew cask install virtualbox minikube</pre><p>In this case you could get some kind of inconclusive installation error related to virtualbox installation, especially on Mojave and probably everything afterwards.</p><p>Whatever it says, it is most probably a new security feature in MacOS X that is in your way.</p><p>Go to <strong>System Preferences &gt; Security &amp; Privacy </strong>and on <strong>General </strong>screen you will see one (or few) messages about some software needing approval to install. You should carefully review the list if there is more then one and allow installation of software you need — in this case software by <strong>Oracle.</strong></p><p>After that is done you can re-run the command above and when it is done you should be ready for next steps.</p><h3>Running and accessing the cluster</h3><p>Starting it would be as easy as</p><pre>$ minikube start</pre><p>In order to optimally utilize your local machine’s resources I would suggest stopping it when you do not need it any more… With VirtualBox in center of it, it will go through you laptop’s battery pretty quickly. Starting it again later will get you back where you left off:</p><pre>$ minikube stop</pre><p>Kubernetes dashboard is also available to you (while minikube is running):</p><pre>$ minikube dashboard</pre><p>I will assume you have <a href="https://kubernetes.io/docs/tasks/tools/install-kubectl/"><strong>kubectl</strong></a><strong> </strong>installed locally and that you are already using it for some remote clusters so you got multiple contexts. In this case, you need to list contexts and switch to minikube one (in following commands assuming default name that is, ofc, “minikube”)</p><pre>$ kubectl config get-contexts<br>$ kubectl config use-context minikube</pre><p>Now you are in the context of your local k8s cluster that runs on minikube and you can do all the k8s things in it.</p><h3>Ingress controller</h3><p>To run your deployments that have ingress (and I assume most of them will), you will need ingress add-on:</p><pre>minikube addons enable ingress</pre><p>Make sure that you setup ingress based on your local hosts. It basically means that whatever you set as host in your ingress rules needs to be set up in your /etc/hosts file</p><pre>[minikube ip] your.host</pre><p>Where “[minikube ip]” should be replaced with actual minikube ip. It also works with multiple, space separated local hosts after minikube ip.</p><p>Here is shortcut to do it in bash:</p><pre>$ echo &quot;$(minikube ip) local.host&quot; | sudo tee -a /etc/hosts</pre><h3>Container registry — i.e. Docker registry</h3><blockquote>Reality of the real container registry usage in local environment is a rough one, so I will provide easy, quick and dirty option that makes it quite easy to deploy your local work to your local k8s, but deprives you of really important experience of using proper container registry.</blockquote><h4><strong>Local container registry</strong></h4><p>Get your local docker context to point to minikube context:</p><pre>$ eval $(minikube docker-env)</pre><blockquote>To revert:$ eval $(docker-machine env -u)</blockquote><p><strong>When in minikube context, to start local docker registry:</strong></p><pre>$ docker run -d -p 5000:5000 --restart=always --name registry registry:2</pre><p>So, now you have local registry to push stuff to (as long as your docker is in context of minikube).</p><p><strong>You can now do:</strong></p><pre>$ docker build . -t &lt;your_tag&gt;<br>$ docker tag &lt;your_tag&gt; localhost:5000/&lt;your_tag&gt;:&lt;version&gt;</pre><p>At this point you can use <strong>localhost:5000/&lt;your_tag&gt;</strong>: as image in your deployment and that is it.</p><h4>Using remote container repo</h4><p>To use remote container repo locally you need to provide way to authenticate, which is through k8s <a href="https://kubernetes.io/docs/concepts/configuration/secret/">secrets</a>.</p><p>For local secrets management for ECR, GCR and Docker registry I recommend using minikube addon called <a href="https://github.com/upmc-enterprises/registry-creds">registry-creds</a>. I do n ot consider it safe enough to be used anywhere but in local env.</p><pre>$ minikube addons configure registry-creds<br>$ minikube addons enable registry-creds</pre><blockquote><strong>Note on ECR setup: </strong><em>Make sure that, if you are setting it for AWS ECR, and you do not have role arn you want to use (you usually wont have and it is optional), you set it as something random as “changeme” or smt… It requires value, if you just press enter (since it is optional) deployment of creds pod will fail and make your life miserable.</em></blockquote><p>In case of AWS ECR, that will let you pull from your repo directly setting url as container image and adding pull secret named awsecr-cred:</p><pre>imagePullSecrets:<br>      - name: awsecr-cred</pre><blockquote>I have to note here that running this locally worked quite chaotically for me and every session was new experience and new hack to make it work… Not a happy path.</blockquote><h3>Helm</h3><p>Helm is package manager for k8s, and is often used for configuration management across deployments. With high popularity of the tool and raising adoption, I want to end this guide with the note about adding helm to your local k8s env.</p><p>It is quite easy at this point, just have minikube up and:</p><pre>$ brew install kubernetes-helm<br>$ helm init</pre><blockquote>This should be deprecated information pretty soon, but in current case helm uses backend called <strong>Tiller </strong>and that is what gets installed/deployed during <strong>helm init </strong>execution.</blockquote><blockquote>You should check tiller deployment with: $ kubectl describe deploy tiller-deploy — namespace=kube-system</blockquote><p>Valuable read: <a href="https://docs.helm.sh/using_helm/">https://docs.helm.sh/using_helm/</a></p><blockquote>Now you have full k8s local environment able to accept all of your test deployments before you decide to put them in the cloud (or “raw iron” server somewhere).</blockquote><blockquote><strong>HAPPY SCALING</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RPlkI7b6Ek85uyaQ3qPS8w.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eeeb1cbdc0b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building general purpose blockchain]]></title>
            <link>https://metaphorical.medium.com/building-general-purpose-blockchain-71ffb8511ce?source=rss-d0d24ba520a9------2</link>
            <guid isPermaLink="false">https://medium.com/p/71ffb8511ce</guid>
            <category><![CDATA[blockchain-technology]]></category>
            <category><![CDATA[blockchain-purpose]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[blockchain]]></category>
            <dc:creator><![CDATA[Rastko Vukasinovic]]></dc:creator>
            <pubDate>Mon, 09 Jul 2018 13:46:01 GMT</pubDate>
            <atom:updated>2019-08-30T23:05:54.118Z</atom:updated>
            <content:encoded><![CDATA[<h4>towards blockchain solution for any micro service ecosystems</h4><p>As I have spent some time playing and working with blockchain solutions, at some point I decided to try and build general purpose blockchain system myself. So, to go beyond “your first blockchain” tutorial. I wanted to do it as an experiment, for purely academic purposes and see what comes out of it. Here is the idea behind it and general overview of what came out of my time spent on analyzing, designing and building product like this…</p><p>The idea is to understand what is needed to create blockchain that can be supplementary component for your backend — a service that plays nicely with your multi / micro service backend setup and provides blockchain functionality.</p><p>If built properly this can be generally useful to anyone wanting to remove single point of failure and strengthen data security by having decentralized, replicated and secure database. Ideally, this solution should reduce throughput of the system as little as possible, but if your data handling is so special that you need blockchain, performance is small price to pay.</p><h3>What is general purpose blockchain</h3><p>For more proper answer we got to look at long list of resources I prepared <a href="https://github.com/metaphorical/quantum-blockchain/blob/master/documentation/blockchain-digest.md">here</a>. Just a glance at content of given document tells you that world is still figuring out what is blockchain and what exactly this new piece of tech is able to solve for us. And if you exclude money, what are the actual use cases to optimize for?</p><p>On the other hand, what is basic set of features general purpose blockchain should have?</p><p>To answer this, we need to first look at were it may fit in your system (if it is not your core competency, product, use case and business):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nkgZXGF3RpvFZC2eOMNbBw.jpeg" /><figcaption>Your microservice high level architecture diagram</figcaption></figure><p>Blockchain storage solution is just another storage solution with specific use case — storing transaction and sensitive data in immutable way. If you look at simplified microservice system depicted above, it is just additional store that command and event bus use to store certain information to.</p><h3>Base set of features needed to be generally applicable</h3><h4>It should be a tool, not a star of the system</h4><p>Obviously, if we are building something to be generally used as a suplement to applications, we should have something that is:</p><ul><li>Easy to deploy</li><li>Highly configurable (it should adopt to needs of a system, not vice versa)</li><li>Integration into the system should be super simple (e.g. you deploy and configure it and after that it all goes through simple RESTful API)</li></ul><h4>It should have multiple consensus algorithms or work distribution systems you can choose from</h4><p>Due the fact that blockchains are distributed systems, consensus algorithms (AKA proof systems) are important part of blockchain functionality. They provide certainty around data consistency on the network, thus providing security to data storage, are responsible for data integrity and, in some cases, ensure proper work distribution.</p><p>General purpose blockchain should provide set of initial approaches to finding consensus in scalable way. Now, there is a few established consensus algorithms but using some for this case, make sense more then using others</p><p>I suggest few from following set of algorithms</p><ul><li>PoW — Proof-of-work algorithm (maybe not that needed for private industrial blockchain, but so basic that can be considered requirement)</li><li>DPoS — Delegated proof-of-stake algorithm</li><li>DBC — Delegated block creation</li></ul><p><strong>PoW — </strong>Simply put — mining machine (the one creating the lock) creates certain proof that is hard to create initially (needs some computational work), but is easy to check (like number coming after last proof and is dividable by last proof and 255)</p><p><strong>DPoS — </strong>Generally this means that several nodes in the system choose representative and then that representative represents their interest on the network. Going deeper into implementation — network is designed is such a way that there are clusters of nodes that all connect to single mining node and that mining node is on the networks whitelist so it can broadcast newly created block. In network like this, I would recommend developing work scheduling system as well to avoid data write conflicts. <strong>Note:</strong> in fully distributed system (like crypto ledger), we would also pay attention to stuff like voting system to elect mining nodes, anonymity etc, in case of industrial blockchain it is all about use case and optimal way to achieve or maintain consensus.</p><p><strong>DBC — </strong>this type of consensus algorithm is only to be used in private networks and is pure scheduled resource consumption — you basically broadcast transactions to all the nodes and execute every next block creation in next node on the list, or rather next healthy node on the list. Maintaining replication, stability, consensus and work distribution across the network. If done like this, other distributed computing algorithms can be used on this system, like searches and such.</p><blockquote>It would be a plus if solution additionally provides a way to write and use new algorithm (as a plugin, or being easily added by community), tho I think each of these would be heavily embedded into the structure of a system.</blockquote><h4>Multi chain</h4><p>It should be able to spawn multiple chains withe configurable structure of blocks.</p><p>In order for it to truly be general purpose we have to cover two obvious cases, first of them being the need for multiple and differently structured chains being maintained the same app.</p><h4>Denormalization</h4><p>Second important need to be aware of is denormalizing the data in certain form, from blockchain to some form of conventional database or reporting system.</p><h4>It should offer content encryption out of the box and on by default</h4><p>It would be lovely if we can just add key to encrypt and then be and then define fields to be encrypted in particular chain.</p><h4>It should be easily deployable and integratabtle in devops systems</h4><p>Provide Dockerfiles, manifests and other receipts out of the box, to make integration easier. Maybe also provide versioned api contracts to maintain and monitor stability.</p><h4>It should have some kind of admin UI</h4><p>Really thin UI layer built in something lite that just allows manipulation with small set of features. Basic features:</p><ul><li>Adding users</li><li>Forking chains</li><li>Managing encryption and keys</li></ul><h4>Basic analytics dashboard</h4><p>Looking at usage, chain sizes, performance. Analyzing performance impact of chain growth etc.</p><p>Provide at least plug to send data to Google Analytics, and plan to build more.</p><h3>Nice to have</h3><ul><li>Custom code execution (aka “contracts” capability)</li><li>Sockets integration (as an option, but possibly one of the technologies to use for scheduling control for DBC)</li><li>CLI client</li></ul><h3>How to do it?</h3><p>Not completely sure yet, but I know how to design it, and I do have some programming skills, so I have experimented with what has to be done in order for open source general purpose blockchain to make sense.</p><p>Altho done purely for academic purposes, in order to satisfy hunger for knowledge and hands on experience with thing like this, I think it has all the potential to grow further.</p><p>For this reason, I have spent some effort to add structure around the project, provide enough documentation (still some work in progress) and also add the Trello board and the license. Project can be found in the repository <a href="https://github.com/metaphorical/quantum-blockchain">https://github.com/metaphorical/quantum-blockchain</a>, I have described my idea of contribution process in readme and I want to include anyone who wants to contribute, so feel free to jump onboard if this project is interesting enough for you.</p><p>I will keep developing the project in my free time as I want to take it to version 1.0 with rounded set of features (MVP that is still undefined).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=71ffb8511ce" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>