<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ahmad Iqbal Ali on Medium]]></title>
        <description><![CDATA[Stories by Ahmad Iqbal Ali on Medium]]></description>
        <link>https://medium.com/@ahmadiq?source=rss-7d9450e34ec2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*jxsiivAe5Pv6pttgm0dZww.jpeg</url>
            <title>Stories by Ahmad Iqbal Ali on Medium</title>
            <link>https://medium.com/@ahmadiq?source=rss-7d9450e34ec2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 04:49:56 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ahmadiq/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Monitoring on Kubernetes]]></title>
            <link>https://medium.com/stakater/monitoring-on-kubernetes-ac5231c8167f?source=rss-7d9450e34ec2------2</link>
            <guid isPermaLink="false">https://medium.com/p/ac5231c8167f</guid>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[monitoring]]></category>
            <dc:creator><![CDATA[Ahmad Iqbal Ali]]></dc:creator>
            <pubDate>Mon, 13 May 2019 06:30:18 GMT</pubDate>
            <atom:updated>2019-05-13T06:30:18.562Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VYQSbSoWjcP3KgqnnKTDKw.jpeg" /></figure><p>Monitoring is an important component of observability. With all our applications running along-with the various tools, we need to keep an eye on each as well as the underlying frameworks and infrastructure. This helps us ensure and stay in the know of how each component is performing and whether they are achieving the availability and performance expectations that they are supposed to. Monitoring helps us gain insight into cloud-native systems, and combined with alerting it allows DevOps engineers to observe scaled-out applications</p><p>Cloud-native systems by nature are ephemeral. They may be recycled at any time, be scaled up or down, triggering events in each case. And given the scale that they can grow to, the monitoring should not just treat any specific component independently, but rather be able to aggregate the results of the components.</p><h3>Metrics</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/746/1*ERST8jrPcVCfLeQJRDXvVQ.jpeg" /></figure><p>Metrics are the measurements that help give us a picture of the system is behaving and help us make informed decisions of whether there needs to be any change in the system to optimize the performance of the application as per expectations. Metrics are a combination of data from measuring multiple events. Cloud-native monitoring tools cater to different types of measurements by having various metrics. Some of these metrics may be cumulative metrics that can only ever increase. For others the value may go up and down over time and only a point-in-time value will be required. The idea is to aggregate similar events to gain a higher level of insight. Metrics are generally time based, therefore we usually collect metrics periodically, such as once per second.</p><h3>Prometheus</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/114/1*IAkVYhV3VTF2UBptrVGdJA.png" /></figure><p>At Stakater, we use Prometheus as our monitoring tool. Prometheus is an open source toolkit for monitoring and alerting under the Cloud-native Computing Foundation (CNCF). Some of the key features that we like about it are that it pulls metric data from services and each service doesn’t need to push metrics to Prometheus itself. Therefore Services need to only expose endpoints where they publish their metrics, and prometheus will call this endpoint and collect the metrics by itself. Additionally service targets can be discovered either using service discovery or via static configuration. Prometheus’ powerful query language, PromQL, is helpful for selecting and aggregating time series data in real time. Prometheus itself has a rudimentary web UI where the status of the monitored services can be viewed, however Prometheus also exposes it’s monitoring results via a HTTP api which makes it easy for an external system such as a more advanced visualization tool to consume and process the metrics. One such tool is Grafana which we will discuss a bit later.</p><p>Deploying and configuring Prometheus can seem a bit daunting at first but using Prometheus Operator it becomes quite easy.</p><h4>Prometheus Operator</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/126/1*-69z4lKwxdklZK165ronkQ.png" /></figure><p>An Operator is a convenient method of deploying and managing an application on Kubernetes. Using some Custom Resource Definitions, we are able to easily configure the deployment we would like, and the <a href="https://coreos.com/operators/">Operator</a> takes care of the heavy lifting of actual deployment work.</p><p>Prometheus Operator in this way makes the Prometheus configuration Kubernetes native and manages and operates Prometheus clusters. We can easily provide monitoring definitions for Kubernetes services and deployment and management of Prometheus instances. With the operator, we can easily create or destroy Prometheus instances, configure our Prometheus deployment with settings such as versions, persistence, retention policies, etc. And we can also use it to automatically generate monitoring configurations based on Kubernetes labels, and easily define the targets we would like to start monitoring.</p><p>The Operator ensures at all times that a deployment matching the Prometheus resource definition is running. The Operator also automatically generates Prometheus scrape configuration based on the definition of ServiceMonitor. And in case we need to modify the Prometheus stack deployment, instead of modifying each component Deployment or StatefulSet, we simply customize the CRDs and let the operator handle the orchestration.</p><h3>Monitoring Applications</h3><h4>Monitoring Services</h4><p>Prometheus consumes metrics that are exposed by services through http(s). We therefore don’t need to install any service agent, just expose a web port to reach our service’s endpoint and Prometheus server will regularly scrape the endpoint. The metrics format itself is in json which is easily readable by a human.</p><p>For any service that may not be natively integrable, it can be easily adapted using a Prometheus exporter. An example of this is the Node exporter that we will discuss shortly, which is used to expose hardware/os metrics from a node for prometheus to consume.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/674/1*NoPwTF56GsOdnDZlb7lbtg.png" /></figure><p>The Prometheus object filters and selects ServiceMonitor objects, which in turn, filter and select Prometheus metrics endpoints. If there is a new metrics endpoint that matches the ServiceMonitor criteria, this target will be automatically added to all the Prometheus servers that select that ServiceMonitor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iI5PQLNW8Oo5EixV" /></figure><h4>Monitoring short-lived jobs</h4><p>While any regular applications will always be alive, and therefore their service will be available to be scraped periodically, a short-lived job may not be available long enough for prometheus to scrape it at the configured interval. For such cases, we have the Prometheus pushgateway which is a metrics cache. Any short-lived jobs can push their metrics to the pushgateway, from where they can be scraped by Prometheus server at the next scraping interval. Client libraries are available for a few platforms such as java, go, etc. so relevant jobs can easily push their metrics to pushgateway. The Prometheus text protocol however makes it so simple to push metrics that a script can even use a command-line HTTP tool like curl to send metrics.</p><h3>Monitoring Cluster</h3><p>Prometheus can be used in pro-actively monitoring the kubernetes cluster. This provides cluster visibility and help with capacity planning. The cluster activity can be available on a metrics dashboard, and can subsequently also be used to trigger alerts and notification</p><p>The following Cluster components can be monitored:</p><ul><li>Orchestration level metrics — Deployment state, resource requests, scheduling and api server latency, etc.</li><li>Kubernetes nodes — classical sysadmin metrics such as cpu, load, disk, memory, etc.</li></ul><h4>Monitoring Kubernetes Objects</h4><p>`kube-state-metrics` is a simple service that listens to the Kubernetes API server and generates metrics about the state of the objects. It listens to the Kubernetes API server and generates metrics about the state of the objects such as deployments, nodes and pods, which the prometheus server can scrape. This is useful to present orchestration metadata i.e. statuses of deployments, pods, replica statuses, etc.</p><h4>Monitoring Kubernetes nodes</h4><p>`Node_exporter`is a Prometheus exporter that exposes hardware and OS metrics for Prometheus server to consume. Some collectors are enabled by default in the Node exporter which may be enough to gather the required memory, Cpu and disk usage metrics for our requirements. The Node exporter is deployed as a daemonset when we use the Prometheus operator, so that it automatically covers all nodes in the cluster at any point in time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/656/1*EkiFgTiNl_Wr8Q0nu8n9KA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/725/1*bqBWk_mspI4-bh6SA_JFcg.png" /><figcaption>Example: pod loads</figcaption></figure><h3>Monitoring logging stack</h3><p>At stakater we use Elastic stack for monitoring, and it can be quite important to monitor the performance and availability of the logging stack. Since logs are an important part of observability, we need assurance that there isn’t any gap in capturing them. Monitoring the logging stack will provide that visibility and transparency as to the performance and availability of the tools involved in the logging stack.</p><h4>Monitoring Fluentd</h4><p>Fluentd has specific plugins that can be used for making metrics available for prometheus scraping. The `prometheus` Filter plugin is used to read and process the incoming log records, and the `prometheus` Output plugin is used to do the same with the records being forwarded, in our case to elasticsearch. And the `prometheus` input plugin is used to exposed the metrics for consumption. Since fluentd will be running as a daemonset, each instance will expose the metrics of its current node. The metrics that we can export from fluentd can be number of total records, queue length, emit count, retry count, error count, etc. These metrics can give us an idea of the health of each fluentd instance relative to the load that it is meant to process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Jktn-EACL8BzaUSA" /><figcaption>Example: fluentd metrics in prometheus</figcaption></figure><h4>Monitoring Elasticsearch</h4><p>Elasticsearch is used to index the logs, and we also persist all our logs in the elasticsearch instance, it is therefore important to monitor it. We use a third party prometheus exporter for exposing Elasticsearch metrics. We capture metrics such as free space on block device, count of request cache hit/miss, percent CPU used by process, etc.</p><h3>Visualizing Metrics</h3><p>Prometheus metrics can be used for alerting based on specific thresholds, however visualizing the metrics even if they are not triggering alerting thresholds, is also useful. Grafana is one of the most popular visualization platforms and add-ons in the Prometheus solution. It pulls data from the Prometheus server and can display that in range of visualizations. The visualizations can also be grouped into custom dashboards. With a visualization tool such as this, the state of the system can easily be viewed by all stakeholders in the project team.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kAwVOP7TVc5ROUyc" /><figcaption>Example: Resources consumption of a particular deployment (CPU, Memory, Volume, etc.)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ukQQ5UFF8Je5ClTG" /><figcaption>Example: Overall cluster resources consumption (CPU, Memory, Volume, etc.)</figcaption></figure><h3>Summing up</h3><p>The overall monitoring architecture will look as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Uwywd9lRbHu51Djd" /><figcaption>Bird’s eye view: Monitoring architecture</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*0FANFMBaw8Q8aLuPQJLOOA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ac5231c8167f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/stakater/monitoring-on-kubernetes-ac5231c8167f">Monitoring on Kubernetes</a> was originally published in <a href="https://medium.com/stakater">Stakater</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Auto-cleanup resources on Kubernetes and Openshift]]></title>
            <link>https://medium.com/stakater/auto-cleanup-resources-on-kubernetes-and-openshift-bbf5c5e2ae61?source=rss-7d9450e34ec2------2</link>
            <guid isPermaLink="false">https://medium.com/p/bbf5c5e2ae61</guid>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[namespaces]]></category>
            <category><![CDATA[cleanup]]></category>
            <category><![CDATA[openshift]]></category>
            <dc:creator><![CDATA[Ahmad Iqbal Ali]]></dc:creator>
            <pubDate>Mon, 29 Apr 2019 11:37:30 GMT</pubDate>
            <atom:updated>2019-05-07T08:12:23.982Z</atom:updated>
            <content:encoded><![CDATA[<p>Dangling resources in a deployment environment such as Kubernetes or Openshift can be a problem. It adds to clutter and may also consume resources unnecessarily. This is especially a problem when there are multiple administrators/developers that are managing the system, and the knowledge of the system is to some extent distributed between them. No one person may be able to definitively say why a particular empty or even non-empty namespace was created in the first place or why it has not been deleted since. This could be a temporary project created on a dev Openshift cluster for prototyping and testing a new feature, or it could be a temporary kubernetes namespace on a staging cluster to demo an application to a particular audience. Either way the scenario indicates that the namespace will be temporary, and all deployments being performed within that will not be required after a particular time. Following the principles of GitOps one can indeed perform all such system changes such as namespace creation and deployments, via Git functions, which inherently provides traceability of these changes. Hence one may be able to trace the creation of such a resource back to the intent as may be described in the commit message, or directly inquire from the committer about it’s purpose and validity. However such an effort to trace the timeline of these events from the git history, can quickly become a complex feat leading one away from more important tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/734/1*s0AtC1auZF1ssMfWc2pDgA.png" /></figure><h3>Jamadar</h3><p>At Stakater we love to think of ways where we can automate processes and tasks that take away the tedious workload and help. For this particular scenario we have created <a href="https://github.com/stakater/Jamadar">Jamadar</a>, a tool to help clean up dangling resources in Kubernetes and Openshift. <a href="https://github.com/stakater/Jamadar">Jamadar</a> is a Kubernetes controller that watches kubernetes resources at a configured time interval, and identifies dangling resources based on the age of the resource, and configuration provided in the annotations. Jamadar will help us execute scenarios such as deleting namespaces that are one month old. The resources that we would like to persist, and should not be recycled by Jamadar should have a particular annotation with value. Any resources without the annotation will be recycled by Jamadar based on the configured time interval. Additionally there are other actions that can be configured to be performed once the resource clean-up has taken place.</p><h3>Configuring</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/555/1*dteCw7nUSgSS_Yo3YtfZag.jpeg" /></figure><p>Jamadar can be configured with the following options at deployment time, in the configuration yaml in the ConfigMap.</p><pre><strong>pollTimeInterval</strong>: The time interval after which the controller will poll and look for dangling resources, The value can be in “ms”, “s”, “m”, “h” or even combined like “2h45m”<br></pre><pre><strong>age</strong>: The minimum age of the dangling resource that should be considered for clean-up. The value unit can be any one of “d”, “w”, “m”, “y”. e.g. “7d” for 7 days<br></pre><pre><strong>resources</strong>: The resources that you want to be taken care of by Jamadar. At the moment only supports `namespaces`<br></pre><pre><strong>actions</strong>: The Array of actions that you want to take. At the moment only supports `default` and `slack`<br></pre><pre><strong>restrictedNamespaces</strong>: The list of namespaces names that should be ignored</pre><p>Following is an example configuration in full:</p><pre>pollTimeInterval: 12h  # Values: &quot;ms&quot;, &quot;s&quot;, &quot;m&quot;, &quot;h&quot;.<br>age: 1m                # Values: &quot;d&quot; or &quot;w&quot; or &quot;m&quot; or &quot;y&quot;.<br>resources:<br>  - namespaces<br>actions:<br>  - name: default<br>  - name: slack<br>    params:<br>      token: &lt;token&gt;<br>      channel: &lt;channel-name&gt;<br>restrictedNamespaces: <br>  - kube-system<br>  - default<br>  - kube-public<br>  - prod<br>  - tools</pre><p><strong>Supported Resources</strong></p><p>At the moment Jamadar supports only namespaces in the resources that can be configured for clean-up. This also corresponds to projects in Openshift.</p><p>We will be adding support for other Resources as well in the future.</p><p><strong>Supported Actions</strong></p><p>At the moment Jamdar supports the following Actions with their parameters,</p><ul><li>Default: No parameters needed, it will just log to console the details.</li><li>Slack: you need to provide token and Channel Name as Parameters in the yaml file</li></ul><p>We will be adding support for other Actions as well in the future</p><h3>Deploying Jamadar</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/1*kx3oITzEx3fuhsbzXw8BVQ.jpeg" /></figure><p>The kubernetes deployment files are included in the Jamadar repository in the deployments folder. The deployment files include options to deploy using a single yaml manifest, multiple yaml manifests one for each resource type, or Helm chart. The configuration should be edited in the ConfigMap and deployment can be made as per one’s preferred method.</p><p>For more information you can visit the <a href="https://github.com/stakater/Jamadar">Jamadar repository on Github</a>.</p><figure><a href="http://stakater.com/?utm_medium=blog&amp;utm_source=auto-cleanup-resources-on-kubernetes-and-openshift-bbf5c5e2ae61"><img alt="" src="https://cdn-images-1.medium.com/max/700/1*0FANFMBaw8Q8aLuPQJLOOA.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bbf5c5e2ae61" width="1" height="1" alt=""><hr><p><a href="https://medium.com/stakater/auto-cleanup-resources-on-kubernetes-and-openshift-bbf5c5e2ae61">Auto-cleanup resources on Kubernetes and Openshift</a> was originally published in <a href="https://medium.com/stakater">Stakater</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stakater Forecastle: Automated Service dashboard for Kubernetes]]></title>
            <link>https://medium.com/stakater/stakater-forecastle-automated-service-dashboard-for-kubernetes-9ff78d728048?source=rss-7d9450e34ec2------2</link>
            <guid isPermaLink="false">https://medium.com/p/9ff78d728048</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[ingress]]></category>
            <category><![CDATA[dashboard]]></category>
            <dc:creator><![CDATA[Ahmad Iqbal Ali]]></dc:creator>
            <pubDate>Thu, 18 Apr 2019 12:58:48 GMT</pubDate>
            <atom:updated>2019-04-18T13:01:28.274Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*JVKqC5q4ulZxEN9GgpNJkQ.png" /></figure><p>In an environment where we have multiple tools and applications including micro-frontends, it may get a little tedious to keep track of the URLs of all of the services. It can get additionally confusing if there are multiple instances of the same tools deployed on different environments or kubernetes namespaces. Keeping track of these URLs and sharing them within a team or other group of consumers can become tiresome. We would like to have a central place where we can easily look for and access our applications running on Kubernetes. We would also like to have our apps on kubernetes to be dynamically discovered. For an easier way to manage this, the Stakater team has developed a Kubernetes controller, Forecastle.</p><p>Forecastle is a Kubernetes controller and a web dashboard. It watches Ingresses and looks for specific annotations on them to indicate whether the Ingress needs to be registered on the dashboard or not. If the relevant configuration is discovered in the annotations of the ingress, the Ingress URL is added to the Forecastle dashboard. The dashboard displays all the running applications that are registered with it, and provides a simple way to access them.</p><p>Following is an example of the Forecastle dashboard. The applications are automatically separated into sections based on the kubernetes namespace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/689/0*E7V5X6t4LgL32taC" /></figure><p>The dashboard page also has a search bar which can be used to filter the application list to find a particular application without having to scroll through the whole list.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/0*UT3DJvh9MAvu9Pzk" /></figure><h3>How it works</h3><h4>Configuration</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*oQ0zip4KX5SYSBExKWQzDA.jpeg" /></figure><p>Forecastle watches for specific annotations on ingresses. To make an app/ingress be discoverable and added to the Forecastle dashboard, the forecastle.stakater.com/expose annotation with true value should be added to the ingress. There are other annotations which can be optionally used to customize how the application is displayed in the dashboard, but without the expose annotation mentioned they will be ignored.</p><p>With forecastle.stakater.com/icon we can specify the URL of an image that should be used for the application button on the dashboard.</p><p>forecastle.stakater.com/appName can be used to specify a custom display name for the application on the dashboard. If this annotation isn’t specified, the name of the ingress is used by default.</p><p>A group/section name can be specified using forecastle.stakater.com/group. This is the section on the dashboard page under which the application will be listed. If the annotation is not used, the application will be displayed under the section titled after its namespace by default.</p><p>When an Ingress with the required annotations is created in the cluster, the ingress is added to the Forecastle dashboard. Likewise when the ingress is removed from the cluster, it’s link button is removed from the dashboard.</p><h4>Authentication</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*9LFeiRzoZMQhY0RN91xj0Q.jpeg" /></figure><p>While the Forecastle dashboard does not have a login mechanism built-in or support yet for integration with OAuth providers, etc., at Stakater we do use Keycloak single sign-on with Keycloak gatekeeper to protect the Forecastle dashboard. Keycloak gatekeeper redirects any unauthenticated requests to the Keycloak server, and once an authenticated session is established it forwards the requests to the target Forecastle container.</p><p>It is therefore recommended to use some external authentication mechanism to protect access to the Forecastle dashboard.</p><h4>Running multiple instances</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*RaLtDSVWgkciEJRmw5o8GA.jpeg" /></figure><p>If needed, we can run multiple instances of forecastle by deploying multiple instances and configure the specific list of namespaces that Forecastle should watch for ingresses. The list of namespaces to watch is configured in the ConfigMap of the deployment.</p><pre>data:<br>  config.yaml: |-<br>    namespaces:<br>    - qa-1<br>    - qa-2</pre><p>We can also have multiple Forecastle instances work within the same namespace and watch ingresses in the same namespace, but have applications register to only a selected number of the dashboards based on the instance name of the Forecastle deployment. The instanceName parameter is configured in the Forecastle deployment ConfigMap as follows:</p><pre>data:<br>  config.yaml: |-<br>    instanceName: dev-forecastle</pre><p>On the Ingress, this configuration will be specified using the forecastle.stakater.com/instance annotation. The annotation value can be a comma separated list of the forecastle instances where we want to register the ingress. This way we can have the application display on a single or multiple dashboards based on the value we specify in the instance annotation.</p><p>Having multiple instances of Forecastle can be useful from an authorization point-of-view. For example if we want dev team members to only have access to applications deployed for the dev team, they can have authorization to use the relevant Forecastle dashboard only. Similarly the QA team can have access to only a QA relevant Forecastle dashboard. And it’s possible there may be some tools for which we are using a single instance across different environments, e.g. if we have a single instance of Jenkins working for both the dev and qa environments. In such a case we can have Jenkins be displayed on both instances of Forecastle dashboard.</p><figure><a href="http://stakater.com/?utm_medium=blog&amp;utm_source=stakater-forecastle-automated-service-dashboard-for-kubernetes-9ff78d728048"><img alt="" src="https://cdn-images-1.medium.com/max/700/1*0FANFMBaw8Q8aLuPQJLOOA.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9ff78d728048" width="1" height="1" alt=""><hr><p><a href="https://medium.com/stakater/stakater-forecastle-automated-service-dashboard-for-kubernetes-9ff78d728048">Stakater Forecastle: Automated Service dashboard for Kubernetes</a> was originally published in <a href="https://medium.com/stakater">Stakater</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Alerting in Kubernetes]]></title>
            <link>https://medium.com/stakater/alerting-in-kubernetes-5df357c9b036?source=rss-7d9450e34ec2------2</link>
            <guid isPermaLink="false">https://medium.com/p/5df357c9b036</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[alerting]]></category>
            <dc:creator><![CDATA[Ahmad Iqbal Ali]]></dc:creator>
            <pubDate>Tue, 09 Apr 2019 13:17:51 GMT</pubDate>
            <atom:updated>2019-04-09T13:19:17.949Z</atom:updated>
            <content:encoded><![CDATA[<p>Alerting is one of the pillars of observability in DevOps, and closely related to monitoring and logging. Even though monitoring and logging provide a way to actively view and get insight on the state of the system, any person cannot remain glued to the screen to keep a look out for an error level log statement, low memory issue or other troublesome event. We use alerting to let us know of events of interest, i.e. events that indicate issues or potential issues, as and when they occur.</p><p>The concept of alerting is simple. The system is continuously monitored, and when a predefined threshold of some metric is crossed, an alert is sent out to pre-configured recipients. Thresholds can be set to not only indicate the occurrence of an issue with the system, but rather also proactively indicate that the system might be headed towards an issue. Alerting on such thresholds may give the stakeholders adequate indication and time to monitor the system more closely and even start some mitigation activities that will lower the probability of issue occurrence. The exact thresholds for alerting will no doubt have to be thought out, as having low thresholds that are reached frequently but have little impact, will lead to very noisy alerting; i.e. We will receive frequent alerts for events that do not require immediate attention. On the other hand if the thresholds are too high, we may not receive alerts until the issue has already occurred, or it is too late to mitigate in any way. The monitoring data will of course all be there in the required fine-grained detail to allow for effective root cause analysis, but the alerts should only be triggered within the sweet-spot of the threshold range that actually warrant attention from human stakeholders.</p><p>At Stakater we try to use open-source software throughout our deployment environment such as with Kubernetes because of the relevant benefits. This is also the case for our monitoring and alerting, for which we use a Prometheus based stack.</p><h3>Prometheus monitoring</h3><p>Let’s take a quick look at how we can use prometheus for monitoring on a Kubernetes cluster.</p><p>A convenient way of deploying Prometheus on Kubernetes is by using the <a href="https://github.com/coreos/prometheus-operator">Prometheus Operator</a>. With Prometheus there are a few components involved at the center of which we have the Prometheus Server. Prometheus server essentially scrapes metric data that other services expose. Each Kubernetes node exposes certain Services like Node Exporter &amp; Kubelet, which contain system level metrics. Node Exporter collects OS level metrics of a node through Docker host and Kubelet contains cadvisor which collects container metrics from the Docker Engine. For Kubernetes monitoring, Prometheus scrapes metrics from each Kubelet and Node Exporter from all nodes.</p><p>There may be some services such as ephemeral and batch jobs that Prometheus server cannot reliably scrape because of their ephemeral nature. For such a case we have the Prometheus Pushgateway which is able to have such jobs or services push their metrics to it, and in turn Pushgateway exposes these metrics to Prometheus for scraping.</p><p>For visualization we have Grafana which queries Prometheus and groups the results and displays it in Dashboards.</p><h3>Prometheus Alert Manager</h3><p>For alerting, Prometheus Server triggers alerts on Prometheus AlertManager based on the rules defined within AlertManager. The alert trigger sends notifications through a desired notification channel.</p><p>Prometheus Operator creates/configures/manages Prometheus atop Kubernetes and makes Kubernetes native Prometheus configuration in the form of Customer Resource Definitions (CRDs). Two of these CRDs are for PrometheusRule and AlertManager.</p><p>We can define multiple alerting rules using PrometheusRule which are actually Prometheus queries. When the rule query is satisfied, it fires the alert to AlertManager. Each rule can have labels. AlertManager has Routes, which can be defined using these labels and each Route can have multiple Receivers, and each receiver can send the notification to a specific app like Slack or email. We can also set a time period during which a rule is satisfied, for the alert to be triggered, e.g. we want the alert to trigger if Kubelet is down for 2 minutes.</p><p>Following is an example configuration which shows the configuration of 2 alerts based on the same metric, i.e. the number of responsive kubelets. The threshold for the percentage of kubelets however is different and are therefore labelled differently.</p><pre>- alert: K8SKubeletDown<br>  expr: count(up{job=&quot;kubelet&quot;} == 0) / count(up{job=&quot;kubelet&quot;}) &gt; 0.03<br>  for: 5m<br>  labels:<br>    severity: warning<br>    kind: infra<br>  annotations:<br>    description: Prometheus failed to scrape {{ $value }}% of kubelets.<br>    summary: Many Kubelets cannot be scraped<br>- alert: K8SKubeletDown<br>  expr: absent(up{job=&quot;kubelet&quot;} == 1) or count(up{job=&quot;kubelet&quot;} == 0) / count(up{job=&quot;kubelet&quot;}) &gt; 0.1<br>  for: 5m<br>  labels:<br>    severity: critical<br>    kind: infra<br>  annotations:<br>    description: Prometheus failed to scrape {{ $value }}% of kubelets, or all Kubelets have disappeared from service discovery.<br>    summary: Many Kubelets cannot be scraped</pre><p>In the first alert configured above, if the percentage of unresponsive kubelets stays above 3% for 5 minutes, an alert of severity Warning is triggered. In second alert, if the percentage of unresponsive kubelets stays above 10% for 5 minutes, an alert of severity Error is triggered.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/539/0*geVciNjn1E5vrtdT" /><figcaption>Example: Slack notification for pod restarting</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/517/0*X0Hfr0zk08jih58x" /><figcaption>Example: Slack notification for pod memory</figcaption></figure><h3>Uptime checker alerts</h3><p>Apart from Prometheus which provides us monitoring from within our kubernetes cluster, we also employ third party cloud services for uptime monitoring such as Uptime Robot. Such a service allows monitoring of application connection from multiple locations across the globe, and additionally alert in case the application is not reachable. Services like Uptime Robot also provide alerts using multiple notification channels such as slack, email, etc.</p><p>For the purpose of easy automation of configuring monitors and alerts with the Uptime checker, we have developed an open-source Kubernetes controller, <a href="https://github.com/stakater/IngressMonitorController">IngressMonitorController</a>. The IngressMonitorController automatically adds/removes monitors and alerts against ingresses in the uptime checker such as UptimeRobot, Pingdom, Statuscake, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/623/1*KQxL4qBcrtN7_9eejY6jOg.png" /><figcaption>Example: Slack notifications from UptimeRobot</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NesYWMOV9HFX1R-g" /><figcaption>Alerting components</figcaption></figure><figure><a href="http://stakater.com/?utm_medium=blog&amp;utm_source=alerting-in-kubernetes-5df357c9b036"><img alt="" src="https://cdn-images-1.medium.com/max/700/1*0FANFMBaw8Q8aLuPQJLOOA.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5df357c9b036" width="1" height="1" alt=""><hr><p><a href="https://medium.com/stakater/alerting-in-kubernetes-5df357c9b036">Alerting in Kubernetes</a> was originally published in <a href="https://medium.com/stakater">Stakater</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Simple Kubernetes alerting with Stakater Chowkidar]]></title>
            <link>https://medium.com/stakater/simple-kubernetes-alerting-with-stakater-chowkidar-dec95d8eb287?source=rss-7d9450e34ec2------2</link>
            <guid isPermaLink="false">https://medium.com/p/dec95d8eb287</guid>
            <category><![CDATA[slack]]></category>
            <category><![CDATA[alerting]]></category>
            <category><![CDATA[resources]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Ahmad Iqbal Ali]]></dc:creator>
            <pubDate>Sun, 31 Mar 2019 08:12:50 GMT</pubDate>
            <atom:updated>2019-05-08T10:58:15.351Z</atom:updated>
            <content:encoded><![CDATA[<h3>Custom Kubernetes Events alerting with Stakater Chowkidar</h3><p>Alerting is one of the components of DevOps that allows stakeholders to easily observe important system events. Alerting is indispensable for modern DevOps teams due to the needs of continuous deployment of multiple applications composed of microservices. Alerting helps with reliability by informing us about the system at important states. That way we have confidence by having insight into the exact state of the system, and knowing when there are any issues that might be occurring, or could be imminent. Alerting systems can be configured to notify at the occurrence of specific system events (e.g. service down), or when thresholds of certain metrics are reached (e.g. 80% memory consumed). And notifications of such events can be connected various channels such as email, Instant Message, or even Text message on phones.</p><p>In this blog we will be exploring Stakater Chowkidar, an open-source Kubernetes controller which can be used for some specific alerting needs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*1qO21qPaRR6sB4t3Tm6vqQ.png" /></figure><h3>Stakater Chowkidar</h3><p>Chowkidar is an Urdu word for watchman, and generically Stakater Chowkidar is implemented to have configuration for certain actions to be performed on certain events in the kubernetes cluster. One of the current event-actions that Chowkidar supports and that we are going to discuss today is the notification on slack at the creation of a Pod without a resource request and limit defined.</p><p>Let us first take a look at what resource limit and requests are for containers and pods.</p><h4>Scenario background</h4><p><strong>Container Resource limits</strong></p><p>By default a container has no resource limits and can use as much of a resource as the host allows. The container can however be configured to set a limit to how much memory or CPU a container can use.</p><p><strong>Memory</strong></p><p>It is good practice to keep a limit on how much host memory a running container can consume. On linux, when there is a shortage of memory to perform important system functions the kernel kills processes to free up memory. Any process can be killed for this purpose, including Docker. Docker does try to adjust the priority of the docker daemon with respect to the out of memory exception however it does not do the same for containers. An individual container may therefore likely be killed in such an event..</p><p>In docker we can force limits on the memory consumption of containers. A hard limit can be set to disallow the container to use no more than a given amount of memory. A soft limit allows the container to use as much memory as needed as long as the system does not have other memory consumption requirements. The following options can be used with the docker run command to enforce these memory limits:</p><p>--memory This is the maximum memory the container can use, i.e. the hard limit</p><p>--memory-reservation This is the memory limit imposed on the container when docker detects low memory on the host machine, i.e. the soft limit</p><p><strong>CPU</strong></p><p>A container by default has unlimited access to the host’s CPU. You can set various constraints to limit a given container’s access to the host machine’s CPU. Some of the docker run flags for configuring the amount of CPU resources the container has are as follows:</p><p>--cpus This is the limit of the number of CPUs the container is allowed to use, i.e. a hard limit</p><p>--cpu-shares This is the container’s weight for CPU access and is considered only when CPU cycles are constrained. So in a way this is a soft limit.</p><h3>Container resources in Kubernetes</h3><p>The docker configuration options that we reviewed above can be specified in Kubernetes on the containers in pods. Each Container of a Pod can specify one or more of the following configuration:</p><ul><li>spec.containers[].resources.limits.cpu</li><li>spec.containers[].resources.limits.memory</li><li>spec.containers[].resources.requests.cpu</li><li>spec.containers[].resources.requests.memory</li></ul><p>When the kubelet starts a Container of a Pod, it passes the CPU and memory limits to the container runtime. In the case of docker these configuration map directly or indirectly as follows:</p><ul><li>spec.containers[].resources.requests.cpu → --cpu-shares</li><li>spec.containers[].resources.limits.memory → --memory</li></ul><h3>Implementing Chowkidar</h3><p>Specifying requests and limits for memory/cpu in the container deployments is a good practice since otherwise there is a possibility of having a container eat up too many resources with not enough left for more important containers/processes, or in extreme cases getting important containers/processes killed off by the OS to relieve stress on the constrained resources. We would like to implement this best practice in all our deployments, and in case any container without this best practice is deployed, we would like to have this container be brought to our notice.</p><p>Chowkidar will watch all the Pods being deployed, and if there is any without a resource request/limit specified, we will configure to be notified via slack.</p><p>Chowkidar allows you to have multiple controllers that will continuously watch types in all the namespaces and automatically perform any actions given in the yaml file. With this, you can easily check for any criteria on your Pods/other types and take corresponding actions.</p><p>We will specify the following configuration on Chowkidar:</p><p>type: The type of Resource you want to monitor, i.e. Pods</p><p>watchcriterion: The criterion for which you want to take actions for the controller i.e. resourceExists. resourceExists will make sure Pods should have Resources and if not found, it will take action.</p><p>actions: The Array of actions that you want to take, i.e. slack. For Slack you need to provide token and Channel Name as Parameters in the yaml file</p><p>Following are the yaml files for deploying Chowkidar with our required configuration:</p><pre>---<br># Source: chowkidar/templates/deployment.yaml<br>apiVersion: extensions/v1beta1<br>kind: Deployment<br>metadata:<br>  annotations:<br>    configmap.reloader.stakater.com/reload: chowkidar<br>  labels:<br>    app: chowkidar<br>    group: com.stakater.platform<br>    provider: stakater<br>    version: v0.0.31<br>    chart: &quot;chowkidar-v0.0.31&quot;<br>    release: &quot;RELEASE-NAME&quot;<br>    heritage: &quot;Tiller&quot;<br>  name: chowkidar<br>spec:<br>  replicas: 1<br>  revisionHistoryLimit: 2<br>  selector:<br>    matchLabels:<br>      app: chowkidar<br>      group: com.stakater.platform<br>      provider: stakater<br>  template:<br>    metadata:<br>      labels:<br>        app: chowkidar<br>        group: com.stakater.platform<br>        provider: stakater<br>    spec:<br>      containers:<br>      - env:<br>        - name: KUBERNETES_NAMESPACE<br>          valueFrom:<br>            fieldRef:<br>              fieldPath: metadata.namespace<br>        - name: CONFIG_FILE_PATH<br>          value: /configs/config.yaml<br>        image: &quot;stakater/chowkidar:v0.0.31&quot;<br>        imagePullPolicy: IfNotPresent<br>        name: chowkidar<br>        volumeMounts:<br>        - mountPath: /configs<br>          name: config-volume<br>      serviceAccountName: chowkidar<br>      volumes:<br>      - configMap:<br>          name: chowkidar<br>        name: config-volume</pre><pre>---<br># Source: chowkidar/templates/configmap.yaml<br>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  labels:<br>    app: chowkidar<br>    group: com.stakater.platform<br>    provider: stakater<br>    version: v0.0.31<br>    chart: &quot;chowkidar-v0.0.31&quot;<br>    release: &quot;RELEASE-NAME&quot;<br>    heritage: &quot;Tiller&quot;<br>  name: chowkidar<br>data:<br>  config.yaml: |-<br>    controllers:<br>    - type: pods<br>      watchcriterion:<br>        operator: and<br>        identifiers:<br>        - resourceExists<br>      actions:<br>      - name: slack<br>        params:<br>          channel: channel-name<br>          token: your-token</pre><p>We will also need to provide Chowkidar with the required access control:</p><pre>---<br># Source: chowkidar/templates/rbac.yaml<br>apiVersion: v1<br>kind: ServiceAccount<br>metadata:<br>  labels:<br>    app: chowkidar<br>    group: com.stakater.platform<br>    provider: stakater<br>    version: v0.0.31<br>    chart: &quot;chowkidar-v0.0.31&quot;<br>    release: &quot;RELEASE-NAME&quot;<br>    heritage: &quot;Tiller&quot;<br>  name: chowkidar<br>---<br>apiVersion: rbac.authorization.k8s.io/v1beta1<br>kind: ClusterRole<br>metadata:<br>  labels: <br>    app: chowkidar<br>    group: com.stakater.platform<br>    provider: stakater<br>    version: v0.0.31<br>    chart: &quot;chowkidar-v0.0.31&quot;<br>    release: &quot;RELEASE-NAME&quot;<br>    heritage: &quot;Tiller&quot;<br>  name: chowkidar-role<br>rules:<br>  - apiGroups:<br>      - &quot;&quot;<br>    resources:      <br>      - pods<br>    verbs:<br>      - list<br>      - get<br>      - watch<br>---<br>apiVersion: rbac.authorization.k8s.io/v1beta1<br>kind: ClusterRoleBinding<br>metadata:<br>  labels: <br>    app: chowkidar<br>    group: com.stakater.platform<br>    provider: stakater<br>    version: v0.0.31<br>    chart: &quot;chowkidar-v0.0.31&quot;<br>    release: &quot;RELEASE-NAME&quot;<br>    heritage: &quot;Tiller&quot;<br>  name: chowkidar-role-binding<br>roleRef:<br>  apiGroup: rbac.authorization.k8s.io<br>  kind: ClusterRole<br>  name: chowkidar-role<br>subjects:<br>  - kind: ServiceAccount<br>    name: chowkidar<br>    namespace: default</pre><figure><a href="http://stakater.com/?utm_medium=blog&amp;utm_source=simple-kubernetes-alerting-with-stakater-chowkidar-dec95d8eb287"><img alt="" src="https://cdn-images-1.medium.com/max/700/1*0FANFMBaw8Q8aLuPQJLOOA.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dec95d8eb287" width="1" height="1" alt=""><hr><p><a href="https://medium.com/stakater/simple-kubernetes-alerting-with-stakater-chowkidar-dec95d8eb287">Simple Kubernetes alerting with Stakater Chowkidar</a> was originally published in <a href="https://medium.com/stakater">Stakater</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Security in Kubernetes and the cloud]]></title>
            <link>https://medium.com/stakater/security-in-kubernetes-and-the-cloud-81e94c6eed5a?source=rss-7d9450e34ec2------2</link>
            <guid isPermaLink="false">https://medium.com/p/81e94c6eed5a</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Ahmad Iqbal Ali]]></dc:creator>
            <pubDate>Sun, 24 Mar 2019 21:54:53 GMT</pubDate>
            <atom:updated>2019-04-01T05:16:51.273Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article we will be discussing some of the security related practices we can follow across the cloud infrastructure or on kubernetes cluster, that can help us create a secure environment for our production systems.</p><h3>Micro-segmentation</h3><p>Micro-segmentation enables consolidation of workloads with differing security needs. An example could be running workloads of a production environment with a test environment, or within an application environment, running public and private components.</p><p>This is implemented by enforcing security policies around each individual workload in the environment. By placing security controls next to the workloads themselves, security policies become asset-specific and as a result, workloads at different security levels can now share common infrastructure, enabling much greater consolidation and agility.</p><p>Micro-segmentation is an emerging security best practice that offers a number of advantages over more established approaches. The added granularity that micro-segmentation offers is essential at a time when many organizations are adopting cloud services and new deployment options like containers that make traditional perimeter security less relevant.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/731/0*xHpDzyUQiff6jIkI" /></figure><h3>Single sign-on (SSO)</h3><p>Single Sign-On (SSO) allows users to log in using a single set of credentials, e.g. username and password, so they can easily access a set of applications. SSO. SSO saves time and energy for users because they do not have to repeatedly log into multiple applications. This provides a smooth user experience, and makes it less likely to have access problems because of lost or forgotten credentials, locked out accounts, etc. Instead of having individual authentication on various tools, a more effective strategy is to use single sign-on for all tools, i.e. a centralized authentication mechanism that can allow or reject access to a set of tools based on a single set of credentials per user. Additionally some tools may not have authentication built into them at all, and may be reliant on an external authentication server in any case. An external authentication server with single sign-on capability can therefore prove to be the way to go in such a situation.</p><p>You can read a discussion on this <a href="https://medium.com/stakater/proxy-injector-enabling-sso-with-keycloak-on-kubernetes-a1012c3d9f8d">here</a>.</p><h3>IP whitelisting</h3><p>IP whitelisting allows you to create lists of trusted IP addresses or IP ranges from which your users can access your domains. On AWS, this can be implemented with security groups which act as virtual firewalls for your instance to control inbound and outbound traffic. Security groups act at the instance level, not the subnet level. Therefore, each instance in a subnet in your VPC could be assigned to a different set of security groups. For each security group, you add <em>rules</em> that control the inbound traffic to instances, and a separate set of rules that control the outbound traffic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/964/0*o3EmIqTKH9uc6hhP" /></figure><h3>Secrets Management in Kubernetes</h3><p>Sensitive information such as a database password or an API key should not be stored in clear text. And these should not be written into the container image. The 12-factor app philosophy says that configuration including secrets should come from the environment. Kubernetes provides the resource type Secrets for this. Kubernetes Secrets provide a mechanism to use such information in a safe and reliable way</p><ul><li>Secrets are namespaced objects, i.e. they exist in the context of a namespace</li><li>You can access them via a volume or an environment variable from a container running in a pod</li><li>The API server stores secrets as plaintext in etcd</li></ul><p>Applying the Principle of Least Privilege, We want to ensure that containerized code can read only the secrets that it needs. And also a good idea is to have a different set of secrets for different environments (like production, development, and testing). The development and test credentials can be shared with a wider set of team members without necessarily giving them full access to the production credentials.</p><p>There are three ways to get secrets (or any other kind of information) into a container so that they are accessible to the application:</p><ul><li>Building them into the image itself</li><li>Passing them into the container as environment variables</li><li>Mounting a volume into a container so that code can read the information out of a file on that volume</li></ul><p>Kubernetes secrets support the last two of these approaches, although the third option of mounting a volume is generally considered the safest.</p><p>Following GitOps principles, we would like to maintain our configuration and secrets in version control with respect to the relevant environment. However committing secrets into version control in plane text is not a good idea. But we still want them to be versioned and maintained with the environment configuration, and be accessible to multiple developers/ops or target systems.</p><p>For this we can commit encrypted secrets into our version control, and decrypt them when we need to use. This becomes easy when using Helm, since Helm has a plugin for this, Helm secrets.</p><p>Helm secrets:</p><ul><li>allows encryption of values, so that they can be checked into version control</li><li>Track which secrets were changed at what time in the version control system</li><li>Values files are encrypted with a secret key.</li><li>The plugin supports the yaml structure encryption per value, i.e. it only encrypts the values</li></ul><p>The following diagram illustrates this flow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aS_MZn4AElalvosm" /></figure><h3>Role Based Access Control</h3><p>Role-based access control (RBAC) is a method of regulating access to computer or network resources based on the roles of individual users. All resources are modeled API objects in Kubernetes, belonging to API Groups. These resources allow operations such as Create, Read, Update, and Delete (CRUD). RBAC is writing rules to allow or deny operations on resources by users, roles or groups.</p><p>Rules are operations which can act upon an API group.</p><p>Roles are a group of rules which affect, or scope, a single namespace</p><ul><li>ClusterRoles have a scope of the entire cluster.</li></ul><p>Each operation can act upon one of three subjects</p><ul><li>User Accounts</li><li>Service Accounts</li><li>Groups</li></ul><p>Here is a summary of the RBAC process:</p><ul><li>Determine or create namespace</li><li>Create certificate credentials for user</li><li>Set the credentials for the user to the namespace using a context</li><li>Create a role for the expected task set</li><li>Bind the user to the role</li><li>Verify the user has limited access.</li></ul><figure><a href="http://stakater.com/?utm_medium=blog&amp;utm_source=security-in-kubernetes-and-the-cloud-81e94c6eed5a"><img alt="" src="https://cdn-images-1.medium.com/max/700/1*0FANFMBaw8Q8aLuPQJLOOA.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=81e94c6eed5a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/stakater/security-in-kubernetes-and-the-cloud-81e94c6eed5a">Security in Kubernetes and the cloud</a> was originally published in <a href="https://medium.com/stakater">Stakater</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Proxy Injector: Enabling SSO with Keycloak on Kubernetes]]></title>
            <link>https://medium.com/stakater/proxy-injector-enabling-sso-with-keycloak-on-kubernetes-a1012c3d9f8d?source=rss-7d9450e34ec2------2</link>
            <guid isPermaLink="false">https://medium.com/p/a1012c3d9f8d</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[keycloak]]></category>
            <category><![CDATA[single-sign-on]]></category>
            <dc:creator><![CDATA[Ahmad Iqbal Ali]]></dc:creator>
            <pubDate>Tue, 12 Mar 2019 20:13:56 GMT</pubDate>
            <atom:updated>2019-03-14T07:24:33.434Z</atom:updated>
            <content:encoded><![CDATA[<p>A DevOps team may be accessing multiple applications and tools in a single product environment in support of their DevOps processes such as CI/CD server, Centralized log, Kubernetes dashboard, Monitoring software, Artifact repositories, Admin tools, etc. All of these tools will require authentication mechanisms for security purposes, and for a user to maintain and remember their authentication credentials on so many softwares can quickly become cumbersome. And in the event of lost credentials, it can be a tedious process for both user and admins to recover the required credentials.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GomBNngXp9eN9bPT" /></figure><p>Instead of having individual authentication on various tools, a more effective strategy is to use single sign-on for all tools, i.e. a centralized authentication mechanism that can allow or reject access to a set of tools based on a single set of credentials per user. Additionally some tools may not have authentication built into them at all, and may be reliant on an external authentication server in any case. An external authentication server with single sign-on capability can therefore prove to be the way to go in such a situation.</p><h3>Single Sign-On</h3><p>Single Sign-On (SSO) allows users to log in using a single set of credentials, e.g. username and password, so they can easily access a set of applications. SSO. SSO saves time and energy for users because they do not have to repeatedly log into multiple applications. This provides a smooth user experience, and makes it less likely to have access problems because of lost or forgotten credentials, locked out accounts, etc.</p><h3>Keycloak</h3><p>Keycloak is an Open Source Identity and Access Management solution. It provides an easy way to add authentication including Single Sign-on to applications and services with minimum effort. Keycloak handles persistence and user authentication all out of the box.</p><p>Instead of having to login to individual applications, users authenticate with Keycloak rather than individual applications. This means that the individual applications don’t have to implement their own login forms, authentication, and storage of users and sessions. Once logged-in to Keycloak, users don’t have to login again to access a different application. And similarly once logged-out from Keycloak, users don’t have to log out of individual applications. Enabling login with social networks is also easy. The configuration for these can be added via Keycloak’s admin console. No code or changes are required to individual applications.</p><p>Keycloak supports OpenID Connect and SAML protocols. OpenID Connect (OIDC) is an extension of the OAuth 2 authentication protocol. While OAuth 2.0 is a framework for building authorization protocols, and OIDC is the full implementation of a authentication and authorization protocol. SAML 2.0 is similar to OIDC but a lot older and consequently more mature. It has its roots in SOAP and works by exchanging XML documents between the authentication server and the application, so it tends to be a bit more verbose than OIDC. In most cases OIDC is recommended by Keycloak.</p><h3>OpenID Connect</h3><p>OIDC uses the Json Web Token (JWT) set of standards. There are really two types of use cases when using OIDC. Our relevant use case is where the application asks the Keycloak server to authenticate a user for them. After a successful login, the application will receive an identity token and an access token. The identity token contains information about the user such as username, email, and other profile information. The access token is digitally signed by the realm and contains access information, e.g. user roles, that the application can use to determine what resources the user is allowed to access on the application.</p><h3>Keycloak setup</h3><p>A <a href="https://github.com/helm/charts/tree/master/stable">stable chart</a> for keycloak is available in the official <a href="https://kubernetes-charts.storage.googleapis.com/">public helm chart repository</a>. The chart deploys a StatefulSet for Keycloak and also includes a deployment of PostgreSQL which it uses as the database for Keycloak. The chart can also be configured to use an external instance of PostgreSQL instead of the bundled deployment, such as AWS RDS.</p><p>Going by the philosophy of GitOps and maintaining all our configuration in code, we use a json file to configure our keycloak realm and import it into keycloak at deployment time. Following is an example configuration of the Keycloak chart. While the standalone operating mode is not recommended for a production setup as it becomes a single point of failure for applications, the example is show for a standalone operating mode to keep things simple while introducing the rest of the required configuration. The realm json is loaded into a configmap and then referenced in the Keycloak chart through the configmap.</p><pre>replicas: 1</pre><pre>image:<br>  repository: jboss/keycloak<br>  tag: 4.8.3.Final<br>  pullPolicy: IfNotPresent</pre><pre>persistence:<br>  # If true, the Postgres chart is deployed<br>  deployPostgres: false</pre><pre># The database vendor. Can be either &quot;postgres&quot;, &quot;mysql&quot;, &quot;mariadb&quot;, or &quot;h2&quot;<br>  dbVendor: postgres</pre><pre>## The following values only apply if &quot;deployPostgres&quot; is set to &quot;false&quot;</pre><pre># Specifies an existing secret to be used for the database password<br>  existingSecret: &quot;keycloak-secrets&quot;<br>  # The key in the existing secret that stores the password<br>  existingSecretKey: database.password</pre><pre>dbName: kc-database<br>  dbHost: keycloak-postgresql.rds<br>  dbPort: 5432<br>  dbUser: keycloak</pre><pre>extraArgs: &gt;<br>  -Djgroups.bind_addr=global<br>  -Dkeycloak.migration.action=import<br>  -Dkeycloak.migration.provider=singleFile<br>  -Dkeycloak.migration.file=/opt/jboss/keycloak/standalone/configuration/import/realm.json<br>  -Dkeycloak.migration.strategy=IGNORE_EXISTING</pre><pre>## Allows the specification of additional environment variables for Keycloak<br> extraEnv: |<br>  - name: OPERATING_MODE<br>  value: standalone</pre><pre>## Add additional volumes and mounts, e. g. for custom themes<br> extraVolumes: |<br>  - name: keycloak-config<br>  configMap:<br>   name: keycloak-configmap<br>   items:<br>   - key: realm.json<br>    path: realm.json<br> extraVolumeMounts: |<br>  - name: keycloak-config<br>  mountPath: /opt/jboss/keycloak/standalone/configuration/import/realm.json<br>  subPath: realm.json</pre><pre>podAnnotations:<br>  fluentdConfiguration: &gt;<br>  [<br>   {<br>    &quot;containers&quot;:<br>    [<br>     {<br>      &quot;expression&quot;: &quot;/^\\S*\\s-\\s-\\s\\[(?&lt;time&gt;\\S*)[\\S\\s]*\\]\\s(?&lt;message&gt;[\\S\\s]*)/&quot;,<br>      &quot;expressionFirstLine&quot;: &quot;/^\\:\\:f{4}:[0-9]+.[0-9]+\\.[0-9]+\\.[0-9]+/&quot;,<br>      &quot;timeFormat&quot;: &quot;%d/%b/%Y:%H:%M:%S&quot;,<br>      &quot;containerName&quot;: &quot;keycloak&quot;<br>     }<br>    ]<br>   }<br>  ]</pre><pre>service:<br>  labels:<br>  expose: &quot;true&quot;<br>  port: 80<br>  annotations:<br>  xposer.stakater.com/annotations: |-<br>   kubernetes.io/ingress.class: external-ingress<br>   ingress.kubernetes.io/force-ssl-redirect: true<br>   forecastle.stakater.com/expose: true<br>   forecastle.stakater.com/icon: <a href="https://www.keycloak.org/resources/images/keycloak_logo_480x108.png">https://www.keycloak.org/resources/images/keycloak_logo_480x108.png</a><br>   forecastle.stakater.com/appName: keycloak<br>   exposeIngressUrl: globally<br>  config.xposer.stakater.com/Domain: company.com<br>  config.xposer.stakater.com/IngressNameTemplate: &#39;{{.Service}}&#39;<br>  config.xposer.stakater.com/IngressURLTemplate: &#39;keycloak.{{.Namespace}}.{{.Domain}}&#39;</pre><p>The postgresql details are provided in the persistence section including the server details, database username and password (referenced from a kubernetes secret).</p><p>The realm json is loaded from the configmap into a volume, under extraVolumeMounts, which is mounted in Keycloak. In the extraArgs of the Keycloak container we specify the path of the mounted realm json which we want to import.</p><pre>keycloak-configmap:<br>  ConfigMapName: &quot;keycloak-configmap&quot;<br>  Data:<br>    realm.json: |-<br>      {<br>        &quot;realm&quot;: &quot;stakater&quot;,<br>        &quot;enabled&quot;: true,<br>        &quot;loginTheme&quot;: &quot;keycloak&quot;,<br>        &quot;sslRequired&quot;: &quot;external&quot;,<br>         ...<br>      }</pre><p>The operating mode is specified with the environment variables under extraEnv of the container.</p><p>It is necessary to create or obtain a client configuration for any application to be able to use Keycloak. You usually configure a new client for each new application hosted on a unique host name. When an application interacts with Keycloak, the application identifies itself with a client ID so Keycloak can provide a login page, single sign-on (SSO) session management, and other services.</p><p>The traditional method in the OAuth2 specification for authentication is to use a client id and secret. The client has a secret, which needs to be known to both the application and the Keycloak server. You can generate the secret for a particular client in the Keycloak administration console or then specify the secret in the keycloak realm json.</p><h3>Keycloak Gatekeeper</h3><p>Keycloak Gatekeeper is an adapter which integrates with the Keycloak authentication service. We deploy it on a per-application instance basis. So usually this will be a sidecar container deployed with the application container on the kubernetes pod. We configure the kubernetes service of the application so that it points to the gatekeeper rather than the application itself, so that the gatekeeper can act as a proxy for incoming requests. The gatekeeper then verifies from the Keycloak server if an active authenticated session exists or not. If not, it redirects the client to the Keycloak login page. If the session exists, it allows the incoming request to pass through to the application container.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*L6GAQ_uhKQGRVzcH" /></figure><p>Using the Keycloak Gatekeeper allows us to have zero authentication configuration within the application itself. The session verification, redirection to Keycloak in case of an invalid session, and pass through to the application in case of a valid session, are all handled by the gatekeeper.</p><h3>Stakater Proxy Injector</h3><p>Deploying a sidecar container for Keycloak Gatekeeper with all our applications can be a hassle. So we want to automatically inject a keycloak gatekeeper container in a pod, for any deployment that requires to connect to keycloak, instead of manually adding a sidecar container with each deployment. This Proxy Injector controller will continuously watch deployments in specific or all namespaces, and automatically add a sidecar container for keycloak gatekeeper. Configuration for the keycloak gatekeeper is done through annotations of the respective deployment or with ConfigMap of the ProxyInjector.</p><p>Some common configuration can either be added to the proxy injector config.yaml in the ConfigMap, or can be overridden at the application deployment level using annotations on the target deployments. Following are the core configuration properties</p><pre>listen: this is the interface the gatekeeper should be listening on e.g. &quot;0.0.0.0:80&quot; to listen on the localhost pod.</pre><pre>upstream-url: url for the upstream endpoint you wish to proxy</pre><pre>resources: list of resources to proxy uri, methods, roles</pre><pre>client-id: client id used to authenticate to the oauth service</pre><pre>client-secret: client secret used to authenticate to the oauth service</pre><p>All of the <a href="https://www.keycloak.org/docs/latest/securing_apps/index.html#configuration-options">available Gatekeeper configuration</a> can also be specified on ProxyInjector annotations.</p><figure><a href="http://stakater.com/?utm_medium=blog&amp;utm_source=proxy-injector-enabling-sso-with-keycloak-on-kubernetes-a1012c3d9f8d"><img alt="" src="https://cdn-images-1.medium.com/max/700/1*0FANFMBaw8Q8aLuPQJLOOA.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a1012c3d9f8d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/stakater/proxy-injector-enabling-sso-with-keycloak-on-kubernetes-a1012c3d9f8d">Proxy Injector: Enabling SSO with Keycloak on Kubernetes</a> was originally published in <a href="https://medium.com/stakater">Stakater</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Implementing Compliance as Code for Github]]></title>
            <link>https://medium.com/stakater/implementing-compliance-as-code-for-github-39c250a272a4?source=rss-7d9450e34ec2------2</link>
            <guid isPermaLink="false">https://medium.com/p/39c250a272a4</guid>
            <category><![CDATA[git]]></category>
            <category><![CDATA[compliance]]></category>
            <category><![CDATA[jenkins]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Ahmad Iqbal Ali]]></dc:creator>
            <pubDate>Wed, 06 Mar 2019 13:25:18 GMT</pubDate>
            <atom:updated>2019-05-09T00:02:57.823Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/564/1*Dz0va1NLtfmma2oe4bCO_w.png" /></figure><p>Compliance is also an important aspect of DevOps. The organization may have various policies that need to be adhered to. This is one of the gaps that has historically put Dev and Ops organizations at odds with each other. Dev teams need to publish their features as quickly as possible, while Ops teams need to verify that all changes being made whether software or hardware, are within the company’s compliance parameters.</p><p>We discussed in an earlier article the concept of GitOps, i.e. using Git for managing operations and describing the source of truth of a system. Using the concept of Deployment Configuration as Code, we are able to describe the deployment usually within a yaml text file, and manage it in git.</p><p>The relevant progression for compliance therefore is Compliance as Code. In this blog we will discuss how we use Compliance as code for our Git repositories. The examples will mainly be using Github, but of course can be extended to any Git management service.</p><h3>Github as Code</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*ZSdkGAMhD-KhgoJLqD-ooA.jpeg" /></figure><p>We use Terraform for implementing Github as code. Terraform comes with the Github provider which is used to interact with GitHub resources. The provider allows us to manage resources such as Github organizations, its teams and members, repositories, etc. The provider needs to be configured with the authentication information, i.e. an access token, and the Github organization to use as the context. The account corresponding to the token will not only need to be a valid Github token but will also need owner access for this organization.</p><p>We perform a few github operations related to these resources as per our requirements, in our DevOps processes.</p><h4>Organization Membership</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*CRTw9Aj8N4MNHCk5P485Dg.jpeg" /></figure><p>We can add collaborators to an organization so that they have access to all the repositories within that organization. Users can either have individual access, or be part of a team which is assigned access on the organization. For creating teams, we use the github_team resource of the github provider. In it we can specify the parameters for the team such as the name, description and privacy level.</p><p>We can see examples of user definitions below including individual members, teams and lastly the association of a member to a team.</p><pre>module “user_username” {<br> source = “github.com/stakater/terraform-module-github.git//modules/user”<br> username = “individual_user”<br>}</pre><pre>resource “github_team” “team_name” {<br> name = “team_name”<br> description = “A cool team of developers”<br> privacy = “closed”<br>}</pre><pre>module “user_username” {<br> source = “github.com/stakater/terraform-module-github.git//modules/user”<br> username = “team_user”<br> team_id = “${github_team.developers.id}”<br>}</pre><h4>Create Repository</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/453/1*gm_-CuR8tDoGbhgKn8vDqQ.jpeg" /></figure><p>Firstly we use terraform to automate creation of a repository. So any time we want to create a new repository whether it is for a new kubernetes controller, a docker image, or other application, we do so using our automated process. This of course enables us to encode some base configuration if required, and present custom configuration declaratively within a terraform module. Such configuration becomes easy to read, and centralized in usually a single terraform module, rather than across multiple web UI screens. And most importantly, they get managed using Git operations. Any time we want to create a new repository, we add a new terraform module, to make any updates we simply make changes and merge, and if needed we can always undo changes using a git revert. The following example shows configuration for the Stakater ProxyInjector git repository.</p><pre>module &quot;stakater_proxyinjector&quot; {<br>  source         = &quot;github.com/stakater/terraform-module-github.git//modules/repository?ref=1.0.10&quot;<br>  name           = &quot;ProxyInjector&quot;<br>  require_status_checks = true<br>  enable_branch_protection = true<br>  enforce_admins = false<br>  private = false<br>  description = &quot;A Kubernetes controller to inject an authentication proxy container to relevant pods - [✩Star] if you&#39;re using it!&quot;<br>  topics = [&quot;stakater&quot;, &quot;kubernetes&quot;, &quot;openshift&quot;, &quot;k8s&quot;, &quot;proxy&quot;, &quot;authentication&quot;, &quot;keycloak&quot;]<br>  homepage_url = &quot;https://www.stakater.com/projects-overview.html&quot;<br>  license_template = &quot;apache-2.0&quot;<br>  webhooks = [<br>    {<br>      url = &quot;https://gitwebhookproxy.company.com/github-webhook/&quot;,<br>      events = &quot;push,pull_request&quot;<br>      secret = &quot;dummysecret&quot;<br>    }<br>  ]<br>}</pre><p>Let’s review some of the configuration parameters.</p><p><strong>Private repository</strong></p><p>We have parameterized whether the repository should be private or public. The private variable can be set accordingly to true or false.</p><p><strong>Webhooks</strong></p><p>Webhooks allow integration for subscribing to events on GitHub. We use Github webhooks to integrate with our Jenkins CI server to trigger pipelines on Github events such as pull request creation and merges into master branch. Since the Jenkins is behind a firewall for security reasons, the only way to reach it is via the Git Webhook Proxy (as shown in the example).</p><p><strong>Branch Protection</strong></p><p>The branch protection feature of Github is very useful. It allows to specify our process to ensure only code changes that fulfill specific criteria can be merged into the trunk, i.e. master branch. This ensures low quality code does not get shipped to production. Using this feature, Github will disallow any direct commits to the master branch by default, and only allow code changes to be merged via pull requests. We can also specify other status checks that we would like to be satisfied before a pull request can be merged into master. We usually utilize two checks at Stakater; 1. The pull request should be reviewed and approved by a peer, and 2. The CI pipeline for the pull request should have run and passed successfully. We can also specify if we would like to enforce or ignore the status checks on team members with Admin access on the repository. The require_status_checks, enable_branch_protection and enforce_admins variables are used for these configurations.</p><p><strong>Miscellaneous</strong></p><p>We can also specify some repository metadata such as description, topics, project URL, and license type. While description provides quick detail about what the repository is for, topics can be used for search on Github.</p><h3>CD with Jenkins</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/150/1*zydp9-XZf2omda7JqOjgxw.png" /></figure><p>We have a Jenkins pipeline variable, <a href="https://github.com/stakater/stakater-pipeline-library/blob/master/vars/terraformValidateAndApply.groovy">terraformValidateAndApply</a>, defined in the Stakater pipeline library to help apply the terraform script. The variable has 3 stages:</p><ol><li>Validate: which essentially runs terraform init and terraform validate</li><li>Plan and apply: which runs terraform plan and terraform apply</li><li>Notify: which adds a comment on the git pull request whether the terraform validate was successful or not.</li></ol><p>The variable of course requires a terraform installation to run, and therefore will need to run on a Jenkins slave with a relevant docker image running. You can take a look at the tools image we have developed and use for our terraform pipelines here: <a href="https://hub.docker.com/r/stakater/pipeline-tools">https://hub.docker.com/r/stakater/pipeline-tools</a></p><p>A Jenkinsfile for the terraform configuration repository will be simple enough in case the terraformValidateAndApply is run without any custom configuration. It may look as follows. By default the stakater/pipeline-tools image is used, but a different image can also be used by assigning the relevant value to the toolsImage parameter.</p><pre>#!/usr/bin/groovy<br>@Library(‘github.com/stakater/stakater-pipeline-library@v2.13.0’) _</pre><pre>terraformValidateAndApply {<br> <br>}</pre><figure><a href="http://stakater.com/?utm_medium=blog&amp;utm_source=implementing-compliance-as-code-for-github-39c250a272a4"><img alt="" src="https://cdn-images-1.medium.com/max/700/1*0FANFMBaw8Q8aLuPQJLOOA.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=39c250a272a4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/stakater/implementing-compliance-as-code-for-github-39c250a272a4">Implementing Compliance as Code for Github</a> was originally published in <a href="https://medium.com/stakater">Stakater</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GitOps for Kubernetes with Jenkins]]></title>
            <link>https://medium.com/stakater/gitops-for-kubernetes-with-jenkins-7db6304216e0?source=rss-7d9450e34ec2------2</link>
            <guid isPermaLink="false">https://medium.com/p/7db6304216e0</guid>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[jenkins]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[gitops]]></category>
            <category><![CDATA[git]]></category>
            <dc:creator><![CDATA[Ahmad Iqbal Ali]]></dc:creator>
            <pubDate>Mon, 25 Feb 2019 06:52:27 GMT</pubDate>
            <atom:updated>2019-02-26T06:03:12.443Z</atom:updated>
            <content:encoded><![CDATA[<p>While GitOps outlines several principles, it is possible due to certain constraints of time or resource you may not be able to follow all of them. As with everything, you can choose to prioritize to fulfill the essence, or core principles first and then later down the road complete a full implementation. In this blog we’ll first take a quick look at the what and why of GitOps, how to implement it into your DevOps processes, and finally an approach to implement GitOps in a light manner. There are tools such as ArgoCD and Weave Flux, that are purpose built for GitOps. We also have Jenkins-x, while technically purpose built for CI/CD on Kubernetes, but also has the ability for implementation of GitOps processes. In today’s blog however we are looking at a GitOps implementation using good old Jenkins. Yes that may see a few steps backwards especially after making mention of the other tools I just did. However, we understand sometimes it is difficult to change your CI/CD tool, and Jenkins being one of the most popular ones, it’s a good idea to take a look at how we can implement GitOps using Jenkins or other comparable CI/CD tool. We will mainly look at a particular set of tools that we use at Stakater, however the concepts should translate easily to other tools of the same nature.</p><h3>GitOps</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/234/1*o-S7NIeNhcA-5cm3GRcpvQ.png" /></figure><p>GitOps is a methodology of Continuous Delivery, at the heart of which is the principle that Git be used as the source of truth for declarative infrastructure and applications.</p><h4>Declarative description of system</h4><p>Any and all changes made to the application, environment, deployment and infrastructure should all be handled via git. This therefore means that we should declare all these as code and maintain in a git repository. Code for the whole system will include the following:</p><p><strong>Infrastructure </strong>can be declared in the form of Terraform modules, Cloudformation script, or other. At Stakater, we also use Kops for AWS, which in addition to setting up the Kubernetes cluster, also creates the underlying AWS infrastructure prior. This is declared using a yaml file.</p><p><strong>Kubernetes configuration</strong> includes details of deployments, replicas, services and container images to deploy etc. It can be declared at the least via vanilla yaml manifests. A better approach that we use at Stakater is to use the Helm package manager for Kubernetes. Helm encapsulates the configuration in the form of reusable Charts, which are groups of yaml and value files.</p><p><strong>Environment configuration</strong> is also handled within the Kubernetes configuration, and handled via the Kubernetes API object, ConfigMap. The ConfigMap is also declared in yaml, and can be expressed inside a Helm chart as such for a particular deployment, or in an independant chart for itself as well.</p><p><strong>Application code</strong> is of course traditionally maintained in git already, and needs to be packaged within docker image, which is declaratively expressed in the form of a Dockerfile. The Dockerfile should be included within the Application code repository.</p><h4>Changes through Pull Request</h4><p>Since our system is declared as code and maintained in Git, the consequent principle to follow is that all changes that need to be made, must be done so through a pull request. Using trunk based development, a <em>master</em> git branch is used for reflecting the current state of the system. Any updates that may be needed can simply be done by opening a new pull request (PR) on the master branch. Once the PR is merged, the CD/GitOps pipeline is triggered, and deploys the required changes to the kubernetes cluster. In case any change needs to be rolled back, to a recent or even older state, that will also be performed by a pull request. We will explore this in more detail in a subsequent section.</p><h4>Self-healing</h4><p>Since the git repository is the source of truth for the system, at any time we can see the intended state of the system by scanning the git repository. In case any application pod crashes, or the system is unintentionally modified manually, the GitOps pipeline will rectify to keep the system state in sync with the declaration in the repository. This principle is built into one of the purpose-built GitOps solutions, however in our case we will see this will not be implemented with Jenkins due to its complexity.</p><h3>GitOps with Jenkins</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*gXC3h7TaOhU8UBpUnnByTQ.jpeg" /></figure><h4>Anti-pattern?</h4><p>You may see discussions that implementing deployment with a CI server is an anti-pattern, however, with some tweaks we will be able to workaround some of the reasons for that argument, which isn’t to say that it will be enough for a proper GitOps implementation.</p><p>Firstly, it is argued that the CI pipeline should not directly deploy. Indeed we will see that we do not take this route. We essentially maintain separate repositories for the application code, and the cluster configuration. And in doing so we also have separate pipelines for each, the latter being responsible for the deployment.</p><p>And secondly, keeping a clear security boundary between the code build pipeline, and config deploy pipeline can be achieved by using two instances of jenkins.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*DtUgxszcMPLHYlZTD5o68g.jpeg" /></figure><h4>Implementation detail</h4><p>The first thing to do is to have your code and configuration repositories separate. We will take a look at one of Stakater’s opensource projects, <a href="https://github.com/stakater/Xposer/">Xposer</a> as an example, to illustrate the points that are being discussed. You can follow the <a href="https://github.com/stakater/Xposer/">github link</a> to get a better view of the individual files.</p><p><strong>Code repository</strong></p><p>The code repo will have the application code including a Dockerfile and helm chart. The reason for this is that the Dockerfile and Helm Chart at this point are serving only as a packaging tool. The responsibility and knowledge of how the application code should be packaged should be included within the Application code repository. In the following screenshot we can see the code repository for Xposer. As visible, apart from the application code, we also have the Dockerfile, Kubernetes manifests (for completeness sake), Helm Charts, and Jenkinsfile.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/0*dHdd-SEeQkx4Cqmz" /></figure><p><strong>Configuration Repository</strong></p><p>Next we need a configuration repository. This in fact can in most cases be used for a group of deployments that are related either by namespace, or other dependencies. For this we use the umbrella chart pattern of Helm. The repository will have a list of required charts in yaml, and for each chart the custom values that will be used for the deployment. The required charts are listed in a yaml file as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/0*zdI7ITqfdges_Vtd" /></figure><p>The customer values of the charts are defined in individual value yaml files as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/0*kDqm8uhnIWXnr8PQ" /></figure><p>The following example is from a different project, and shows how other deployment configuration where parameterized, can be customized in an umbrella chart value file. This way we customize the number of replicas, the health checks, the tolerations, etc. for a particular deployment in our configuration repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/344/0*G3NIiZze7V8lw3FG" /></figure><p><strong>Updates</strong></p><p>When a change is merged into the Code repository, The code pipeline runs in Jenkins, and upon success it pushes the packaged docker image and helm chart to the artifact repositories, Nexus and ChartMuseum respectively in our case. This pipeline will us a pre-baked builder docker image on the Jenkins slave to build the artifacts. This builder image may include tools such as the JDK, npm, go releaser, etc. depending on the project. At this point while our new project version is packaged into our artifact repository, it has in no way been indicated that it should also be deployed. That part will be handled by the Configuration repo pipeline.</p><p>As discussed, the configuration repository has a separate pipeline, probably that will be run in a separate Jenkins instance. To deploy our new project version, we will need to create a pull request and merge changes into our configuration repository. In the case of our Xposer project, we can create a pull request to update the chart version in the requirements.yaml. If any configuration values also need to be updated, for the new deployment, we can do that in the values file, xposer.yaml, and create a single pull request for these changes. Once the pull request is merged, the Jenkins pipeline runs and on success, it updates the deployment on the cluster. This pipeline will use a pre-baked deployer docker image on the Jenkins slave to deploy the charts. This builder image may include tools such as the kubectl client or helm client.</p><p>The image below illustrates the flow discussed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rKfptZqzblkNuQvl" /><figcaption>Bird’s eye view: GitOps with Jenkins</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*N7Jtd1lYLKagAYN39uXd1w.jpeg" /></figure><h3>Recap</h3><p>Looking at the diagram we can get a bird’s eye view of the implementation. As discussed we want to maintain everything in git as per GItOps principles, and all updates will be done with git operations, such as pull requests. Our code and configuration are defined in separate repositories since the scope of each is different. The code repository merely defines the software artifact. It is the configuration repository declares where and how the deployment of this artifact should take place.</p><p>Additionally, the two repository CI and CD pipelines are ideally handled by independent Jenkins servers. This allows each server to receive only the required security level. The Jenkins instance for the code repository will have access to the code repository obviously, and write access to the artifact repositories, and it will not need any access on the configuration repository or deployment related access level on the cluster. The Jenkins instance for the configuration repository will not need access to the code repository, and will have access to the configuration repository, read-only access to the artifact repositories and deployment access on the cluster.</p><p>This way we do satisfy the core principles of GitOps. One thing that remains missing is for the deployment configuration to be self-healing. In case someone manually updates the deployment on the cluster, using kubectl client for instance, our Jenkins instance will not be aware of this, as it is not watching the cluster status. The deployment will only be fixed in such a case when the Jenkins pipeline for the configuration repo is rerun manually, or runs automatically triggered by a new updated merged to the configuration repository.</p><figure><a href="http://stakater.com/?utm_medium=blog&amp;utm_source=gitops-for-kubernetes-with-jenkins-7db6304216e0"><img alt="" src="https://cdn-images-1.medium.com/max/700/1*0FANFMBaw8Q8aLuPQJLOOA.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7db6304216e0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/stakater/gitops-for-kubernetes-with-jenkins-7db6304216e0">GitOps for Kubernetes with Jenkins</a> was originally published in <a href="https://medium.com/stakater">Stakater</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Logging in Kubernetes: From container to visualization]]></title>
            <link>https://medium.com/stakater/logging-in-kubernetes-from-container-to-visualization-99524f733d8d?source=rss-7d9450e34ec2------2</link>
            <guid isPermaLink="false">https://medium.com/p/99524f733d8d</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[fluentd]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[logging]]></category>
            <dc:creator><![CDATA[Ahmad Iqbal Ali]]></dc:creator>
            <pubDate>Wed, 13 Feb 2019 08:21:46 GMT</pubDate>
            <atom:updated>2019-02-26T06:13:56.091Z</atom:updated>
            <content:encoded><![CDATA[<p>In our <a href="https://medium.com/stakater/dynamic-log-processing-with-fluentd-konfigurator-and-slack-935a5de4eddb">previous blog</a>, we took a look at some basics of logging in kubernetes and how we can use Fluentd and Stakater Konfigurator for dynamic log parsing. In this blog let’s take a look at the processes and tools to build a broader logging ecosystem for our kubernetes cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*eKZIBE2muvsOdaisEQ0uNQ.png" /></figure><h3>Log file generation</h3><p>Let’s first take a look at how a log file is generated on the cluster. All applications are run in docker containers, so how they generate logs is important. At Stakater we follow the best practice of writing our logs to STDOUT stream in the container. This is compliant with the <a href="https://12factor.net/logs">twelve-factor app</a> principle in that, the application running within the container should not concern itself with how to write, buffer or store the logs. It should simply write its stream of logs to STDOUT. It should be the responsibility of the execution environment to capture, process and route the logs in whichever way is suitable. Also, we do not need to separately add volume mounts to map our log files outside of the container to have them be processed, which would add the path of the log file into the equation. Should there be any change in the file path in future, you will need to handle that in volume mount and maybe even any downstream system. Another benefit of writing to STDOUT is that the logs are automatically captured by docker logs, which is not the case for logs that are not written to STDOUT, such as to a custom log file. Docker of course captures these logs based on the logging driver that it is configured with. Using the default json driver means that the logs are stored in json format on disk in a specific file path.</p><p>In case you are using a third party application or tool which writes its logs to a log file rather than STDOUT, you need not worry. There is always a workaround, and one suitable workaround for this is adding a symbolic link (symlink) from the log file to /dev/stdout. An example of this is Nginx, where the server itself writes its logs to a logfile, but the officially provided <a href="https://github.com/nginxinc/docker-nginx/blob/8921999083def7ba43a06fabd5f80e4406651353/mainline/jessie/Dockerfile#L21">dockerfile</a> adds the symlink.</p><p>Outside of the containers, Kubernetes creates symlinks to the docker log files at /var/log/containers/*.log on the Node. We can therefore easily locate and capture logs within our fluentd daemonset.</p><p>We do not handle persistent storage of log files at this point since we have our fluentd daemonset forwarding the logs in real-time. Persistence of the log files will handled at a later point in the logging flow as we will see soon.</p><h3>Logging flow</h3><h4>Fluentd</h4><p>Fluentd is an open source data collector for a unified logging layer. It has a flexible plugin architecture, allowing easy extension of its functionality. We deploy it as a DaemonSet on our cluster to ensure that all Nodes run a copy of the fluentd Pod. Any new node that is added to the cluster will also get a fluentd pod automatically.</p><p>The fluentd input plugin has the responsibility for reading in data from these log sources, and generating a Fluentd event against it. We use the in_tail Input plugin which allows Fluentd to read events from the tail of text files. This position from where fluentd has read a particular log file is recorded in a position file. And next time fluentd will pick up reading from this position in the file. The input is parsed, based on the configuration provided to the input plugin.</p><p>The fluentd event contains information such as where an event comes from, the time of the event, and the actual log content. We use the kubernetes_metadata_filter plugin to enrich the log event with kubernetes pod and namespace metadata. The plugin gets basic metadata about the container that emitted a given log record using the source of the log record.</p><p>These log events are matched to an Output plugin type in the Fluentd configuration. We use the out_elasticsearch Output plugin to forward these records to Elasticsearch.</p><h4>Logrotate</h4><p>Log rotation essentially renames the old log file, and creates a new log file in its place for continued capturing of logs. This is done so that the file size remains manageable. At Stakater we use the linux logrotate utility for this, also as a daemonset so that it runs on every node like fluentd. At the moment there isn’t a public chart available in the official helm repository, we are using our own which you can find <a href="https://github.com/stakater-charts/logrotate">here</a>.</p><p>We can see an example configuration as follows. The schedule is maintained through cron and the rest of the parameters such as path and size of file, number of rotations, etc. are specified in the configuration:</p><pre>environment:<br> cronSchedule: 0 */12 * * *</pre><pre>config:<br> k8sRotatorConf: |-<br> /var/lib/docker/containers/*/*.log {<br> rotate 5<br> copytruncate<br> missingok<br> notifempty<br> compress<br> maxsize 200M<br> daily<br> create 0644 root root<br> }</pre><h4>Elasticsearch</h4><p>Elasticsearch is basically a NoSQL database that became popular in the Logging domain as part of the ELK/ElasticStack. As we saw before, Fluentd forwards the log streams to Elasticsearch, which then goes on to index and store the log information. Elasticsearch basically allows us to easily search logs based on the metadata that was added by Fluentd. We can easily get results from its REST API to look at logs of a particular application, or apps on a particular node, or even search for all ERROR logs that may have been generated throughout the cluster. A RESTful api however is not a user friendly way of viewing log information, and can also be restricting to one query at a time. A more powerful use of this is to be able to see the log data in terms of trends, and also aggregations. This is where Kibana comes in as we will see in a subsequent section.</p><p>With Elasticsearch we would like to have persistent storage, so that our log data and indices are safe in case of a server crash. We there back the Elasticsearch deployment with a Persistent Volume Claim (PVC) on Kubernetes. This way the logs are persisted across Node restarts.</p><h4>ES Curator</h4><p>We may also like to archive our Elasticsearch indices instead of having them all maintained since the beginning of time in our EBS. Old logs may not be needed and we can easily move them to a cheaper form of storage, such as an AWS S3 bucket with glacial storage. In such cases we use ES Curator. It helps us to curate our Elasticsearch indices. Following is an example configuration of how the curator cron schedule and other properties can be configured.</p><pre>config:<br> cronSchedule: 0 0 * * *<br> esHost: elasticsearch<br> esPort: 9200<br> snapshotName: stackator-es<br> bucketName: 12345678912-stakater-store<br> bucketBasePath: /es-indices-snapshots/<br> awsRegion: us-east-2<br> timeUnit: days<br> olderThan: 30<br> loglevel: INFO</pre><h4>Cerebro</h4><p>Cerebro is a tool we use for web administration of Elasticsearch. It can be useful to get a web view of the elasticsearch nodes, indices, shards, resource usage etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_2dlO31NZALjLP2U" /><figcaption>Cerebro</figcaption></figure><h4>Kibana</h4><p>Kibana, is a visualization tool which helps us to view queries on Elasticsearch. We can view the text log data as-is, filter and query it based on certain labels to view just a subset, and we can also view it in the form of a chart, graph or other visualization. Kibana comes with some visualizations out of the box, and we can also build custom visualizations and dashboards as per our requirement. Visualizations can be helpful in indicating trends such as number of warning or error logs throughout the day time, which could be correlated to changing server load or other factors. Such analyses can then help identify issues pro-actively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FGXQvhvKrarJWfAM" /><figcaption>Kibana</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*56jD1R8u2Sljdu1ZTDaUVQ.png" /><figcaption>Bird’s eye view: Logging Flow</figcaption></figure><figure><a href="http://stakater.com/?utm_medium=blog&amp;utm_source=logging-in-kubernetes-from-container-to-visualization-99524f733d8d"><img alt="" src="https://cdn-images-1.medium.com/max/700/1*0FANFMBaw8Q8aLuPQJLOOA.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=99524f733d8d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/stakater/logging-in-kubernetes-from-container-to-visualization-99524f733d8d">Logging in Kubernetes: From container to visualization</a> was originally published in <a href="https://medium.com/stakater">Stakater</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>