<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stakater - Medium]]></title>
        <description><![CDATA[Looking for a highway to Kubernetes? With our Kubernetes Support Subscription, we offer the tools and support you need for rapid Kubernetes adoption. - Medium]]></description>
        <link>https://medium.com/stakater?source=rss----c05e1bafc3be---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Stakater - Medium</title>
            <link>https://medium.com/stakater?source=rss----c05e1bafc3be---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 03:58:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/stakater" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Mutual TLS with Istio]]></title>
            <link>https://medium.com/stakater/mutual-tls-with-istio-ed82dc3d1711?source=rss----c05e1bafc3be---4</link>
            <guid isPermaLink="false">https://medium.com/p/ed82dc3d1711</guid>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[istio]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[tls]]></category>
            <category><![CDATA[service-mesh]]></category>
            <dc:creator><![CDATA[Waleed Malik]]></dc:creator>
            <pubDate>Tue, 17 Mar 2020 13:53:53 GMT</pubDate>
            <atom:updated>2020-03-17T13:53:53.042Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Service mesh</strong> is a decentralized application-networking infrastructure that allows applications to be secure, resilient, observable and controllable. The service mesh exists to make your distributed applications behave reliably in any environment e.g. production. There are multiple open-source products available like linkerd, istio, Conduit etc. <br><a href="https://istio.io/docs/concepts/what-is-istio/"><strong>Istio</strong></a> is an open-source implementation of a service mesh that provides several capabilities for traffic monitoring, access control, discovery, security, resiliency, and other useful things to a bundle of services. Istio does all that at a platform level so it requires minimal, if any, code changes to the applications.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/895/1*SaLs1F9vUaDePSV1lkIihQ.png" /><figcaption><a href="https://istio.io/docs/concepts/what-is-istio/#what-is-a-service-mesh">https://istio.io/docs/concepts/what-is-istio/#what-is-a-service-mesh</a></figcaption></figure><p>Istio deploys <a href="https://www.envoyproxy.io/"><strong>envoy proxy</strong></a> as a sidecar alongside each application instance. They are responsible for handling all incoming and outgoing traffic from the application. This approach makes it possible to have consistent network policies throughout the service mesh. These sidecars can take the responsibility of encrypting the traffic without the application having to do anything or even know.<br>Transport Layer Security(TLS) ensures that communication between services is encrypted. In mTLS the client and server both verify each other’s certificates and use them to encrypt traffic using TLS. Istio takes care of certificate generation and maintenance using <a href="https://istio.io/docs/concepts/security/"><strong>Citadel</strong></a> and propagates these certificates to all the istio envoy proxies.</p><p>Istio requires us to use a <a href="https://istio.io/docs/tasks/security/authentication/authn-policy/"><strong>Policy</strong></a> object to instruct a service, namespace, or mesh to recieve mTLS traffic. We then use <a href="https://istio.io/docs/reference/config/networking/destination-rule/"><strong>DestinationRule</strong></a> to define policy that ensures that all traffic intended for the service(s) uses mTLS i.e. client services make mTLS connection with target services using appropriate certificates.</p><h4>mTLS types:</h4><p>1. Permissive: Allow both http and mutual TLS traffic<br>2. Strict: Allow only mutual TLS traffic i.e. it won’t accept plain text data<br>3. Disabled: Plain text data will be sent in and out of the mesh</p><h3>To enable mTLS first create mTLS authentication policy:</h3><ol><li>Mesh-wide:</li></ol><pre>apiVersion: authentication.istio.io/v1alpha1<br>kind: MeshPolicy<br>metadata:<br>  name: default<br>spec:<br>  peers:<br>  - mtls: {}</pre><p>2. Namespace-wide:</p><pre>apiVersion: authentication.istio.io/v1alpha1<br>kind: Policy<br>metadata:<br>  name: default<br>  namespace: namespace-name<br>spec:<br>  peers:<br>  - mtls: {}</pre><p>3. For a specific service:</p><pre>apiVersion: authentication.istio.io/v1alpha1<br>kind: Policy<br>metadata:<br>  name: default<br>  namespace: namespace-name<br>spec:<br>  targets:<br>  - name: service-name<br>  peers:<br>  - mtls: {}</pre><h3>Create destination rule:</h3><p>The “Policy” informs the running services to expect any incoming traffic to use mTLS. But for “Mutual” TLS, we need to inform the clients to use mTLS when they call a service. DestinationRule serves this purpose:</p><pre>apiVersion: networking.istio.io/v1alpha3<br>kind: DestinationRule<br>metadata:<br>  name: default<br>  namespace: namespace-name<br>spec:<br>  host: &quot;service-name.namespace-name.svc.cluster.local&quot;<br>  trafficPolicy:<br>    tls:<br>      mode: ISTIO_MUTUAL</pre><p>Use*.namespace-name.svc.cluster.local for forcing mTLS on all services in that particular namespace and *.local for mesh-wide mTLS.</p><h4>Health checks:</h4><p>Requests for health checks, against liveness and readiness probes, are sent from Kubelet which does not have Istio issued certificates. So when mutual TLS is enabled, the health check requests will fail. An easy fix for this is to let istio handle it, istio can rewrite the applications PodSpec readiness/liveness probe, such that the probe request will be sent to <a href="https://istio.io/docs/reference/commands/pilot-agent/">Pilot agent</a>. Pilot agent then redirects the request to the application, and strips the response body only returning the response code.</p><p>To use this, use the following value with istio helm chart(or deployment):<br>values.sidecarInjectorWebhook.rewriteAppHTTPProbe=true</p><p>Or, annotate the pod with sidecar.istio.io/rewriteAppHTTPProbers: &quot;true&quot;</p><h3>Demo:</h3><p>Estimated time: 5 mins</p><p>Now let’s go through a short practical demonstration for mTLS. We’ll use <a href="https://istio.io/docs/examples/bookinfo/">bookinfo</a> which is an application based on micro-services architecture.</p><figure><img alt="https://istio.io/docs/examples/bookinfo/" src="https://cdn-images-1.medium.com/max/870/1*1RaC4aTRgzNsr5q3wJS4lw.png" /><figcaption><a href="https://istio.io/docs/examples/bookinfo/">https://istio.io/docs/examples/bookinfo/</a></figcaption></figure><p>1. Deploy BookInfo</p><pre>kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.4/samples/bookinfo/platform/kube/bookinfo.yaml</pre><p>2. Verify that the application is accepting plain-text requests:</p><pre>kubectl exec $(kubectl get pod -l app=productpage -o jsonpath={.items..metadata.name}) -c istio-proxy -- curl http://reviews:9080/health -o /dev/null -s -w &#39;%{http_code}\n&#39;</pre><p><strong>Output</strong>: <em>200</em></p><p>3. Apply policy and destination rule(namespace-wide for demo):</p><pre>cat &lt;&lt;EOF | kubectl apply -f -<br>apiVersion: authentication.istio.io/v1alpha1<br>kind: Policy<br>metadata:<br>  name: default<br>  namespace: default<br>spec:<br>  peers:<br>  - mtls: {}<br>---<br>apiVersion: networking.istio.io/v1alpha3<br>kind: DestinationRule<br>metadata:<br>  name: default<br>  namespace: default<br>spec:<br>  host: &quot;*.default.svc.cluster.local&quot;<br>  trafficPolicy:<br>    tls:<br>      mode: ISTIO_MUTUAL<br>EOF</pre><p>4. Plain-text requests will fail now with:</p><pre>kubectl exec $(kubectl get pod -l app=productpage -o jsonpath={.items..metadata.name}) -c istio-proxy -- curl http://reviews:9080/health -o /dev/null -s -w &#39;%{http_code}\n&#39;</pre><p><strong>Output</strong>: <em>000 command terminated with exit code 56</em></p><p>5. Verify that the application is accepting TLS requests:</p><pre>kubectl exec $(kubectl get pod -l app=productpage -o jsonpath={.items..metadata.name}) -c istio-proxy -- curl https://reviews:9080/health -o /dev/null -s -w &#39;%{http_code}\n&#39; --key /etc/certs/key.pem --cert /etc/certs/cert-chain.pem --cacert /etc/certs/root-cert.pem -k</pre><p><strong>Output</strong>: <em>200</em></p><h3>Relevant links:</h3><p><a href="https://github.com/stakater/til/tree/master/istio">https://github.com/stakater/til/tree/master/istio</a></p><p><a href="https://istio.io/docs/tasks/security/authentication/mutual-tls/">https://istio.io/docs/tasks/security/authentication/mutual-tls</a>/</p><p><a href="https://istio.io/docs/concepts/">https://istio.io/docs/concepts/</a></p><p><a href="https://redhat-developer-demos.github.io/istio-tutorial/istio-tutorial/1.3.x/8mTLS.html">https://redhat-developer-demos.github.io/istio-tutorial/istio-tutorial/1.3.x/8mTLS.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ed82dc3d1711" width="1" height="1" alt=""><hr><p><a href="https://medium.com/stakater/mutual-tls-with-istio-ed82dc3d1711">Mutual TLS with Istio</a> was originally published in <a href="https://medium.com/stakater">Stakater</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Openshift DeploymentConfig for Stakater Reloader]]></title>
            <link>https://medium.com/stakater/openshift-deploymentconfig-for-stakater-reloader-b124745b4e0a?source=rss----c05e1bafc3be---4</link>
            <guid isPermaLink="false">https://medium.com/p/b124745b4e0a</guid>
            <category><![CDATA[configuration]]></category>
            <category><![CDATA[openshift]]></category>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[reloader]]></category>
            <category><![CDATA[kubernetes-controller]]></category>
            <dc:creator><![CDATA[Irtiza]]></dc:creator>
            <pubDate>Tue, 03 Mar 2020 16:19:40 GMT</pubDate>
            <atom:updated>2020-03-17T14:03:06.848Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>OpenShift deployment is a replication controller based on a user-defined template called a deployment configuration. A template describes a set of <a href="https://docs.okd.io/latest/architecture/core_concepts/index.html#architecture-core-concepts-index">objects</a> that can be parameterized and processed to produce a list of objects like <a href="https://docs.okd.io/latest/architecture/core_concepts/pods_and_services.html#services">services</a>, <a href="https://docs.okd.io/latest/architecture/core_concepts/builds_and_image_streams.html#builds">build configurations</a>, and <a href="https://docs.okd.io/latest/architecture/core_concepts/deployments.html#deployments-and-deployment-configurations">deployment configurations</a> for creation by OpenShift. It can also be used to assign <a href="https://docs.okd.io/latest/architecture/core_concepts/pods_and_services.html#labels">labels</a> to every object defined in a template. An <a href="https://docs.openshift.com/enterprise/3.0/architecture/core_concepts/index.html#architecture-core-concepts-index">object</a> is an entity like pod, service and deployment etc, that can be used to create resources in OpenShift, most of these objects have been derived from Kubernetes and extended by OpenShift to provide a more feature-rich development lifecycle platform. OpenShift Deployments are created manually or in response to triggered events. While the Kubernetes <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/">deployment</a> controller provides a declarative way to create and update pods and replicaSets.</p><h3>Comparison</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*o39HcHMH0i2hBS650_fOng.png" /></figure><p>OpenShift Deployment configuration differs from Kubernetes deployment in the following ways:</p><ol><li><strong>Dependency</strong></li></ol><p>Deployment configuration uses a <em>replication controller</em> while Kubernetes deployment uses a <em>replicaSet controller</em>.</p><p><strong>2. Rollback</strong></p><p>Deployment configuration controller will rollback to the previous state in case of a failure which Kubernetes deployment does not support this feature.</p><p><strong>3. Triggers</strong></p><p>Deployment configuration has an implicit <strong>ConfigChange</strong> trigger that gets triggered due to change in pod template of deployment for a new rollout. It also provides <strong>ImageChange</strong> trigger. While Kubernetes deployment triggers new rollout on <strong>ConfigChange.</strong></p><p><strong>4. Lifecycle Hooks</strong></p><p>Deployment configuration supports <a href="https://docs.openshift.com/container-platform/3.4/dev_guide/deployments/deployment_strategies.html#lifecycle-hooks">lifecycle hooks</a>, while it is not supported in Kubernetes deployment.</p><p><strong>5. Canary Deployments</strong></p><p>Deployment configuration supports <a href="https://docs.openshift.com/container-platform/3.3/dev_guide/deployments/deployment_strategies.html#canary-deployments">canary deployment</a>, while it is not supported in Kubernetes deployment.</p><p><strong>6. Rollover</strong></p><p>In the deployment process, Kubernetes deployments use a controller loop, while OpenShift deployment configuration uses deployer pods for every new rollout. This means that a Kubernetes deployment can have as many active replica sets as possible, and eventually, the deployment controller will scale down all old replica sets and scale up the newest one.</p><p>OpenShift deployment configuration can have at most one deployer pod running, otherwise, multiple deployers end up fighting with each other trying to scale up what they think should be the newest replication controller. Because of this, only two replication controllers can be active at any point in time. Ultimately, this translates to faster rapid rollouts for Kubernetes deployments.</p><p><strong>7. Proportional Scaling</strong></p><p>Kubernetes deployment can scale ongoing rollouts because additional replicas are distributed proportionally based on the size of each replica set. While it is not possible in OpenShift deployment controller because it will end up fighting with the deployer process about the size of the new replication controller</p><p><strong>8. Pausing Rollout</strong></p><p>Kubernetes deployments ongoing rollout can be paused at any time. While the OpenShift deployment controller cannot be paused during rollout because the deployer process will not be affected until it finishes its job.</p><h3>How it works</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*fb5P5g4_qAmIiPrVzQD7zQ.png" /></figure><p>Kubernetes applications need configuration, whether related to authentication with parameters such as API keys, tokens, or other credentials, or general parameters such as logging configuration, environment variables, or other flags. In Kubernetes, these are usually handled by either <em>ConfigMaps</em> or <em>Secrets</em>. And configuration can be updated, whether it be for credential updates or rotation, or toggling logging on or off, or updating a particular environment parameter. Updates may take place frequently or rarely depending on the application, and there will be cases where we would like such updates to be reflected immediately within our applications. At this point, Reloader plays its part by continuously watching <em>Secrets</em> and <em>ConfigMaps</em>. Whenever there is an update in either, Reloader will then check whether any <em>Deployment</em>, <em>DaemonSet</em> or <em>StatefulSet</em> is dependent on it. Once dependent resources are identified, Reloader will trigger automatic updates for all resources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/270/1*msx8-mOM_cT8Ih71FXWCtA.png" /></figure><h3>Improvements</h3><p>Reloader’s previous version only supported the Kubernetes environment but now it also supports OpenShift. Now Reloader can be deployed using the deployment configuration controller.</p><h3>In Closing</h3><p>Stakater Reloader has significant improvements compared to other similar open-source controllers. With its support for different resource types, open-source license, and stable code base, it is a good controller to use when pods need to be automatically restarted on configuration update.</p><p>More information is available at the Github repository: <a href="https://github.com/stakater/reloader">https://github.com/stakater/reloader</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*2V7WZxl6H6-EbBeI3YyeLQ.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b124745b4e0a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/stakater/openshift-deploymentconfig-for-stakater-reloader-b124745b4e0a">Openshift DeploymentConfig for Stakater Reloader</a> was originally published in <a href="https://medium.com/stakater">Stakater</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting Started with AWS EKS using Terraform]]></title>
            <link>https://medium.com/stakater/getting-started-with-aws-eks-using-terraform-7487ffeac48f?source=rss----c05e1bafc3be---4</link>
            <guid isPermaLink="false">https://medium.com/p/7487ffeac48f</guid>
            <category><![CDATA[ek]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[Ali Kahoot]]></dc:creator>
            <pubDate>Tue, 03 Mar 2020 10:36:54 GMT</pubDate>
            <atom:updated>2020-03-03T10:36:54.091Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*EW-5IJ3TJX1q8GnRNI_P-Q.jpeg" /></figure><p>Kubernetes (K8s) is an open-source container orchestration system for automating deployment, scaling, and management of containerized applications. Following is the architecture of a Kubernetes cluster</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*BbrbBu7KiY09r9RiptRRhg.png" /><figcaption>Source: <a href="https://www.guru99.com/kubernetes-tutorial.html">https://www.guru99.com/kubernetes-tutorial.html</a></figcaption></figure><p>There are different ways to provision a Kubernetes cluster</p><ul><li><a href="https://kubernetes.io/docs/setup/learning-environment/minikube/">Minikube</a>: Easily create a local, single-node Kubernetes cluster for<br>development and testing</li><li><a href="https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/">kubeadm</a>: Bootstrap a Kubernetes cluster conforming to best practices. Can be installed on various types of platforms and can easily integrate<br>with provisioning systems such as Terraform or Ansible</li><li><a href="https://kubernetes.io/docs/setup/production-environment/tools/kops/">Kops</a>: Create, destroy, upgrade and maintain production-grade, highly<br>available Kubernetes clusters from the command line on AWS only</li><li>Hosted/Managed Solutions: Almost all clouds provide managed Kubernetes services like Azure offers AKS, IBM cloud offers IKS, Google offers GKE and AWS offers EKS.</li></ul><p>In Managed Kubernetes, you don’t have to manage the master node components of K8s, rather the cloud manages that, and you just provision the worker nodes and the cloud will handle the uptime and related features. You just start deploying your workloads on the cluster.</p><p>In this post, I will be discussing how to provision an AWS EKS cluster using Terraform. Terraform is a tool to manage your infrastructure as code so rather than provisioning an EKS cluster and related things manually, you can use terraform to provision them for you.</p><p>You can install terraform by following this <a href="https://learn.hashicorp.com/terraform/getting-started/install.html">link</a>. And you can verify it by running</p><pre>$ terraform<br>Usage: terraform [--version] [--help] &lt;command&gt; [args]</pre><pre>The available commands for execution are listed below.<br>The most common, useful commands are shown first, followed by<br>less common or more advanced commands. If you&#39;re just getting<br>started with Terraform, stick with the common commands. For the<br>other commands, please read the help and docs before usage.</pre><pre>Common commands:<br>    apply              Builds or changes infrastructure<br>    console            Interactive console for Terraform interpolations<br># ...</pre><p>And confirm the version by running</p><pre>terraform version</pre><h3><strong>Assumptions</strong></h3><ul><li>You’ve created a Virtual Private Cloud (VPC) and subnets where you intend to put the EKS resources. The VPC satisfies <a href="https://docs.aws.amazon.com/eks/latest/userguide/network_reqs.html">EKS requirements</a>.</li><li>You have aws configured locally i.e. your ACCESS_KEY and SECRET_ACCESS_KEY is set and you have a user that has <a href="https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/iam-permissions.md">these permissions</a></li><li>If manage_aws_auth = true, it&#39;s required that both <a href="https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl">kubectl</a> (&gt;=1.10) and <a href="https://github.com/kubernetes-sigs/aws-iam-authenticator#4-set-up-kubectl-to-use-authentication-tokens-provided-by-aws-iam-authenticator-for-kubernetes">aws-iam-authenticator</a> are installed and on your shell&#39;s PATH.</li></ul><h3><strong>Steps</strong></h3><ol><li>Get the VPC and subnet ids and the region in which you want to create an EKS cluster.</li></ol><p>2. Create a file <strong>main.tf </strong>in which we will define the cluster manifest and for simplicity, instead of using variables, we will be using <strong>locals </strong>and defining customizable variables there.</p><p>Create a locals section in the file and copy following manifest in it and replace the variables with your values</p><pre>locals{<br>    vpc_id = &quot;vpc-1234ab&quot;<br>    region = eu-west-1<br>    cluster_name = &quot;dev-eks-cluster&quot;<br>    subnets = [&quot;subnet-1234&quot;, &quot;subnet-1235&quot;, &quot;subnet-1236&quot;]<br>    cluster_enabled_log_types = [&quot;api&quot;, &quot;audit&quot;, &quot;authenticator&quot;, &quot;controllerManager&quot;, &quot;scheduler&quot;]<br>    asg_desired_capacity = 1<br>    asg_max_size = 3<br>    asg_min_size = 1<br>    instance_type = &quot;m4.large&quot;<br>    spot_price = &quot;0.20&quot;<br>}</pre><p>These are the customizable variables that will be used in the terraform module. The variables are self explanatory like VPC id and subnet ids where your EKS cluster will be running. The variable <strong>cluster_enabled_log_types </strong>is the list of the desired control plane logs to enable. These logs will be available in Cloudwatch -&gt; LogGroups. The other variables are regarding the Autoscaling Group Instance counts.</p><p>We also focus on cost reduction so we are using Spot Instances rather than dedicated instances. So you can set the <strong>spot_price </strong>that you want to bid for the instances. Spot instances are almost 80–90% cheaper than normal dedicated instances. For details about Spot Instances vs Dedicated Instances see <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-purchasing-options.html">AWS Instance Purchasing Options</a></p><p>Add a <strong>provider aws</strong> and <strong>terraform</strong> section to specify region and version.</p><pre>provider &quot;aws&quot; {<br>  region  = local.region<br>}<br>terraform {<br>  required_version = &quot;&gt;= 0.12.7&quot;<br>}</pre><p>And now setup the EKS module to be used</p><pre>module &quot;eks-cluster&quot; {<br>  source       = &quot;terraform-aws-modules/eks/aws&quot;<br>  version = &quot;v7.0.1&quot;<br>  cluster_name = local.cluster_name<br>  subnets = local.subnets<br>  vpc_id  = local.vpc_id</pre><pre>  worker_groups = [<br>    {<br>      spot_price = locals.spot_price<br>      asg_desired_capacity = locals.asg_desired_capacity<br>      asg_max_size = locals.asg_max_size<br>      asg_min_size = locals.asg_min_size<br>      instance_type = locals.instance_type<br>      name = &quot;worker-group&quot;<br>      additional_userdata = &quot;Worker group configurations&quot;<br>      tags = [{<br>          key                 = &quot;worker-group-tag&quot;<br>          value               = &quot;worker-group-1&quot;<br>          propagate_at_launch = true<br>      }]<br>    }<br>  ]</pre><pre>  map_users = [<br>      {<br>        userarn = &quot;arn:aws:iam::AWS_ACCOUNT:user/AWS_USERNAME&quot;<br>        username = &quot;AWS_USERNAME&quot;<br>        groups = [&quot;system:masters&quot;]<br>      },<br>    ]<br>  cluster_enabled_log_types = local.cluster_enabled_log_types<br>  tags = {<br>    environment = &quot;dev-env&quot;<br>  }<br>}</pre><p>Your final file should have all of the above manifests and should look like</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/72ae510239d1954f3e1ac49fd71a6d52/href">https://medium.com/media/72ae510239d1954f3e1ac49fd71a6d52/href</a></iframe><p>3. Replace the above placeholders with your values. You can set the worker nodes autoscaling group desired, minimum and maximum capacity and the spot price for the nodes in the <strong>worker_groups</strong> section</p><p>In the <strong>map_users</strong> section, you can add any other AWS user that you want to have access to the cluster that you created.</p><p>These are the simplest variables that you can set to create your cluster, for other variables and their details, see <a href="https://github.com/terraform-aws-modules/terraform-aws-eks#inputs">Terraform-AWS-EKS-Inputs</a></p><p>4. After this run</p><pre>terraform init<br>terraform apply</pre><p>This will show the resources that the terraform module will create and will ask you to confirm them, type <strong>yes </strong>so terraform will start creating. <br>It takes sometime to create the cluster almost 10 minutes, and after that if everything goes smooth hopefully, terraform will show</p><pre>Apply complete! Resources: N added, 0 changed, 0 destroyed.</pre><p>It will also add the <strong>kubeconfig</strong> of your cluster in the folder directory. And a configmap, which will contain the user that you had given permission while creating the cluster. That user/users and the user that created the cluster can access the cluster using the kubeconfig.</p><p>5. Run the following commands to verify your access to the cluster</p><pre>kubectl get nodes --kubeconfig kubeconfig_dev-eks-cluster<br>kubectl get pods --all-namespaces --kubeconfig kubeconfig_dev-eks-cluster</pre><p>And your EKS cluster is ready, you can start deploying your workloads and take use of managed Kubernetes from AWS.</p><h4>Cleanup</h4><p>Now to destroy the workloads, run</p><pre>terraform destroy</pre><p>And it will show resources that will be deleted, acknowledge it by entering yes and the resources will be deleted.</p><h4><strong>Cost</strong></h4><p>Cost of Running an EKS cluster is $0.10 per hour, so for a month(almost 744 hours) it becomes almost 75$ per month per EKS cluster(the master instance that AWS manages for you) in AWS. Other than this, the cost for worker nodes is dependent on number &amp; type of EC2 instances that you create and their cost is the same as you create a normal EC2 instance.</p><p>We deployed <a href="https://github.com/stakater/stakaterplatform">StakaterPlatform</a>(A consolidated solution that helps the community to adopt Security, Monitoring, Logging, Tracing, Alerting and CI/CD processes for their workloads) and <a href="https://stakater.com/">Stakater</a>’s production grade applications which needs almost 8 VCPUs &amp; 32 GB of RAM(2 * t2.xlarge spot instances) and it cost us around <strong>5-6$ per day</strong> which is quite reasonable keeping in view that the master is managed by AWS and they will guarantee the uptime of master components.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*Eb2RuyjrL1wsw1gbPHxS5A.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7487ffeac48f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/stakater/getting-started-with-aws-eks-using-terraform-7487ffeac48f">Getting Started with AWS EKS using Terraform</a> was originally published in <a href="https://medium.com/stakater">Stakater</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monitoring on Kubernetes]]></title>
            <link>https://medium.com/stakater/monitoring-on-kubernetes-ac5231c8167f?source=rss----c05e1bafc3be---4</link>
            <guid isPermaLink="false">https://medium.com/p/ac5231c8167f</guid>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[monitoring]]></category>
            <dc:creator><![CDATA[Ahmad Iqbal Ali]]></dc:creator>
            <pubDate>Mon, 13 May 2019 06:30:41 GMT</pubDate>
            <atom:updated>2019-05-13T06:30:18.562Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VYQSbSoWjcP3KgqnnKTDKw.jpeg" /></figure><p>Monitoring is an important component of observability. With all our applications running along-with the various tools, we need to keep an eye on each as well as the underlying frameworks and infrastructure. This helps us ensure and stay in the know of how each component is performing and whether they are achieving the availability and performance expectations that they are supposed to. Monitoring helps us gain insight into cloud-native systems, and combined with alerting it allows DevOps engineers to observe scaled-out applications</p><p>Cloud-native systems by nature are ephemeral. They may be recycled at any time, be scaled up or down, triggering events in each case. And given the scale that they can grow to, the monitoring should not just treat any specific component independently, but rather be able to aggregate the results of the components.</p><h3>Metrics</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/746/1*ERST8jrPcVCfLeQJRDXvVQ.jpeg" /></figure><p>Metrics are the measurements that help give us a picture of the system is behaving and help us make informed decisions of whether there needs to be any change in the system to optimize the performance of the application as per expectations. Metrics are a combination of data from measuring multiple events. Cloud-native monitoring tools cater to different types of measurements by having various metrics. Some of these metrics may be cumulative metrics that can only ever increase. For others the value may go up and down over time and only a point-in-time value will be required. The idea is to aggregate similar events to gain a higher level of insight. Metrics are generally time based, therefore we usually collect metrics periodically, such as once per second.</p><h3>Prometheus</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/114/1*IAkVYhV3VTF2UBptrVGdJA.png" /></figure><p>At Stakater, we use Prometheus as our monitoring tool. Prometheus is an open source toolkit for monitoring and alerting under the Cloud-native Computing Foundation (CNCF). Some of the key features that we like about it are that it pulls metric data from services and each service doesn’t need to push metrics to Prometheus itself. Therefore Services need to only expose endpoints where they publish their metrics, and prometheus will call this endpoint and collect the metrics by itself. Additionally service targets can be discovered either using service discovery or via static configuration. Prometheus’ powerful query language, PromQL, is helpful for selecting and aggregating time series data in real time. Prometheus itself has a rudimentary web UI where the status of the monitored services can be viewed, however Prometheus also exposes it’s monitoring results via a HTTP api which makes it easy for an external system such as a more advanced visualization tool to consume and process the metrics. One such tool is Grafana which we will discuss a bit later.</p><p>Deploying and configuring Prometheus can seem a bit daunting at first but using Prometheus Operator it becomes quite easy.</p><h4>Prometheus Operator</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/126/1*-69z4lKwxdklZK165ronkQ.png" /></figure><p>An Operator is a convenient method of deploying and managing an application on Kubernetes. Using some Custom Resource Definitions, we are able to easily configure the deployment we would like, and the <a href="https://coreos.com/operators/">Operator</a> takes care of the heavy lifting of actual deployment work.</p><p>Prometheus Operator in this way makes the Prometheus configuration Kubernetes native and manages and operates Prometheus clusters. We can easily provide monitoring definitions for Kubernetes services and deployment and management of Prometheus instances. With the operator, we can easily create or destroy Prometheus instances, configure our Prometheus deployment with settings such as versions, persistence, retention policies, etc. And we can also use it to automatically generate monitoring configurations based on Kubernetes labels, and easily define the targets we would like to start monitoring.</p><p>The Operator ensures at all times that a deployment matching the Prometheus resource definition is running. The Operator also automatically generates Prometheus scrape configuration based on the definition of ServiceMonitor. And in case we need to modify the Prometheus stack deployment, instead of modifying each component Deployment or StatefulSet, we simply customize the CRDs and let the operator handle the orchestration.</p><h3>Monitoring Applications</h3><h4>Monitoring Services</h4><p>Prometheus consumes metrics that are exposed by services through http(s). We therefore don’t need to install any service agent, just expose a web port to reach our service’s endpoint and Prometheus server will regularly scrape the endpoint. The metrics format itself is in json which is easily readable by a human.</p><p>For any service that may not be natively integrable, it can be easily adapted using a Prometheus exporter. An example of this is the Node exporter that we will discuss shortly, which is used to expose hardware/os metrics from a node for prometheus to consume.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/674/1*NoPwTF56GsOdnDZlb7lbtg.png" /></figure><p>The Prometheus object filters and selects ServiceMonitor objects, which in turn, filter and select Prometheus metrics endpoints. If there is a new metrics endpoint that matches the ServiceMonitor criteria, this target will be automatically added to all the Prometheus servers that select that ServiceMonitor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iI5PQLNW8Oo5EixV" /></figure><h4>Monitoring short-lived jobs</h4><p>While any regular applications will always be alive, and therefore their service will be available to be scraped periodically, a short-lived job may not be available long enough for prometheus to scrape it at the configured interval. For such cases, we have the Prometheus pushgateway which is a metrics cache. Any short-lived jobs can push their metrics to the pushgateway, from where they can be scraped by Prometheus server at the next scraping interval. Client libraries are available for a few platforms such as java, go, etc. so relevant jobs can easily push their metrics to pushgateway. The Prometheus text protocol however makes it so simple to push metrics that a script can even use a command-line HTTP tool like curl to send metrics.</p><h3>Monitoring Cluster</h3><p>Prometheus can be used in pro-actively monitoring the kubernetes cluster. This provides cluster visibility and help with capacity planning. The cluster activity can be available on a metrics dashboard, and can subsequently also be used to trigger alerts and notification</p><p>The following Cluster components can be monitored:</p><ul><li>Orchestration level metrics — Deployment state, resource requests, scheduling and api server latency, etc.</li><li>Kubernetes nodes — classical sysadmin metrics such as cpu, load, disk, memory, etc.</li></ul><h4>Monitoring Kubernetes Objects</h4><p>`kube-state-metrics` is a simple service that listens to the Kubernetes API server and generates metrics about the state of the objects. It listens to the Kubernetes API server and generates metrics about the state of the objects such as deployments, nodes and pods, which the prometheus server can scrape. This is useful to present orchestration metadata i.e. statuses of deployments, pods, replica statuses, etc.</p><h4>Monitoring Kubernetes nodes</h4><p>`Node_exporter`is a Prometheus exporter that exposes hardware and OS metrics for Prometheus server to consume. Some collectors are enabled by default in the Node exporter which may be enough to gather the required memory, Cpu and disk usage metrics for our requirements. The Node exporter is deployed as a daemonset when we use the Prometheus operator, so that it automatically covers all nodes in the cluster at any point in time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/656/1*EkiFgTiNl_Wr8Q0nu8n9KA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/725/1*bqBWk_mspI4-bh6SA_JFcg.png" /><figcaption>Example: pod loads</figcaption></figure><h3>Monitoring logging stack</h3><p>At stakater we use Elastic stack for monitoring, and it can be quite important to monitor the performance and availability of the logging stack. Since logs are an important part of observability, we need assurance that there isn’t any gap in capturing them. Monitoring the logging stack will provide that visibility and transparency as to the performance and availability of the tools involved in the logging stack.</p><h4>Monitoring Fluentd</h4><p>Fluentd has specific plugins that can be used for making metrics available for prometheus scraping. The `prometheus` Filter plugin is used to read and process the incoming log records, and the `prometheus` Output plugin is used to do the same with the records being forwarded, in our case to elasticsearch. And the `prometheus` input plugin is used to exposed the metrics for consumption. Since fluentd will be running as a daemonset, each instance will expose the metrics of its current node. The metrics that we can export from fluentd can be number of total records, queue length, emit count, retry count, error count, etc. These metrics can give us an idea of the health of each fluentd instance relative to the load that it is meant to process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Jktn-EACL8BzaUSA" /><figcaption>Example: fluentd metrics in prometheus</figcaption></figure><h4>Monitoring Elasticsearch</h4><p>Elasticsearch is used to index the logs, and we also persist all our logs in the elasticsearch instance, it is therefore important to monitor it. We use a third party prometheus exporter for exposing Elasticsearch metrics. We capture metrics such as free space on block device, count of request cache hit/miss, percent CPU used by process, etc.</p><h3>Visualizing Metrics</h3><p>Prometheus metrics can be used for alerting based on specific thresholds, however visualizing the metrics even if they are not triggering alerting thresholds, is also useful. Grafana is one of the most popular visualization platforms and add-ons in the Prometheus solution. It pulls data from the Prometheus server and can display that in range of visualizations. The visualizations can also be grouped into custom dashboards. With a visualization tool such as this, the state of the system can easily be viewed by all stakeholders in the project team.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kAwVOP7TVc5ROUyc" /><figcaption>Example: Resources consumption of a particular deployment (CPU, Memory, Volume, etc.)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ukQQ5UFF8Je5ClTG" /><figcaption>Example: Overall cluster resources consumption (CPU, Memory, Volume, etc.)</figcaption></figure><h3>Summing up</h3><p>The overall monitoring architecture will look as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Uwywd9lRbHu51Djd" /><figcaption>Bird’s eye view: Monitoring architecture</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*0FANFMBaw8Q8aLuPQJLOOA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ac5231c8167f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/stakater/monitoring-on-kubernetes-ac5231c8167f">Monitoring on Kubernetes</a> was originally published in <a href="https://medium.com/stakater">Stakater</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Easy Kubernetes Development with Google Skaffold]]></title>
            <link>https://medium.com/stakater/easy-kubernetes-development-with-google-skaffold-cbddb7e40519?source=rss----c05e1bafc3be---4</link>
            <guid isPermaLink="false">https://medium.com/p/cbddb7e40519</guid>
            <category><![CDATA[development]]></category>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[skaffold]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Stakater]]></dc:creator>
            <pubDate>Fri, 03 May 2019 09:45:25 GMT</pubDate>
            <atom:updated>2019-05-03T09:54:20.768Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*ZMGYxvJqigG_HiDWXMPopQ.jpeg" /></figure><p>Skaffold is a command line tool from Google that brings automation to the deployment workflow of Kubernetes applications. This tool is tailored to provide developers with an experience of iteration and deployment which closely mirrors production. What this essentially means is that the developers will now be able to locally iterate on application source code without worrying about how it is updated in the later stages of validation and testing in the Kubernetes clusters.</p><p>Skaffold automates the traditional workflow and therefore handles building, pushing and deploying of applications running on Kubernetes clusters. Any directory with a Dockerfile can be deployed to Kubernetes cluster with this command line tool. What happens technically is that Skaffold builds the docker image locally, pushes it to a registry and deploys it using the skaffold CLI tool. For that reason, the work of developers becomes quite easy since they only need to launch Skaffold, run it in the background and develop their code while Skaffold updates the application continuously without the intervention of typing in any extra commands. At the same time, it can be employed when applications are being moved to production in the processes of continuous integration and continuous delivery due to its powerful tools. That is the inherent power that comes with Skaffold.</p><h3>Main features</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*wQ00NtXfM2Ivx39mRF-3Kg.jpeg" /></figure><p>Skaffold solves a problem for developers and to better understand how it works, let us look at the features that it offers:</p><ul><li>Skaffold is an automation tool and has the capability of detecting changes made on the source code while it runs and proceeds with the process of building, pushing and deploying automatically.</li><li>It supports multiple application components: Skaffold supports the workflows and tools that are already in your environment.</li><li>During implementation, Skaffold updates image tags in Kubernetes manifests which alleviates the developer from doing it manually. Thanks to the image tag management feature that comes with Skaffold.</li><li>Skaffold automatically detect changes in your source code and automatically build/push/deploy.</li><li>What is more interesting about Skaffold is that it has no server-side component. This reduces the configuration process and cluster overhead.</li></ul><h4>Where Skaffold saves the day</h4><ul><li>Deployment of applications on Kubernetes</li><li>Definition of build, push and deployment pipelines in the Continuous integration and Continuous delivery flow.</li><li>Quite useful in iterative builds in a loop of continuous build-deploy process.</li></ul><h4>Advantages of Skaffold</h4><p>Skaffold guarantees easy app deployment compared to the traditional way. This shortens the time of moving from Development to Production.</p><p>The architecture that Skaffold is built upon is a pluggable type of architecture. What this allows one to have is the freedom to pick the tools in the development workflow that work best. This provides flexibility and high usability compared to other tools such as Draft.</p><p>When compared to other tools, Skaffold has the source-code watching capability which triggers build and push automatically without further commands to facilitate it. This is one of the major reasons one should consider using Skaffold.</p><p>Skaffold saves invaluable time in deployment which can be spent on other activities to improve the economic status of an organization.</p><h3>Installing Skaffold</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*f5dkRxZT8pAdSWF1ioLEHg.jpeg" /></figure><p>For this setup, you need a working Kubernetes cluster or alternatively use Minikube. Check out our previous guide on how to install and configure minikube on your local machine. Once you have Kubernetes environment ready, proceed to install Skaffold.</p><p>Installation of Skaffold has the following prerequisites:</p><ul><li>Docker Engine</li><li>Docker image registry: Your docker client should be configured to push to an external docker image repository. If you’re using a minikube or Docker for Desktop cluster, you can skip this requirement.</li><li>kubectl — Kubernetes command-line tool</li></ul><p>Installing Skaffold is a straightforward process since the package is distributed as a binary file. You only need to download the package, place it under your <strong>$PATH</strong> and make it executable. You can use <em>wget</em> or <em>curl</em> to pull the binary file.</p><h4>Installing on Linux</h4><p>Download the package:</p><pre># wget https://storage.googleapis.com/skaffold/releases/latest/skaffold-linux-amd64  -O /usr/local/bin/skaffold</pre><p>Make the file executable:</p><pre># chmod +x  /usr/local/bin/skaffold</pre><h4>Installing on macOS</h4><p>Download the package.</p><pre># wget https://storage.googleapis.com/skaffold/releases/latest/skaffold-darwin-amd64  -O /usr/local/bin/skaffold<br># chmod +x /usr/local/bin/skaffold</pre><p>Confirm that the directory with newly added skaffold package is in your PATH environment. You can confirm using:</p><pre># echo $PATH</pre><h3>Demo</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*zlfk6qprxAxMr7RSYYHm-g.jpeg" /></figure><p>Now that you have everything you need to start playing with Skaffold, let’s work on a demo project to help you grasp all the concepts.</p><h4>Create Dockerfile</h4><p>Skaffold needs Dockerfile to be used for building Docker image.</p><pre>$ mkdir ~/skaffold-demo<br>$ cd ~/skaffold-demo<br>$ vim Dockerfile</pre><p>The contents of my Dockerfile are as below:</p><pre>FROM nginx<br><strong>COPY</strong> index.html /usr/share/nginx/html/<br><strong>WORKDIR</strong> /usr/share/nginx/html</pre><p>The content of index.html file:</p><pre>$ cat index.html<br>Hello world - Test Page 1</pre><p>This will pull official Docker image for Nginx, copies <em>index.html</em> file to Nginx default document root, and finally sets working directory to /usr/share/nginx/html.</p><h4>Create Kubernetes deployment</h4><p>Your next step is to create Kubernetes deployment yaml file. The contents of my file is:</p><pre>$ cat k8s-deployment.yaml<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>    name: hello-world<br>spec:<br>    selector:<br>      matchLabels:<br>        app: hello-world<br>    template:<br>      metadata:<br>        labels:<br>            app: hello-world<br>      spec:<br>        containers:<br>        - name: hello-world<br>          image: nginx-skaffold</pre><h4>Create Kubernetes Service</h4><pre>$ cat k8s-service.yaml <br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: hello-world<br>spec:<br>  selector:<br>    app: hello-world<br>  ports:<br>  - port: 8080<br>    protocol: TCP<br>    targetPort: 80<br>  type: LoadBalancer</pre><h4>Create Skaffold configuration</h4><p>The Skaffold configuration file is where the workflow is configured. In this file you define the tools you will be using and their configuration. Skaffold configuration file used in this demo has the following content.</p><pre>$ cat skaffold.yaml <br>apiVersion: skaffold/v1alpha2<br>kind: Config<br>build:<br>  artifacts:<br>  - imageName: nginx-skaffold<br>    workspace: .<br>  local: {}<br><br>deploy:<br>  kubectl:<br>    manifests:<br>      - k8s-*</pre><p>From the file definitions, it can be seen that Skaffold workflow has three main phases.</p><p>1. <strong>Build</strong></p><p>In the build phase, Skaffold builds an artifact using the tool of your choice. Artifacts are Docker images that you would like to deploy into your Kubernetes cluster when changes are made to them.</p><p>2. <strong>Push</strong></p><p>In the push phase, Skaffold ensures that your image is uploaded to the registry referenced in the image name. Before running Skaffold you should ensure that you are able to push to your configured image registry.</p><p>When using local Kubernetes clusters like Minikube or Docker for Desktop, the push is skipped because the image is already available locally.</p><p>3. <strong>Deploy</strong></p><p>The deploy step ensures that the most recent artifacts are running in the cluster. You can use different tools for deployment, for example kubectl or helm. Each deployment type has parameters that allow you to define how you want your app to be installed and updated.</p><p>You have to run skaffold dev command to initiate skaffold deployment on your local Kubernetes environment.</p><pre>$ skaffold dev</pre><p>You should receive an output like below:</p><pre>$  skaffold dev          <br>Starting build...<br>Found [minikube] context, using local docker daemon.<br>Sending build context to Docker daemon  8.704kBStep 1/4 : FROM nginx ---&gt; c5c4e8fa2cf7<br>Step 2/4 : COPY index.html /usr/share/nginx/html/index.html<br> ---&gt; Using cache<br> ---&gt; b2743e4ff68a<br>Step 3/4 : WORKDIR /usr/share/nginx/html<br> ---&gt; Using cache<br> ---&gt; 4ee8631f078b<br>Step 4/4 : EXPOSE 80<br> ---&gt; Using cache<br> ---&gt; 51a516564d41<br>Successfully built 51a516564d41<br>Successfully tagged 87de02f02ad14494edd89e60779062fd:latest<br>Successfully tagged nginx-skaffold:51a516564d41fc9a39d714abcba1464ee301c5feb9de6ab858bc047a74ad3a79<br>Build complete in 231.654336ms<br>Starting deploy...<br>Deploying k8s-deployment.yaml...<br>Deploying k8s-service.yaml...<br>Deploy complete in 685.787798ms<br>Watching for changes...</pre><p>To access your running service, you need to run:</p><pre>$ minikube service [service-name]</pre><p>In our example, this should be:</p><pre>$ minikube service hello-world</pre><p>This opens nginx page on your default browser and displays the content that was added to index.html file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MG2EcZs3XHJtDwlg" /></figure><p>Whenever you make any changes to the index.html file, Skaffold should update your deployed application continually:</p><pre>$ cat index.html<br><br>Hello world - Test Page 2</pre><p>Upon saving the changes, you should see Skaffold doing redeployment automatically</p><pre>Starting build...<br>Found [minikube] context, using local docker daemon.<br>Building [nginx-skaffold]...<br>Sending build context to Docker daemon  3.072kB<br>Step 1/3 : FROM nginx<br> ---&gt; 06144b287844<br>Step 2/3 : COPY index.html /usr/share/nginx/html/<br> ---&gt; fd751a98ac09<br>Step 3/3 : WORKDIR /usr/share/nginx/html<br> ---&gt; 0a4009d6de20<br>Successfully built 0a4009d6de20<br>Successfully tagged f15dd87578a0fb660a2e66cc3da0cf96:latest<br>WARN[1350] Using digest instead of git <strong>commit</strong>: Running [git rev-<strong>parse</strong> --short HEAD]: stdout , stderr: fatal: not a git repository (or any of the parent directories): .git<br>, err: <strong>exit</strong> <strong>status</strong> 128: <strong>exit</strong> <strong>status</strong> 128 <br><strong>Build</strong> <strong>complete</strong> <strong>in</strong> 1.012115486s<br><strong>Starting</strong> deploy...<br>deployment.apps/hello-world configured<br>service/hello-world unchanged<br>Deploy <strong>complete</strong> <strong>in</strong> 1.073736335s<br>Watching <strong>for</strong> changes...</pre><p>Reload the webpage to see new changes</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8ibWXglo1enEg55r" /></figure><h3>Conclusion</h3><p>Skaffold is a command line tool designed to facilitate continuous development of Kubernetes applications. By using Skaffold, it is easy to iterate on your application source code locally and later deploy to a remote or local Kubernetes clusters.</p><p>Leverage this tool to improve your productivity by ensuring you have quick automated deployments of Kubernetes applications. You can read more about Skaffold on its <a href="https://skaffold.dev/">website</a> or <a href="https://github.com/GoogleContainerTools/skaffold">Github repository</a></p><figure><a href="http://stakater.com/?utm_medium=blog&amp;utm_source=easy-kubernetes-development-with-google-skaffold-cbddb7e40519"><img alt="" src="https://cdn-images-1.medium.com/max/700/1*0FANFMBaw8Q8aLuPQJLOOA.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cbddb7e40519" width="1" height="1" alt=""><hr><p><a href="https://medium.com/stakater/easy-kubernetes-development-with-google-skaffold-cbddb7e40519">Easy Kubernetes Development with Google Skaffold</a> was originally published in <a href="https://medium.com/stakater">Stakater</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Auto-cleanup resources on Kubernetes and Openshift]]></title>
            <link>https://medium.com/stakater/auto-cleanup-resources-on-kubernetes-and-openshift-bbf5c5e2ae61?source=rss----c05e1bafc3be---4</link>
            <guid isPermaLink="false">https://medium.com/p/bbf5c5e2ae61</guid>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[namespaces]]></category>
            <category><![CDATA[cleanup]]></category>
            <category><![CDATA[openshift]]></category>
            <dc:creator><![CDATA[Ahmad Iqbal Ali]]></dc:creator>
            <pubDate>Mon, 29 Apr 2019 11:37:42 GMT</pubDate>
            <atom:updated>2019-05-07T08:12:23.982Z</atom:updated>
            <content:encoded><![CDATA[<p>Dangling resources in a deployment environment such as Kubernetes or Openshift can be a problem. It adds to clutter and may also consume resources unnecessarily. This is especially a problem when there are multiple administrators/developers that are managing the system, and the knowledge of the system is to some extent distributed between them. No one person may be able to definitively say why a particular empty or even non-empty namespace was created in the first place or why it has not been deleted since. This could be a temporary project created on a dev Openshift cluster for prototyping and testing a new feature, or it could be a temporary kubernetes namespace on a staging cluster to demo an application to a particular audience. Either way the scenario indicates that the namespace will be temporary, and all deployments being performed within that will not be required after a particular time. Following the principles of GitOps one can indeed perform all such system changes such as namespace creation and deployments, via Git functions, which inherently provides traceability of these changes. Hence one may be able to trace the creation of such a resource back to the intent as may be described in the commit message, or directly inquire from the committer about it’s purpose and validity. However such an effort to trace the timeline of these events from the git history, can quickly become a complex feat leading one away from more important tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/734/1*s0AtC1auZF1ssMfWc2pDgA.png" /></figure><h3>Jamadar</h3><p>At Stakater we love to think of ways where we can automate processes and tasks that take away the tedious workload and help. For this particular scenario we have created <a href="https://github.com/stakater/Jamadar">Jamadar</a>, a tool to help clean up dangling resources in Kubernetes and Openshift. <a href="https://github.com/stakater/Jamadar">Jamadar</a> is a Kubernetes controller that watches kubernetes resources at a configured time interval, and identifies dangling resources based on the age of the resource, and configuration provided in the annotations. Jamadar will help us execute scenarios such as deleting namespaces that are one month old. The resources that we would like to persist, and should not be recycled by Jamadar should have a particular annotation with value. Any resources without the annotation will be recycled by Jamadar based on the configured time interval. Additionally there are other actions that can be configured to be performed once the resource clean-up has taken place.</p><h3>Configuring</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/555/1*dteCw7nUSgSS_Yo3YtfZag.jpeg" /></figure><p>Jamadar can be configured with the following options at deployment time, in the configuration yaml in the ConfigMap.</p><pre><strong>pollTimeInterval</strong>: The time interval after which the controller will poll and look for dangling resources, The value can be in “ms”, “s”, “m”, “h” or even combined like “2h45m”<br></pre><pre><strong>age</strong>: The minimum age of the dangling resource that should be considered for clean-up. The value unit can be any one of “d”, “w”, “m”, “y”. e.g. “7d” for 7 days<br></pre><pre><strong>resources</strong>: The resources that you want to be taken care of by Jamadar. At the moment only supports `namespaces`<br></pre><pre><strong>actions</strong>: The Array of actions that you want to take. At the moment only supports `default` and `slack`<br></pre><pre><strong>restrictedNamespaces</strong>: The list of namespaces names that should be ignored</pre><p>Following is an example configuration in full:</p><pre>pollTimeInterval: 12h  # Values: &quot;ms&quot;, &quot;s&quot;, &quot;m&quot;, &quot;h&quot;.<br>age: 1m                # Values: &quot;d&quot; or &quot;w&quot; or &quot;m&quot; or &quot;y&quot;.<br>resources:<br>  - namespaces<br>actions:<br>  - name: default<br>  - name: slack<br>    params:<br>      token: &lt;token&gt;<br>      channel: &lt;channel-name&gt;<br>restrictedNamespaces: <br>  - kube-system<br>  - default<br>  - kube-public<br>  - prod<br>  - tools</pre><p><strong>Supported Resources</strong></p><p>At the moment Jamadar supports only namespaces in the resources that can be configured for clean-up. This also corresponds to projects in Openshift.</p><p>We will be adding support for other Resources as well in the future.</p><p><strong>Supported Actions</strong></p><p>At the moment Jamdar supports the following Actions with their parameters,</p><ul><li>Default: No parameters needed, it will just log to console the details.</li><li>Slack: you need to provide token and Channel Name as Parameters in the yaml file</li></ul><p>We will be adding support for other Actions as well in the future</p><h3>Deploying Jamadar</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/1*kx3oITzEx3fuhsbzXw8BVQ.jpeg" /></figure><p>The kubernetes deployment files are included in the Jamadar repository in the deployments folder. The deployment files include options to deploy using a single yaml manifest, multiple yaml manifests one for each resource type, or Helm chart. The configuration should be edited in the ConfigMap and deployment can be made as per one’s preferred method.</p><p>For more information you can visit the <a href="https://github.com/stakater/Jamadar">Jamadar repository on Github</a>.</p><figure><a href="http://stakater.com/?utm_medium=blog&amp;utm_source=auto-cleanup-resources-on-kubernetes-and-openshift-bbf5c5e2ae61"><img alt="" src="https://cdn-images-1.medium.com/max/700/1*0FANFMBaw8Q8aLuPQJLOOA.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bbf5c5e2ae61" width="1" height="1" alt=""><hr><p><a href="https://medium.com/stakater/auto-cleanup-resources-on-kubernetes-and-openshift-bbf5c5e2ae61">Auto-cleanup resources on Kubernetes and Openshift</a> was originally published in <a href="https://medium.com/stakater">Stakater</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[K8s: Deployments vs StatefulSets vs DaemonSets]]></title>
            <link>https://medium.com/stakater/k8s-deployments-vs-statefulsets-vs-daemonsets-60582f0c62d4?source=rss----c05e1bafc3be---4</link>
            <guid isPermaLink="false">https://medium.com/p/60582f0c62d4</guid>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[statefulsets]]></category>
            <category><![CDATA[pods]]></category>
            <dc:creator><![CDATA[Ali Kahoot]]></dc:creator>
            <pubDate>Tue, 23 Apr 2019 07:39:22 GMT</pubDate>
            <atom:updated>2021-06-09T09:56:42.821Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*dSL05n34oOKHwFMJJHETRw.jpeg" /><figcaption>Containers Ship Source: <a href="https://pixabay.com/photos/container-ship-container-ship-537724/">https://pixabay.com/photos/container-ship-container-ship-537724/</a></figcaption></figure><p>I have explained this blog in detail with live example on <a href="https://www.youtube.com/watch?v=7Tnm9hRyoYU">Data on Kubernetes Community Meetup 49 Deployments vs StatefulSets vs Daemonsets</a>.</p><p>Kubernetes (K8s) is an open-source container orchestration system for automating deployment, scaling, and management of containerized applications.</p><p>Kubernetes provides a basic resource called Pod. A pod is the smallest deployable unit in Kubernetes which is actually a wrapper around containers. A pod can have one or more containers and you can pass different configuration to the container(s) using the pod’s configuration e.g. passing environment variables, mounting volumes, having health checks, etc. For more details about pods, check <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod/">Pod</a>.</p><p>In this post, I will be discussing three different ways to deploy your application(pods) on Kubernetes using different Kubernetes resources. Below are 3 different resources that Kubernetes provides for deploying pods.</p><ol><li>Deployments</li><li>StatefulSets</li><li>DaemonSets</li></ol><p>There is one other type ReplicationController but Kubernetes now favors Deployments as Deployments configure ReplicaSets to support replication.</p><p>For detailed differences between the 3 resources, I will be deploying a <a href="https://github.com/kahootali/counter-app">sample counter app</a>, which logs and increments the count from a counter file like 1,2,3,…. I am using the counter file from a Persistent Volume to detail the differences between the Deployments, StatefulSets and DaemonSets. The manifests files to deploy the following resources can be found in the <a href="https://github.com/kahootali/counter-app/tree/master/kubernetes">counter app</a>.</p><h3>Deployments</h3><p>Deployment is the easiest and most used resource for deploying your application. It is a Kubernetes controller that matches the current state of your cluster to the desired state mentioned in the Deployment manifest. e.g. If you create a deployment with 1 replica, it will check that the desired state of ReplicaSet is 1 and current state is 0, so it will create a ReplicaSet, which will further create the pod. If you create a deployment with name <strong>counter</strong>, it will create a ReplicaSet with name <strong>counter-&lt;replica-set-id&gt;</strong>, which will further create a Pod with name <strong>counter-&lt;replica-set-&gt;-&lt;pod-id&gt;.</strong></p><p>Deployments are usually used for stateless applications. However, you can save the state of deployment by attaching a Persistent Volume to it and make it stateful, but all the pods of a deployment will be sharing the same Volume and data across all of them will be same.</p><p>For deploying the <a href="https://github.com/kahootali/counter-app">sample counter app</a> using a deployment, we will be using the following manifest, you can deploy it by copying the below manifest and saving it in a file e.g. deployment.yaml, and then applying by</p><p>kubectl apply -f deployment.yaml</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/670900cf1a442b2a10b3255fb9644153/href">https://medium.com/media/670900cf1a442b2a10b3255fb9644153/href</a></iframe><p>If you deploy the above deployment, and see the logs of the pod, you will be able to see the log in order like 1,2,3,…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Cv6hmDbeWQKZzSEEnL37hg.png" /><figcaption>The logs from the 1st pod, Note the name of pod, counter-c9d778cf7-jd9bw,<strong> counter</strong> — deployment name, <strong>c9d778cf7</strong> — ReplicaSet id, <strong>jd9b2</strong> — Pod id</figcaption></figure><figure><img alt="Persistence in Deployments" src="https://cdn-images-1.medium.com/max/1024/1*0WtBPfINnpP6b4Q_cnog0A.png" /></figure><p>Now if you scale the deployment to 2 by running</p><pre>kubectl scale deployment counter --replicas=2</pre><p>you can see a new pod created, if you check the logs of the new pod, its logs will not start from 1 rather it will start from the last number of the 1st pod.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RLUVkPAliz_rAAU70R504g.png" /><figcaption>The logs from the 2nd pod, Note the name of pod, <strong>counter-c9d778cf7-jd9bw. </strong>Deployment and replicaset id are same only pod id is different than previous pod.</figcaption></figure><p>If you see the logs, they are starting from 73, meaning that the previous pod had written till 72 in the file and they both are sharing the same file and volume and data is shared across all pods of a Deployment. Also if you check the Persistent Volume Claims(PVCs), only one PVC will be created that both the pods will be sharing so it can cause Data Inconsistency.</p><figure><img alt="Persistence for replicas in Deployments" src="https://cdn-images-1.medium.com/max/1024/1*m1BoqSmhxGeGHmeexJ_mZA.png" /><figcaption>Persistence for Deployments sharing single Volume can cause Data Inconsistency</figcaption></figure><p>Deployments, as discussed, creates a ReplicaSet which then creates a Pod so whenever you update the deployment using RollingUpdate(default) strategy, a new ReplicaSet is created and the Deployment moves the Pods from the old ReplicaSet to the new one at a controlled rate. Rolling Update means that the previous ReplicaSet doesn’t scale to 0 unless the new ReplicaSet is up &amp; running ensuring 100% uptime. If an error occurs while updating, the new ReplicaSet will never be in <strong>Ready</strong> state, so old ReplicaSet will not terminate again ensuring 100% uptime in case of a failed update. In Deployments, you can also manually roll back to a previous ReplicaSet, if needed in case if your new feature is not working as expected.</p><h3>StatefulSets</h3><p>StatefulSet(stable-GA in k8s v1.9) is a Kubernetes resource used to manage stateful applications. It manages the deployment and scaling of a set of Pods, and provides guarantee about the ordering and uniqueness of these Pods.</p><p>StatefulSet is also a Controller but unlike Deployments, it doesn’t create ReplicaSet rather itself creates the Pod with a unique naming convention. e.g. If you create a StatefulSet with name <strong>counter, </strong>it will create a pod with name <strong>counter-0, </strong>and for multiple replicas of a statefulset, their names will increment like <strong>counter-0</strong>, <strong>counter-1</strong>, <strong>counter-2, etc</strong></p><p>Every replica of a stateful set will have its own state, and each of the pods will be creating its own PVC(Persistent Volume Claim). So a statefulset with 3 replicas will create 3 pods, each having its own Volume, so total 3 PVCs.</p><p>For deploying the <a href="https://github.com/kahootali/counter-app">sample counter app</a> using a statefulset, we will be using the following manifest. you can deploy it by copying the below manifest and saving it in a file e.g. statefulset.yaml, and then applying by</p><p>kubectl apply -f statefulset.yaml</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f888c9481ae228c5040157faf09955be/href">https://medium.com/media/f888c9481ae228c5040157faf09955be/href</a></iframe><p>If you deploy the above statefulset, and see the logs of the pod, you will be able to see the log in order like 1,2,3,…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yQtGqDw54yitDPgT7-O7qQ.png" /><figcaption>The logs from the 1st pod. Note the name of the pod is <strong>counter-0</strong></figcaption></figure><p>Here, you can see the logs start from 1. Now if we scale up the statefulset to 3 replicas by run</p><pre>kubectl scale statefulsets counter --replicas=3</pre><p>it will first create a new pod <strong>counter-1, </strong>and once that pod is ready, then another pod <strong>counter-2. </strong>The new pods will have their own Volume and if you check the logs, the count will again start from 1 for the new pods, unlike in Deployments as we saw earlier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nM-hkzuUqrh4ulBcJmCFyA.png" /><figcaption>The logs from 2nd pod. Note the name is <strong>counter-1.</strong></figcaption></figure><p>Here, the logs are again starting from 1, as this pod has its own Volume, so it doesn’t read the file of 1st pod. And if we see the Persistent Volume Claims,their will be 3 claims created as we had scaled the replicas to 3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*307Wn2ioKOYwA3guEvZycw.png" /><figcaption>PVC</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t0TyiZMRyJCPOGg2XphzwA.png" /><figcaption>Persistence for StatefulSets each having its own Volume</figcaption></figure><p>StatefulSets don’t create ReplicaSet or anything of that sort, so you cant rollback a StatefulSet to a previous version. You can only delete or scale up/down the Statefulset. If you update a StatefulSet, it also performs RollingUpdate i.e. one replica pod will go down and the updated pod will come up, then the next replica pod will go down in same manner e.g. If I change the image of the above StatefulSet, the <strong>counter-2 </strong>will terminate and once it terminates completely, then <strong>counter-2</strong> will be recreated and <strong>counter-1</strong> will be terminated at the same time, similarly for next replica i.e. counter-0. If an error occurs while updating, so only <strong>counter-2</strong> will be down, counter-1 &amp; counter-0 will still be up, running on previous stable version. Unlike Deployments, you cannot roll back to any previous version of a StatefulSet.</p><p>StatefulSets are useful in case of Databases especially when we need Highly Available Databases in production as we create a cluster of Database replicas with one being the primary replica and others being the secondary replicas. The primary will be responsible for read/write operations and secondary for read only operations and they will be syncing data with the primary one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0b7yevdOXKmluG-ElZ2H6Q.png" /><figcaption>Using StatefulSets to provision Postgres as Highly Available Database</figcaption></figure><p>If the primary goes down, any of the secondary replica will become primary and the StatefulSet controller will create a new replica in account of the one that went down, which will now become a secondary replica.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dh5O8yRAbdXHjmVnu9IfzA.png" /><figcaption>In case if postgres-0 went down, now postgres-1 became the primary replica</figcaption></figure><h3>DaemonSet</h3><p>A DaemonSet is a controller that ensures that the pod runs on all the nodes of the cluster. If a node is added/removed from a cluster, DaemonSet automatically adds/deletes the pod.</p><p>Some typical use cases of a DaemonSet is to run cluster level applications like:</p><ul><li><strong>Monitoring Exporters</strong>: You would want to monitor all the nodes of your cluster so you will need to run a monitor on all the nodes of the cluster like NodeExporter.</li><li><strong>Logs Collection Daemon:</strong> You would want to export logs from all nodes so you would need a DaemonSet of log collector like Fluentd to export logs from all your nodes.</li></ul><p>However, Daemonset automatically doesn’t run on nodes which have a taint e.g. Master. You will have to specify the tolerations for it on the pod.</p><p>Taints are a way of telling the nodes to repel the pods i.e. no pods will be schedule on this node unless the pod tolerates the node with the same toleration. The master node is already tainted by:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b113486ca4b409d5fb269f46f207f6e3/href">https://medium.com/media/b113486ca4b409d5fb269f46f207f6e3/href</a></iframe><p>Which means it will repel all pods that do not tolerate this taint, so for daemonset to run on all nodes, you would have to add following tolerations on DaemonSet</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9dd0c3294a1333718cea8180c701ffeb/href">https://medium.com/media/9dd0c3294a1333718cea8180c701ffeb/href</a></iframe><p>which means that it should tolerate all nodes.</p><p>For deploying the <a href="https://github.com/kahootali/counter-app">sample counter app</a> using a daemonset, we will be using the following manifest. you can deploy it by copying the below manifest and saving it in a file e.g. daemonset.yaml, and then applying by</p><p>kubectl apply -f daemonset.yaml</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/62feaeedc58e798c5624da26b3b7aa8c/href">https://medium.com/media/62feaeedc58e798c5624da26b3b7aa8c/href</a></iframe><p>When you deploy the daemonset, it will create pods equal to the number of nodes. In terms of behavior, it will behave the same as Deployments i.e. all pods will share the same Persistent Volume.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-4H5HgNp5hOIwasX5-nYWg.png" /><figcaption>Logs of a pod of DaemonSet</figcaption></figure><p>These are the logs of a pod of DaemonSet, you can see the logs are not in order, meaning that all pods are sharing the same Volume. Also only one PVC will be created that all pods will be sharing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xrAwDE0tFymV62BiAwDyGw.png" /><figcaption>Similar to ReplicaSet, but DaemonSets run one replica per node in the cluster</figcaption></figure><p>If you update a DaemonSet, it also performs RollingUpdate i.e. one pod will go down and the updated pod will come up, then the next replica pod will go down in same manner e.g. If I change the image of the above DaemonSet, one pod will go down, and when it comes back up with the updated image, only then the next pod will terminate and so on. If an error occurs while updating, so only one pod will be down, all other pods will still be up, running on previous stable version. Unlike Deployments, you cannot roll back your DaemonSet to a previous version.</p><p>That’s all, these are the main resources to deploy your applications (containers) on Kubernetes.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=60582f0c62d4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/stakater/k8s-deployments-vs-statefulsets-vs-daemonsets-60582f0c62d4">K8s: Deployments vs StatefulSets vs DaemonSets</a> was originally published in <a href="https://medium.com/stakater">Stakater</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stakater Forecastle: Automated Service dashboard for Kubernetes]]></title>
            <link>https://medium.com/stakater/stakater-forecastle-automated-service-dashboard-for-kubernetes-9ff78d728048?source=rss----c05e1bafc3be---4</link>
            <guid isPermaLink="false">https://medium.com/p/9ff78d728048</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[ingress]]></category>
            <category><![CDATA[dashboard]]></category>
            <dc:creator><![CDATA[Ahmad Iqbal Ali]]></dc:creator>
            <pubDate>Thu, 18 Apr 2019 13:00:21 GMT</pubDate>
            <atom:updated>2019-04-18T13:01:28.274Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*JVKqC5q4ulZxEN9GgpNJkQ.png" /></figure><p>In an environment where we have multiple tools and applications including micro-frontends, it may get a little tedious to keep track of the URLs of all of the services. It can get additionally confusing if there are multiple instances of the same tools deployed on different environments or kubernetes namespaces. Keeping track of these URLs and sharing them within a team or other group of consumers can become tiresome. We would like to have a central place where we can easily look for and access our applications running on Kubernetes. We would also like to have our apps on kubernetes to be dynamically discovered. For an easier way to manage this, the Stakater team has developed a Kubernetes controller, Forecastle.</p><p>Forecastle is a Kubernetes controller and a web dashboard. It watches Ingresses and looks for specific annotations on them to indicate whether the Ingress needs to be registered on the dashboard or not. If the relevant configuration is discovered in the annotations of the ingress, the Ingress URL is added to the Forecastle dashboard. The dashboard displays all the running applications that are registered with it, and provides a simple way to access them.</p><p>Following is an example of the Forecastle dashboard. The applications are automatically separated into sections based on the kubernetes namespace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/689/0*E7V5X6t4LgL32taC" /></figure><p>The dashboard page also has a search bar which can be used to filter the application list to find a particular application without having to scroll through the whole list.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/0*UT3DJvh9MAvu9Pzk" /></figure><h3>How it works</h3><h4>Configuration</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*oQ0zip4KX5SYSBExKWQzDA.jpeg" /></figure><p>Forecastle watches for specific annotations on ingresses. To make an app/ingress be discoverable and added to the Forecastle dashboard, the forecastle.stakater.com/expose annotation with true value should be added to the ingress. There are other annotations which can be optionally used to customize how the application is displayed in the dashboard, but without the expose annotation mentioned they will be ignored.</p><p>With forecastle.stakater.com/icon we can specify the URL of an image that should be used for the application button on the dashboard.</p><p>forecastle.stakater.com/appName can be used to specify a custom display name for the application on the dashboard. If this annotation isn’t specified, the name of the ingress is used by default.</p><p>A group/section name can be specified using forecastle.stakater.com/group. This is the section on the dashboard page under which the application will be listed. If the annotation is not used, the application will be displayed under the section titled after its namespace by default.</p><p>When an Ingress with the required annotations is created in the cluster, the ingress is added to the Forecastle dashboard. Likewise when the ingress is removed from the cluster, it’s link button is removed from the dashboard.</p><h4>Authentication</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*9LFeiRzoZMQhY0RN91xj0Q.jpeg" /></figure><p>While the Forecastle dashboard does not have a login mechanism built-in or support yet for integration with OAuth providers, etc., at Stakater we do use Keycloak single sign-on with Keycloak gatekeeper to protect the Forecastle dashboard. Keycloak gatekeeper redirects any unauthenticated requests to the Keycloak server, and once an authenticated session is established it forwards the requests to the target Forecastle container.</p><p>It is therefore recommended to use some external authentication mechanism to protect access to the Forecastle dashboard.</p><h4>Running multiple instances</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*RaLtDSVWgkciEJRmw5o8GA.jpeg" /></figure><p>If needed, we can run multiple instances of forecastle by deploying multiple instances and configure the specific list of namespaces that Forecastle should watch for ingresses. The list of namespaces to watch is configured in the ConfigMap of the deployment.</p><pre>data:<br>  config.yaml: |-<br>    namespaces:<br>    - qa-1<br>    - qa-2</pre><p>We can also have multiple Forecastle instances work within the same namespace and watch ingresses in the same namespace, but have applications register to only a selected number of the dashboards based on the instance name of the Forecastle deployment. The instanceName parameter is configured in the Forecastle deployment ConfigMap as follows:</p><pre>data:<br>  config.yaml: |-<br>    instanceName: dev-forecastle</pre><p>On the Ingress, this configuration will be specified using the forecastle.stakater.com/instance annotation. The annotation value can be a comma separated list of the forecastle instances where we want to register the ingress. This way we can have the application display on a single or multiple dashboards based on the value we specify in the instance annotation.</p><p>Having multiple instances of Forecastle can be useful from an authorization point-of-view. For example if we want dev team members to only have access to applications deployed for the dev team, they can have authorization to use the relevant Forecastle dashboard only. Similarly the QA team can have access to only a QA relevant Forecastle dashboard. And it’s possible there may be some tools for which we are using a single instance across different environments, e.g. if we have a single instance of Jenkins working for both the dev and qa environments. In such a case we can have Jenkins be displayed on both instances of Forecastle dashboard.</p><figure><a href="http://stakater.com/?utm_medium=blog&amp;utm_source=stakater-forecastle-automated-service-dashboard-for-kubernetes-9ff78d728048"><img alt="" src="https://cdn-images-1.medium.com/max/700/1*0FANFMBaw8Q8aLuPQJLOOA.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9ff78d728048" width="1" height="1" alt=""><hr><p><a href="https://medium.com/stakater/stakater-forecastle-automated-service-dashboard-for-kubernetes-9ff78d728048">Stakater Forecastle: Automated Service dashboard for Kubernetes</a> was originally published in <a href="https://medium.com/stakater">Stakater</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Alerting in Kubernetes]]></title>
            <link>https://medium.com/stakater/alerting-in-kubernetes-5df357c9b036?source=rss----c05e1bafc3be---4</link>
            <guid isPermaLink="false">https://medium.com/p/5df357c9b036</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[alerting]]></category>
            <dc:creator><![CDATA[Ahmad Iqbal Ali]]></dc:creator>
            <pubDate>Tue, 09 Apr 2019 13:18:06 GMT</pubDate>
            <atom:updated>2019-04-09T13:19:17.949Z</atom:updated>
            <content:encoded><![CDATA[<p>Alerting is one of the pillars of observability in DevOps, and closely related to monitoring and logging. Even though monitoring and logging provide a way to actively view and get insight on the state of the system, any person cannot remain glued to the screen to keep a look out for an error level log statement, low memory issue or other troublesome event. We use alerting to let us know of events of interest, i.e. events that indicate issues or potential issues, as and when they occur.</p><p>The concept of alerting is simple. The system is continuously monitored, and when a predefined threshold of some metric is crossed, an alert is sent out to pre-configured recipients. Thresholds can be set to not only indicate the occurrence of an issue with the system, but rather also proactively indicate that the system might be headed towards an issue. Alerting on such thresholds may give the stakeholders adequate indication and time to monitor the system more closely and even start some mitigation activities that will lower the probability of issue occurrence. The exact thresholds for alerting will no doubt have to be thought out, as having low thresholds that are reached frequently but have little impact, will lead to very noisy alerting; i.e. We will receive frequent alerts for events that do not require immediate attention. On the other hand if the thresholds are too high, we may not receive alerts until the issue has already occurred, or it is too late to mitigate in any way. The monitoring data will of course all be there in the required fine-grained detail to allow for effective root cause analysis, but the alerts should only be triggered within the sweet-spot of the threshold range that actually warrant attention from human stakeholders.</p><p>At Stakater we try to use open-source software throughout our deployment environment such as with Kubernetes because of the relevant benefits. This is also the case for our monitoring and alerting, for which we use a Prometheus based stack.</p><h3>Prometheus monitoring</h3><p>Let’s take a quick look at how we can use prometheus for monitoring on a Kubernetes cluster.</p><p>A convenient way of deploying Prometheus on Kubernetes is by using the <a href="https://github.com/coreos/prometheus-operator">Prometheus Operator</a>. With Prometheus there are a few components involved at the center of which we have the Prometheus Server. Prometheus server essentially scrapes metric data that other services expose. Each Kubernetes node exposes certain Services like Node Exporter &amp; Kubelet, which contain system level metrics. Node Exporter collects OS level metrics of a node through Docker host and Kubelet contains cadvisor which collects container metrics from the Docker Engine. For Kubernetes monitoring, Prometheus scrapes metrics from each Kubelet and Node Exporter from all nodes.</p><p>There may be some services such as ephemeral and batch jobs that Prometheus server cannot reliably scrape because of their ephemeral nature. For such a case we have the Prometheus Pushgateway which is able to have such jobs or services push their metrics to it, and in turn Pushgateway exposes these metrics to Prometheus for scraping.</p><p>For visualization we have Grafana which queries Prometheus and groups the results and displays it in Dashboards.</p><h3>Prometheus Alert Manager</h3><p>For alerting, Prometheus Server triggers alerts on Prometheus AlertManager based on the rules defined within AlertManager. The alert trigger sends notifications through a desired notification channel.</p><p>Prometheus Operator creates/configures/manages Prometheus atop Kubernetes and makes Kubernetes native Prometheus configuration in the form of Customer Resource Definitions (CRDs). Two of these CRDs are for PrometheusRule and AlertManager.</p><p>We can define multiple alerting rules using PrometheusRule which are actually Prometheus queries. When the rule query is satisfied, it fires the alert to AlertManager. Each rule can have labels. AlertManager has Routes, which can be defined using these labels and each Route can have multiple Receivers, and each receiver can send the notification to a specific app like Slack or email. We can also set a time period during which a rule is satisfied, for the alert to be triggered, e.g. we want the alert to trigger if Kubelet is down for 2 minutes.</p><p>Following is an example configuration which shows the configuration of 2 alerts based on the same metric, i.e. the number of responsive kubelets. The threshold for the percentage of kubelets however is different and are therefore labelled differently.</p><pre>- alert: K8SKubeletDown<br>  expr: count(up{job=&quot;kubelet&quot;} == 0) / count(up{job=&quot;kubelet&quot;}) &gt; 0.03<br>  for: 5m<br>  labels:<br>    severity: warning<br>    kind: infra<br>  annotations:<br>    description: Prometheus failed to scrape {{ $value }}% of kubelets.<br>    summary: Many Kubelets cannot be scraped<br>- alert: K8SKubeletDown<br>  expr: absent(up{job=&quot;kubelet&quot;} == 1) or count(up{job=&quot;kubelet&quot;} == 0) / count(up{job=&quot;kubelet&quot;}) &gt; 0.1<br>  for: 5m<br>  labels:<br>    severity: critical<br>    kind: infra<br>  annotations:<br>    description: Prometheus failed to scrape {{ $value }}% of kubelets, or all Kubelets have disappeared from service discovery.<br>    summary: Many Kubelets cannot be scraped</pre><p>In the first alert configured above, if the percentage of unresponsive kubelets stays above 3% for 5 minutes, an alert of severity Warning is triggered. In second alert, if the percentage of unresponsive kubelets stays above 10% for 5 minutes, an alert of severity Error is triggered.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/539/0*geVciNjn1E5vrtdT" /><figcaption>Example: Slack notification for pod restarting</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/517/0*X0Hfr0zk08jih58x" /><figcaption>Example: Slack notification for pod memory</figcaption></figure><h3>Uptime checker alerts</h3><p>Apart from Prometheus which provides us monitoring from within our kubernetes cluster, we also employ third party cloud services for uptime monitoring such as Uptime Robot. Such a service allows monitoring of application connection from multiple locations across the globe, and additionally alert in case the application is not reachable. Services like Uptime Robot also provide alerts using multiple notification channels such as slack, email, etc.</p><p>For the purpose of easy automation of configuring monitors and alerts with the Uptime checker, we have developed an open-source Kubernetes controller, <a href="https://github.com/stakater/IngressMonitorController">IngressMonitorController</a>. The IngressMonitorController automatically adds/removes monitors and alerts against ingresses in the uptime checker such as UptimeRobot, Pingdom, Statuscake, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/623/1*KQxL4qBcrtN7_9eejY6jOg.png" /><figcaption>Example: Slack notifications from UptimeRobot</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NesYWMOV9HFX1R-g" /><figcaption>Alerting components</figcaption></figure><figure><a href="http://stakater.com/?utm_medium=blog&amp;utm_source=alerting-in-kubernetes-5df357c9b036"><img alt="" src="https://cdn-images-1.medium.com/max/700/1*0FANFMBaw8Q8aLuPQJLOOA.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5df357c9b036" width="1" height="1" alt=""><hr><p><a href="https://medium.com/stakater/alerting-in-kubernetes-5df357c9b036">Alerting in Kubernetes</a> was originally published in <a href="https://medium.com/stakater">Stakater</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Simple Kubernetes alerting with Stakater Chowkidar]]></title>
            <link>https://medium.com/stakater/simple-kubernetes-alerting-with-stakater-chowkidar-dec95d8eb287?source=rss----c05e1bafc3be---4</link>
            <guid isPermaLink="false">https://medium.com/p/dec95d8eb287</guid>
            <category><![CDATA[slack]]></category>
            <category><![CDATA[alerting]]></category>
            <category><![CDATA[resources]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Ahmad Iqbal Ali]]></dc:creator>
            <pubDate>Sun, 31 Mar 2019 19:33:35 GMT</pubDate>
            <atom:updated>2019-05-08T10:58:15.351Z</atom:updated>
            <content:encoded><![CDATA[<h3>Custom Kubernetes Events alerting with Stakater Chowkidar</h3><p>Alerting is one of the components of DevOps that allows stakeholders to easily observe important system events. Alerting is indispensable for modern DevOps teams due to the needs of continuous deployment of multiple applications composed of microservices. Alerting helps with reliability by informing us about the system at important states. That way we have confidence by having insight into the exact state of the system, and knowing when there are any issues that might be occurring, or could be imminent. Alerting systems can be configured to notify at the occurrence of specific system events (e.g. service down), or when thresholds of certain metrics are reached (e.g. 80% memory consumed). And notifications of such events can be connected various channels such as email, Instant Message, or even Text message on phones.</p><p>In this blog we will be exploring Stakater Chowkidar, an open-source Kubernetes controller which can be used for some specific alerting needs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*1qO21qPaRR6sB4t3Tm6vqQ.png" /></figure><h3>Stakater Chowkidar</h3><p>Chowkidar is an Urdu word for watchman, and generically Stakater Chowkidar is implemented to have configuration for certain actions to be performed on certain events in the kubernetes cluster. One of the current event-actions that Chowkidar supports and that we are going to discuss today is the notification on slack at the creation of a Pod without a resource request and limit defined.</p><p>Let us first take a look at what resource limit and requests are for containers and pods.</p><h4>Scenario background</h4><p><strong>Container Resource limits</strong></p><p>By default a container has no resource limits and can use as much of a resource as the host allows. The container can however be configured to set a limit to how much memory or CPU a container can use.</p><p><strong>Memory</strong></p><p>It is good practice to keep a limit on how much host memory a running container can consume. On linux, when there is a shortage of memory to perform important system functions the kernel kills processes to free up memory. Any process can be killed for this purpose, including Docker. Docker does try to adjust the priority of the docker daemon with respect to the out of memory exception however it does not do the same for containers. An individual container may therefore likely be killed in such an event..</p><p>In docker we can force limits on the memory consumption of containers. A hard limit can be set to disallow the container to use no more than a given amount of memory. A soft limit allows the container to use as much memory as needed as long as the system does not have other memory consumption requirements. The following options can be used with the docker run command to enforce these memory limits:</p><p>--memory This is the maximum memory the container can use, i.e. the hard limit</p><p>--memory-reservation This is the memory limit imposed on the container when docker detects low memory on the host machine, i.e. the soft limit</p><p><strong>CPU</strong></p><p>A container by default has unlimited access to the host’s CPU. You can set various constraints to limit a given container’s access to the host machine’s CPU. Some of the docker run flags for configuring the amount of CPU resources the container has are as follows:</p><p>--cpus This is the limit of the number of CPUs the container is allowed to use, i.e. a hard limit</p><p>--cpu-shares This is the container’s weight for CPU access and is considered only when CPU cycles are constrained. So in a way this is a soft limit.</p><h3>Container resources in Kubernetes</h3><p>The docker configuration options that we reviewed above can be specified in Kubernetes on the containers in pods. Each Container of a Pod can specify one or more of the following configuration:</p><ul><li>spec.containers[].resources.limits.cpu</li><li>spec.containers[].resources.limits.memory</li><li>spec.containers[].resources.requests.cpu</li><li>spec.containers[].resources.requests.memory</li></ul><p>When the kubelet starts a Container of a Pod, it passes the CPU and memory limits to the container runtime. In the case of docker these configuration map directly or indirectly as follows:</p><ul><li>spec.containers[].resources.requests.cpu → --cpu-shares</li><li>spec.containers[].resources.limits.memory → --memory</li></ul><h3>Implementing Chowkidar</h3><p>Specifying requests and limits for memory/cpu in the container deployments is a good practice since otherwise there is a possibility of having a container eat up too many resources with not enough left for more important containers/processes, or in extreme cases getting important containers/processes killed off by the OS to relieve stress on the constrained resources. We would like to implement this best practice in all our deployments, and in case any container without this best practice is deployed, we would like to have this container be brought to our notice.</p><p>Chowkidar will watch all the Pods being deployed, and if there is any without a resource request/limit specified, we will configure to be notified via slack.</p><p>Chowkidar allows you to have multiple controllers that will continuously watch types in all the namespaces and automatically perform any actions given in the yaml file. With this, you can easily check for any criteria on your Pods/other types and take corresponding actions.</p><p>We will specify the following configuration on Chowkidar:</p><p>type: The type of Resource you want to monitor, i.e. Pods</p><p>watchcriterion: The criterion for which you want to take actions for the controller i.e. resourceExists. resourceExists will make sure Pods should have Resources and if not found, it will take action.</p><p>actions: The Array of actions that you want to take, i.e. slack. For Slack you need to provide token and Channel Name as Parameters in the yaml file</p><p>Following are the yaml files for deploying Chowkidar with our required configuration:</p><pre>---<br># Source: chowkidar/templates/deployment.yaml<br>apiVersion: extensions/v1beta1<br>kind: Deployment<br>metadata:<br>  annotations:<br>    configmap.reloader.stakater.com/reload: chowkidar<br>  labels:<br>    app: chowkidar<br>    group: com.stakater.platform<br>    provider: stakater<br>    version: v0.0.31<br>    chart: &quot;chowkidar-v0.0.31&quot;<br>    release: &quot;RELEASE-NAME&quot;<br>    heritage: &quot;Tiller&quot;<br>  name: chowkidar<br>spec:<br>  replicas: 1<br>  revisionHistoryLimit: 2<br>  selector:<br>    matchLabels:<br>      app: chowkidar<br>      group: com.stakater.platform<br>      provider: stakater<br>  template:<br>    metadata:<br>      labels:<br>        app: chowkidar<br>        group: com.stakater.platform<br>        provider: stakater<br>    spec:<br>      containers:<br>      - env:<br>        - name: KUBERNETES_NAMESPACE<br>          valueFrom:<br>            fieldRef:<br>              fieldPath: metadata.namespace<br>        - name: CONFIG_FILE_PATH<br>          value: /configs/config.yaml<br>        image: &quot;stakater/chowkidar:v0.0.31&quot;<br>        imagePullPolicy: IfNotPresent<br>        name: chowkidar<br>        volumeMounts:<br>        - mountPath: /configs<br>          name: config-volume<br>      serviceAccountName: chowkidar<br>      volumes:<br>      - configMap:<br>          name: chowkidar<br>        name: config-volume</pre><pre>---<br># Source: chowkidar/templates/configmap.yaml<br>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  labels:<br>    app: chowkidar<br>    group: com.stakater.platform<br>    provider: stakater<br>    version: v0.0.31<br>    chart: &quot;chowkidar-v0.0.31&quot;<br>    release: &quot;RELEASE-NAME&quot;<br>    heritage: &quot;Tiller&quot;<br>  name: chowkidar<br>data:<br>  config.yaml: |-<br>    controllers:<br>    - type: pods<br>      watchcriterion:<br>        operator: and<br>        identifiers:<br>        - resourceExists<br>      actions:<br>      - name: slack<br>        params:<br>          channel: channel-name<br>          token: your-token</pre><p>We will also need to provide Chowkidar with the required access control:</p><pre>---<br># Source: chowkidar/templates/rbac.yaml<br>apiVersion: v1<br>kind: ServiceAccount<br>metadata:<br>  labels:<br>    app: chowkidar<br>    group: com.stakater.platform<br>    provider: stakater<br>    version: v0.0.31<br>    chart: &quot;chowkidar-v0.0.31&quot;<br>    release: &quot;RELEASE-NAME&quot;<br>    heritage: &quot;Tiller&quot;<br>  name: chowkidar<br>---<br>apiVersion: rbac.authorization.k8s.io/v1beta1<br>kind: ClusterRole<br>metadata:<br>  labels: <br>    app: chowkidar<br>    group: com.stakater.platform<br>    provider: stakater<br>    version: v0.0.31<br>    chart: &quot;chowkidar-v0.0.31&quot;<br>    release: &quot;RELEASE-NAME&quot;<br>    heritage: &quot;Tiller&quot;<br>  name: chowkidar-role<br>rules:<br>  - apiGroups:<br>      - &quot;&quot;<br>    resources:      <br>      - pods<br>    verbs:<br>      - list<br>      - get<br>      - watch<br>---<br>apiVersion: rbac.authorization.k8s.io/v1beta1<br>kind: ClusterRoleBinding<br>metadata:<br>  labels: <br>    app: chowkidar<br>    group: com.stakater.platform<br>    provider: stakater<br>    version: v0.0.31<br>    chart: &quot;chowkidar-v0.0.31&quot;<br>    release: &quot;RELEASE-NAME&quot;<br>    heritage: &quot;Tiller&quot;<br>  name: chowkidar-role-binding<br>roleRef:<br>  apiGroup: rbac.authorization.k8s.io<br>  kind: ClusterRole<br>  name: chowkidar-role<br>subjects:<br>  - kind: ServiceAccount<br>    name: chowkidar<br>    namespace: default</pre><figure><a href="http://stakater.com/?utm_medium=blog&amp;utm_source=simple-kubernetes-alerting-with-stakater-chowkidar-dec95d8eb287"><img alt="" src="https://cdn-images-1.medium.com/max/700/1*0FANFMBaw8Q8aLuPQJLOOA.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dec95d8eb287" width="1" height="1" alt=""><hr><p><a href="https://medium.com/stakater/simple-kubernetes-alerting-with-stakater-chowkidar-dec95d8eb287">Simple Kubernetes alerting with Stakater Chowkidar</a> was originally published in <a href="https://medium.com/stakater">Stakater</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>