<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sergey Nuzhdin on Medium]]></title>
        <description><![CDATA[Stories by Sergey Nuzhdin on Medium]]></description>
        <link>https://medium.com/@SergeyNuzhdin?source=rss-5c5da0e5f3bb------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*bQVKrurN9ZSpE_9X.jpg</url>
            <title>Stories by Sergey Nuzhdin on Medium</title>
            <link>https://medium.com/@SergeyNuzhdin?source=rss-5c5da0e5f3bb------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 21:10:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@SergeyNuzhdin/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Agola — One CI To Run Them All]]></title>
            <link>https://medium.com/@SergeyNuzhdin/agola-one-ci-to-run-them-all-f2f50c4e3d78?source=rss-5c5da0e5f3bb------2</link>
            <guid isPermaLink="false">https://medium.com/p/f2f50c4e3d78</guid>
            <category><![CDATA[cicd]]></category>
            <category><![CDATA[gitlab]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[github]]></category>
            <dc:creator><![CDATA[Sergey Nuzhdin]]></dc:creator>
            <pubDate>Thu, 30 Jan 2020 00:00:00 GMT</pubDate>
            <atom:updated>2020-01-31T16:31:02.628Z</atom:updated>
            <content:encoded><![CDATA[<h3>Agola — One CI To Run Them All</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/631/0*m0KAxQpYopiZ7za0.png" /></figure><p>I’ve been looking for a good Continues Integration(CI) system for quite some time now. System, that could work with all the major repository providers and run on my hardware. The need for multi-provider support comes from the fact that my code is spread across:</p><ul><li>Github</li><li>Gitlab, cloud and <a href="https://blog.lwolf.org/post/fully-automated-gitlab-installation-on-kubernetes-including-runner-and-registry/">self-hosted</a></li><li>Gitea — I’m currently replacing my self-hosted Gitlab with it. I find it much simpler and faster.</li></ul><p>This means I can neither have a single view of all my tests nor have a unified way of testing. Of course, one could pay for some cloud-based CI/CD system and solve the problem. In my case, it doesn’t make any sense to pay for cloud solutions. I have more than enough computing power in my home-lab to run any CI. The code that I need to test is either open-source or for personal projects. There are OSS solutions out there that could do the work, but I didn’t find a single one that could tick all my boxes:</p><ul><li><a href="https://gitlab.com/">Gitlab</a> supports integration with different external providers, but only on a paid plan</li><li><a href="https://drone.io/">Drone</a> only works with the single integration at a time, so you need to run one instance per provider.</li><li><a href="https://jenkins-x.io/">Jenkins</a> — not sure if it can, but I’d rather not go this way.</li></ul><p>That’s why I got excited when I’ve found out about the release of <a href="https://agola.io/">Agola</a>.</p><h3>Overview</h3><p>Agola is a new CI system developed by <a href="https://github.com/sgotti">@sgotti</a>, the guy who built <a href="https://blog.lwolf.org/post/how-to-deploy-ha-postgressql-cluster-on-kubernetes/">stolon</a> — cloud-native HA PostgreSQL manager. I like the approach he used building stolon. He took great software — PostgreSQL and made it cloud-native. Not by stockpiling a bunch of bash scripts and voodoo magic like others, but by writing proper code.</p><p>Apart from the author, a few of the main selling points for me were:</p><ul><li><strong>Multiple git sources integrations</strong> — a single CI for all my repositories.</li><li><strong>User direct runs</strong> — ability to run CI tests against your local code without committing it to the server.</li><li><strong>Advanced workflows</strong> — like matrix builds, fan-in, fan-out, multi-arch.</li><li><strong>Golang and OSS</strong> — should be very resource-efficient and easy to contribute</li></ul><p>You can find a full list of features in this introductory <a href="https://sgotti.dev/post/agola-introduction/">blog post</a>.</p><h3>Installation</h3><p>This is my second attempt trying to get it up and running. The first one happened a few weeks after the initial release — v0.1, and I couldn’t make it work. I didn’t have enough time to dig into details, so I decided to give it another try later, This time, with version 0.4, I managed to get it up and running and using it for a month now.</p><p>Installation could be a bit tricky because information about it is split between the <a href="https://agola.io/tryit/">try-it</a> section and actual <a href="https://agola.io/doc/">Documentation</a>. If you, like me, never use those everything-in-one-container setups and skip the try-it section, you can spend a lot of time trying to figure out why your <a href="https://github.com/agola-io/agola/issues/72">login/register buttons do not work</a> and you can’t log in.</p><p>To get Agola up and running in a Kubernetes cluster, assuming you do not want to use <a href="https://github.com/agola-io/agola/tree/master/examples/kubernetes/simple">all-in-one demo container</a>, you need to:</p><ol><li>Deploy <a href="https://github.com/etcd-io/etcd">Etcd</a> if you don’t have one already.</li><li>Deploy <a href="https://github.com/helm/charts/tree/master/stable/minio">MinIO</a> or any other S3-compatible object storage.</li><li>Deploy Agola from the example in the repo</li></ol><p>I’m not going to cover how to install etcd, because it varies wildly based on how resilient the system should be, or where you&#39;re running it.</p><h3>Adding Providers</h3><p>After you’ve managed to install the system, you need to add at least one repository provider. Otherwise, you’ll get an empty page trying to login or register. Agola does not have it’s own authentication mechanisms and relies on the providers. Which makes perfect sense, you need to connect at least one repository provider to use it anyways.</p><p>The workflow of adding a repository provider is the same. You need to get ClientID and Client Secret and use Agola CLI to add it. Below are the steps needed to get those ClientID and Secret for different providers.</p><h3>Github</h3><p>On Github you need to go to settings/applications to add an OAuth application:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/614/0*ioMSt5e1FYC3fujb.png" /></figure><h3>Gitlab</h3><p>In Gitlab you need to open the profile/applications section. You can also configure the level of permissions you want to give to the new application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gykJarLlMDK7A4x4.png" /></figure><h3>Gitea</h3><p>To create a new application in Gitea, go to /user/settings/applications page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GVrLX2KJ-TM4vFWO.png" /></figure><p>Step by step guide on how to add oauth2 provider could be found in gitea <a href="https://docs.gitea.io/en-us/oauth2-provider/">documentation page</a></p><h3>Agola side</h3><p>To add a new provider you need to exec to the agola&#39;s gateway pod and use Agola&#39;s CLI. You also need $ADMIN_TOKEN, which you set during the installation in the <a href="https://github.com/agola-io/agola/blob/master/examples/kubernetes/distributed/agola.yml#L88">configuration YAML</a></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e4f66b26e1b3f955c4094e8ac10f633a/href">https://medium.com/media/e4f66b26e1b3f955c4094e8ac10f633a/href</a></iframe><p>Now you can register and login using any of those providers in the web UI.</p><h3>Direct runs</h3><p>Direct runs is a unique feature of the Agola and so far my favorite. Direct runs allow you to run the CI pipeline without pushing the code to the remote. So, you don&#39;t have to create tens of the commits to the repo labeled testing the CI. You can test your local code in the CI environment, using a simple command:</p><pre>agola --gateway-url=https://agola.yourhost.address --token=&lt;YOUR_TOKEN&gt; directrun start</pre><p>You can create a token from the user/settings page in the Agola Web UI. Under the hood the directrun start command will commit everything into a temporary git branch and push it to an internal git repository, then it&#39;ll run build as if it were normal.</p><h3>Secrets and variables</h3><p>Agola, like any other CI system, allows you to add secrets and variables to your jobs. It has some useful features, like group-&gt;project <strong>inheritance</strong> or dynamic <strong>per-branch</strong> secrets, but I’d say the implementation is not very polished. For example, I find the way how secrets are currently implemented very inconvenient. The only way to create one is by using Agola CLI, which means you can’t have these things under version control.</p><h3>Wrap up</h3><p>I only use it in a few of my projects now, but I plan to migrate all personal repositories to it. So far I like it a lot and I hope it’ll get more traction among the community.</p><p><em>Originally published at </em><a href="https://blog.lwolf.org/post/agola-one-ci-to-run-them-all/"><em>https://blog.lwolf.org</em></a><em> on January 30, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f2f50c4e3d78" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Switching to Istio as the primary ingress]]></title>
            <link>https://medium.com/@SergeyNuzhdin/switching-to-istio-as-the-primary-ingress-cb7a4f95efe0?source=rss-5c5da0e5f3bb------2</link>
            <guid isPermaLink="false">https://medium.com/p/cb7a4f95efe0</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[istio]]></category>
            <dc:creator><![CDATA[Sergey Nuzhdin]]></dc:creator>
            <pubDate>Mon, 08 Apr 2019 00:00:00 GMT</pubDate>
            <atom:updated>2019-04-08T18:24:58.976Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GESn4jCuqR3_oJJmch-ltw.png" /></figure><p>I’ve been following the news about <a href="https://istio.io/">istio</a> since it’s first alpha release in 2017. I think this project has a great future, because it solves a lot of pain points in the microservice based architecture, like auth, observability, fault-injection, etc.</p><p>But the fact is, I never actually tried it. Prior to v1.0 there were too many bugs and limitations to put it into production. So, when they finally released “production-ready” version I got quite exciting. It still took me a few months to find free time, but I finally gave it a try.​​</p><p>I decided to start my experimentations with Istio not from the usual bookstore example, but by making it the main ingress controller, responsible for all the traffic, incoming to my cluster. Previously, I used ingress-nginx, more precisely I had 2 ingress-controllers: one exposed to the Internet and one for internal-only services. This post is about replacing both of them with Istio.</p><h3>Install</h3><blockquote><em>This post is referencing Istio version </em><a href="https://github.com/istio/istio/releases/tag/1.1.0-rc.0"><em>1.1.0-rc.0</em></a><em>, unless explicitly stated otherwise</em></blockquote><p>It is extremely easy and straight forward to install istio using helm chart, at least if you’re not trying to understand what all of the configuration options mean. I started by copying the default values.yaml file and enabling all the components (ingress, grafana, servicegraph, tracing, kiali, istiocoredns, istio_cni).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/546d7344a9e6b9f63f660ac35cf28cbc/href">https://medium.com/media/546d7344a9e6b9f63f660ac35cf28cbc/href</a></iframe><p>Assuming there were no problems during the installation, you now have a new namespace filled with containers.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c18b62dc8243a87b39d55b4290fcf4b2/href">https://medium.com/media/c18b62dc8243a87b39d55b4290fcf4b2/href</a></iframe><p>and of course a lot of new services</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7d11fc45c78a1262282408bb61721521/href">https://medium.com/media/7d11fc45c78a1262282408bb61721521/href</a></iframe><blockquote><em>Services with the type LoadBalancer got their External IPs from the </em><a href="https://github.com/google/metallb"><em>metallb</em></a><em> load balancer set up in the cluster.</em></blockquote><p>some of the questions one may have at this point are:</p><ul><li>What’s the difference between <strong>istio-ingress</strong> and <strong>istio-ingressgateway</strong>, which one should you set in your DNS?</li><li>How to access default istio services e.g kiali, grafana, prometheus?</li><li>How to expose something to the outside world?</li></ul><h3>Ingressing traffic: istio-ingress and istio-ingressgateway</h3><p>The Ingress controller is, basically, a reverse-proxy that runs in a cluster and configures routing rules according to Ingress resources. Istio provides two ways of ingressing traffic into your cluster.</p><p>First one, <strong>istio-ingress</strong>, is a traditional ingress controller like nginx-ingress, traefik or controur. This controller runs in your cluster and listens to all the changes to Ingress resources from the Kubernetes API and sends incoming traffic according to these rules. You can easily swap one ingress controller to another by just changing the annotation on your ingress objects:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ed585f80a8076f8699faa0df7b5f5745/href">https://medium.com/media/ed585f80a8076f8699faa0df7b5f5745/href</a></iframe><p>Istio documentation discourages use of this method as a “legacy way” and suggests using the second one.</p><p>The second one, <strong>istio-ingressgateway,</strong> is also an ingress controller, but unlike traditional ones, it does not rely on native Kubernetes Ingress objects. It uses its own custom resources: Gateway, VirtualService, DestinationRule, etc. Using custom resources provides more flexibility in configuring network policies, but also makes you write a few more YAMLs for each service. In this post, I’m going to use the second one.</p><h3>Exposing services using Istio-ingressgateway</h3><p>To expose a service using ingressgateway you have to create at least 2 objects — Gateway and a VirtualService. As an example, let’s expose the internal istio services. Assuming that we have domain <strong>example.com</strong> we need to create a following gateway:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/40972c5f23a75aeefefcf5b0c0caac3e/href">https://medium.com/media/40972c5f23a75aeefefcf5b0c0caac3e/href</a></iframe><p>Here we create a rule for the ingressgateway with the selector <strong>istio: ingressgateway</strong> to accept HTTP traffic on port 80 if SNI host matches the host in the list.</p><p>Few things to keep in mind:</p><ul><li>the selector is used by istio to select the ingressgateway. Keep this in mind if you have multiple ingressgateways</li><li>Better to <a href="https://github.com/istio/istio/issues/11509">avoid</a> using *<strong>.domain</strong> in hosts, unless you have only a single gateway.</li><li>port naming is important, it is used in some routing logic <a href="https://istio.io/help/faq/traffic-management/">inside the istio</a>.</li></ul><blockquote><em>The port names must be of the form </em><em>protocol-</em><em>suffix with </em><em>grpc, </em><em>http, </em><em>http2, </em><em>https, </em><em>mongo, </em><em>redis, </em><em>tcp, </em><em>tls or </em><em>udp as the </em><em>protocol in order to take advantage of Istio’s routing features.</em></blockquote><blockquote><em>For example, </em><em>name: http2-foo or </em><em>name: http are valid port names, but </em><em>name: http2foo is not. If the port name does not begin with a recognized prefix or if the port is unnamed, traffic on the port will be treated as plain TCP traffic (unless the port explicitly uses Protocol: UDP to signify a UDP port).</em></blockquote><p>Next, we need to create a VirtualService for each of the services we want to expose. Here is a one for the Grafana:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8a58857fd5570b616d6224302c6e010b/href">https://medium.com/media/8a58857fd5570b616d6224302c6e010b/href</a></iframe><p>Things to keep in mind here:</p><ul><li>If VirtualService and Gateway are located in the different namespaces, make sure to set gateway in the format of <strong>gateway-name.namespace,</strong> otherwise, you’ll be getting 404.</li></ul><h3>Istio and HTTPS</h3><p>Exposing services to the world is cool, basics are working. But in 2019 everything should be exposed using HTTPS. With Istio exposing site using HTTPS is in fact much more complicated and limited comparably with other ingresses. Let’s start with exposing our service using HTTPS and then I’ll list all the issues and limitations I faced with possible workarounds.</p><p>To update our gateway to listen to HTTPS traffic we need to have a valid certificate. I’m a big fan of <a href="https://letsencrypt.org/">LetsEncrypt</a> and <a href="https://github.com/jetstack/cert-manager">cert-manager</a>, so I’m going to use it here. You can install cert-manager as part of istio chart or you can use a dedicated chart for that. I already have cert-manager installed, so I will just add another certificate to the cluster.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cfcac04d6eb0f94216b254ed3a4d7e10/href">https://medium.com/media/cfcac04d6eb0f94216b254ed3a4d7e10/href</a></iframe><p>Next step is to add this certificate to the igressgateway deployment.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4d40c784651379d7f719cf458f695f7c/href">https://medium.com/media/4d40c784651379d7f719cf458f695f7c/href</a></iframe><p>Now, we can update our gateway configuration to use HTTPS:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c865fedddbcfe33a2c6921902ff5b2f2/href">https://medium.com/media/c865fedddbcfe33a2c6921902ff5b2f2/href</a></iframe><p>If everything went well, we can now access <a href="https://grafana.com/">Grafana</a> using HTTPS with a valid certificate from the <a href="https://letsencrypt.org/">LetsEncrypt</a>.</p><h3>Issues</h3><p>Now, let’s talk about the issues I have encountered during that short period of time I spent working with istio.</p><ul><li>[FIXED in 1.1] Manual update of the certificates. This is probably the most annoying thing. Having to update the deployment for each certificate, and restart ingressgateway to pick up it is not cool. This was finally fixed in 1.1 release. It is now possible to tell istio to use Kubernetes secret. <a href="https://istio.io/docs/tasks/traffic-management/secure-ingress/sds/">Detailed how-to is here</a>.</li></ul><ul><li><a href="https://github.com/istio/istio/issues/9030">Istio Ingress integration with cert-manager · Issue #9030 · istio/istio</a></li><li><a href="https://github.com/istio/istio/pull/11496">Use credentialName to specify credential resource name and support mTLS for external cert management at ingress gateway. by JimmyCYJ · Pull Request #11496 · istio/istio</a></li></ul><ul><li>[FIXED in 1.1] Another issue popped up when I tried to use the same port 443 for both PASSTHROUGH mode and SSL-termination, which is a perfectly reasonable scenario.</li></ul><ul><li><a href="https://discuss.istio.io/t/tls-modes-passthrough-and-simple/852">TLS modes PASSTHROUGH and SIMPLE</a></li><li><a href="https://github.com/istio/istio/issues/11509">IngressGateway only respects first port 443 Gateway definition · Issue #11509 · istio/istio</a></li></ul><ul><li>[FIXED in <a href="https://github.com/kiali/kiali/releases/tag/v0.16.2">Kiali v0.16.2</a>] I spent some time trying to figure out the correct way of specifying Gateway in the VirtualService. At some point, I thought that you can’t have Gateway and VirtualService in different namespaces. What made it even more confusing is <a href="https://www.kiali.io/">Kiali</a>. Whenever I tried to point VirtualService to the gateway in the different namespace I saw a validation error “virtualservice is pointing to non-existing gateway”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/0*qcYGxVziOp5NE_2l.png" /></figure><p>This issue is fixed in the <a href="https://github.com/kiali/kiali/releases/tag/v0.16.2">Kiali v0.16.2</a>, which is not yet updated in istio chart.</p><h3>Conclusion</h3><p>Istio is still a very young project. Reaching the 1.0 version was a huge step, but it still requires a lot of work. I only touched the ingress part of it and found that it lacks some of the basic functionality that one would expect from the Ingress controller.<br> On the other hand, it is in a very active development phase. All the issues that I had in 1.0.* version was addressed and fixed in 1.1.0 release. Another nice piece of software that comes with the Istio is Kiali, it looks really great and it should be very useful when I start using istio as a service mesh, not just an ingress.</p><p><em>Originally published at </em><a href="https://blog.lwolf.org/post/switching_to_istio_as_the_primary_ingress/"><em>blog.lwolf.org</em></a><em> on April 8, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cb7a4f95efe0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to deploy multi-arch Kubernetes cluster using Kubespray]]></title>
            <link>https://medium.com/@SergeyNuzhdin/how-to-deploy-multi-arch-kubernetes-cluster-using-kubespray-d394874fd4fc?source=rss-5c5da0e5f3bb------2</link>
            <guid isPermaLink="false">https://medium.com/p/d394874fd4fc</guid>
            <category><![CDATA[multiarch]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Sergey Nuzhdin]]></dc:creator>
            <pubDate>Sat, 02 Feb 2019 00:00:00 GMT</pubDate>
            <atom:updated>2019-02-13T18:36:08.013Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*20Tpl_a8swX5hjnDxQtNnA.png" /></figure><p>I recently bought 3 <a href="https://www.hardkernel.com/shop/odroid-hc1-home-cloud-one/">ODROID-HC1</a> devices to add a dedicated storage cluster to my home Kubernetes. I thought that it’s a good excuse to spend some time redeploying the cluster. Usually, I would’ve gone with CoreOS, since I’m a big fan of their immutable OS. Unfortunately, that is not an option if you have ARM nodes. So I had to choose between manual provisioning and Ansible. I chose Ansible.</p><p>I knew about <a href="https://github.com/kubernetes-incubator/kubespray">Kubespray</a>, but still, I decided to spend some time looking for alternatives. I found a few other projects: some targeted towards ARMs, some with a “simplified” approach. In the end, I decided to go with Kubespray. Partly, because it is a part of official k8s tooling, partly because they started to use kubeadm under the hood.</p><p>Kubespray is the oldest project aimed to automate Kubernetes cluster provisioning. There are a lot of good manuals on how to deploy a simple k8s cluster using Kubespray, like <a href="https://dzone.com/articles/kubespray-10-simple-steps-for-installing-a-product">this</a> or <a href="https://medium.com/@iamalokpatra/deploy-a-kubernetes-cluster-using-kubespray-9b1287c740ab">this</a>. So, I’m not going to repeat any of this here. Instead, I’ll try to highlight the steps required to make things work in a multi-arch cluster.</p><h3>Multi-arch deployment</h3><p>Kubernetes’s support for non-amd64 platforms improved dramatically during the last 2 years. Thanks to the popularity of cheap ARM boards like RaspberryPi and the community. Kubespray does not yet work with multi-arch setup out of the box, but it’s getting there with every release. As it turned out it’s not very difficult to make it work. In general, the work could be split into a few categories:</p><ul><li>change some Kubespray configs where architecture is still hardcoded.</li><li>choose overlay network that works with different architectures and setup it</li><li>configures arch-specific software to be scheduled to corresponding nodes using <strong>nodeSelector</strong></li></ul><h3>Fine-tuning settings</h3><p>When I started experimenting with Kubespray and multi-arch deployment, around 2 months ago, I had to change a lot of files. Lots of URLs for binaries and docker images had a hardcoded amd64 part. Luckily, at the moment of writing only a few such places left in the repo.</p><ol><li>One of such hardcoded arch is still present in the latest 2.8 release, but it’s already fixed in master. So we need to change it.</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3117cf996f39bd10883cb7ccce5d149d/href">https://medium.com/media/3117cf996f39bd10883cb7ccce5d149d/href</a></iframe><p>Previously I had to do such changes all over the place.</p><ol><li>Another thing that you have to deal with is the current implementation of checksums map. Kubespray was not meant to run in a hybrid environment, so it assumes that a binary could only have a single checksum. That is not a case in a multi-arch environment, where you have one binary per architecture.</li><li>It might change in the future. For example, a simple map could be replaced with some sort of multileveled map. But for now, the easiest way is to comment out sha256 checksums for the binaries you’re gonna use.</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/252c2e7b7cde9851852296541d006c90/href">https://medium.com/media/252c2e7b7cde9851852296541d006c90/href</a></iframe><ol><li>The last thing left in this section is to extend the architecture group with ARM variable.</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/60ae40505a42d2b69be4ab896c09653d/href">https://medium.com/media/60ae40505a42d2b69be4ab896c09653d/href</a></iframe><h3>Overlay network</h3><p>The main choice one has to make is what overlay network to use. Here are the main options I considered:</p><ul><li><strong><em>cilium</em></strong> was my first choice. I heard a lot of great stuff about it and for a long time, I want to try it on a real system. Unfortunately, it <a href="https://github.com/cilium/cilium/issues/1104">does not support ARM yet</a>.</li><li><strong>calico</strong> — is another popular solution which I didn’t try yet. But, again, as far as I can tell, it still does not support multi-arch containers, but they are <a href="https://github.com/projectcalico/calico/issues/1865">working on it</a>. [<strong>UPDATE</strong>: the ticket is closed now, and you can use it on amd64/arm64 and ppc64 architectures.]</li><li><strong><em>flannel</em></strong> — I’ve been using flannel in most of my k8s clusters for a few years now and it turned out that I’m gonna use it for a much longer period. It is the only CNI that supports all the architectures. Well, maybe not all, but all architectures that I would ever need.</li></ul><h3>Flannel</h3><p>Deploying flannel is very straight-forward. Flannel does not yet support multi-arch containers. You need to add a DaemonSet for each architecture you use and limit the nodes using NodeSelector. For example, to make it work on ARM64, all you need to do is copy original manifest and update a few lines.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/668a9bfe052ed885c3e415d5540a2ac3/href">https://medium.com/media/668a9bfe052ed885c3e415d5540a2ac3/href</a></iframe><p>When I added all the changes for the flannel and tried to provision, playbook kept failing with the following error.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b57fe47551fe5918b493c8328a2cf41e/href">https://medium.com/media/b57fe47551fe5918b493c8328a2cf41e/href</a></iframe><p>Looking into failing step, it does not seem to be related to the networking. But in the logs of the several scheduled containers, it’s clear that the problem is in CNI plugins.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/22386074d38b8590207165f7546fcee8/href">https://medium.com/media/22386074d38b8590207165f7546fcee8/href</a></iframe><p>After some googling, I found a few issues in the flannel repo mentioning missing plugins in the flannel 0.10. Especially the absence of the <a href="https://github.com/coreos/flannel/issues/890#issuecomment-349448629">portmap plugin</a>. The workaround for this is either downgrade to flannel 0.9.1 or install <a href="https://github.com/containernetworking/plugins">CNI plugins</a> manually. I decided to install the plugins using Ansible. After this, provisioning succeeded.</p><p>When you’re dealing with a multi-arch setup on daily basis at some point you’ll get tired of gating deployments. I mean for each deployment you create, you need to add NodeSelector so it will be able to run.</p><p>Docker’s manifests are still not very popular, and only a few projects are using. As far as I know, only Docker and GKE registries support them.</p><p><a href="https://github.com/lwolf/docker-multiarch">lwolf/docker-multiarch</a></p><p>So I created a <a href="https://github.com/lwolf/docker-multiarch">repo on GitHub</a> with a bunch of dockerfiles and build steps to build manifest-based containers for the software I use. The build is automatic, it checks releases daily and runs build on new releases. Currently, I have flannel, flannel-cni, kubernetes-dashboard, Prometheus and helm. Thanks to the manifest-based flannel container, I removed my custom one-per-architecture flannel DaemonSet. Now I use an upstream version of it. So, instead of a bunch of DaemonSets, I need to replace a few CNI related variables:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4193e6ea9ddd06ca0502ec0256b39ad9/href">https://medium.com/media/4193e6ea9ddd06ca0502ec0256b39ad9/href</a></iframe><h3>Upgrading K8S versions</h3><p>During the time have this setup, a few versions of Kubernetes were released. So I had a good chance to test upgrades between releases. I started with v1.12 and went through the upgrades up to v1.12.5. In general, everything went well, I didn’t have any problems with the core components. The only thing that was causing provision to fail from time to time was nginx-ingress-controller provided by Kubespray. For some reason, Kubespray is trying to delete it and reinstall each time. I ended up disabling it and other charts. I wasn’t planning on managing helm charts with Kubespray anyways. I find this workflow a bit strange: Ansible variables -&gt; jinja -&gt; helm templates -&gt; helm install.</p><p>After this change, updates became stable.</p><h3>Conclusions</h3><p>It’s great to see that Kubernetes is getting good support for a multi-arch setup. I am very pleased with the number of hacks that I had to do to Kubespray to make it work seamlessly in a hybrid cluster.</p><p>I planned to cover the deployment of GlusterFS in this post as well since it was the reason for this whole deployment. But GlusterFS, as usual, caused a lot of trouble to deploy and deserves a separate post.</p><p><em>Originally published at </em><a href="https://blog.lwolf.org/post/how-to-deploy-multi-arch-kubernetes-cluster-using-kubespray/"><em>blog.lwolf.org</em></a><em> on February 2, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d394874fd4fc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Going open-source in monitoring, part V: Collecting errors from production using Sentry]]></title>
            <link>https://medium.com/@SergeyNuzhdin/going-open-source-in-monitoring-part-v-collecting-errors-from-production-using-sentry-add449d3775a?source=rss-5c5da0e5f3bb------2</link>
            <guid isPermaLink="false">https://medium.com/p/add449d3775a</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[sentry]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[monitoring]]></category>
            <dc:creator><![CDATA[Sergey Nuzhdin]]></dc:creator>
            <pubDate>Wed, 06 Sep 2017 00:00:00 GMT</pubDate>
            <atom:updated>2017-09-06T07:40:31.649Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bkrA1FzPYeS2HzakHp1eig.png" /><figcaption>sentry project overview</figcaption></figure><p>In this part of the series, I’m going to add Sentry to the monitoring stack. For those who do not know, <a href="https://sentry.io/"><strong>Sentry</strong></a> is an open-source cross-platform crash reporting and aggregation platform written in Python. I know about it for several years now, since the time it supported only a few languages. A lot has changed since my last use of it: new domain, new languages, new integrations. Today it has integrations with most of the modern languages and frameworks like Elixir, React, React-Native, Go. Not to mention olds like Java, PHP or Python. The full list of languages is available on <a href="https://github.com/getsentry/sentry#official-sentry-sdks">Github</a> and <a href="https://sentry.io/platforms/">official site</a>.</p><p>This post is one of a series of posts about monitoring of infrastructure and services. Other posts in the series:</p><p><a href="https://medium.com/@SergeyNuzhdin/going-open-source-in-monitoring-part-0-intro-dffa2b40665f"><strong>0. Intro</strong></a></p><ol><li><a href="https://medium.com/p/going-open-source-in-monitoring-part-i-deploying-prometheus-and-grafana-to-kubernetes-e3d44460cec6"><strong>Deploying Prometheus and Grafana to Kubernetes</strong></a></li></ol><p><a href="https://medium.com/@SergeyNuzhdin/going-open-source-in-monitoring-part-ii-creating-the-first-dashboard-in-grafana-ada59a4ced2e"><strong>2. Creating the first dashboard in Grafana</strong></a></p><p><strong>3. </strong><a href="https://medium.com/@SergeyNuzhdin/going-open-source-in-monitoring-part-iii-10-most-useful-grafana-dashboards-to-monitor-kubernetes-7d22ac4645db"><strong>10 most useful Grafana dashboards to monitor Kubernetes and services</strong></a></p><p>4. Configuring alerts in Prometheus and Grafana</p><p>5. <strong>Collecting errors from production using Sentry (this article)</strong></p><p>6. Making sense of logs with ELK stack</p><p>7. Replacing commercial APM monitoring</p><p>8. SLA, SLO, SLI and other useful abstractions</p><h3>Installation on k8s</h3><p>Sentry has several moving parts: server, workers, sql database and Redis. You can find a <a href="https://github.com/kubernetes/charts/tree/master/stable/sentry">good chart</a> in the kubernetes-charts repository which will work for most scenarios. The only missing features for me in this chart were the impossibility of using s3-like storage for persistence out of the box and the ability to use already existing databases. It was not hard to add it, but now I have to maintain my copy of the chart. Aside from this, configuration and installation are straightforward.</p><h3>Sentry integrations</h3><p>As I already mentioned sentry can collect errors from most of modern languages and frameworks. It also integrates with a bunch of applications and services. Some of it is shown in the screenshot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lV2rkBvUVKSH_8pv.png" /></figure><p>In my current <a href="https://librer.io/">project</a> I have:</p><ul><li><strong>Python/Flask</strong> — most of the backend APIs so far</li><li><strong>Python/Scrapy </strong>— the best scraping framework out there</li><li><strong>Golang</strong> for some services</li><li><strong>Universal/Isomorphic React</strong> as frontend</li></ul><p>I’m going to add sentry monitoring to all these systems, also I’m going to add integrations with Gitlab and Slack.</p><h3>Collect reports from Python/Flask ✔</h3><p>Integration with Flask is usually the easiest example of integration. You just add</p><pre>from raven.contrib.flask import Sentry<br>sentry = Sentry(app, dsn=&#39;your dsn&#39;)</pre><p>and you will start receiving all of your tracebacks. But this time I had to spend some extra time trying to debug SSL issues. It took way more time to debug since I was facing another SSL issue with a python-kubernetes client the same time. It turned out to be two separate problems with the same error :)</p><pre>In [1]: from raven import Client<br>In [2]: client = Client(&#39;https://key:secret@sentry.example.com/2&#39;)<br>In [3]: client.captureMessage(&#39;hello, world!&#39;)<br>Out[3]: &#39;4ba5df0f8621406bbbce47a18b987c60&#39;<br>In [4]: Sentry responded with an error: &lt;urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)&gt; (url: https://sentry.example.com/api/2/store/)<br>Traceback (most recent call last):<br>...<br>ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)<br>...<br>urllib.error.URLError: &lt;urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)&gt;</pre><p>In the case with sentry, the <a href="https://community.letsencrypt.org/t/problems-with-sentry-and-letsencrypt/19948/3">problem was</a> because raven (Python client for sentry) does not support <a href="https://en.wikipedia.org/wiki/Server_Name_Indication">SNI</a>.</p><blockquote><strong><em>Server Name Indication (SNI)</em></strong><em> is an extension to the TLS computer networking protocol by which a client indicates which hostname it is attempting to connect to at the start of the handshaking process. This allows a server to present multiple certificates on the same IP address and TCP port number and hence allows multiple secure (HTTPS) websites (or any other Service over TLS) to be served by the same IP address without requiring all those sites to use the same certificate. It is the conceptual equivalent to HTTP/1.1 name-based virtual hosting, but for HTTPS.</em></blockquote><p>Luckily it supports <a href="https://raven.readthedocs.io/en/stable/transports.html">overriding of the transport protocol</a>,</p><pre>from raven.transport.requests import RequestsHTTPTransport<br>SENTRY_CONFIG = {<br>    &#39;dsn&#39;: os.environ.get(&#39;SENTRY_DSN&#39;),<br>    &#39;transport&#39;: RequestsHTTPTransport,<br>}</pre><p>After I found this fix, everything started to work.</p><h3>Collect reports from Python/Scrapy ✔</h3><p>Adding Sentry to Scrapy is similar. Just add a few settings to the config after installing the <a href="https://github.com/llonchj/scrapy-sentry">extension</a></p><pre># sentry dsn<br>SENTRY_DSN = &#39;http://public:secret@example.com/1&#39;<br>EXTENSIONS = {<br>    &quot;scrapy_sentry.extensions.Errors&quot;:10,<br>}</pre><h3>Collect reports from isomorphic/universal react app ✔</h3><p>Integration with client-side JS or with backend-side is easy. Making it work in Universal App is a bit trickier. As a solution for <a href="https://github.com/kriasoft/react-starter-kit">ReactStarterKit based</a> project, I end up with something like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/322/0*0UTH5--HuwuT3MwH.png" /></figure><p>Have a package named raven which imports either raven.client.js or raven.server.js based on environment. And then use it the same way in both server.js and client.js</p><pre># client/server<br>import Raven from &#39;./services/shared/raven&#39;;<br>import { sentry } from &#39;./config&#39;;<br><br>if (sentry.dsn) {<br>  Raven.config(sentry.dsn).install();<br>}</pre><h3>Collect reports from Golang ✔</h3><pre>package main<br><br>import &quot;github.com/getsentry/raven-go&quot;<br><br>func init() {<br>    raven.SetDSN(&quot;https://&lt;key&gt;:&lt;secret&gt;@sentry.io/&lt;project&gt;&quot;)<br>}</pre><h3>Integrations</h3><p>Sentry supports more than 25 integrations with other platforms, for now, I’m going to enable only two: gitlab and slack. In Sentry, you can enable a different set of apps for each project. To do it you need to go the project settings <strong>Open Project → Project Setting</strong> and select <strong>All Integrations</strong> in the side menu. You will get a list of all available integration to select. After saving you will have your integrations listed in the side menu.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/193/0*kk6ifBztZOnsM0SK.png" /></figure><h3>Gitlab</h3><p>After enabling Gitlab you need to configure it. You will need to <a href="https://docs.gitlab.com/ce/user/profile/personal_access_tokens.html">create an <strong>Access Token</strong></a> in Gitlab and enter it in the Configuration page of Sentry.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/995/0*OEiFi3acmdbmmz_H.png" /></figure><p>At the end, you will have a nice Gitlab button on the toolbar. It gives you ability to associate Sentry stacktrace with a new or Existing Gitlab issue in a few clicks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/0*xjd6D104XEZJ3bvh.png" /></figure><h3>Slack</h3><p>Connecting slack is easy as everywhere. First, you need to create new webhook configuration in team settings on Slack: <a href="https://your-team.slack.com/apps/manage/custom-integrations">https://your-team.slack.com/apps/manage/custom-integrations</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/0*o6arf1HopDpSyn3V.png" /></figure><p>You’ll get a webhook URL that you’ll need to enter in Sentry on the Slack settings page. It has few other common settings like selecting a channel to post to, customising tags or bot name.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/501/0*iddcS7SaYNgm366d.png" /></figure><h3>Wrap up</h3><p>Sentry became a very mature product with a lot of supported frameworks and integrations. It is really easy to configure, at least in languages I’ve tried, and use. Excluding the SNI issue, it took me just a few hours to install it in Kubernetes and add it to all my projects. I believe it is the best opensource solution in the field available today.</p><p>Stay tuned.</p><h4>Like this article?</h4><p>Click the 💚 below so other people will see it here on Medium.<br><a href="http://eepurl.com/cdUha1"><strong>Subscribe</strong></a> to get new stories delivered to your inbox or <a href="https://twitter.com/SergeyNuzhdin">follow me on twitter</a>.</p><p><em>Originally published at </em><a href="http://blog.lwolf.org/post/going-open-source-in-monitoring-part-v-collecting-errors-from-production-using-sentry/"><em>blog.lwolf.org</em></a><em> on September 6, 2017.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=add449d3775a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Making long running sitemap generation kubernetes-friendly using k8s-api and Minio]]></title>
            <link>https://medium.com/@SergeyNuzhdin/making-long-running-sitemap-generation-kubernetes-friendly-using-k8s-api-and-minio-a3b803302557?source=rss-5c5da0e5f3bb------2</link>
            <guid isPermaLink="false">https://medium.com/p/a3b803302557</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[minio]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Sergey Nuzhdin]]></dc:creator>
            <pubDate>Sun, 03 Sep 2017 00:00:00 GMT</pubDate>
            <atom:updated>2017-09-04T06:51:20.864Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/687/1*z8z8-3EdDDWLMFLi6SSizQ.png" /></figure><p>Generation of sitemaps is usually not a problem, at least not until you have several millions of items in the database to iterate through. Currently, I have slightly more than 4M documents, so it takes time to regenerate. Previously all my sitemaps were stored on the disk and were served using simple Nginx container. It worked like this:</p><ul><li>Sitemap-related containers have nodeSelector to assign to particular node with hostPath volume</li><li>sitemaps are served from /data/sitemaps folder</li><li>periodic task generates new sitemaps to /data/sitemaps-temp</li><li>on completion, folders is atomically switched</li></ul><p>It was fine in my previous VM-based cluster, which was on a single physical machine. After migration to the new multi-node cluster, I did not want to bind it to the single node to avoid downtime during node upgrade or maintenance. Since I do not have distributed block storage, I decided to serve it from the Minio which I already setup in a distributed fashion.</p><p>I thought that it is also a great opportunity to play with Kubernetes API in Python.</p><p>So the workflow became like this:</p><ul><li>Always have ingress pointing to sitemaps bucket in Minio</li><li>Generate sitemaps inside the container</li><li>Create bucket named sitemaps-</li><li>Upload all of the sitemaps to the new bucket</li><li>Make bucket public</li><li>Switch ingress programmatically to point to the new bucket</li><li>Delete old bucket</li></ul><h3>Ingress</h3><p>Ingress part is pretty usual, except for a few changes that I made to the metadata section. First one is the ingress.kubernetes.io/rewrite-target rule. What it does is replace URL part specified in the rule. For example in my case, the request like <a href="https://librer.io/sitemaps/sitemap-books.xml">https://librer.io/sitemaps/sitemap-books.xml</a> will be sent to Minio service as /sitemaps-20170803/sitemap-books.xml. The second is the label with the bucket name put into labels, for easy filtering.</p><pre>apiVersion: extensions/v1beta1<br>kind: Ingress<br>metadata:<br>  name: libr-static<br>  labels:<br>    app: librerio<br>    role: static<br>    bucket: sitemaps-20170803<br>  annotations:<br>    ingress.kubernetes.io/rewrite-target: /sitemaps-20170803<br>    kubernetes.io/ingress.class: tectonic<br>    kubernetes.io/tls-acme: &#39;true&#39;<br>spec:<br>  tls:<br>  - hosts:<br>    - librer.io<br>    secretName: librerio-tls<br>  rules:<br>  - host: librer.io<br>    http:<br>      paths:<br>      - path: /sitemaps<br>        backend:<br>          serviceName: clustered-minio<br>          servicePort: 9000</pre><h3>Kubernetes API in Python</h3><p>Since most of the backend written in Python, python-client is the usual choice to access the API.</p><p>The first thing we need to do is access Kubernetes API. So we need to install the client.</p><pre>pip install kubernetes</pre><p>Official client is hosted in the <a href="https://github.com/kubernetes-incubator/client-python/">kubernetes-incubator</a> and has a lot of examples. Also, this <a href="https://www.linux.com/learn/kubernetes/enjoy-kubernetes-python">post</a> could be useful if you’re trying k8s API for the first time.</p><p>Anyway, here are the basics.</p><p>To access the API we need to configure the client. For this, we need to import k8s config. There are two separate ways to access the cluster: using kubeconfig or serviceAccounts. First one is used mostly when you’re trying to access the cluster from the outside world, it is the same as kubectl.</p><pre>from kubernetes import config <br>config.load_kube_config()</pre><p>Another one, which you’ll be using more often, is using serviceAccounts from the pod running inside the cluster.</p><pre>from kubernetes import config<br>config.load_incluster_config()</pre><p>These are helper functions to make configuration easier. But, if you need, you can configure the client by setting option manually.</p><p>For testing, you can even run kubectl proxy and access the cluster like this</p><pre>&gt;&gt;&gt; from kubernetes import client,config<br>&gt;&gt;&gt; client.Configuration().host=&quot;http://localhost:8001&quot;</pre><p>Using the client you can access all of the <a href="https://github.com/kubernetes-incubator/client-python/tree/master/kubernetes#documentation-for-api-endpoints">API resources</a> available:</p><pre># for example to get all nodes <br>&gt;&gt;&gt; v1=client.CoreV1Api() <br>&gt;&gt;&gt; v1.list_node() # or to get all namespace names <br>&gt;&gt;&gt; for ns in v1.list_namespace().items:<br>        print(ns.metadata.name)</pre><h3>Back to the real-world</h3><p>For this task we need to read and update ingress object. Make sure to give your serviceAccount corresponding roles/rolebindings in the RBAC-enabled cluster. steps related to k8s API are:</p><ul><li>read the ingress</li><li>extract current bucket name</li><li>update ingress with the new bucket name</li></ul><p>Ingresses are versioned as extensions/v1beta1 so we need to create API instance with the corresponding version.</p><pre>kube_api = client.ExtensionsV1beta1Api()</pre><p>Reading of the ingress by name and namespace could be done like this:</p><pre>ingress = kube_api.read_namespaced_ingress(ingress_name, namespace)</pre><p>If you print this object you’ll see a big JSON, but all we need from it is just the name of the current bucket.</p><pre>old_bucket = ingress.metadata.labels.get(&#39;bucket&#39;)</pre><p>Finally, we need to update the ingress with the information about new bucket name. For this, we’re going to use patch_namespaced_ingress method with a dictionary containing the fields we want to change.</p><pre>def update_bucket_in_ingress(bucket_name):<br>    body = {<br>        &quot;metadata&quot;: {<br>            &quot;labels&quot;: {<br>                &quot;bucket&quot;: bucket_name<br>            },<br>            &quot;annotations&quot;: {<br>                &quot;ingress.kubernetes.io/rewrite-target&quot;: &quot;/{}&quot;.format(bucket_name)<br>            }<br>        },<br>    }<br>    kube_api.patch_namespaced_ingress(ingress_name, namespace, body=body)</pre><p>Here is the complete snippet of the code with Minio and k8s API related code.</p><pre>from kubernetes import config, client<br><br>def create_bucket(minio_client, bucket_name):<br>    try:<br>        minio_client.make_bucket(<br>            bucket_name,<br>            location=Config.OBJECT_STORAGE_S3_BUCKET_LOCATION<br>        )<br>    except BucketAlreadyOwnedByYou as err:<br>        pass<br>    except BucketAlreadyExists as err:<br>        pass<br>    except ResponseError as err:<br>        raise<br><br>def update_bucket_in_ingress(namespace, ingress_name, bucket_name):<br>    body = {<br>        &quot;metadata&quot;: {<br>            &quot;labels&quot;: {<br>                &quot;bucket&quot;: bucket_name<br>            },<br>            &quot;annotations&quot;: {<br>                &quot;ingress.kubernetes.io/rewrite-target&quot;: &quot;/{}&quot;.format(bucket_name)<br>            }<br>        },<br>    }<br>    kube_api.patch_namespaced_ingress(ingress_name, namespace, body=body)<br><br>config.load_incluster_config()<br>kube_api = client.ExtensionsV1beta1Api()<br>namespace = os.environ[&#39;SITEMAPS_INGRESS_NAMESPACE&#39;]<br>ingress_name = os.environ[&#39;SITEMAPS_INGRESS_NAME&#39;]<br><br>##### Steps not related to k8s api<br>minio_client = Minio(host, access_key, secret_key)<br><br>bucket_name = &quot;sitemaps-{}&quot;.format(today)<br>create_bucket(minio_client, bucket_name)<br><br>tmp_store = &#39;/tmp/sitemaps/{}&#39;.format(today)<br>os.makedirs(tmp_store, exist_ok=True)<br><br>log.info(&quot;Going to generate sitemaps&quot;)<br>for model, prefix in SITEMAPS:<br>    log.info(&quot;Generating sitemaps for %s...&quot; % prefix)<br>    generate_items_sitemap(model, tmp_store, prefix)<br>    generate_sitemap_wrapper(tmp_store, prefix)<br><br>log.info(&quot;Sitemaps generating completed. uploading to object storage...&quot;)<br>for fl in glob.glob(os.path.join(tmp_store, &#39;*&#39;)):<br>    minio_client.fput_object(<br>        bucket_name=bucket_name,<br>        object_name=fl.split(&#39;/&#39;)[-1],<br>        file_path=fl,<br>    )<br>minio_client.set_bucket_policy(bucket_name, &quot;*&quot;, Policy.READ_ONLY)<br>###<br><br>ingress = kube_api.read_namespaced_ingress(ingress_name, namespace)<br>old_bucket = ingress.metadata.labels.get(&#39;bucket&#39;)<br>update_bucket_in_ingress(bucket_name)<br><br># Do not delete todays bucket on second run<br>if old_bucket != bucket_name:<br>    try:<br>        minio_client.remove_bucket(old_bucket)<br>    except NoSuchBucket:<br>        log.info(&quot;Unable to delete old bucket, not found&quot;)</pre><h4>Like this article?</h4><p>Click the 💚 below so other people will see it here on Medium.<br><a href="http://eepurl.com/cdUha1"><strong>Subscribe</strong></a> to get new stories delivered to your inbox or <a href="https://twitter.com/SergeyNuzhdin">follow me on twitter</a>.</p><p><em>Originally published at </em><a href="http://blog.lwolf.org/post/making-long-running-sitemap-generation-kubernetes-friendly-using-k8s-api-and-minio/"><em>blog.lwolf.org</em></a><em> on September 3, 2017.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a3b803302557" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Going open-source in monitoring, part III: 10 most useful Grafana dashboards to monitor Kubernetes…]]></title>
            <link>https://medium.com/@SergeyNuzhdin/going-open-source-in-monitoring-part-iii-10-most-useful-grafana-dashboards-to-monitor-kubernetes-7d22ac4645db?source=rss-5c5da0e5f3bb------2</link>
            <guid isPermaLink="false">https://medium.com/p/7d22ac4645db</guid>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Sergey Nuzhdin]]></dc:creator>
            <pubDate>Fri, 21 Jul 2017 00:00:00 GMT</pubDate>
            <atom:updated>2017-09-06T07:26:10.050Z</atom:updated>
            <content:encoded><![CDATA[<h3>Going open-source in monitoring, part III: 10 most useful Grafana dashboards to monitor Kubernetes and services</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E_pWuevJPpzPx-U3BAQfdg.png" /></figure><p>There are dozens of ready dashboards available on <a href="https://grafana.net/dashboards">grafana.net/dashboards</a> and Github. Many of them just work, but many do not. In most cases, you just need to fix template variables, but some require deeper involvement. For example <a href="https://grafana.com/dashboards/455">dashboard</a> for PostgreSQL monitoring. After import, it welcomes you with the error message “Datasource named ${DS_PROMETHEUS} was not found”. Setting the correct datasource name in dashboard settings does not help because it has an error in __inputs declaration. The easiest way to fix it is to edit JSON before import.</p><p>It takes time to find useful dashboards, make them work for you and then choose one. For example, if you’ll search for Kubernetes dashboards with datasource Prometheus you’ll get, among others, 5 results with the same name.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/901/0*WcuOGZt3oG10cRqa.png" /></figure><p>In this post, I want to cover some of the most useful dashboards available to help you monitor your Kubernetes cluster and services deployed on it.</p><p>This post is one of a series of posts about monitoring of infrastructure and services. Other posts in the series:</p><p><a href="https://medium.com/@SergeyNuzhdin/going-open-source-in-monitoring-part-0-intro-dffa2b40665f"><strong>0. Intro</strong></a></p><p><strong>1.Deploying Prometheus and Grafana to Kubernetes </strong>(this article)</p><p><a href="https://medium.com/@SergeyNuzhdin/going-open-source-in-monitoring-part-ii-creating-the-first-dashboard-in-grafana-ada59a4ced2e"><strong>2. Creating the first dashboard in Grafana</strong></a></p><p><strong>3. </strong><a href="https://medium.com/p/going-open-source-in-monitoring-part-i-deploying-prometheus-and-grafana-to-kubernetes-e3d44460cec6"><strong>10 most useful Grafana dashboards to monitor Kubernetes and services</strong></a></p><p>4. Configuring alerts in Prometheus and Grafana</p><p>5. <a href="https://medium.com/@SergeyNuzhdin/going-open-source-in-monitoring-part-v-collecting-errors-from-production-using-sentry-add449d3775a"><strong>Collecting errors from production using Sentry</strong></a></p><p>6. Making sense of logs with ELK stack</p><p>7. Replacing commercial APM monitoring</p><p>8. SLA, SLO, SLI and other useful abstractions</p><h3>Cluster view</h3><p><em>Kubernetes cluster overview </em>— <a href="https://grafana.com/dashboards/1621">#1621</a> or <a href="https://grafana.com/dashboards/315">#315</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Sz7TrpggkJSWVDYf.png" /></figure><p>This dashboard usually the first one you search for after deploying Kubernetes + Prometheus. Among the 5 shown in the screenshot above only 2 is really good. Actually, it is the same dashboard saved by two people.</p><h3>Node view</h3><p>Detailed node overview — <a href="https://grafana.com/dashboards/1860">#1860</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*laCEblrOLhgbGElQ.png" /></figure><p>This one is great when you need to understand what is wrong with the particular node. It has much more stats compared to the cluster overview.</p><h3>Deployment metrics monitoring</h3><p><em>Kubernetes Deployment metrics </em>— <a href="https://grafana.com/dashboards/741">#741</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SaFNrULu8FQp4XEe.png" /></figure><p>Next level of monitoring after the node — deployment. This dashboard will show you everything about selected deployment.</p><h3>Pod metrics monitoring</h3><p><em>Kubernetes Pod Metrics </em>— <a href="https://grafana.com/dashboards/747">#747</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uw6d1rALKLD7sEDv.png" /></figure><p>Pod level monitoring. Shows mostly the same stats as the deployment dashboard but on Pod level. Pod info row shows some random values in Pod container and Pod IP Address places, but other values seem fine.</p><h3>Application metrics monitoring</h3><p><em>Kubernetes App Metrics </em>— <a href="https://grafana.com/dashboards/1471">#1471</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mdd6M7wvanN2_ydxmHratg.png" /></figure><p>This one is good as an example of how to monitor your application deployed to Kubernetes. It has metrics like request rate, error rate and response times for your application alongside with different resource usage stats.</p><h3>PostgreSQL</h3><p><em>Postgres Overview </em>— <a href="https://grafana.com/dashboards/455"><em>#455</em></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AjPZZNxcvMtZ8iw2.png" /></figure><p>After you fix the broken variable declaration — it just works :) I saved fixed one <a href="https://github.com/lwolf/kube-monitoring/blob/master/dashboards/postgres-overview.json">here</a></p><h3>ElasticSearch</h3><p>Elasticsearch cluster overview <a href="https://grafana.com/dashboards/266">#266</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bTpwiPGJiUWECX2r.png" /></figure><p>The best one so far is <a href="https://grafana.com/dashboards/266">266</a>. It supports most of the metrics needed and required least amount of tweaks to make it work. There are several other options — <a href="https://grafana.com/dashboards/2322">2322</a>, <a href="https://grafana.com/dashboards/2347">2347</a>, <a href="https://grafana.com/dashboards/718">718</a>. But they either lack graphs or did not work at all for me.</p><h3>Redis</h3><p><em>Prometheus Redis </em>— <a href="https://grafana.com/dashboards/763">#763</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8RAwaIWfJNKdlY0F.png" /></figure><h3>Memcached</h3><p>Memcached node — <a href="https://grafana.com/dashboards/37">#37</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*o0ezeo3A0z9TS-S2.png" /></figure><h3>Etcd</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NTAsHBYDL09Eduz_.png" /></figure><p>Monitoring of Etcd is covered on CoreOS website <a href="https://coreos.com/etcd/docs/latest/op-guide/monitoring.html">here</a>. Dashboard suggested in this manual has hardcoded datasource, so you have to edit it before import. So I made datasource configurable and saved it in the repo <a href="https://github.com/lwolf/kube-monitoring/blob/master/dashboards/etcd-overview.json">here</a>. By default, Etcd is not monitored by Prometheus at all. So, you need to tell it where your Etcd lives. If you have self-hosted Etcd annotations to the Etcd service will work, otherwise, you need to update prometheus config with the location on your Etcd nodes.</p><pre>- job_name: etcd<br>  static_configs:<br>  - targets:   [&#39;10.240.0.32:2379&#39;,&#39;10.240.0.33:2379&#39;,&#39;10.240.0.34:2379&#39;]</pre><h3>Wrap Up</h3><p>That’s it. At this point, we can monitor health of the Kubernetes cluster and several common services.</p><p>Sources for PostgreSQL and Etcd dashboards, with changes I did, are available on <a href="https://github.com/lwolf/kube-monitoring/tree/master/dashboards">GitHub</a>.</p><p>Stay tuned.</p><h4>Like this article?</h4><p>Click the 💚 below so other people will see it here on Medium.<br><a href="http://eepurl.com/cdUha1"><strong>Subscribe</strong></a> to get new stories delivered to your inbox or <a href="https://twitter.com/SergeyNuzhdin">follow me on twitter</a>.</p><p><em>Originally published at </em><a href="http://blog.lwolf.org/post/going-open-source-in-monitoring-part-iii-10-most-useful-grafana-dashboards-to-monitor-kubernetes-and-services/"><em>blog.lwolf.org</em></a><em> on July 21, 2017.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7d22ac4645db" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Going open-source in monitoring, part II: Creating the first dashboard in Grafana]]></title>
            <link>https://medium.com/@SergeyNuzhdin/going-open-source-in-monitoring-part-ii-creating-the-first-dashboard-in-grafana-ada59a4ced2e?source=rss-5c5da0e5f3bb------2</link>
            <guid isPermaLink="false">https://medium.com/p/ada59a4ced2e</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[monitoring]]></category>
            <dc:creator><![CDATA[Sergey Nuzhdin]]></dc:creator>
            <pubDate>Mon, 19 Jun 2017 00:00:00 GMT</pubDate>
            <atom:updated>2017-09-06T07:25:25.810Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JPxYGWD_SiZ2KfncOp2nqw.png" /></figure><p>This post is one of a series of posts about monitoring of infrastructure and services. Other posts in the series:</p><p><strong>0. </strong><a href="https://medium.com/p/going-open-source-in-monitoring-part-0-intro-dffa2b40665f"><strong>Intro</strong></a></p><p><strong>1.</strong><a href="https://medium.com/p/going-open-source-in-monitoring-part-i-deploying-prometheus-and-grafana-to-kubernetes-e3d44460cec6"><strong>Deploying Prometheus and Grafana to Kubernetes</strong></a></p><p><strong>2. Creating the first dashboard in Grafana</strong> (this article)</p><p><strong>3. </strong><a href="https://medium.com/@SergeyNuzhdin/going-open-source-in-monitoring-part-iii-10-most-useful-grafana-dashboards-to-monitor-kubernetes-7d22ac4645db"><strong>10 most useful Grafana dashboards to monitor Kubernetes and services</strong></a></p><p>4. Configuring alerts in Prometheus and Grafana</p><p>5. <a href="https://medium.com/@SergeyNuzhdin/going-open-source-in-monitoring-part-v-collecting-errors-from-production-using-sentry-add449d3775a"><strong>Collecting errors from production using Sentry</strong></a></p><p>6. Making sense of logs with ELK stack</p><p>7. Replacing commercial APM monitoring</p><p>8. SLA, SLO, SLI and other useful abstractions</p><p>Having <a href="https://grafana.net/dashboards">grafana.net/dashboards</a> with dozens of dashboards to choose from is awesome and helps get started with monitoring, but creating your own dashboards from scratch is also very helpful. When I first deployed Grafana, I imported a lot of ready dashboards, really a lot. Many of them worked out of the box, but many did not. In most cases, the fix was in change of template variables, but some required deeper involvement and understanding of how everything works. Such understanding could be acquired by creating your own dashboards.</p><p>Since one of the goals of the whole project is to replace NewRelic, the first dashboard will replicate its basic server monitoring page. In case you do not know, it looks like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ViaOD21VE1vqu2pm.png" /></figure><p>The corresponding dashboard we building is shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cjG0EQcJ6NLqtXi_.png" /></figure><p>It’s time to create the dashboard. If you just installed Grafana you’ll have “Create your first dashboard” button on the homepage. Otherwise, click “create new” from the top dropdown. I’m not going to copy the <a href="http://docs.grafana.org/guides/getting_started/">gettings-started guide</a> from grafana docs, so if you do not know grafana basics, like how to add a graph to the dashboards — read the <a href="http://docs.grafana.org/guides/getting_started/">guide</a> or watch the 10min <a href="https://www.youtube.com/watch?v=sKNZMtoSHN4&amp;index=7&amp;list=PLDGkOdUX1Ujo3wHw9-z5Vo12YLqXRjzg2">beginners guide to building dashboards</a> to get a quick intro to setting up Dashboards and Panels.</p><p><strong>Templating</strong></p><p>One of the essential features of Grafana is an ability to create variables. It allows you to create interactive and dynamic dashboards. You can have a single variable, like server selector, or a chain of variables each new populated based on previous selection, like namespace -&gt; deployment -&gt; Pod.</p><p>Templating is usually not the first thing covered in manuals. The common way is to create all graphs with hardcoded values and then replace it with the variable. I’m going to cover templating first and use variables from the beginning.</p><p>For this dashboard, we need only one variable — server selector.</p><p>To add one go to <strong>Manage dashboards</strong> → <strong>Templating</strong> → <strong>Variables</strong>, and click <strong>+New.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CSG_qOre_SiB4z9A.png" /></figure><p>You’ll get to the edit variable tab. First, set <strong>Name</strong> and <strong>Label</strong> of the new variable. I prefer to have identical label and name, it helps you not to keep the mapping between label and its name in mind.</p><p>Next, we need to set query options. This is the place where we can select data to populate dropdown.</p><p><strong>Data source</strong>: prometheus data source you’ve created</p><p><strong>Refresh</strong>: one of: Never / On Dashboard Load / On Time Range change. Choose what is suitable for your setup.</p><p><strong>Query</strong>: label_values(node_boot_time{component=”node-exporter”}, instance). What does this query say. Take node_boot_time metric, filter it by label component=“node-exporter” and return a list of values of the label instance . <a href="http://docs.grafana.org/features/datasources/prometheus/#query-variable">More about queries</a>.</p><p><strong>Regex</strong>: Regular expression is optional, but sometimes is very helpful. For example, in my case, the result of the query is IP:PORT, but I want to show only IP in the dropdown. With regex you can make required transformations.</p><p>During edit of the query, you will see a realtime result in the bottom, under the “Preview of the values” block.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5OprCLwOIUcQsppj.png" /></figure><p>More detailed manual about templating could be found in the <a href="http://docs.grafana.org/reference/templating/">docs</a></p><p>Now, when we have our node variable, with all servers in it, we can add some graphs.</p><p><strong>CPU usage</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/0*U3nTiVZq6Jdrmwm2.png" /><figcaption>NewRelic CPU usage</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aA3aLR4OGqk_Vesf.png" /><figcaption>Grafana CPU usage</figcaption></figure><p>Let’s walk through creating of this one in details. To build this graph we need 4 queries, all using node_cpu metric from node-exporter. Here are the queries.</p><pre><strong>system:</strong> (avg by (instance) (irate(node_cpu{instance=~&quot;$node:.*&quot;,mode=~&quot;system|irq|softirq&quot;}[5m])) * 100) <br><strong>user: </strong>(avg by (instance) (irate(node_cpu{instance=~&quot;$node:.*&quot;,mode=&quot;user&quot;}[5m])) * 100) <strong>iowait:</strong> (avg by (instance) (irate(node_cpu{instance=~&quot;$node:.*&quot;,mode=&quot;iowait&quot;}[5m])) * 100) <strong>idle:</strong> (avg by (instance) (irate(node_cpu{instance=~&quot;$node:.*&quot;,mode=&quot;idle&quot;}[5m])) * 100)</pre><p>All these queries differ only by the mode filter, so let’s look only at the first one — system. What is happening here: <em>node_cpu</em> metric is filtered by instance using our previous template variable and by mode. Since it is counters we need to use irate(rate) function to calculate the per-second rate. This will produce the list of results, one for each CPU/core on the machine, so we need to aggregate it to get a single per-node value. At the end, we need to multiply it by 100 to get percents of the total load.</p><p>Here is a good article about the <a href="https://www.robustperception.io/irate-graphs-are-better-graphs/">rate vs irate</a> difference and <a href="https://www.robustperception.io/understanding-machine-cpu-usage/">this one</a> is about CPU usage calculation.</p><p><strong>SystemLoad</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/459/0*6nSC7s91G4rCKgyF.png" /><figcaption>NewRelic System Load</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lD4OOhFsXwuTQPPJ.png" /><figcaption>Grafana System Load</figcaption></figure><pre><strong>load5: </strong>node_load5{instance=~&quot;$node:.*&quot;}</pre><p><strong>Memory</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/0*vujQVtZOAsCSQPBr.png" /><figcaption>NewRelic Memory Usage</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0RP6ZRWqM8P_XSf2.png" /><figcaption>Grafana Memory Usage</figcaption></figure><pre><strong>total:</strong> node_memory_MemTotal{instance=~&quot;$node:.*&quot;} <br><strong>used:</strong> node_memory_MemTotal{instance=~&quot;$node:.*&quot;} - node_memory_MemFree{instance=~&quot;$node:.*&quot;} - node_memory_Cached{instance=~&quot;$node:.*&quot;} - node_memory_Buffers{instance=~&quot;$node:.*&quot;} - node_memory_Slab{instance=~&quot;$node:.*&quot;} swap: (node_memory_SwapTotal{instance=~&quot;$node:.*&quot;} - node_memory_SwapFree{instance=~&quot;$node:.*&quot;})</pre><p><strong>Network I/O</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/322/0*sNfD_vDc763Ft3lY.png" /><figcaption>NewRelic network IO</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sb6l3c5WeMF0CqO4.png" /><figcaption>Grafana Network IO</figcaption></figure><pre><strong>received: </strong>sum(irate(node_network_receive_bytes{instance=~&quot;$node.*&quot;}[5m]))<br><strong>transmitted: </strong>sum(irate(node_network_transmit_bytes{instance=~&quot;$node.*&quot;}[5m]))</pre><p><strong>Disk I/O</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/328/0*NFzz9qJ418c1npZX.png" /><figcaption>NewRelic Disk IO</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*k5ogK_94TjZNbVj_.png" /><figcaption>Grafana Disk IO</figcaption></figure><pre><strong>read_by_device:</strong> irate(node_disk_reads_completed{instance=~&quot;$node:.*&quot;, device!~&quot;dm-.*&quot;}[5m])<br><strong>write_by_device:</strong> irate(node_disk_writes_completed{instance=~&quot;$node:.*&quot;, device!~&quot;dm-.*&quot;}[5m])</pre><p><strong>Top 10 memory hungry pods</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*o2ZdOixgmx4jMPsA.png" /></figure><p>For this graph, we’re going to use two new functions — <strong>sort_desc</strong> and <strong>topk</strong>. topk is used to limit results, while sort_desc is to sort it. by (pod_name) - will group results by pod_name</p><pre><strong>metrics:</strong> sort_desc(topk(10, sum(container_memory_usage_bytes{pod_name!=&quot;&quot;}) by (pod_name)))</pre><p><strong>Top 10 CPU hungry pods</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*17KtmZDRKA0KRcZk.png" /></figure><pre><strong>metrics:</strong> sort_desc(topk(10, sum by (pod_name)( rate(container_cpu_usage_seconds_total{pod_name!=&quot;&quot;}[1m] ) )))</pre><h3>Node overview</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JluuJEHlUwf5lpqF.png" /></figure><p>This row is a bit different from all the others. While all previous stats were using Graph type, here we’re going to use Singlestats **and Table.</p><p><strong>Total RAM:</strong></p><pre><strong>type:</strong> singlestat <br><strong>metrics:</strong> node_memory_MemTotal{instance=~&quot;$node:.*&quot;} <br><strong>options:</strong> stat=current, unit=bytes, decimals=1</pre><p><strong>CPU cores:</strong></p><pre><strong>type:</strong> singlestat<br><strong>metrics:</strong> count(count(node_cpu{instance=~&quot;$node:.*&quot;}) by (cpu))</pre><p><strong>Uptime:</strong></p><pre><strong>type:</strong> singlestat<br><strong>metrics:</strong> node_time{instance=~&quot;$node:.*&quot;} - node_boot_time{instance=~&quot;$node:.*&quot;} <br><strong>options:</strong> stat=current, unit=seconds, decimals=1</pre><p><strong>Disk Space Usage:</strong></p><pre><strong>type:</strong> singlestat<br><strong>metrics:</strong> sum((node_filesystem_size{instance=~&quot;$node:.*&quot;,mountpoint=&quot;/rootfs&quot;} - node_filesystem_free{instance=~&quot;$node:.*&quot;,mountpoint=&quot;/rootfs&quot;}) / node_filesystem_size{instance=~&quot;$node:.*&quot;,mountpoint=&quot;/rootfs&quot;}) * 100</pre><p><strong>FileSystem Fill Up Time</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/416/0*XgRrLDzvcBmznSYp.png" /></figure><p>This one is a bit more sophisticated. It uses simple linear regression to predict time until disks fill up. We’re using avg by (device) to show each device only once, otherwise, we will have one line for each mountpoint.</p><pre><strong>type:</strong> table metrics: avg by (device) (deriv(node_filesystem_free{device=~&quot;/dev/sd.*&quot;,instance=~&quot;$node:.*&quot;}[4h]) &gt; 0)</pre><blockquote><em>In case you only have 3 mountpoints (/etc/hosts, /etc/hostsname, /etc/resolv.conf) exposed by node-exporter — double check that your node-exporter pods have all the rootfs directories mounted. As stated </em><a href="https://github.com/prometheus/node_exporter/blob/master/README.md#using-docker"><em>here</em></a><em>.</em></blockquote><h4>Wrap Up</h4><p>That’s it. Now we have a node overview dashboard which shows all the basic information available in NewRelic server view. As a bonus, we’ve got disk fullness prediction and information about most RAM/CPU hungry pods. Source of this dashboard is available on <a href="https://github.com/lwolf/kube-monitoring/blob/master/dashboards/server-overview.json">GitHub</a>.</p><p>Stay tuned.</p><h4>Like this article?</h4><p>Click the 💚 below so other people will see it here on Medium.<br><a href="http://eepurl.com/cdUha1"><strong>Subscribe</strong></a> to get new stories delivered to your inbox or <a href="https://twitter.com/SergeyNuzhdin">follow me on twitter</a>.</p><p><em>Originally published at </em><a href="http://blog.lwolf.org/post/going-open-source-in-monitoring-part-ii-creating-the-first-dashboard-in-grafana/"><em>blog.lwolf.org</em></a><em> on June 19, 2017.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ada59a4ced2e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Going open-source in monitoring, part I: Deploying Prometheus and Grafana to Kubernetes]]></title>
            <link>https://medium.com/@SergeyNuzhdin/going-open-source-in-monitoring-part-i-deploying-prometheus-and-grafana-to-kubernetes-e3d44460cec6?source=rss-5c5da0e5f3bb------2</link>
            <guid isPermaLink="false">https://medium.com/p/e3d44460cec6</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Sergey Nuzhdin]]></dc:creator>
            <pubDate>Mon, 29 May 2017 00:00:00 GMT</pubDate>
            <atom:updated>2017-09-06T07:25:55.323Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xzOqAySHPvtJKVuW.png" /></figure><p>I’ve been keeping my eye on <a href="https://prometheus.io/">Prometheus</a> for some time. What I like about it — is its pull nature and great integration with service discovery and Kubernetes.</p><p>With pull system, you do not have to tell every service you have where the monitoring system is located. You just need to expose your metrics as a plain text using HTTP and Prometheus will access this endpoint and pull the data. The system will never fail because your monitoring service ran out of space or overloaded.</p><p>Among other advantages, for me, are: it is written in Go, easy to deploy and inspired by Google’s Borgmon system. It’s not that I like stuff created by Google, but I think it really makes sense to use Borgmon-inspired system to monitor Borg-inspired container management system — Kubernetes :)</p><p>The Prometheus system consists of several components:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/793/0*7WWK4D_lvXHB-tSX.png" /><figcaption>prometheus architecture</figcaption></figure><ul><li>Prometheus server which scrapes and stores metrics</li><li>Exporters which expose metrics for well-known services</li><li>Alertmanager to send notifications and alerts on events</li><li>Push gateway, client libraries to build custom monitoring solutions.</li></ul><p>This post is one of a series of posts about monitoring of infrastructure and services. Other posts in the series:</p><p><a href="https://medium.com/@SergeyNuzhdin/going-open-source-in-monitoring-part-0-intro-dffa2b40665f"><strong>0. Intro</strong></a></p><p><strong>1.Deploying Prometheus and Grafana to Kubernetes </strong>(this article)</p><p><a href="https://medium.com/@SergeyNuzhdin/going-open-source-in-monitoring-part-ii-creating-the-first-dashboard-in-grafana-ada59a4ced2e"><strong>2. Creating the first dashboard in Grafana</strong></a></p><p><strong>3. </strong><a href="https://medium.com/@SergeyNuzhdin/going-open-source-in-monitoring-part-iii-10-most-useful-grafana-dashboards-to-monitor-kubernetes-7d22ac4645db"><strong>10 most useful Grafana dashboards to monitor Kubernetes and services</strong></a></p><p>4. Configuring alerts in Prometheus and Grafana</p><p>5. <a href="https://medium.com/@SergeyNuzhdin/going-open-source-in-monitoring-part-v-collecting-errors-from-production-using-sentry-add449d3775a"><strong>Collecting errors from production using Sentry</strong></a></p><p>6. Making sense of logs with ELK stack</p><p>7. Replacing commercial APM monitoring</p><p>8. SLA, SLO, SLI and other useful abstractions</p><h3>Planning resources</h3><p><strong>Memory</strong> usage could be configured by setting storage.local.target-heap-size flag. Its default value is 2GiB which corresponds to about 200,000 time series simultaneously present in memory. This value should be set to 2⁄3 of the physical memory limit. That means that we should set resource limit for the container to 3GiB.</p><blockquote><em>Prior to v1.6, there was no flag </em><em>storage.local.target-heap-size. Instead, the number of chunks kept in memory had to be configured using the flags </em><em>storage.local.memory-chunks and </em><em>storage.local.max-chunks-to-persist. These parameters are covered in this </em><a href="https://www.robustperception.io/how-much-ram-does-my-prometheus-need-for-ingestion/"><em>article</em></a><em>.</em></blockquote><p><strong>Storage.</strong> The size of the storage needed for the system depends on 2 parameters: retention rate and the number of time series. After some observations in the staging environment I ended up with such numbers: ~90k metrics collected for 2 weeks have a size of around 40G on disk. To be on the safe side I have set retention to 1 month and created 100G volume for Prometheus data. Retention rate could be set using storage.local.retention setting.</p><p>More information about configurations could be found in the <a href="https://prometheus.io/docs/operating/storage/">operations/storage section</a> of the docs. This page also has link to great talks about Prometheus — <a href="https://www.youtube.com/watch?v=HbnGSNEjhUc">The Prometheus Time Series Database</a>, <a href="https://www.youtube.com/watch?v=hPC60ldCGm8">Configuring Prometheus for High Performance</a>.</p><h3>Installation on Kubernetes</h3><p>There are several ways to install Prometheus on Kubernetes: <a href="https://github.com/kubernetes/charts/tree/master/stable/prometheus">Helm chart</a>, <a href="https://github.com/coreos/prometheus-operator">CoreOS’s operator</a> and of course the manual way.</p><p>In this post, I’m going to use the helm chart to install and configure both Prometheus and Grafana.</p><p>First of all, make sure you have helm installed and configured. If not — <a href="https://github.com/kubernetes/helm/blob/master/docs/install.md">install it</a>.</p><p>Now, we need to create a configuration file for our chart. To not start from scratch, let’s download default values from the chart</p><pre>$ mkdir prometheus &amp;&amp; cd prometheus<br># save the default values<br>$ wget <a href="https://github.com/kubernetes/charts/blob/master/stable/prometheus/values.yaml">https://github.com/kubernetes/charts/blob/master/stable/prometheus/values.yaml</a></pre><p>This file consists of a several parts:</p><ul><li><strong>server + serverFiles </strong>— configurations related to Prometheus itself</li><li><strong>alertmanager + alertmanagerFiles </strong>— configurations related to alertmanager. Alertmanager is not covered in this post. A separate post is planned for it.</li><li><a href="https://github.com/prometheus/node_exporter"><strong>nodeExporter</strong></a> and <a href="https://github.com/kubernetes/kube-state-metrics"><strong>kubeStateMetrics</strong></a> — configurations for two exporters shipped with the chart. One will expose various metrics about the nodes, the second — kubernetes related metrics. Their defaults are fine and do not require any changes.</li><li><strong>configmapReload</strong> — settings for confimap reload sidecar. <a href="https://github.com/jimmidyson/configmap-reload">configmapReload</a> is a small binary to trigger a reload of the service when Kubernetes ConfigMaps are updated. We’re not going to change defaults here.</li></ul><p>Let’s start with the server. Here is the corresponding part of the current default.</p><pre>server:<br>  ## Prometheus server container name<br>  ##<br>  name: server<br><br>  ## Prometheus server container image<br>  ##<br>  image:<br>    repository: prom/prometheus<br>    tag: v1.5.2<br>    pullPolicy: IfNotPresent<br><br>  ## (optional) alertmanager URL<br>  ## only used if alertmanager.enabled = false<br>  alertmanagerURL: &quot;&quot;<br><br>  ## Additional Prometheus server container arguments<br>  ##<br>  extraArgs: {}<br><br>  ingress:<br>    ## If true, Prometheus server Ingress will be created<br>    ##<br>    enabled: false<br><br>    ## Prometheus server Ingress annotations<br>    ##<br>    annotations: {}<br>    #   kubernetes.io/ingress.class: nginx<br>    #   kubernetes.io/tls-acme: &#39;true&#39;<br><br>    ## Prometheus server Ingress hostnames<br>    ## Must be provided if Ingress is enabled<br>    ##<br>    hosts: []<br>    #   - prometheus.domain.com<br><br>    ## Prometheus server Ingress TLS configuration<br>    ## Secrets must be manually created in the namespace<br>    ##<br>    tls: []<br>    #   - secretName: prometheus-server-tls<br>    #     hosts:<br>    #       - prometheus.domain.com<br><br>  ## Node labels for Prometheus server pod assignment<br>  ## Ref: https://kubernetes.io/docs/user-guide/node-selection/<br>  nodeSelector: {}<br><br>  persistentVolume:<br>    ## If true, Prometheus server will create/use a Persistent Volume Claim<br>    ## If false, use emptyDir<br>    ##<br>    enabled: true<br><br>    ## Prometheus server data Persistent Volume access modes<br>    ## Must match those of existing PV or dynamic provisioner<br>    ## Ref: http://kubernetes.io/docs/user-guide/persistent-volumes/<br>    ##<br>    accessModes:<br>      - ReadWriteOnce<br><br>    ## Prometheus server data Persistent Volume annotations<br>    ##<br>    annotations: {}<br><br>    ## Prometheus server data Persistent Volume existing claim name<br>    ## Requires server.persistentVolume.enabled: true<br>    ## If defined, PVC must be created manually before volume will be bound<br>    existingClaim: &quot;&quot;<br><br>    ## Prometheus server data Persistent Volume mount root path<br>    ##<br>    mountPath: /data<br><br>    ## Prometheus server data Persistent Volume size<br>    ##<br>    size: 8Gi<br><br>    ## Prometheus server data Persistent Volume Storage Class<br>    ## If defined, volume.beta.kubernetes.io/storage-class: &lt;storageClass&gt;<br>    ## Default: volume.alpha.kubernetes.io/storage-class: default<br>    ##<br>    storageClass: &quot;&quot;<br><br>    ## Subdirectory of Prometheus server data Persistent Volume to mount<br>    ## Useful if the volume&#39;s root directory is not empty<br>    ##<br>    subPath: &quot;&quot;<br><br>  ## Annotations to be added to Prometheus server pods<br>  ##<br>  podAnnotations: {}<br>    # iam.amazonaws.com/role: prometheus<br><br>  replicaCount: 1<br><br>  ## Prometheus server resource requests and limits<br>  ## Ref: http://kubernetes.io/docs/user-guide/compute-resources/<br>  ##<br>  resources: {}<br>    # limits:<br>    #   cpu: 500m<br>    #   memory: 512Mi<br>    # requests:<br>    #   cpu: 500m<br>    #   memory: 512Mi<br><br>  service:<br>    annotations: {}<br>    clusterIP: &quot;&quot;<br><br>    ## List of IP addresses at which the Prometheus server service is available<br>    ## Ref: https://kubernetes.io/docs/user-guide/services/#external-ips<br>    ##<br>    externalIPs: []<br><br>    loadBalancerIP: &quot;&quot;<br>    loadBalancerSourceRanges: []<br>    servicePort: 80<br>    type: ClusterIP<br><br>  ## Prometheus server pod termination grace period<br>  ##<br>  terminationGracePeriodSeconds: 300</pre><p>What we need to change now is volume size, resource limits, ingress and set retention rate.</p><p><strong>persistentVolume</strong> configuration will be something like this for me. As discussed in the section planning resources, the size of the volume will be 100G.</p><pre>persistentVolume:<br>  enabled: true<br>  accessModes:<br>    - ReadWriteOnce<br>  annotations: {}<br>  existingClaim: &quot;&quot;<br>  mountPath: /data<br>  size: 100Gi<br>  storageClass: &quot;&quot;<br>  subPath: &quot;&quot;</pre><p><strong>resources.</strong> Since I’m going to stay with Prometheus’s default heap size (2G) I need to set max memory limit for the container to 3G</p><pre>resources:<br>  limits:<br>    memory: 3Gi<br>  requests:<br>    memory: 3Gi</pre><p><strong>ingress</strong>. I’m using <a href="https://github.com/jetstack/kube-lego">kube-lego</a> for automatic management of lets encrypt certificates. So here is the settings for the assumed domain example.com. I also added an annotation to allow only the local access to it. It’s really nice way to deploy internal services. Service is internal but it has proper SSL.</p><pre>ingress:<br>  enabled: true<br>  annotations:<br>    kubernetes.io/ingress.class: nginx<br>    kubernetes.io/tls-acme: &#39;true&#39;<br>    ingress.kubernetes.io/whitelist-source-range: &quot;192.168.1.0/24&quot;<br>  hosts:<br>    - prometheus.example.com<br>  tls:<br>    - secretName: prometheus-server-tls<br>      hosts:<br>        - prometheus.example.com</pre><p><strong>retention rate</strong>. The last setting we’re going to set now is retention rate to 1 month. Just like a most Prometheus settings, this one should to be set as command argument. <strong>extraArgs</strong> is used to populate these arguments.</p><pre>extraArgs:<br>  storage.local.retention: 744h</pre><p>This is enough for a simple setup. Let’s deploy it.</p><pre>$ helm install -f values.yaml stable/prometheus</pre><p>When all components are in the Running state, try to open <a href="https://prometheus.example.com/">https://prometheus.example.com</a>. You should see the Prometheus UI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Iy5e44WrnzmlDbGb.png" /></figure><p>Make sure to check that everything is green at <a href="https://prometheus.example.com/targets">https://prometheus.example.com/targets</a>. It lists all services known to Prometheus.</p><p>Using the UI you can explore different metrics exposed by your system. For example, you can easily inspect average node load by selecting one of the metrics — node_load1, node_load5, node_load15. Below is the example of node_load5 on my cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RqPGdwEUu3srflX0.png" /></figure><p>You can use most of the usual arithmetic operations, like addition or division to transform your data. A full list of operators could be found <a href="https://prometheus.io/docs/querying/operators/">here</a>. Here you can see memory usage in percents, constructed using 2 metrics — node_memory_MemTotal and node_memory_MemFree.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jrMqotArQdgj5x8Q.png" /></figure><p>This UI is primarily useful for ad-hoc queries and debugging. For day-to-day use, we are going to deploy <a href="https://grafana.com/">Grafana</a>.</p><h4>Grafana</h4><p>I‘m going to install Grafana from charts also. The process in straightforward. Let’s grab default values config and update it a little.</p><pre>$ mkdir grafana &amp;&amp; cd grafana<br>$ wget https://github.com/kubernetes/charts/blob/master/stable/grafana/values.yaml</pre><p>Similar to the Prometheus setup we’re going to configure ingress. But in this case, it should be accessible to the outside world.</p><pre># update ingress to use SSL<br>ingress:<br>  enabled: true<br>  annotations:<br>    kubernetes.io/ingress.class: nginx<br>    kubernetes.io/tls-acme: &#39;true&#39;<br>  hosts:<br>    - grafana.example.com<br>  tls:<br>    - secretName: grafana-server-tls<br>      hosts:<br>        - grafana.example.com</pre><p>The chart also provides the ability to automatically install <a href="https://grafana.com/plugins">plugins</a>. As an example, we’re going to add a few.</p><pre>server:<br>  installPlugins: grafana-clock-panel,grafana-piechart-panel</pre><p>That’s basically it. Let’s deploy.</p><pre>&gt; $ helm install -f values.yaml stable/grafana</pre><p>After successful deploy, you should be able to access Grafana on <a href="https://grafana.example.com/">https://grafana.example.com</a> using admin: admin as credentials. You should change default login/password immediately.</p><blockquote><em>IMPORTANT: Change the default login/password in grafana!!!</em></blockquote><p>Now, we need to tell Grafana where our Prometheus lives. To do this we have to <a href="https://prometheus.io/docs/visualization/grafana/#creating-a-prometheus-data-source">add datasource</a>. Make sure to correctly set Url in Http settings. It should point to your internal prometheus address — <a href="http://release_name-prometheus-server.default.svc.cluster.local/">http://release_name-prometheus-server.default.svc.cluster.local</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/0*20YuADbiEfpDKFMz.png" /></figure><p>Selecting Prometheus as a type for datasource will add <a href="https://grafana.com/dashboards/358">default Prometheus dashboard</a> to the suggested dashboards (the second tab). Importing it will give you dashboard similar to the one shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XCbGyMyaKUFPKDns.png" /></figure><p>At this point, you should have a working Prometheus and Grafana. Next time I will go through a creating a node monitoring dashboard from scratch. At the end, I will post a list of the existing dashboards from grafana.net that I found useful.</p><p>Stay tuned.</p><h4>Like this article?</h4><p>Click the 💚 below so other people will see it here on Medium.<br><a href="http://eepurl.com/cdUha1"><strong>Subscribe</strong></a> to get new stories delivered to your inbox or <a href="https://twitter.com/SergeyNuzhdin">follow me on twitter</a>.</p><p><em>Originally published at </em><a href="http://blog.lwolf.org/post/going-open-source-in-monitoring-part-i-deploying-prometheus-and-grafana-to-kubernetes/"><em>blog.lwolf.org</em></a><em> on May 29, 2017.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e3d44460cec6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Going open-source in monitoring, part 0: Intro]]></title>
            <link>https://medium.com/@SergeyNuzhdin/going-open-source-in-monitoring-part-0-intro-dffa2b40665f?source=rss-5c5da0e5f3bb------2</link>
            <guid isPermaLink="false">https://medium.com/p/dffa2b40665f</guid>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Sergey Nuzhdin]]></dc:creator>
            <pubDate>Fri, 12 May 2017 00:00:00 GMT</pubDate>
            <atom:updated>2017-09-06T07:25:40.370Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4aTagNr036snBlABKpXQRw.jpeg" /></figure><p>This post is one of a series of posts about monitoring of infrastructure and services. Other posts in the series:</p><p>0. <strong>Intro (this article)</strong></p><p><a href="https://medium.com/p/going-open-source-in-monitoring-part-i-deploying-prometheus-and-grafana-to-kubernetes-e3d44460cec6"><strong>1.Deploying Prometheus and Grafana to Kubernetes</strong></a></p><p><strong>2. </strong><a href="https://medium.com/@SergeyNuzhdin/going-open-source-in-monitoring-part-ii-creating-the-first-dashboard-in-grafana-ada59a4ced2e"><strong>Creating the first dashboard in Grafana</strong></a></p><p><strong>3. </strong><a href="https://medium.com/@SergeyNuzhdin/going-open-source-in-monitoring-part-iii-10-most-useful-grafana-dashboards-to-monitor-kubernetes-7d22ac4645db"><strong>10 most useful Grafana dashboards to monitor Kubernetes and services</strong></a></p><p>4. Configuring alerts in Prometheus and Grafana</p><p>5. <a href="https://medium.com/@SergeyNuzhdin/going-open-source-in-monitoring-part-v-collecting-errors-from-production-using-sentry-add449d3775a"><strong>Collecting errors from production using Sentry</strong></a></p><p>6. Making sense of logs with ELK stack</p><p>7. Replacing commercial APM monitoring</p><p>8. SLA, SLO, SLI and other useful abstractions</p><p>Monitoring of the infrastructure is an essential part of any product. But it’s not uncommon for companies to postpone monitoring for the later period. Having it in “nice-to-have” bucket. That’s one of the reasons why they spend a lot of time reacting to the problems after service disruption. The uptime of the infrastructure is as important to the product as the product itself. Monitoring is especially crucial in modern cloud based systems. Containers and even nodes could die any minute, and you will not be able to analyze logs afterwords. Monitoring as a base for any service is perfectly described in a Dickerson’s Hierarchy of Reliability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/0*rrPb3P5Bn2nW-5Ho.jpg" /></figure><p>Monitoring is hard. But collecting the metrics is only a part of the problem, not the hardest though. Interpreting the data and using monitoring systems is much more difficult. It does not matter if you collect all possible metrics if you do not have proper dashboards and alert systems.</p><p>For a long time, I was using systems like NewRelic for monitoring my infrastructures and applications. One of the main reasons for it was ease of install and ready to use dashboards.</p><p>Recently, I decided to stop using commercial monitoring systems in favor of the open-source. It’s not the first attempt, though. Previously I already tried to use Prometheus and ELK stack (ElasticSearch, Logstash, Kibana). But, I never actually did a full switch. Even having the system in place I did not use it very often. So, sooner or later, the disk space or other resources were needed, and since there was NewRelic, these systems were deleted.</p><p>The main reason for the failures, I believe, was the default place to go. You can’t have several systems doing almost the same. One will always be <strong>the monitoring system</strong>. One that you will open when something is wrong. In my case it was NewRelic.</p><p>Another reason is not enough time spent configuring. Since you’re trying to replace an existing system, which works, you’re not committed. You will always have an excuse like “it didn’t work for me”, or “I didn’t have enough time to configure it”.</p><p>This time I decided to stop using NewRelic, and make Grafana my default place to go.</p><p>In this series of posts, I’m planning to describe my migration path. The end goal is to have complete monitoring of the Kubernetes based infrastructure. Having alert system in place. Configure and visualize logs from the infrastructure and applications.</p><p>And the final and the hardest step is to replace APM.</p><p>Stay tuned.</p><h4>Like this article?</h4><p>Click the 💚 below so other people will see it here on Medium.<br><a href="http://eepurl.com/cdUha1"><strong>Subscribe</strong></a> to get new stories delivered to your inbox or <a href="https://twitter.com/SergeyNuzhdin">follow me on twitter</a>.</p><p><em>Originally published at </em><a href="http://blog.lwolf.org/post/going-open-source-in-monitoring-part-0-Intro/"><em>blog.lwolf.org</em></a><em> on May 12, 2017.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dffa2b40665f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to recover data from broken GlusterFS cluster]]></title>
            <link>https://medium.com/@SergeyNuzhdin/how-to-recover-data-from-broken-glusterfs-cluster-2898172228e7?source=rss-5c5da0e5f3bb------2</link>
            <guid isPermaLink="false">https://medium.com/p/2898172228e7</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[cloud-storage]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Sergey Nuzhdin]]></dc:creator>
            <pubDate>Fri, 31 Mar 2017 00:00:00 GMT</pubDate>
            <atom:updated>2017-09-04T06:55:04.693Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1Qp5zxrw1ewbfztfSZ1Wwg.png" /></figure><p>A few days ago when I tried to install helm chart in my Kubernetes cluster I noticed that all new pods that required storage were in pending state. After a quick check of the logs, I found out that pods were unable to get PVC from GlusterFS. I recently wrote about my experience <a href="http://blog.lwolf.org/post/how-i-deployed-glusterfs-cluster-to-kubernetes/">deploying GlusterFS cluster</a>. This time I will go through recovering data from the broken GlusterFS cluster, and some problems I faced deploying new cluster.</p><h3>State: Peer Rejected</h3><p>The first thing I noticed was the bad state of the peers.</p><pre>$ kubectl exec -ti glusterfs0-2272744551-a4ghp gluster peer status<br>Number of Peers: 2<br><br>Hostname: XXX<br>Uuid: fd499492-e26f-4c3f-919a-57ebced1439a<br>State:   State: Peer Rejected (Connected)</pre><p>This case is covered in GlusterFS <a href="https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Resolving%20Peer%20Rejected/">documentation</a>.</p><blockquote><strong><em>NOTE</em></strong><em>: If you’ll find your peers in </em><em>Accepted peer request state after following the manual - restart glusterd one more time.</em></blockquote><p>After all my nodes were in good state I tried to use storage again. It didn’t work, but I found this errors in heketi logs (I don’t remember if they where there before) :</p><pre>[heketi] ERROR 2017/03/24 12:24:51 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Unable to execute command on glusterfs-6bjq2: volume create: vol_9d00fda936724e9f0500575f734b8fe3: failed: Brick: 192.168.11.69:/var/lib/heketi/mounts/vg_d16a4532c7879392d4546009d7dab6d0/brick_2830bc82e8836bb5913949ca32d0f8d4/brick not available. Brick may be containing or be contained by an existing brick.<br><br>[sshexec] ERROR 2017/03/24 12:24:48 /src/github.com/heketi/heketi/executors/sshexec/volume.go:139: Unable to delete volume vol_9d00fda936724e9f0500575f734b8fe3: Unable to execute command on glusterfs-6bjq2: volume delete: vol_9d00fda936724e9f0500575f734b8fe3: failed: Volume vol_9d00fda936724e9f0500575f734b8fe3 does not exist</pre><p>I spent several hours trying to heal volumes using all possible commands from GlusterFS mailing lists, and different manual[<a href="https://access.redhat.com/documentation/en-US/Red_Hat_Storage/2.0/html/Administration_Guide/sect-User_Guide-Managing_Volumes-Self_heal.html">1</a>][<a href="http://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Features/heal-info-and-split-brain-resolution/">2</a>], but it didn’t help. Heal command showed that everything was fine. But old volumes were empty inside the PVC.</p><p>At some point, I realized that it’s easier to recreate the whole cluster. So I started to look how to get data from my volumes.</p><h3>LVM intro</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/0*mLepTbteal0WoIpu.png" /></figure><p>(<a href="http://www.tecmint.com/rhcsa-exam-create-format-resize-delete-and-encrypt-partitions-in-linux/">image source</a>)</p><p><strong>Volume group(VG)</strong> — the highest level of abstraction in lvm. It gathers collection of LV and PV</p><p><strong>Physical Volume(PV)</strong> — usually hard disk or device that ‘looks’ like a hard disk</p><p><strong>Logical Volume(LV)</strong> — an equivalent of a disk partition in usual system.</p><p>Here are some useful resources about logical volumes:</p><ul><li><a href="http://www.tldp.org/HOWTO/LVM-HOWTO/index.html">LVM-Howto</a></li><li><a href="https://www.howtogeek.com/howto/40702/how-to-manage-and-use-lvm-logical-volume-management-in-ubuntu/">How to Manage and Use LVM</a></li></ul><h3>Recovering data</h3><p>For recovery, we need to ssh to the data node.<br>We will only need information about Logical Volumes, but for better understanding, I added similar commands for PV and VG.</p><p><strong>Physical Volumes</strong></p><pre># List all physical volumes<br>$ pvscan<br>  PV /dev/sdd   VG vg_af24fae4b067b6e1ff353d36b6a6918d   lvm2 [1023.87 GiB / 861.94 GiB free]<br>  Total: 1 [1023.87 GiB] / in use: 1 [1023.87 GiB] / in no VG: 0 [0   ]<br><br># Display various attributes of physical volumes<br>$ pvdisplay<br>  --- Physical volume ---<br>  PV Name               /dev/sdd<br>  VG Name               vg_af24fae4b067b6e1ff353d36b6a6918d<br>  PV Size               1.00 TiB / not usable 132.00 MiB<br>  Allocatable           yes<br>  PE Size               4.00 MiB<br>  Total PE              262111<br>  Free PE               220657<br>  Allocated PE          41454<br>  PV UUID               eHxHEy-bMhv-rzF1-HOQn-ds1f-fs3T-GChR1p</pre><p><strong>Volume groups</strong></p><pre>$ vgscan<br>  Reading all physical volumes.  This may take a while...<br>  Found volume group &quot;vg_af24fae4b067b6e1ff353d36b6a6918d&quot; using metadata type lvm2<br>  <br>$ vgdisplay<br>  --- Volume group ---<br>  VG Name               vg_af24fae4b067b6e1ff353d36b6a6918d<br>  System ID<br>  Format                lvm2<br>  Metadata Areas        1<br>  Metadata Sequence No  194<br>  VG Access             read/write<br>  VG Status             resizable<br>  MAX LV                0<br>  Cur LV                26<br>  Open LV               13<br>  Max PV                0<br>  Cur PV                1<br>  Act PV                1<br>  VG Size               1023.87 GiB<br>  PE Size               4.00 MiB<br>  Total PE              262111<br>  Alloc PE / Size       41454 / 161.93 GiB<br>  Free  PE / Size       220657 / 861.94 GiB<br>  VG UUID               Vhnkxf-HCHU-1USc-Y1fo-sNej-PNzr-fZXd3A</pre><p><strong>Logical Volumes</strong></p><pre>$ lvscan<br>  ACTIVE            &#39;/dev/vg_af24fae4b067b6e1ff353d36b6a6918d/tp_32a08cbefd56fbdb4bcceba3b20ab4aa&#39; [2.00 GiB] inherit<br>  ACTIVE            &#39;/dev/vg_af24fae4b067b6e1ff353d36b6a6918d/brick_32a08cbefd56fbdb4bcceba3b20ab4aa&#39; [2.00 GiB] inherit<br>  ACTIVE            &#39;/dev/vg_af24fae4b067b6e1ff353d36b6a6918d/tp_8a72359bb8912053e87af7cf411282ab&#39; [15.00 GiB] inherit<br>  ACTIVE            &#39;/dev/vg_af24fae4b067b6e1ff353d36b6a6918d/brick_8a72359bb8912053e87af7cf411282ab&#39; [15.00 GiB] inherit<br>  ACTIVE            &#39;/dev/vg_af24fae4b067b6e1ff353d36b6a6918d/tp_b9b58b673d04adc56531bc4553c566e8&#39; [20.00 GiB] inherit<br>  ACTIVE            &#39;/dev/vg_af24fae4b067b6e1ff353d36b6a6918d/brick_b9b58b673d04adc56531bc4553c566e8&#39; [20.00 GiB] inherit<br>  <br>$ lvdisplay<br>  --- Logical volume ---<br>  LV Name                tp_32a08cbefd56fbdb4bcceba3b20ab4aa<br>  VG Name                vg_af24fae4b067b6e1ff353d36b6a6918d<br>  LV UUID                ySroXS-reeP-L3mh-PJ8F-JzXS-8IWE-BubioM<br>  LV Write Access        read/write<br>  LV Creation host, time server.example.com, 2017-03-25 14:54:02 +0100<br>  LV Pool metadata       tp_32a08cbefd56fbdb4bcceba3b20ab4aa_tmeta<br>  LV Pool data           tp_32a08cbefd56fbdb4bcceba3b20ab4aa_tdata<br>  LV Status              available<br>  # open                 2<br>  LV Size                2.00 GiB<br>  Allocated pool data    0.71%<br>  Allocated metadata     0.33%<br>  Current LE             512<br>  Segments               1<br>  Allocation             inherit<br>  Read ahead sectors     auto<br>  - currently set to     256<br>  Block device           252:2<br><br>  --- Logical volume ---<br>  LV Path                /dev/vg_af24fae4b067b6e1ff353d36b6a6918d/brick_32a08cbefd56fbdb4bcceba3b20ab4aa<br>  LV Name                brick_32a08cbefd56fbdb4bcceba3b20ab4aa<br>  VG Name                vg_af24fae4b067b6e1ff353d36b6a6918d<br>  LV UUID                QHUP2i-n2Tv-zz89-xpke-dr34-4jAJ-SJ286F<br>  LV Write Access        read/write<br>  LV Creation host, time server.example.com, 2017-03-25 14:54:03 +0100<br>  LV Pool name           tp_32a08cbefd56fbdb4bcceba3b20ab4aa<br>  LV Status              available<br>  # open                 1<br>  LV Size                2.00 GiB<br>  Mapped size            0.71%<br>  Current LE             512<br>  Segments               1<br>  Allocation             inherit<br>  Read ahead sectors     auto<br>  - currently set to     256<br>  Block device           252:4<br><br>  --- Logical volume ---<br>  LV Name                tp_8a72359bb8912053e87af7cf411282ab<br>  VG Name                vg_af24fae4b067b6e1ff353d36b6a6918d<br>  LV UUID                C7TByo-VdOq-mL1y-sdyY-7Pza-4IxI-1k3kxs<br>  LV Write Access        read/write<br>  LV Creation host, time server.example.com, 2017-03-25 16:43:06 +0100<br>  LV Pool metadata       tp_8a72359bb8912053e87af7cf411282ab_tmeta<br>  LV Pool data           tp_8a72359bb8912053e87af7cf411282ab_tdata<br>  LV Status              available<br>  # open                 2<br>  LV Size                15.00 GiB<br>  Allocated pool data    0.12%<br>  Allocated metadata     0.07%<br>  Current LE             3840<br>  Segments               1<br>  Allocation             inherit<br>  Read ahead sectors     auto<br>  - currently set to     256<br>  Block device           252:7<br><br>  --- Logical volume ---<br>  LV Path                /dev/vg_af24fae4b067b6e1ff353d36b6a6918d/brick_8a72359bb8912053e87af7cf411282ab<br>  LV Name                brick_8a72359bb8912053e87af7cf411282ab<br>  VG Name                vg_af24fae4b067b6e1ff353d36b6a6918d<br>  LV UUID                zuUtZB-oRJP-lmZt-Qmn2-j8ta-LR3k-4rH2VA<br>  LV Write Access        read/write<br>  LV Creation host, time server.example.com, 2017-03-25 16:43:07 +0100<br>  LV Pool name           tp_8a72359bb8912053e87af7cf411282ab<br>  LV Status              available<br>  # open                 1<br>  LV Size                15.00 GiB<br>  Mapped size            0.12%<br>  Current LE             3840<br>  Segments               1<br>  Allocation             inherit<br>  Read ahead sectors     auto<br>  - currently set to     256<br>  Block device           252:9</pre><p>What we need from all of this is the output of the lvdisplay command. Specifically its LV Path . Because we can mount it as a usual disk.</p><pre>$ mount /dev/vg_d16a4532c7879392d4546009d7dab6d0/brick_be8a8a7cbf5239caa185cb8d00ae4cfd /mount/volume1</pre><p>Since I had more than 30 of GlusterFS volumes, I ended up with this bash one-liner. It mounts each volume to its own directory.</p><pre>$ cd your-target-directory<br>$ for i in $(lvdisplay | grep /dev/ | cut -d&quot; &quot; -f20); do DIR=&quot;${i##/*/}&quot;; echo $DIR; mkdir $DIR; mount $i $DIR; done<br>$ ls<br>brick_be8a8a7cbf5239caa185cb8d00ae4cfd <br>brick_8a72359bb8912053e87af7cf411282ab <br>...</pre><p>Now you can inspect volumes and copy your data from the bricks as usual.</p><p>All data save and disks are clean. Time to deploy GlusterFS again. I was very interested to see whether anything has changed since my <a href="http://blog.lwolf.org/post/how-i-deployed-glusterfs-cluster-to-kubernetes/">previous experience</a>. So I decided to follow the <a href="https://github.com/heketi/heketi/wiki/Kubernetes-Integration">installation manual</a> from the beginning.</p><p>I was pleasantly surprised that I didn’t face any of the problems I had before. It’s probably due to the non-fresh system. Instead, I faced a new problem because of the this.</p><pre>&gt; $ heketi-cli topology load --json=topology-sample.json                                       Found node node-03 on cluster b55122d933d2a850f69524d9758c7c49<br>                Adding device /dev/sdb ... Unable to add device: Unable to execute command on glusterfs-b0n9m:   WARNING: Not using lvmetad because config setting use_lvmetad=0.<br>  WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).<br>  Can&#39;t open /dev/sdb exclusively.  Mounted filesystem?<br>        Found node node-02 on cluster b55122d933d2a850f69524d9758c7c49<br>                Adding device /dev/sdb ... Unable to add device: Unable to execute command on glusterfs-xnftq:   WARNING: Not using lvmetad because config setting use_lvmetad=0.<br>  WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).<br>  Can&#39;t open /dev/sdb exclusively.  Mounted filesystem?<br>        Found node node-01 on cluster b55122d933d2a850f69524d9758c7c49<br>                Adding device /dev/sdd ... Unable to add device: Unable to execute command on glusterfs-x4kdf:   WARNING: Not using lvmetad because config setting use_lvmetad=0.<br>  WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).<br>  Can&#39;t open /dev/sdd exclusively.  Mounted filesystem?</pre><p>Running pvscan --cache as suggested didn’t help. Neither as deleting all LV, VG, PV. It didn’t work even after a reboot. At the end, I found the solution on <a href="http://stackoverflow.com/questions/10889433/lvm2-failing-to-pvcreate-a-block-device">stackoverflow</a>.</p><pre>$ dmsetup remove sdX</pre><p><strong>dmsetup</strong> is some kind of low-level utility to work with lvm. More information could be found <a href="https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Logical_Volume_Manager_Administration/dmsetup.html">here</a></p><p>Anyway, I continued to follow the manual</p><pre>$ heketi-cli setup-openshift-heketi-storage <br>$ kubectl create -f heketi-storage.json</pre><p>At this point everything was fine. Running heketi-cli topology info returned me all my nodes and volumes. Then I deleted all helper objects and deployed heketi-deployment</p><pre># kubectl delete all,service,jobs,deployment,secret --selector=&quot;deploy-heketi&quot; <br># kubectl create -f heketi-deployment.json <br>service &quot;heketi&quot; created<br>deployment &quot;heketi&quot; created</pre><p>After this, heketi-cli cluster info returned me nothing.</p><p>I thought that may be I didn’t wait until the end of deployment job to finish. Following the manual for the second time gave me the same result.</p><p>After some debugging, I traced the problem to the empty heketi.db in heketi-db-backup secret.</p><p>Setting heketi.db value from the generated heketi-storage.json file seems to fix the issue.</p><p>I don’t know what caused the problem with my storage. One of the guesses is some network related issue. I didn’t lose any of the data but there were some scary moments :).</p><p>I will continue to use GlusterFS on staging cluster, but for now, I’m not ready to use it in production.</p><h4>Like this article?</h4><p>Click the 💚 below so other people will see it here on Medium.<br><a href="http://eepurl.com/cdUha1"><strong>Subscribe</strong></a> to get new stories delivered to your inbox or <a href="https://twitter.com/SergeyNuzhdin">follow me on twitter</a>.</p><p><em>Originally published at </em><a href="http://blog.lwolf.org/post/how-to-recover-data-from-broken-glusterfs-cluster/"><em>blog.lwolf.org</em></a><em> on March 31, 2017.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2898172228e7" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>