<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Miron Gofer on Medium]]></title>
        <description><![CDATA[Stories by Miron Gofer on Medium]]></description>
        <link>https://medium.com/@mirong?source=rss-befab45cfdfe------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*c85_cSLF0HDvN_KV</url>
            <title>Stories by Miron Gofer on Medium</title>
            <link>https://medium.com/@mirong?source=rss-befab45cfdfe------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 14:30:50 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mirong/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How I teach DevOps using ‘ponysays’]]></title>
            <link>https://medium.com/israeli-tech-radar/how-i-teach-devops-using-ponysays-524512f97a43?source=rss-befab45cfdfe------2</link>
            <guid isPermaLink="false">https://medium.com/p/524512f97a43</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[trends]]></category>
            <category><![CDATA[devops-training]]></category>
            <dc:creator><![CDATA[Miron Gofer]]></dc:creator>
            <pubDate>Fri, 03 Mar 2023 01:00:40 GMT</pubDate>
            <atom:updated>2023-03-03T01:00:40.053Z</atom:updated>
            <content:encoded><![CDATA[<p>I love teaching</p><p>While teaching, I use catchphrases and quotes to deliver a message.<br>A good catchphrases can capture a concept or basic idea and anchor memory. It establishes common lingo and mutual understanding. The best quotes and catchphrases are cliches.</p><p>I love cliches.</p><p>I believe every good cliche has a fundamental seed of truth. I use ponysay and fortune to manage (and view) my cliches collection. Ponysay is how my terminal greets me hello every time I start it. It looks like this:</p><figure><img alt="Ponysay ‘Hello World’ examples" src="https://cdn-images-1.medium.com/max/506/1*jprGGN89Ce5Mo4lyx40iZg.png" /><figcaption>Ponysay ‘Hello World’ examples</figcaption></figure><p>If you know what <a href="https://en.wikipedia.org/wiki/Cowsay#:~:text=cowsay%20is%20a%20program%20that,It%20is%20written%20in%20Perl.">cowsay</a>is, <a href="https://github.com/erkin/ponysay">ponysay</a>is the same thing with nicer ASCII art. It is a cool Linux tool to output short snippets of text. I combine it with another tool named <a href="https://en.wikipedia.org/wiki/Fortune_(Unix)">fortune</a> to randomly output text snippets from a given list. I call those snippets <strong>ponysays</strong>.</p><p>I love <strong>ponysays</strong>.</p><p>During the last year, I led a program we named pro-camp. I took four non-experienced DevOps engineers for an intensive month of training and then led them as a devops team for the rest of the year. My campers came with basic knowledge of the tools and technologies we used. So the tech part was easy to teach. My challenge was to bequeath them good practices, mindset, and approach. I wanted them to become knowledgeable engineers and also good DevOps. Such practices and mindset are sometimes as important as knowing the technologies. And more challenging to teach.</p><p>So I used some <strong>ponysays</strong>.</p><p>Catchphrases and quotes helped me capture the basic rules and concepts I think a sound DevOps engineer must follow. And created an internal lingo we used in our daily work as a team after the training. I inserted the best of them into my ponysay database so my terminal could quote, from time to time, those important pieces of truth.</p><p>In this article, I want to share some of them with you as a way to express my point of view of how to become a good DevOps.</p><h3>The only stupid question is the one you chose not to ask</h3><figure><img alt="The only stupid question is the one you chose not to ask" src="https://cdn-images-1.medium.com/max/637/1*8ZOpQmve87c9PJkpR5oicQ.png" /><figcaption>The only stupid question is the one you chose not to ask</figcaption></figure><p>I literally started my first training session with this ponysay. It is an essential rule if you want to learn effectively. But it is more than just a good learning practice. DevOps job involves a lot of transfer of information. You need to learn the new environment, understand the problem at hand, or get the DoD of the task right. The enemy of understanding is the fear of asking a question that will make you look ‘stupid’ or ‘junior’.</p><p>Asking questions to understand the context of a situation is an art form. It is more than just asking for a missing piece of knowledge. The best question uncovers the things you don’t know you don’t know. And they can sound trivial or ‘stupid’ at times.</p><p>Come clean, try to assume nothing, and ask anything that you are not completely sure you understand. And keep asking until you are sure. (Anyone who parented a 3-year-old child will probably know the drill) It can be annoying sometimes. But trust me, it works.</p><h3>The boy scout rule</h3><figure><img alt="The boy scout rule: Leave any piece of code you touch in a better shape than you found it" src="https://cdn-images-1.medium.com/max/616/1*JD_XTP64QuPIcLRRNqHYuA.png" /><figcaption>The boy scout rule: Leave any piece of code you touch in a better shape than you found it</figcaption></figure><p>This is the heart of continuous improvement everyone is talking about. It takes some experience to recognize a ‘bad shape’ of DevOps code. But there are some thumb rules one can use.</p><p>If it takes too much time, effort, and explanations from the code writer to understand — it is not clear enough and at least must be better documented. (re-write the code will probably do a better job, but it is harder and sometimes riskier).</p><p>Another good rule is to run all linting and formatting tools available for the specific tool/language in use. (if not already enforced via your CI) and fix what is not well formatted.</p><p>Look for inconsistencies in naming and patterns. Look for code duplication.</p><p>With time and experience, after reading (and writing) more and more code, a sense of ‘goodness’ is being developed. If the code doesn’t ‘feel’ good, look for ways to improve it.If you have the confidence and the capacity — do it. Make it better.</p><h3>First make it work, then make it look good</h3><figure><img alt="First make it work, then make it look good" src="https://cdn-images-1.medium.com/max/535/1*E2hfqcydXXlqlS9mejfbVg.png" /><figcaption>First make it work, then make it look good</figcaption></figure><p>This ponysay basically says that both are important, but order matters. Writing everything perfectly from the first draft is something even very good engineers do not do. Refactoring and beautifying a working code is a must. A Task is never done when ‘It works!’. But it is easier and faster to do it in two separate steps, in this order.</p><p>What makes code ‘look good’ is a subject for future posts. But you at least have some guidelines in your team.</p><p>This ponysay hides within it another cliche: ‘use baby steps to move fast’. Don’t try to do everything in one giant step. The best approach to a project is the spiral: Make something work, make it look good, then make another part work, make everything look good, and so on.</p><h3>Remember William of Ockham</h3><figure><img alt="Remember William of Ockham. When there is more than one way to do anything, the simplest way is the best." src="https://cdn-images-1.medium.com/max/606/1*G2dqsjbEJUeXOmnGZKfaMg.png" /><figcaption>Remember William of Ockham. When there is more than one way to do anything, the simplest way is the best.</figcaption></figure><p>This is my DevOps take on the famous <a href="https://en.wikipedia.org/wiki/Occam%27s_razor">Ockham razor</a> (AKA KISS). Doing things the simple way is hard. One must fight the urge to use the cool complex solution he just came up with. Over-engineering is an illness that hits us all, but less experienced engineers are more vulnerable.</p><p>A simple, well-written code (every code, but it is even more common in DevOps code) can sometimes look ‘silly’ and, well, simple. The questions ‘Is this everything? Is this what I spent all this time and effort on?’ comes to mind. Don’t worry; simple is good.</p><p>To avoid over-complexity, write only what you need to complete the task at hand. There are always some patterns to follow to prepare your code for the future. But investing too much in preparing your code for, yet unknown, future additions will damage your simplicity. Refactoring will be more efficient when you know exactly what you want to add. The ‘<a href="https://en.wikipedia.org/wiki/Open%E2%80%93closed_principle">open-closed</a>’ principle is your friend.</p><h3>Take care of every penny. The dollar will take care of itself.</h3><figure><img alt="Take care of every penny. The dollar will take care of itself." src="https://cdn-images-1.medium.com/max/612/1*Ajwl27EXvE3_YxJjWmB-4Q.png" /><figcaption>Take care of every penny. The dollar will take care of itself.</figcaption></figure><p>If your charts are well-written, the umbrella will work great. If you test your Terraform modules, your apply will run smoothly. We are working on very complex and large systems with many dependencies and moving parts. The only way to ensure the entire system works is to verify that every small component is as best written as possible.</p><p>Put your chips into writing the small modules right. A good set of ‘Lego blocks’ will enable you to build almost everything. And will make your system flexible, maintainable, robust, and scalable.</p><h3>If it is not in git — it does not exist</h3><figure><img alt="If it is not in git — it does not exist" src="https://cdn-images-1.medium.com/max/611/1*k0cfm8NABXoyr1HRymw2ig.png" /><figcaption>If it is not in git — it does not exist</figcaption></figure><p>We teach it to everybody else. But sometimes forget to follow our own rules. If I run terraform applyor helm updatefrom my laptop, change something in ArgoCD via its GUI, or even just plain simple kubectl edit. It will become a ghost if not committed and pushed to git. You will not know where it comes from, and it will hunt you.</p><p>This post is already too long. So I will leave the last ponysay I chose to share with no extra explanations.</p><figure><img alt="Trust your code, trust your tools, trust your skills. Anything can be done." src="https://cdn-images-1.medium.com/max/583/1*sK4sk2ucM8ZEsGCFFwQfew.png" /><figcaption>Trust your code, trust your tools, trust your skills. Anything can be done.</figcaption></figure><blockquote>Trust your code, trust your tools, trust your skills. Anything can be done.</blockquote><p>Stay good!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=524512f97a43" width="1" height="1" alt=""><hr><p><a href="https://medium.com/israeli-tech-radar/how-i-teach-devops-using-ponysays-524512f97a43">How I teach DevOps using ‘ponysays’</a> was originally published in <a href="https://medium.com/israeli-tech-radar">Israeli Tech Radar</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Develop (and debug) Prometheus configuration locally using data from remote Prometheus database]]></title>
            <link>https://medium.com/israeli-tech-radar/develop-and-debug-prometheus-configuration-locally-using-data-from-remote-prometheus-database-9f84f73e8a62?source=rss-befab45cfdfe------2</link>
            <guid isPermaLink="false">https://medium.com/p/9f84f73e8a62</guid>
            <category><![CDATA[prometheus-operator]]></category>
            <category><![CDATA[externalname-service]]></category>
            <category><![CDATA[endpoints]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Miron Gofer]]></dc:creator>
            <pubDate>Thu, 23 Jul 2020 05:09:54 GMT</pubDate>
            <atom:updated>2020-07-28T15:23:56.691Z</atom:updated>
            <content:encoded><![CDATA[<p>Prometheus operator installed on our GKE cluster, all default metrics and dashboards work great. It is now time to add our own app metrics. A very talented developer added the instrumentation and metric exposure. The service discovery did its magic and all new metrics are in the time-series database. Some cool Grafana dashboards were added and life is good.</p><p>My next task is to add alert configuration, meaning firing alerts and configure Alert-Manager to handle them. This is not a trivial task to me. I don’t have a lot of Prometheus configuration syntax experience and there are a lot of moving parts to sync. I don’t want to do it directly on my running cluster (used by others for dev and QAstuff) but I do want to test my configuration using real application metrics from my ‘live’ cluster.</p><p>So this is what I did:</p><h3>first step: Setup a local Kubernetes cluster</h3><p>I tried all sorts of local kubernetes solutions. I love <strong>k3s</strong> Managed with <strong>k3d</strong>. there are other options as well, such as: Minikube, Docker-desktop etc. As long as you get a working kubernetes cluster you can work with.</p><p>Using <strong>k3d</strong> I run</p><pre>k3d create</pre><p>Since k3d works so fast and easy, I re-create clusters all the time. To avoid adding a new cluster to the kubeconfig every time i recreate one, I just run:</p><pre>export KUBECONFIG=”$(k3d get-kubeconfig — name=’k3s-default’)”</pre><p>and my kubectl knows where to go.</p><p>Actually, this is it.</p><p>But an important note is in place here: I am going to work on 2 Kubernetes clusters in parallel. Therefore it is very important to make sure every kubectl (and helm) command will be executed against the right cluster. I use tmux to separate of environments. On one window I work with the local k3s cluster (exporting this KUBECONFIG) and on another window I work with my ‘live’ cluster where my app is running and the metrics are comming from. (in my case it is a GKE cluster)</p><h4>Install Prometheus-operator</h4><p>I used the same command I used on my GKE cluster. Since I use helm 3 no tiller installation required, Just helm. And then:</p><p>helm install prometheus stable/prometheus-operator</p><p>My base installation had no values.yaml file and used all the defaults of the helm chart. Developing the values.yamlis what I came here to do. I split it to three files, one for Grafana, one for Prometheus and one for AlertManager.</p><p>I wrote my configurations and rules into the values files. So the next command (after writing the values) was:</p><pre>helm upgrade prometheus stable/prometheus-operator \<br>-f ./grafana/values.yaml \<br>-f ./prometheus/values.yaml \<br>-f ./alertManager/values.yaml</pre><p>(I think you can guess my directory layout)</p><p>Now my new configuration (Prometheus alert rules and AlertManager configuration) are installed. I know my yaml syntax is correct and if I want to look at the resulting config I just run:</p><pre>kubectl port-forward service/prometheus-prometheus-oper-alertmanager 9093</pre><p>To look into AlertManager web ui at <a href="http://localhost:9093/">http://localhost:9093</a> and:</p><pre>kubectl port-forward service/prometheus-prometheus-oper-prometheus 9090</pre><p>To look into prometheus web ui at <a href="http://localhost:9090/">http://localhost:9090</a> (tmux can help here, I open 2 panes for the port-forward commands to run and use the 3rd for everything else)</p><p>This is great. But, remember that I want to test it ‘for real’. To see if my alerts works with real app metrics. The local Prometheus db is full of local Kubernetes metrics (and some Prometheus-operator metrics as well) but not my metrics. So nothing will ever trigger my new alerting rules. I need some real data.</p><h4>Create the tunnel to the remote Prometheus (running on GKE cluster)</h4><p>To work on my GKE installation I need to login with my Google account and connect the VPN. Let’s say I did it both. So, on another terminal (tmux window in my case) I use the Kubeconfig file that I got from GCloud by running:</p><pre>gcloud container clusters get-credentials &lt;my-cluster-name&gt; — region &lt;my-region&gt;</pre><p>So now, kubectl works with my GKE cluster. And I can port-forward my Prometheus service to a localhost port. I use the — — address 0.0.0.0 because I need it to be available on an IP that is not the loopback 127.0.0.1 . The loopback address can’t be used by the external-address endpoint I am about to create. Adding this flag ( — — address) enables me to access Prometheus from any IP in my laptop’s root namespace. (I work in a secure environment. I don’t think using this setting with a coffee shop wifi is a good idea. Don’t!)</p><pre>kubectl -n monitoring port-forward — address 0.0.0.0 services/prometheus-prometheus-oper-prometheus 39090:9090</pre><p>On the local cluster I run only Prometheus-operator so I allow myself to run it on the default namespace. On the real GKE cluster it gets its own namespace: monitoring. I use the local port 39090 because port 9090 is already taken by the local Prometheus. (And you can’t open low ports on 0.0.0.0 without being root.)</p><h4>Create external endpoint and ExternalName service</h4><p>Now I’m going to use one of the coolest features of Kubernetes service. Services in Kubernetes are used to route traffic to a group of pods selected by a selector. But, a service can actually route traffic to any endpoint. Including an external IP.<br>So first, I created such a service, using this yaml file:</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: federate<br>  labels:<br>    k8s-app: federate<br>spec:<br>  type: ExternalName<br>  externalName: 10.2.0.44<br>  clusterIP: “”<br>  ports:<br>    - name: prom<br>    port: 39090<br>    protocol: TCP<br>  targetPort: 39090</pre><p>Let’s have a look at it: I named it federate since I am going to use the Prometheus federation to get all metrics from my remote Prometheus db. The type: ExternalName tells kubernetes that this service route to, well, external name (or ip). The externalName: points to the IP of my VPN connection. You can use the IP of your machine. (Again, do not do it if your IP comes from a public, unsecured network). ExternalName service is basically DNS translator. It returns the external name (or IP) instead of the service name. This is why it can not return localhost or 127.0.0.1. You must use a DNS name or some other IP.</p><p>The rest is a regular service configuration. The thing with ExternalName type services is that unlike other type of services (ClusterIP, NodePort or LoadBalancer) Kubernetes will not create an EndPoint object for you. You need to do it yourself. So I add this section to my .yaml:</p><pre>---<br>apiVersion: v1<br>kind: Endpoints<br>metadata:<br>  name: federate<br>  labels:<br>    k8s-app: federate<br>subsets:<br>  - addresses:<br>    - ip: 10.2.0.44<br>ports:<br>  - name: prom<br>  port: 39090<br>  protocol: TCP</pre><p>Now, Kubernetes creates all the necessary routing and I can access Prometheus server that runs on my GKE cluster via a local service in my k3s cluster running on my laptop. (Actually, I don’t really need the service object to connect to an external IP. EndPoint is enough. But, the ServiceMonitor object I am going to set next, needs a service object. EndPoint is not enough for it.) The only thing left to do is config my local Prometheus to get the metrics from the remote one. In Prometheus lingo we say: ‘to scrap the metrics from the federate endpoint of the remote server’. Luckily, we use Prometheus-operator so adding scrap configuration is as easy as adding a ServiceMonitor object to the Kubernetes cluster. Therefore I added this section to my yaml:</p><pre>apiVersion: monitoring.coreos.com/v1<br>kind: ServiceMonitor<br>metadata:<br>  name: federate<br>  labels:<br>    k8s-app: federate<br>    prometheus: kube-prometheus<br>spec:<br>  namespaceSelector:<br>    matchNames:<br>      - default<br>  selector:<br>    matchLabels:<br>      k8s-app: federate<br>  endpoints:<br>    - port: prom<br>    path: /federate<br>    params:<br>      match[]:<br>        - ‘{namespace=”staging”}’<br>  interval: 10s<br>  honorLabels: true</pre><p>ServiceMonitor finds a service to scrap using the namespaceSelector and selector fields. In my case you can see it points to the service I created earlier. The endpoint to scrap is the federation endpoint of Promehteus: /federate. But, to scrap it, a ‘matcher’ must be provided. In my case, I need all the metrics from the staging namespace (on my remote GKE cluster).</p><p>When everything is in place, I can go to the local Prometheus web-UI (remember? <a href="http://localhost:9090)">http://localhost:9090</a>) and run queries on my app metrics. All the metrics are there. And, of course, Now I can test my alert configuration against real data.</p><p>To read more about:</p><p>Kubernetes ExternalName type services — <a href="https://kubernetes.io/docs/concepts/services-networking/service/#externalname">https://kubernetes.io/docs/concepts/services-networking/service/#externalname</a></p><p>Promehteus federation — <a href="https://prometheus.io/docs/prometheus/latest/federation/">https://prometheus.io/docs/prometheus/latest/federation/</a></p><p>Prometheus-operator serviceMonitor CRD — <a href="https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/getting-started.md">https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/getting-started.md</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9f84f73e8a62" width="1" height="1" alt=""><hr><p><a href="https://medium.com/israeli-tech-radar/develop-and-debug-prometheus-configuration-locally-using-data-from-remote-prometheus-database-9f84f73e8a62">Develop (and debug) Prometheus configuration locally using data from remote Prometheus database</a> was originally published in <a href="https://medium.com/israeli-tech-radar">Israeli Tech Radar</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>