<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jonathan Burmester on Medium]]></title>
        <description><![CDATA[Stories by Jonathan Burmester on Medium]]></description>
        <link>https://medium.com/@jonbc?source=rss-c986373fc47d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*HnO7FY_QaksndnxGLIYFXg.jpeg</url>
            <title>Stories by Jonathan Burmester on Medium</title>
            <link>https://medium.com/@jonbc?source=rss-c986373fc47d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 05:39:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jonbc/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Hacking your way to Observability — Part 3]]></title>
            <link>https://jonbc.medium.com/hacking-your-way-to-observability-part-3-56e6ddff1315?source=rss-c986373fc47d------2</link>
            <guid isPermaLink="false">https://medium.com/p/56e6ddff1315</guid>
            <category><![CDATA[tracing]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[jaeger]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[nodejs]]></category>
            <dc:creator><![CDATA[Jonathan Burmester]]></dc:creator>
            <pubDate>Mon, 24 May 2021 13:10:45 GMT</pubDate>
            <atom:updated>2021-05-24T13:33:01.996Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/875/1*1wgW2VP2MMD1Dfo_g9roYw.png" /></figure><h3>Hacking your way to Observability — Part 3</h3><h4>A distributed tracing quick start with Jaeger and OpenTelemetry</h4><p><a href="https://jonbc.medium.com/hacking-your-way-to-observability-part-2-c38baaee6b92?source=friends_link&amp;sk=2cd1e6f5d613bd2ddaf737d6819252fb">Last time</a>, we used Prometheus Alert Manager to configure rules that would send notifications via Slack when triggered. Even though having alerts and notifications it’s great, can metrics help you troubleshoot or explain a problem by themselves?. This is where the problem arises; metrics are good to tell you that something happened with a single instance, according to the boundaries you defined for their values, but as soon as you start working with a distributed system, metrics won’t tell you the story of a request that goes through multiple components. With the microservices boom, systems are becoming more complex, and to understand pathological behavior we need to understand the requests end to end. This is where <strong>distributed tracing</strong> helps you; it captures the activities performed in a request giving you the context missing in metrics and logs.</p><p>In this post, we will extend our application observability capabilities by generating spans and export them into an open-source distributed system named Jaeger. But first, let’s start by defining what a <strong>trace </strong>is.</p><p>You will find all the resources in this repository: <a href="https://github.com/jonathanbc92/observability-quickstart/tree/master/part3">https://github.com/jonathanbc92/observability-quickstart/tree/master/part3</a></p><h3>Tracing</h3><p>A trace is a collection of spans, where each span is a record of an operation performed by a single service; they have a name, start time, duration, context, and additional metadata to bring additional information. Traces allows you to observe the journey of a request as it goes through all the services of a distributed system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/899/1*wCOo8f3A4q5IBRF5uBTQOg.png" /></figure><p>With the analysis of trace data, you can see the behavior of a request, identify issues, bottlenecks, and potential areas for improvement and optimizations. To generate spans, you rely on <strong>instrumentation</strong>, using an API or SDK provided by a tracer <strong>client library.</strong></p><h3>Instrumentation</h3><p>You can manually instrument your application by coding the start and finish of spans in pieces of code that provide meaningful information to you. As an alternative, some frameworks offer automatic instrumentation, which saves time and reduces effort by avoiding the need to modify your codebase.</p><p>Manual and automatic instrumentation are not exclusive; in fact, you might end up combining both; automatic instrumentation to leverage the advantage of using less or no code and manual instrumentation where you need more control within a service.</p><p>There are many tracing solutions out there that include their own client libraries to perform instrumentation in many different languages, some of them are:</p><ul><li><strong>Jaeger</strong>: Developed by Uber and now a CNCF graduated project.</li><li><strong>Zipkin</strong>: Initially developed by Twitter based on <a href="https://static.googleusercontent.com/media/research.google.com/en//archive/papers/dapper-2010-1.pdf">Google Daper</a> paper.</li><li><strong>AWS X-Ray</strong>: AWS Distributed Tracing System.</li><li><strong>Google Cloud Trace</strong>: Distributed Tracing System for Google Cloud (Formerly Stackdriver Trace).</li><li><strong>Azure Application Insights: </strong>Feature of Azure Monitor.</li></ul><p>If you want to go for a more agnostic option, you have <a href="https://opentelemetry.io/">OpenTelemetry, </a>which provides manual and auto instrumentation SDK. It has exporters for Jaeger and Zipkin, and many vendors are working to support it on their platforms.</p><h3>Jaeger</h3><p>Jaeger is an open-source distributed tracing system initially developed by Uber. It is used for monitoring and troubleshooting microservices-based distributed systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*K3VC0EVLC2ZRHmd2VWBgnQ.png" /></figure><p>Jaeger architecture has the following components:</p><ul><li><strong>Jaeger Client</strong>: Implementation of OpenTracing API used to instrument applications.</li><li><strong>Jaeger Agent</strong>: Network daemon that listens for spans sent over UDP.</li><li><strong>Jaeger Collector</strong>: Receives the traces from the agents and runs them through a processing pipeline.</li><li><strong>Storage</strong>: Component on which the traces are stored.</li><li><strong>Jaeger Query</strong>: Service that retrieves traces from the storage and presents them on the UI.</li></ul><h3>Deploying Jaeger</h3><p>There are many strategies to deploy Jaeger in Kubernetes:</p><ul><li><strong>All in One</strong>: All Jaeger components are deployed in a single pod that uses in-memory storage.</li><li><strong>Production</strong>: The components are deployed separately. The collector and query are configured to work with Cassandra or Elasticsearch, being Elasticsearch recommended over Cassandra.</li><li><strong>Streaming</strong>: Replicates the production strategy, but it also includes the streaming capabilities of Kafka; it sits between the collector and storage to reduce the pressure on the storage under high load situations.</li></ul><p>For the sake of simplicity, we will use the all-in-one deployment strategy, using Helm with the Operator, although take into consideration that this is for testing purposes only. If you are in a production deployment, it is highly recommended to deploy with any of the remaining strategies.</p><p>To deploy, we add the helm repository and install Jaeger custom resources with the operator.</p><pre>helm repo add jaegertracing https://jaegertracing.github.io/helm-charts<br>helm repo update<br>helm install jaeger jaegertracing/jaeger-operator -n observability</pre><p>Then, create the Jaeger custom resource with kubectl apply -f</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d6662e7068275c18a5d48732a712291f/href">https://medium.com/media/d6662e7068275c18a5d48732a712291f/href</a></iframe><blockquote>On previous posts we used kubectl port forward command to expose the UIs. From now on, as a persistent alternative, we will create a NodePort service.</blockquote><p>If you are using minikube, to get your node IP run minikube ip</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e_WAHpkCQIbAB5t2gDjr5A.png" /><figcaption>Jaeger UI on port 30007</figcaption></figure><p>We are ready to start instrumenting our NodeJS services, but we need to do some OpenTelemetry preparation first.</p><h3>Configuring OpenTelemetry</h3><p>Fortunately, configuring OpenTelemetry is a straightforward task. First, you need to choose which instrumentation you are going to use and instantiate it.</p><ul><li>For manual instrumentation: use <a href="https://github.com/open-telemetry/opentelemetry-js/tree/main/packages/opentelemetry-tracing">Tracing SDK</a>.</li><li>For automatic instrumentation: use <a href="https://github.com/open-telemetry/opentelemetry-js/tree/main/packages/opentelemetry-node">Node SDK</a>. Automatic instrumentation includes OpenTelemetry API, so we also have the ability to generate custom spans anytime.</li></ul><p>Then, we need to define where we are going to send our spans. We will export the spans to Jaeger, but there are many additional <a href="https://opentelemetry.io/registry/?language=js#">exporters</a> available that you can use. Each exporter has its own configuration; Jaeger Exporter, by default, sends the spans to localhost:6832 which is the jaeger agent URL. In a production deployment, this is expected because you deploy the Jaeger agent alongside your services, but in this case, we will use the agent that is already included within the jaeger pod.</p><p>When we are done configuring the exporter, we need to add it to a span processor and initialize the provider with register(). To automate the instrumentation, you need to register the right modules; for example, if you want to instrument HTTP calls, there’s an <a href="https://github.com/open-telemetry/opentelemetry-js/tree/main/packages/opentelemetry-instrumentation-http">http module</a>; if you want to instrument the MySQL calls, there’s a <a href="https://github.com/open-telemetry/opentelemetry-js-contrib/tree/main/plugins/node/opentelemetry-instrumentation-mysql">mysql module</a>, and many <a href="https://opentelemetry.io/registry/?language=js#">more</a>. This time we will be instrumenting HTTP requests, Express Framework, and MySQL library <strong>automatically</strong>.</p><p>Lastly, your export the tracer to be used in your service.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cd7d40c9cb479c0798f97b439b4880a2/href">https://medium.com/media/cd7d40c9cb479c0798f97b439b4880a2/href</a></iframe><p>If you noticed, we have configured HttpInstrumention to ignore any requests to the /metrics endpoint. Don&#39;t forget that Prometheus pulls the metrics from it; by ignoring it, we avoid having unwanted traces from Prometheus.</p><p>Now, go ahead and test the services:</p><pre>kubectl port-forward service/hello-service-svc -n applications 8080</pre><pre>curl http://localhost:8080/sayHello/iroh #On a different terminal<br>Hey! this is iroh, here is my message to you:  It is important to draw wisdom from many different places</pre><p>Back in Jaeger UI, you should see that the services are available in the Service dropdown. Select hello-service and click on Find Traces.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*1tbw2oOcAIQZg5YFdRT2HQ.png" /><figcaption>Jaeger Seach Form</figcaption></figure><blockquote>If you can’t see the services, double check OpenTelemetry configuration, it should point to Jaeger agent service using UDP port.</blockquote><p>We should see three span colors, one per service. There will be at least two kinds of spans on each of the services, GET spans representing the requests instrumented by the HTTP instrumentation, and Middleware and Route spans instrumented by Express instrumentation. Some of the middleware spans are from the middleware functions we created to gather values for the metrics on <a href="https://jonbc.medium.com/hacking-your-way-to-observability-part-1-cf4cd42fb4dc?source=friends_link&amp;sk=84191bff6613f6d86c33daf5b585028f">part1</a> of this series.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tvMpAVRaNhx_CZg6xod2hg.png" /></figure><p>If we click on any of the spans, we should see some attributes that give us more context about that Span.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PeGuqSDtbv1_EbUA-my2Jg.png" /><figcaption>Jaeger’s HTTP span</figcaption></figure><p>Don&#39;t forget that we also instrumented MySQL. Notice that it includes very useful attributes like the database statement and user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W1DimMeNSwszfrG68VKnMw.png" /><figcaption>Jaeger’s MySQL span</figcaption></figure><p>So far, we have a lot of information available. But what if we want more fine-grained control of what happens in our service? How do we combine automatic and manual tracing?</p><h3>Context Propagation</h3><p>Before creating our custom spans, we need to answer first: <strong>How do you correlate the spans?</strong>. For the spans to be correlated, they should share some information; that information is shared through the context. The context contains information that can be passed between functions within the same process (in-process propagation) and between different processes (inter-process propagation).</p><p>Then, propagation is the mechanism by which a context is moved across different services or processes.</p><p>By now, you already saw both. Did you wonder how the spans from the three services belong to the same trace? Or how the spans from different functions in the same service can be related as parent and child?. The answer: <strong>context propagation.</strong></p><p>There are many protocols for context propagation; the most common are:</p><p><a href="https://w3c.github.io/trace-context/">W3C Trace-Context HTTP Propagator</a></p><p><a href="https://github.com/openzipkin/b3-propagation">B3 Zipkin Propagator</a></p><p>OpenTelemetry can use both, but in some cases, you may be forced to use one specifically. For example, to propagate context in Istio, you are required to use the B3 Headers.</p><h3>Manual Instrumentation</h3><p>To create your first span, you need to import the tracer from the file where we configured OpenTelemetry and call startSpanmethod. The span needs a name, and optionally you can include custom attributes and the context. In this case, we are retrieving the current context with context.active() ; if there is an active context, the span will be created within that context. After you start a span, you do some stuff with your code, and then you must end the span.</p><p>To propagate the context between different functions, either to add attributes to the span or create a child span, you need to wrap up the calls in context.with but also set the desired span in the current context with setSpan. In the code below, we create a <em>Router GET</em> span, and then we set it as the current span before calling the getPerson function. As a result, the <em>getPerson </em>span will be created as a child of <em>Router GET </em>span.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fb473497b36ba8deef549804d69c4613/href">https://medium.com/media/fb473497b36ba8deef549804d69c4613/href</a></iframe><p>If we do some tests, the result will be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RDanFnYT3XY9ld7msM0M9w.png" /></figure><p>This time, we have HTTP, MySQL, and Express Framework spans automatically generated, and custom spans manually generated from the functions in our code using OpenTelemetry.</p><h3>Conclusion</h3><p>Distributed tracing may sound scary initially, but everything will make sense as soon as you get started. Also, the OpenTelemetry community it’s making things even easier for us, and it&#39;s improving very fast. You just saw how easy it is to combine automatic and manual instrumentation to get the best of your services tracing data.</p><p>Happy Tracing 🔎!</p><h3>References</h3><ul><li><a href="https://www.jaegertracing.io/docs/1.22/">https://www.jaegertracing.io/docs/1.22/</a></li><li><a href="https://github.com/jaegertracing/helm-charts">https://github.com/jaegertracing/helm-charts</a></li><li><a href="https://opentelemetry.io/docs/js/getting_started/nodejs/">https://opentelemetry.io/docs/js/getting_started/nodejs/</a></li><li><a href="https://istio.io/latest/docs/tasks/observability/distributed-tracing/overview/">https://istio.io/latest/docs/tasks/observability/distributed-tracing/overview/</a></li></ul><p><a href="https://jonbc.medium.com/hacking-your-way-to-observability-part-1-cf4cd42fb4dc">Hacking your way to Observability — Part 1</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=56e6ddff1315" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hacking your way to Observability — Part 2]]></title>
            <link>https://jonbc.medium.com/hacking-your-way-to-observability-part-2-c38baaee6b92?source=rss-c986373fc47d------2</link>
            <guid isPermaLink="false">https://medium.com/p/c38baaee6b92</guid>
            <category><![CDATA[slack]]></category>
            <category><![CDATA[alerts]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[observability]]></category>
            <dc:creator><![CDATA[Jonathan Burmester]]></dc:creator>
            <pubDate>Wed, 12 May 2021 12:54:26 GMT</pubDate>
            <atom:updated>2021-05-20T04:20:25.808Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/875/1*5w4tlxzeuJQeXM0HpGD5-g.png" /></figure><h3>Hacking your way to Observability — Part 2 : Alerts</h3><h4>Taking advantage of metrics by sending notifications via Slack</h4><p>In my previous <a href="https://jonbc.medium.com/hacking-your-way-to-observability-part-1-cf4cd42fb4dc?source=friends_link&amp;sk=84191bff6613f6d86c33daf5b585028f">post</a>, we deployed Prometheus Operator with the Helm Chart and a set of services to demonstrate how to collect metrics using prom-client and exporters. Don’t forget that the purpose of observability is to infer the status of a system, so for the metrics to serve their purpose, they must communicate something to the right people when the values are not within the boundaries defined by the organization. Alerts can help you with that.</p><p>Alerting in Prometheus is separated into two parts. On Prometheus, you create <strong>alert rules</strong> that define the condition for alerts to be fired. When an alert is fired, Prometheus sends it to an Alert Manager, then the Alert Manager will be able to silence, group, or route the alert to send notifications to different platforms.</p><p>In this post, we will create some alert rules and then send notifications of those alerts via Slack. You can find all the resources used in this post here: <a href="https://github.com/jonathanbc92/observability-quickstart/tree/master/part2">https://github.com/jonathanbc92/observability-quickstart/tree/master/part2</a></p><h3>Alert rules</h3><p>To create an alert rule using the Prometheus Operator, we will use the PrometheusRule custom resource. PrometheusRule requires you to specify the following:</p><p><strong>Groups</strong>: A collection of alerts that are sequentially evaluated.</p><p><strong>Rules</strong>: A condition for an alert to be fired, it includes: the name, the condition, a waiting period and, labels and annotations to include additional information.</p><p>The condition expression of the alert is based on Prometheus expressions; you can use Prometheus expression builder to validate the condition before creating it. In the following example, we have a group of rules named database.rules with a single rule that fires when the mysql_up metric has been <strong>absent for at least 1 minute</strong>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/40345b1916efc8a56a26736f4889b1d4/href">https://medium.com/media/40345b1916efc8a56a26736f4889b1d4/href</a></iframe><p>Create the resource using kubectl apply -f alertrules.yml and go to the Alerts page in Prometheus.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yntyon-vOb9rQD6TPK2AXA.png" /><figcaption>Prometheus MysqlDown Alert</figcaption></figure><p>To test the rule, we need to scale down Mysql Deployment:</p><pre>kubectl scale deployment/mysql --replicas=0 -n applications</pre><p>After a minute or so, we will see the alert firing:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_H9futsDdgWYrQErR3Yo4w.png" /><figcaption>Prometheus MysqlDown Alert firing</figcaption></figure><p>We don’t have to build everything on our own; Kube Prometheus helm deployment already comes with many valuable alerts that we can check to learn about Kubernetes metrics or even create new alerts based on them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I3m84pXERb4Hpx-U6MlZqA.png" /></figure><p>Think about the following, what happens if you have a significant system outage?. Many of your applications may fail, and you will end up with many alerts firing up and many notifications being sent to your team. You can reduce the number of notifications by using Alert Manager grouping capability; it will allow you to group alerts of similar nature to be sent on a single notification.</p><p>To demonstrate that, we will create a simple alert that will fire when the deployments have less than two container replicas and will test this scenario after Slack is configured.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fb29d8d6e49dbb58359f388d0a37777e/href">https://medium.com/media/fb29d8d6e49dbb58359f388d0a37777e/href</a></iframe><h3>Preparing Slack</h3><p>Since we will be sending all our alerts to Slack, we need to do some preparation first. Let’s start by creating a Slack channel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/496/1*_zaLvBzNMik3-hNwQs5VRA.png" /><figcaption>Create channel prompt in Slack</figcaption></figure><p>Create an <a href="https://api.slack.com/apps">App </a>in your workspace. Then, enable Incoming Webhooks on your App and add a new webhook to the workspace. Don’t forget to copy the webhook URL; we will need it later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*zw-9RTLvn-pB9M5zrRq5IQ.png" /><figcaption>Enabling Incoming Webhooks on Slack</figcaption></figure><h3>Configuring AlertManager</h3><p>To configure Alert Manager, we need to create a custom resource called AlertmanagerConfig. It requires us to configure at least one receiver, which is the platform we will send the message to, and a route to any of the receivers.</p><p>The route has some configurations related to grouping.</p><ul><li>The <strong>groupBy</strong> parameter list the labels that Alert Manager will use to group the alerts in a single notification.</li><li><strong>groupWait </strong>indicates how long to wait before sending the initial notification.</li><li><strong>groupInterval </strong>indicates how long to wait before sending a notification update.</li><li><strong>repeatInterval </strong>indicates how long to wait before sending the last notification again.</li></ul><p>Slack receiver configuration has multiple parameters; you can check all of them <a href="https://prometheus.io/docs/alerting/latest/configuration/#slack_config">here</a>.</p><ul><li>Slack configuration requires the webhook URL to be presented as a secret; that secret will be referenced in the <strong>apiURL </strong>parameter.</li><li>The <strong>channel </strong>parameter specifies the slack channel that will be used to receive notifications.</li><li><strong>SendResolved</strong> is a parameter that indicates Alert Manager to send a notification when the condition for the alert is not met anymore.</li><li><strong>Title</strong> and <strong>Text</strong> are parameters to modify the format of the slack message. Both can include a reference to an existing <a href="#ad1e">template</a>.</li></ul><p>In the code below, all the alerts that fire and have the same alert name within 30 seconds will be grouped in a single Slack notification.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4860887e3d1b185a3188f8d546deab21/href">https://medium.com/media/4860887e3d1b185a3188f8d546deab21/href</a></iframe><p>Create the configuration by running kubectl apply -f alertmanagerconfig.yml.</p><p>Go to the Alert Manager status page and you will see all the routes configured. The route we configured has changed, the name is different and it has a match parameter. The <strong>match </strong>parameter specifies the labels the alert needs to have to be routed to the receiver. By default, every route you configure will be modified to include the namespace label in the match parameter, even if we include other labels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/587/1*ODPWjXwDCjz1E814LIvidg.png" /></figure><p>To see if the alert will be routed to the right receiver, we can use the <a href="https://prometheus.io/webtools/alerting/routing-tree-editor/">routing tree editor</a>. Copy the alert manager configuration on the status page, and test with the labels of your alerts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7fhItXbHjckfHwapcl4oIQ.png" /><figcaption>Routing Tree Editor</figcaption></figure><h3>Templates</h3><p>Prometheus supports defining templates for them to be used in the notifications. Templates allow us to standardize a notification message for all our alerts.</p><p>The following piece of code is an example of a template:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fa62c619482f922148af675380dd4eb9/href">https://medium.com/media/fa62c619482f922148af675380dd4eb9/href</a></iframe><p>The <em>define </em>keyword represents a chunk of code that is reusable. The code has three reusable chunks: __title, alert_title, and alert_description.</p><p><strong>__title:</strong> iterates over firing alerts and resolved alerts to print the alert name.</p><p><strong>alert_title:</strong> prints the status in uppercase between square brackets and the count of firing alerts. It also includes the content of __title if there is only one alert firing or one alert resolved.</p><p><strong>alert_description: </strong>if there is only one alert firing or one alert resolved, prints the description of the alert along with the severity and a link to the Prometheus Graph URL. If there is more than one, it prints the list of alerts.</p><p>To include a template file on a Prometheus handled by the Operator, we need to update the AlertManager custom resource; we can do that by passing <a href="https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml">custom values</a> to the helm chart. Since we only need to modify the template files, the following file will be enough.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/681835171d0e77cea80289b14c313492/href">https://medium.com/media/681835171d0e77cea80289b14c313492/href</a></iframe><p>We can update the helm deployment by running the following command.</p><pre>helm upgrade --reuse-values prometheus prometheus-community/kube-prometheus-stack -n observability -f values.yml</pre><h3>Testing Notifications</h3><p>To receive the Slack notifications we should trigger the alerts. Let’s start by scaling down MySQL to receive a single alert notification:</p><p>Scale down MySQL: kubectl scale deployment/mysql --replicas=0 -n applications</p><p>After a couple of minutes, scale up: kubectl scale deployment/mysql --replicas=1 -n applications</p><p>Go to your slack channel to see the notification.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*-hGxRtX0h7g7JarTqo0Ttg.png" /><figcaption>Prometheus Notifications on Slack</figcaption></figure><p>Lastly, we should test a grouped alert notification.</p><p>Scale down all the NodeJs services:</p><pre>kubectl scale deployment/format-service-depl --replicas=1 -n applications<br>kubectl scale deployment/hello-service-depl --replicas=1 -n applications<br>kubectl scale deployment/people-service-depl --replicas=1 -n applications</pre><p>After a couple of minutes, scale up:</p><pre>kubectl scale deployment/format-service-depl --replicas=2 -n applications<br>kubectl scale deployment/hello-service-depl --replicas=2 -n applications<br>kubectl scale deployment/people-service-depl --replicas=2 -n applications </pre><p>Go back to slack and compare the results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/651/1*pPt1qEtuSverumGT5soxSQ.png" /><figcaption>Prometheus grouped alerts on Slack</figcaption></figure><p>As you can see, the alerts are grouped under a single notification. In this case, they belong to the same group because all of them have the same alert name. You can change the group configuration by adding more labels in the future.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*14p3XMQMjfSexPsD-4CNMA.png" /><figcaption>Alerts with the same alertname</figcaption></figure><h3>Conclusion</h3><p>Notifications are very powerful to communicate to your team what’s is happening with your systems; take advantage of the templates to show more accurate messages; providing precise information helps solve issues faster. Don’t forget to group your alerts to avoid a notification hell!</p><p>Happy Alerting 🚨!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c38baaee6b92" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hacking your way to Observability — Part 1]]></title>
            <link>https://jonbc.medium.com/hacking-your-way-to-observability-part-1-cf4cd42fb4dc?source=rss-c986373fc47d------2</link>
            <guid isPermaLink="false">https://medium.com/p/cf4cd42fb4dc</guid>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[metrics]]></category>
            <dc:creator><![CDATA[Jonathan Burmester]]></dc:creator>
            <pubDate>Mon, 03 May 2021 12:41:06 GMT</pubDate>
            <atom:updated>2021-05-20T04:17:18.618Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/875/1*zN_0GWdPAPZf1_YJMN9rJA.png" /></figure><h3>Hacking your way to Observability — Part 1 : Metrics</h3><h4>Starting your journey in observability by gathering metrics with Prometheus</h4><p>Have you ever seen someone trying to find how to solve an issue and a dozen people watching or suggesting to him/her what to look for as if they were trying to solve a mystery? . These situations arise more questions than answers, and trying to get the answers can consume most of your time and in the end, it will have an impact on your company.</p><p>The “unknown” can be hard to spot or even to understand and that’s where Observability comes in, it gives you the ability to understand what is broken or what is failing and may give you an idea of the “why” so you can continuously improve.</p><h3>Observability</h3><p>Observability is defined as a measurement of how well a system’s internal state could be inferred from its outputs (the telemetry). It’s nothing new, we have been trying to understand applications by reading logs since the beginnings of programming, but it became something complex to achieve with the expansion of the cloud, containerization, microservices architectures, polyglot environments, and so on.</p><h3>The Three Pilars</h3><p>When talking about Observability you probably end up hearing something about the three pillars. These pillars as commonly referred to, are the telemetry, they are the formats that can be aggregated and analyzed to understand the state of a system.</p><ol><li>Metrics</li><li>Traces</li><li>Logs</li></ol><p>During this series, I will cover all the pillars with examples. We will start with Metrics and an Open Source software called Prometheus.</p><h3>Metrics</h3><p>A metric is a numeric value recorded over a period of time such as counters, timers, or gauges. These are very cheap to collect because numeric values can be aggregated to reduce the data transmission to the monitoring system. Normally we use metrics to determine the health of a system because are very well suited to describe a resource status. Also, you can instrument your code using libraries like <a href="https://prometheus.io/docs/instrumenting/clientlibs/">prometheus_client</a> or <a href="https://github.com/open-telemetry/opentelemetry-js/tree/main/packages/opentelemetry-exporter-prometheus">prometheus_exporter</a> to emit your custom metrics like Response Time, Active Requests, Error Counters, etc.</p><h3>Prometheus</h3><p>Prometheus is an open-source metrics-based monitoring and alerting system originally developed by SoundCloud and now hosted by the Cloud Native Computing Foundation. It collects metrics about your infrastructure and applications.</p><p>Prometheus consists of multiple components:</p><ul><li><strong>Prometheus server</strong>: component that scrapes the metrics and stores the time series data.</li><li><strong>Client libraries</strong>: libraries to generate metrics on your application.</li><li><strong>Push Gateway</strong>: is an intermediate component that allows you to push metrics from jobs that cannot be scraped.</li><li><strong>Alert Manager</strong>: Handles alerts send by Prometheus. It group, deduplicate and route the alerts to the correct receiver.</li><li><strong>Exporters</strong>: Component that fetches statistics from another system and turns those into Prometheus metrics.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/888/1*t4bkRXng71vuz8knXDNieg.png" /><figcaption>Prometheus Architecture</figcaption></figure><h3>Deployment</h3><p>You can deploy Prometheus in many ways: installing it on your infrastructure, manually on containers or Kubernetes, via helm with or without the operator on Kubernetes. To keep it simple we will deploy Prometheus using the helm chart with the Prometheus operator.</p><pre>kubectl create namespace observability<br>helm repo add prometheus-community <a href="https://prometheus-community.github.io/helm-charts">https://prometheus-community.github.io/helm-charts</a><br>helm repo update<br>helm install prometheus prometheus-community/kube-prometheus-stack --namespace observability</pre><blockquote>At the time of writing, Prometheus is at version 2.26, it has new features that vastly improve the experience. If you want to know more: <a href="https://github.com/prometheus/prometheus/releases/tag/v2.26.0">https://github.com/prometheus/prometheus/releases/tag/v2.26.0</a></blockquote><p>When the deployment finishes, you will see the following pods:</p><pre>NAME                                                     READY   STATUS    RESTARTS   AGE<br>alertmanager-prometheus-kube-prometheus-alertmanager-0   2/2     Running   0          69m<br>prometheus-grafana-5c469b9cbf-kn2zk                      2/2     Running   0          69m<br>prometheus-kube-prometheus-operator-68d59d8d59-gvkzk     1/1     Running   0          69m<br>prometheus-kube-state-metrics-685b975bb7-rhstl           1/1     Running   0          69m<br>prometheus-prometheus-kube-prometheus-prometheus-0       2/2     Running   0          69m<br>prometheus-prometheus-node-exporter-mn68b                1/1     Running   0          69m</pre><p>To access Prometheus, we will forward a local port to the Prometheus service port.</p><pre>kubectl port-forward service/prometheus-operated -n observability 9090<br>Forwarding from 127.0.0.1:9090 -&gt; 9090<br>Forwarding from [::1]:9090 -&gt; 9090</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bvBhxw4ySXiCrj4vs6preA.png" /><figcaption>Prometheus on localhost:9090</figcaption></figure><p>As you can see, we already have some metrics. This is because of <strong>kube-state-metrics </strong>pod, which is a service that listens to the Kubernetes API and generates metrics about the state of Kubernetes objects.</p><p>To interact with these metrics and aggregate them you can use the <a href="https://prometheus.io/docs/prometheus/latest/querying/basics/">PromQL</a> (Prometheus Query Language). The result of the query can be seen as tabular data or as a graph.</p><p>There are 4 types of metrics:</p><ul><li><strong>Counter</strong>: The value can only increase or be reset to 0.</li><li><strong>Gauge</strong>: The value can increase or decrease.</li><li><strong>Histogram</strong>: It exposes multiple time series — Cumulative counters for buckets, Sum of the observed values, and a general counter for the events.</li><li><strong>Summary</strong>: It similar to a histogram, it provides the sum of observed values and a counter for the events but it also provides the calculation for <strong>φ-quantiles.</strong></li></ul><blockquote>it is possible to obtain the φ-quantiles on histograms as well, using the function <a href="https://prometheus.io/docs/prometheus/latest/querying/functions/#histogram_quantile"><em>histogram_quantile</em></a>, but the calculation is done on the <strong>server-side</strong> instead of the <strong>client-side</strong> like in Summaries.</blockquote><h3>Exposing Metrics</h3><p>Ok, that’s enough of theory… how do we make Prometheus gather our applications/services metrics?</p><p>First, we need to expose some metrics, to do that we will use the nodejs Prometheus <a href="https://github.com/siimon/prom-client">client</a>. This client already comes with some default metrics, so we can get something done very fast.</p><p>To expose the metrics we just need to create the registry and tell the client to collect the default metrics.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6885a0707575b07c9ef4de3c16eb6749/href">https://medium.com/media/6885a0707575b07c9ef4de3c16eb6749/href</a></iframe><p>Then, we expose those metrics through the<strong> /metrics</strong> endpoint</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e1365be65bf3d8dc2ae135850192bbdf/href">https://medium.com/media/e1365be65bf3d8dc2ae135850192bbdf/href</a></iframe><p>And… <em>Voila!</em>,<em> </em>we have metrics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/905/1*jesh7y7YqdxY_oqJi-nSDw.png" /><figcaption>/metrics endpoint on localhost:8080</figcaption></figure><p>To create custom metrics we have to decide what type of metric we want and then configure and register it. All the metric types allow you to configure labels, which gives you more context about the metrics and work as filters too. On histograms, you can configure buckets that will group and count the values that fall within the limits of each bucket.</p><p>With the metrics configured, you need a method to observe the values. We have a middleware function per metric to do that for us.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/30555a1a0c62b1456dc9d2354631658a/href">https://medium.com/media/30555a1a0c62b1456dc9d2354631658a/href</a></iframe><p>The middleware functions will then, be loaded by express.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/895f4ba6a27521ca3288be977524ab3d/href">https://medium.com/media/895f4ba6a27521ca3288be977524ab3d/href</a></iframe><h3>Sample Application</h3><p>All the resources used in this series are available in the following github repository: <a href="https://github.com/jonathanbc92/observability-quickstart/tree/master/part1">https://github.com/jonathanbc92/observability-quickstart/tree/master/part1</a></p><p>The application we will use for this entire series will be deployed on minikube. It will be composed of 3 Nodejs containers:</p><ol><li><strong>People Service</strong>: Service with a simple query to get data from a table in MySQL.</li><li><strong>Format Service</strong>: Service that will format a message using the values from the People Service.</li><li><strong>Hello Service</strong>: Service that will call the previous services and send the final message to the user.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/279/1*aOMQueDbtS1YpPXUTi7sOQ.png" /></figure><p>All the microservices have the default and custom metrics already implemented in the <strong>metrics.js</strong> file.</p><p>To deploy everything (including Prometheus), just execute the make command.</p><p>You can test the application by port-forwarding a port to the hello service port 8080 and then make a request to localhost:8080/sayHello/{name}</p><pre>curl <a href="http://localhost:8081/sayHello/iroh">http://localhost:8081/sayHello/iroh</a><br>Hey! this is iroh, here is my message to you:  It is important to draw wisdom from many different places</pre><h3>Service Monitoring</h3><p>Before explaining how to collect the metrics, it is important to understand that deploying Prometheus with or without the Operator changes the configuration required to identify the target services.</p><p>On a Prometheus that is deployed without the Operator, you configure the targets on a configuration file.</p><pre>scrape_configs:<br>  - job_name:       &#39;node&#39;</pre><pre>    # Override the global default and scrape targets from this job every 5 seconds.<br>    scrape_interval: 5s</pre><pre>    static_configs:<br>      - targets: [&#39;localhost:8080&#39;, &#39;localhost:8081&#39;]<br>        labels:<br>          group: &#39;production&#39;</pre><pre>      - targets: [&#39;localhost:8082&#39;]<br>        labels:<br>          group: &#39;canary&#39;</pre><p>When you deploy with the Operator you need to create some custom resources called ServiceMonitors. The Service Monitor specifies how the Kubernetes services are monitored and it will generate the scrape configuration automatically.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*QaGmsz7yJwij0zXFCY9IoQ.png" /><figcaption>Service Monitor Diagram</figcaption></figure><p>The Service Monitor custom resource will match the services with the labels specified in the <strong>matchLabels</strong> parameter within the namespaces specified in the <strong>namespaceSelector</strong>. A service can expose many ports, so it is important to specify the port and path from where the metrics will be collected.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/02b278158f71972e35b8e71862a5c582/href">https://medium.com/media/02b278158f71972e35b8e71862a5c582/href</a></iframe><p>Check the labels included on the ServiceMonitor; it is important to mention that Prometheus won’t select the Service Monitors that don’t have the required labels. Which labels are these? If we check the Prometheus custom resource specs, we will see the following:</p><pre>kubectl get prometheuses prometheus-kube-prometheus-prometheus -n observability -o yaml</pre><pre>alerting:<br>    alertmanagers:<br>    - apiVersion: v2<br>      name: prometheus-kube-prometheus-alertmanager<br>      namespace: observability<br>      pathPrefix: /<br>      port: web<br>  enableAdminAPI: false<br>  externalUrl: <a href="http://prometheus-kube-prometheus-prometheus.observability:9090/">http://prometheus-kube-prometheus-prometheus.observability:9090</a><br>  image: quay.io/prometheus/prometheus:v2.26.0<br>  listenLocal: false<br>  logFormat: logfmt<br>  logLevel: info<br>  paused: false<br>  podMonitorNamespaceSelector: {}<br>  podMonitorSelector:<br>    matchLabels:<br>      release: prometheus<br>  portName: web<br>  probeNamespaceSelector: {}<br>  probeSelector:<br>    matchLabels:<br>      release: prometheus<br>  replicas: 1<br>  retention: 10d<br>  routePrefix: /<br>  ruleNamespaceSelector: {}<br>  ruleSelector:<br>    matchLabels:<br>      app: kube-prometheus-stack<br>      release: prometheus<br>  securityContext:<br>    fsGroup: 2000<br>    runAsGroup: 2000<br>    runAsNonRoot: true<br>    runAsUser: 1000<br>  serviceAccountName: prometheus-kube-prometheus-prometheus<br>  serviceMonitorNamespaceSelector: {}<br>  <strong>serviceMonitorSelector:<br>    matchLabels:<br>      release: prometheus</strong><br>  shards: 1<br>  version: v2.26.0</pre><p>The serviceMonitorSelector parameters tells us which are the required labels for our ServiceMonitor to be able to be selected by Prometheus.</p><p>Now we just need to create the ServiceMonitor:</p><pre>kubectl apply -f k8s/config/prometheus/servicemonitoring.yml</pre><p>If everything worked correctly, on Prometheus’s Status menu, go to Targets. We should see all the pods related to the monitored services.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JRd0kXplcH6VDENwObIwGw.png" /><figcaption>Endpoints beign scraped by Prometheus</figcaption></figure><p>As soon as new metrics are available, you will be able to query them on the expression browser.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8gba0sxCe7ctCVvnct8m3A.png" /><figcaption>NodeJS heap size used on Prometheus</figcaption></figure><h3>Exporters</h3><p>We already covered the NodeJS services, but what happens with MySQL? It is important to collect metrics from the database too. To export existing metrics from 3rd party system as Prometheus metrics we can use exporters.</p><blockquote>There are many exporters available, you can see some of them here: <a href="https://prometheus.io/docs/instrumenting/exporters/">https://prometheus.io/docs/instrumenting/exporters/</a></blockquote><p>We will use the <a href="https://github.com/prometheus/mysqld_exporter">official </a>MySQL exporter image and it will run alongside the MySQL container. The image requires an environment variable DATA_SOURCE_NAME which value will be taken from a secret. Since all the containers within a pod share the network we can use <strong>localhost </strong>to connect to MySQL.</p><pre>kubectl create secret generic mysql-credentials --from-literal=username=root --from-literal=password=admin --from-literal=datasource=&quot;exporter:exportpass@(localhost:3306)/tutorial&quot; -n applications</pre><p>After creating the resources on Kubernetes, in Prometheus Targets we should see the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JXiIoct1Youkh8fDEFJ-GQ.png" /></figure><p>If we check Prometheus expression browser, we will see that we have MySQL metrics available.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/810/1*ysWSzP_5ahpJTwNOvmSSDQ.png" /><figcaption>Mysql Metrics available on Prometheus</figcaption></figure><p>Now we can use the rate function to see the average rate of how many select commands has increased over a period. But these values are only the selects that are being executed by the exporter to gather the metrics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2J77dKCBgIGM83TeaNiEMg.png" /></figure><p>If we do the same with delete commands we will see that it’s 0 because there hasn’t been any delete command executed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WxcLDiE1-fhIYl1qpsmkTg.png" /></figure><p>With the help of Apache Benchmark, we can do some load testing to create an spike.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/1*KGodXOncoMr4YMDsLarQ8A.png" /><figcaption>Running 20k requests with AB</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9YRcdh9ejND6WgviFsmn1g.png" /><figcaption>Select command spike in Prometheus</figcaption></figure><h3>Conclusion</h3><p>As you can see, it’s really easy to start with Prometheus using the helm chart, with a couple of minutes and a minikube you are ready to go. Even though it may be interesting to see the metrics the first time, metrics by themselves don’t do much. The next time we’ll see alerts.</p><p>Happy conding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cf4cd42fb4dc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hello Terraform CDK!]]></title>
            <link>https://jonbc.medium.com/hello-terraform-cdk-47bc8bf9fdcd?source=rss-c986373fc47d------2</link>
            <guid isPermaLink="false">https://medium.com/p/47bc8bf9fdcd</guid>
            <category><![CDATA[ibm-cloud]]></category>
            <category><![CDATA[cdk]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[travis-ci]]></category>
            <dc:creator><![CDATA[Jonathan Burmester]]></dc:creator>
            <pubDate>Mon, 22 Feb 2021 11:48:22 GMT</pubDate>
            <atom:updated>2021-02-22T11:48:22.786Z</atom:updated>
            <content:encoded><![CDATA[<p>It has been a long time since the initial release of Terraform in 2014, we have seen new providers, changes in the CLI and on the functions as well, but I think we are yet to see one of its biggest improvements, the Cloud Development Kit. I won’t assume that you are familiar with Terraform or even with Infrastructure as Code, so I’ll explain it very briefly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*VEloMh8Asn81UZfSZDMvkg.png" /></figure><h3>Infrastructure as Code</h3><p>Working with infrastructure was always a complicated process, people used to put servers in place, configure them and setup the operative system to finally install the dependencies and get an application up and running. Even with documentation and experience, the process was never without its problems and as a result, provisioning was inconsistent, slow, and prone to human error.</p><p>Infrastructure as code came with the idea to define all the configurations in files, in a way that we could have easy-to-read, consistent, and reusable templates. There are multiple solutions that can help us with this, some of them are Azure Resource Manager, AWS Cloudformation, Terraform amongst many others.</p><p>Terraform configuration files are written in HashiCorp Configuration Language (HCL). In those configuration files, you declare the resources you are going to manage within your provider.</p><p>To give you an idea of how is it done, here you have an example of how you declare your resources.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ee21c535c68f00a0ad3a4ebd3099af65/href">https://medium.com/media/ee21c535c68f00a0ad3a4ebd3099af65/href</a></iframe><p>With something as simple as that you can create a VPC and a Subnet, seems easy right? of course you can improve it by adding variables, more resources and even include modules, but that’s not on the scope of this article. You can learn more from Terraform here: <a href="https://learn.hashicorp.com/terraform">https://learn.hashicorp.com/terraform</a>.</p><p>Since last year we have a new alternative to work with Terraform and that’s the Cloud Development Kit (CDK). HashiCorp collaborated with the AWS CDK team to support the ability to use general-purpose programming languages to declare our resources and then generate the Terraform configuration files needed for provisioning.</p><p>In this article, I’ll show you how to use the Terraform CDK with Typescript to create resources on IBM Cloud to finally configure a CI/CD pipeline with Travis CI.</p><blockquote><strong>Disclaimer: </strong>We will be using IBM Cloud, but to show you how to include a backend I won’t be using IBM Schematics — which is the IBM IaC service based on Terraform. If you are following this article, don’t forget to delete everything you create with this article.</blockquote><h3>Creating the Project</h3><p>First, you need to install the CDK CLI to be able to run all the commands, then run <strong>cdktf init</strong> to initialize the project. The init command will create all the files you need to start writing some code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/1*Fcm9Exyi3_Mz-weX-adDcA.png" /></figure><p>Before we continue, this is what we are going to create with the CDK:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/683/1*NouPXfrqBv-Vp62XS9eW5g.png" /></figure><ul><li>1 VPC.</li><li>2 Subnets.</li><li>2 Public Gateways.</li><li>1 Public Load Balancer.</li><li>1 Security Group</li><li>1 Instance group to handle our instances.</li></ul><h3>Setup</h3><p>In the cdk-tutorial folder, open the<strong> cdktf.json</strong>. In this file, we are going to declare the providers we are going to use. In this case IBM Cloud provider.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/754d77f1078117f9bed991d40a88fa86/href">https://medium.com/media/754d77f1078117f9bed991d40a88fa86/href</a></iframe><p>Then, run the <strong>cdktf get</strong> command to download all the dependencies to use IBM Cloud. All the dependencies will be located in the <strong>.gen</strong> folder.</p><h3>Configuring the Provider and creating your first resource</h3><p>With the dependencies in place, your IDE autocomplete feature will make your life easier than ever before. Configurations in CDK are represented as classes, so all we have to do is create instances of the resources we want to work with, inside the MyStack class. Open the <strong>main.ts</strong> file and let’s declare the provider and a VPC.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2b26420424d1542fc94e38273f53811a/href">https://medium.com/media/2b26420424d1542fc94e38273f53811a/href</a></iframe><p>If we compare the typescript code, we’ll see is very similar to what we do with HCL.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a46edc79c1062f3bcdce87d38f1e8082/href">https://medium.com/media/a46edc79c1062f3bcdce87d38f1e8082/href</a></iframe><h3>Using Variables</h3><p>As in HCL, we can use variables here as well, we just need to create a new instance of the class TerraformVariable. This variable will be user in all the resources as a prefix for the name.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/698031cc90c3be95652856d8a7d96cb1/href">https://medium.com/media/698031cc90c3be95652856d8a7d96cb1/href</a></iframe><h3>Referencing Resources</h3><p>To create the public gateways we need the ID of the previously declared VPC, so we will create two instances of the public gateway and assign the ID of the VPC.</p><p>It’s important to mention that we need to assign the instance of the class to a constant only when a reference is needed, otherwise cdk cli will take it as an error.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/99272848f8550426c656f36dbd98d736/href">https://medium.com/media/99272848f8550426c656f36dbd98d736/href</a></iframe><h3>Explicit Dependency</h3><p>We won’t be using the default Address Prefixes for this article, instead we will create two, one for each zone. But, since there is no implicit dependency between the subnet and address prefixes, we need to put it explicitly using the dependsOn property, that way we can be sure that Subnets are created after the Address Prefixes.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/219c917669a7d32a3f6161fdf7be7314/href">https://medium.com/media/219c917669a7d32a3f6161fdf7be7314/href</a></iframe><h3>Data Sources</h3><p>Sometimes, you have resources already created on your Cloud Service Provider account that you might want to reference. We will create an SSH Key required by the instance template we will declare later.</p><p>On the IBM Cloud Console menu click on <strong>VPC Infrastructure — SSH Keys</strong>, then click on the Create button, name your key “<em>tutorial” </em>and complete the rest of the form.</p><p>Data Source declaration is similar to resource declaration, you just need to create an instance of the data source class.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/182c75a9e8ac63760100ef4f283f279d/href">https://medium.com/media/182c75a9e8ac63760100ef4f283f279d/href</a></iframe><h3>Load Balancer and Security Groups</h3><p>For the Load Balancer and Security Groups there’s nothing new to explain in terms of CDK. Our security group will allow outgoing traffic to the ports 80 (HTTP) , 443 (HTTPS) and 53 (DNS).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/91fef1481de617ad16241010fa823a7b/href">https://medium.com/media/91fef1481de617ad16241010fa823a7b/href</a></iframe><h3>Built in Functions?</h3><p>We want to install nginx on instances at the time they launch. The instance templates include a property called User Data which allows us to execute some commands at launch time.</p><p>HCL has some built-in functions that allows us to perform different operations with files, strings, numbers, and so on. Normally we would have a script file that we could use as user-data, and to read the contents of that file we end up using the <a href="https://www.terraform.io/docs/language/functions/file.html">file</a> function.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/54dbfa808aa91f02b9e198717b29e7ed/href">https://medium.com/media/54dbfa808aa91f02b9e198717b29e7ed/href</a></iframe><p>At the time of writing, built-in functions have not been abstracted to the CDK, so we can’t use them, but we can rely on libraries of the language of your choice.</p><p>To read the contents of a script we will use the nodejs filesystem library.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ead7cc5460f4880887a352a56a1b4e8d/href">https://medium.com/media/ead7cc5460f4880887a352a56a1b4e8d/href</a></iframe><h3>Output</h3><p>When everything is created, we might need an easy way to get the load balancer url. We can do that with output values.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c86cb7ee237b2cc9d71ff2f8229fc4d4/href">https://medium.com/media/c86cb7ee237b2cc9d71ff2f8229fc4d4/href</a></iframe><h3>Where’s the backend?</h3><p>We are almost ready, we just need a backend to keep the current state of the resources handled by Terraform. For demonstration purposes, we will create an etcd database.</p><p>On your IBM Cloud account, go to the top search bar and enter <em>databases for etcd </em>and click on the result, then click on the create button. Name your database “<em>Terraform-etcd” </em>and complete the rest of the form<em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qm9QrcuN_xj1n4Bs_6L-UQ.png" /></figure><p>Click on the create button and wait until provisioning is done. Now click on your database, then go to the service credentials, click on New Credential and finally click on Add.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UYDnJ2hDoBjFgJb-xfrpfw.png" /></figure><p>On your new credential take note of the endpoint, username and password, we will use those later.</p><p>Unlike resources and data sources, the backend is not declared inside the MyStack class. The Etcd backend requires to include the endpoints, password and, username of your database, but of course, we don’t want to put it on plain text. The good news is we can use environment variables.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d7d291436448fa1cc71bd943ce77784d/href">https://medium.com/media/d7d291436448fa1cc71bd943ce77784d/href</a></iframe><h3>Deploy</h3><p>Before deploying you must consider the following:</p><p>For Terraform to be able to create resources on your account it needs an API Key.</p><ul><li>In the IBM Cloud console, go to Manage — Access (IAM) — API keys.</li><li>Click Create an IBM Cloud API key.</li><li>Enter a name and description for your API key.</li><li>Click Create.</li><li>Take note of the API Key.</li></ul><p>Finally, set all the required environment variables</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c35fb36d8ebe545f83a7eeb0006339f4/href">https://medium.com/media/c35fb36d8ebe545f83a7eeb0006339f4/href</a></iframe><p>Configuring etcdv3 as backend requires you to enter the cacertpath, certpath and keypath.</p><ul><li>To obtain the CA certificate we can use ibmcloud CLI.</li><li>To obtain the Certificate and its key we can use openssl to generate a self signed certificate.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/54a2dbcd0e91895ca732556e9c916aa3/href">https://medium.com/media/54a2dbcd0e91895ca732556e9c916aa3/href</a></iframe><p>There are two choices to deploy when it comes to CDK:</p><ul><li>We can deploy with a single command using <strong>cdktf deploy</strong>.</li><li>We can run the workflow in smaller steps by running <strong>cdktf synth</strong> to generate the configuration file in JSON format, which then we can use to run terraform commands.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/276/1*wd_kGgkcuCS0lre5fB3BeQ.png" /></figure><h3>Travis CI</h3><p>What if... we go to the next step and set up a pipeline? Here is Travis to help us.</p><p>If you have been following this article, you already have the project almost complete, you are just missing the travis configuration file. If this is your case:</p><ul><li>Create a GitHub repository.</li><li><a href="https://docs.github.com/en/github/importing-your-projects-to-github/adding-an-existing-project-to-github-using-the-command-line">Initialize a local git repository</a> on the cdk-tutorial folder and push its contents to the repository.</li></ul><p>Otherwise, you can import my repository <a href="https://github.com/jonathanbc92/TerraformCDKTutorial">https://github.com/jonathanbc92/TerraformCDKTutorial</a> in your account.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/155/1*jBT7B1LtlmlvHT0DdUEssw.png" /></figure><p>If you don’t have a travis-ci account, you can sign up here: <a href="https://travis-ci.com/">https://travis-ci.com/</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/689/1*nUp8UXb-1gJtJ-4o6KyaUw.png" /></figure><p>When we sign up, we’ll see that Travis CI asks for authorization on our GitHub account.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/532/1*3Skd_jz8LdoqUhjLIru16A.png" /></figure><p>Then, it will ask us to activate all our repositories.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1013/1*25OKg4E-BESjd4ls5nMStA.png" /></figure><p>if you don’t want to have all of them activated, go to your account settings and click on Manage repositories on GiHub.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/711/1*1WopTdoykKvt043gFMH8Hg.png" /></figure><p>On the Repository access section select “Only select repositories” and then select the repositories you want to work with.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*fxRU3Tbf9BAjsIJ340uFhA.png" /></figure><p>Back on the dashboard, let’s see the repository settings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eMN7M6i0cptxQAUAbZbq-g.png" /></figure><p>We need to add the environment variables required by this article.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/961/1*W4KDWh1NZSYflT-sl8UByA.png" /></figure><p>For travis to work with your repository you need to create the build configuration file <strong>.travis.yml</strong>. You can learn more about it here: <a href="https://docs.travis-ci.com/user/customizing-the-build/">https://docs.travis-ci.com/user/customizing-the-build/</a></p><p>In this case, we are setting up an ubuntu 18.04 (bionic) environment to run our build. Also, we set three environment variables, one for the Terraform version, a second one for the terraform apply command options and the last one to set the base_name variable we declared before.</p><p>The branch configuration will tell our build to run only on the master branch.</p><p>We also define the commands that must be executed before the jobs:</p><ul><li>Download and install Terraform.</li><li>Install npm.</li><li>Download and install ibmcloud CLI.</li><li>Login and install the Cloud Databases (cdb)plugin.</li><li>Download the etcd CA certificate and generate our self signed certificate to able to communicate with the etcd backend.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/203601f66f8a91d49d7b33f4e93ec042/href">https://medium.com/media/203601f66f8a91d49d7b33f4e93ec042/href</a></iframe><p>Now we define two jobs:</p><ul><li>The first one, called terraform plan will execute only on pull requests. Let’s say that you work on a development branch, when you create the pull request to merge development branch into the master branch, terraform plan will execute, letting you see the changes in your resources so you can validate if those meet your expectations.</li><li>The second one, called terraform apply will execute on push to master branch, this includes approved pull requests, and direct commits to master branch. In case your expectations were met and you decide to merge development into the master branch, this job will execute applying your changes.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cfc86e71577aca8493df34efee77be23/href">https://medium.com/media/cfc86e71577aca8493df34efee77be23/href</a></iframe><p>To test the pipeline, we just need to create a new branch on the GitHub repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/305/1*VxysHeUHmFLNO8dbPp_Grw.png" /></figure><p>Then, on the <strong>main.ts</strong> file add a comment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ojAm8RgrWinmX7KL6LrvQQ.png" /></figure><p>And commit the changes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wy9DUCl5MvnLOmPg-13vRA.png" /></figure><p>On the development branch click on Compare &amp; pull request to create a new pull request.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W14D7p45Lr6thclqJK1XUQ.png" /></figure><p>Enter some description and click on Create pull request.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/861/1*FRoj-MTkfndUc8Mh3Fr5pQ.png" /></figure><p>If we go back to your travis dashboard, we will see a new build is being created. Wait until the build finishes and then check the log, we should see at the output of the <strong>terraform plan</strong> command at the bottom.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*vgRt_dHmVVuMbRNPNiY9Zw.png" /></figure><p>Back on the pull request window, we should see the build passed. Now we can click on Merge pull request.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/927/1*ydLJ2FHWdxBITOq8hOqEYA.png" /></figure><p>This will trigger the terraform apply job, that will create our resources. On the Travis dashboard we should see another build being created, wait until it finishes and check the logs.</p><p>At the bottom, you will find our output value with the URL of the load balancer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/491/1*MrbJrohLTIyXoYviKM0ykQ.png" /></figure><p>Wait a couple of minutes until the instances are ready and then open the URL on your favorite browser. You should see the nginx welcome page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*Io5ytqh9Oo_pRLk9jvIq-A.png" /></figure><h3>Summary</h3><p>In this post, we walked through the Terraform CDK to create some resources on IBM Cloud and defined a CI/CD pipeline with TravisCI that has two jobs: the first one for pull requests and the second one for pushing directly to master. Terraform CDK is still on alpha, so it’s probably going to be changing quite often but it is a good start.</p><p>I hope you can keep building some great stuff! and give feedback to the community as well.</p><p>Jonathan</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=47bc8bf9fdcd" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>