<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Rafael Durelli on Medium]]></title>
        <description><![CDATA[Stories by Rafael Durelli on Medium]]></description>
        <link>https://medium.com/@RafaelDurelli?source=rss-ca3e83e2e680------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*a7WesDl8oNkswlWt.jpg</url>
            <title>Stories by Rafael Durelli on Medium</title>
            <link>https://medium.com/@RafaelDurelli?source=rss-ca3e83e2e680------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 08:56:05 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@RafaelDurelli/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[End-to-End Observability with Kubernetes — Part 3: Tracing distribuído com Grafana Alloy + Loki]]></title>
            <link>https://medium.com/@RafaelDurelli/end-to-end-observability-with-kubernetes-part-3-tracing-distribu%C3%ADdo-com-grafana-alloy-loki-0338748d6fe2?source=rss-ca3e83e2e680------2</link>
            <guid isPermaLink="false">https://medium.com/p/0338748d6fe2</guid>
            <category><![CDATA[grafana-alloy]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[grafana-loki]]></category>
            <dc:creator><![CDATA[Rafael Durelli]]></dc:creator>
            <pubDate>Wed, 11 Jun 2025 13:20:50 GMT</pubDate>
            <atom:updated>2025-06-11T13:34:16.208Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>End-to-End Observability with Kubernetes — Part 3: Distributed Tracing with Grafana Alloy + Loki</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yVRsekVdDY7oQGQ3Oam4aw.png" /></figure><p>In <a href="https://medium.com/@RafaelDurelli/end-to-end-observability-with-kubernetes-prometheus-grafana-plus-a-spring-boot-bonus-6e97f220da14"><strong>Part 1</strong></a>, we built a local Kubernetes cluster, deployed Prometheus and Grafana using Helm, and instrumented a Spring Boot service to expose metrics.</p><p>In <a href="https://medium.com/@RafaelDurelli/end-to-end-observability-with-kubernetes-parte-2-tracing-distribu%C3%ADdo-com-grafana-alloy-tempo-87da3a467927"><strong>Part 2</strong></a>, we introduced <strong>Grafana Tempo</strong> for distributed tracing and configured <strong>Grafana Alloy</strong> as a unified OpenTelemetry agent for traces.</p><p>Now, in <strong>Part 3</strong>, we’re closing the loop of full observability by integrating <strong>Grafana Alloy with Loki</strong>. This allows us to <strong>collect logs with embedded trace context</strong> and correlate them with distributed traces and metrics — building a truly <strong>end-to-end observability pipeline</strong> on Kubernetes.</p><h3>⚙️ Overview of the Observability Stack</h3><p>Here’s what our current stack looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YA62shUCdGSjrZudAqN2Pw.png" /></figure><h3>🔧 Configuring Grafana Alloy (Traces + Metrics + Logs)</h3><blockquote><em>⚠️ </em><strong><em>Before we begin: </em></strong><em>This section assumes that </em><strong><em>Grafana Alloy is already installed</em></strong><em> and running inside your Kubernetes cluster. If you’re not there yet, check out:</em></blockquote><blockquote><em>🔹 </em><a href="https://medium.com/@RafaelDurelli/end-to-end-observability-with-kubernetes-prometheus-grafana-plus-a-spring-boot-bonus-6e97f220da14"><strong><em>Part 1</em></strong></a><em> — where we installed Prometheus and Grafana via Helm and built our observability foundation</em></blockquote><blockquote><em>🔹 </em><a href="https://medium.com/@RafaelDurelli/end-to-end-observability-with-kubernetes-parte-2-tracing-distribu%C3%ADdo-com-grafana-alloy-tempo-87da3a467927"><strong><em>Part 2</em></strong></a><em> — where we deployed Grafana Alloy and integrated distributed tracing with Grafana Tempo</em></blockquote><blockquote>If you’re not there yet, check out:</blockquote><blockquote>🔹 Part 1 — where we installed Prometheus and Grafana via Helm and built our observability foundation<br>🔹 Part 2 — where we deployed Grafana Alloy and integrated distributed tracing with Grafana Tempo</blockquote><p>Now let’s extend that setup to include <strong>log collection and forwarding to Grafana Loki</strong>, while also collecting metrics and traces through the same agent.</p><h3>📦 Step 1: Installing Grafana Loki</h3><p>We’ll start by installing <strong>Grafana Loki</strong> in distributed mode using Helm:</p><pre>helm upgrade --install loki grafana/loki-distributed \<br>  --namespace monitoring \<br>  --values loki-custom.yaml \<br>  --wait</pre><p>Ensure your loki-custom.yaml configures the distributor correctly and exposes port 3100, which will be used by Alloy to push logs.</p><p>After deploying, the Loki endpoint should look like:</p><pre>http://loki-loki-distributed-distributor.monitoring.svc.cluster.local:3100/loki/api/v1/push</pre><h3>🥈 Step 2: Extending Alloy to Handle Logs</h3><p>Now, take a look at the <strong>full Grafana Alloy configuration file</strong> we used in our setup.<br>👉 However, for the purpose of this section, <strong>we’ll focus only on the logs pipeline</strong>.</p><p>This is the relevant portion:</p><pre>loki.write &quot;default&quot; {<br>  endpoint {<br>    url = &quot;http://loki-loki-distributed-distributor.monitoring.svc.cluster.local:3100/loki/api/v1/push&quot;<br>  }<br>}<br><br>discovery.kubernetes &quot;pod&quot; {<br>  role = &quot;pod&quot;<br>}<br><br>discovery.relabel &quot;pod_logs&quot; {<br>  targets = discovery.kubernetes.pod.targets<br><br>  rule {<br>    source_labels = [&quot;__meta_kubernetes_namespace&quot;]<br>    action = &quot;replace&quot;<br>    target_label = &quot;namespace&quot;<br>  }<br><br>  rule {<br>    source_labels = [&quot;__meta_kubernetes_pod_container_name&quot;]<br>    action = &quot;replace&quot;<br>    target_label = &quot;container&quot;<br>  }<br><br>  rule {<br>    source_labels = [&quot;__meta_kubernetes_pod_uid&quot;, &quot;__meta_kubernetes_pod_container_name&quot;]<br>    action = &quot;replace&quot;<br>    target_label = &quot;__path__&quot;<br>    separator = &quot;/&quot;<br>    replacement = &quot;/var/log/pods/*$1/*.log&quot;<br>  }<br>}<br><br>loki.source.kubernetes &quot;pod_logs&quot; {<br>  targets    = discovery.relabel.pod_logs.output<br>  forward_to = [loki.process.pod_logs.receiver]<br>}<br><br>loki.process &quot;pod_logs&quot; {<br>  stage.static_labels {<br>    values = {<br>      cluster = sys.env(&quot;CLUSTER&quot;),<br>    }<br>  }<br>  forward_to = [loki.write.default.receiver]<br>}</pre><p>This is the complete alloy configuration:</p><pre>alloy:<br>  configMap:<br>    content: |<br>      // ===== TRACES CONFIGURATION =====<br><br>      otelcol.receiver.otlp &quot;otlp_receiver&quot; {<br>        grpc {<br>          endpoint = &quot;0.0.0.0:4317&quot;<br>        }<br>        http {<br>          endpoint = &quot;0.0.0.0:4318&quot;<br>          cors {<br>            allowed_origins = [&quot;*&quot;]<br>            allowed_headers = [&quot;*&quot;]<br>            max_age         = 600<br>          }<br>        }<br>        output {<br>          traces = [otelcol.exporter.otlp.tempo.input]<br>        }<br>      }<br><br>      otelcol.exporter.otlp &quot;tempo&quot; {<br>        client {<br>          endpoint = &quot;tempo:4317&quot;<br>          tls {<br>            insecure = true<br>          }<br>        }<br>      }<br><br>      // ===== METRICS CONFIGURATION =====<br><br>      prometheus.scrape &quot;springboot&quot; {<br>        scrape_interval = &quot;15s&quot;<br>        forward_to = [prometheus.remote_write.mimir.receiver]<br>        targets = [<br>          {<br>            __address__   = &quot;spring-boot.spring-boot.svc.cluster.local:8080&quot;,<br>            job           = &quot;springboot-app&quot;,<br>            metrics_path  = &quot;/actuator/prometheus&quot;,<br>          },<br>        ]<br>      }<br><br>      prometheus.remote_write &quot;mimir&quot; {<br>        endpoint {<br>          url = &quot;http://mimir-nginx.monitoring.svc.cluster.local/api/v1/push&quot;<br>        }<br>      }<br><br>      // ===== LOGS CONFIGURATION =====<br><br>      loki.write &quot;default&quot; {<br>        endpoint {<br>          url = &quot;http://loki-loki-distributed-distributor.monitoring.svc.cluster.local:3100/loki/api/v1/push&quot;<br>        }<br>      }<br><br>      discovery.kubernetes &quot;pod&quot; {<br>        role = &quot;pod&quot;<br>      }<br><br>      discovery.relabel &quot;pod_logs&quot; {<br>        targets = discovery.kubernetes.pod.targets<br><br>        rule {<br>          source_labels = [&quot;__meta_kubernetes_namespace&quot;]<br>          action = &quot;replace&quot;<br>          target_label = &quot;namespace&quot;<br>        }<br><br>        rule {<br>          source_labels = [&quot;__meta_kubernetes_pod_name&quot;]<br>          action = &quot;replace&quot;<br>          target_label = &quot;pod&quot;<br>        }<br><br>        rule {<br>          source_labels = [&quot;__meta_kubernetes_pod_container_name&quot;]<br>          action = &quot;replace&quot;<br>          target_label = &quot;container&quot;<br>        }<br><br>        rule {<br>          source_labels = [&quot;__meta_kubernetes_pod_label_app_kubernetes_io_name&quot;]<br>          action = &quot;replace&quot;<br>          target_label = &quot;app&quot;<br>        }<br><br>        rule {<br>          source_labels = [&quot;__meta_kubernetes_namespace&quot;, &quot;__meta_kubernetes_pod_container_name&quot;]<br>          action = &quot;replace&quot;<br>          target_label = &quot;job&quot;<br>          separator = &quot;/&quot;<br>          replacement = &quot;$1&quot;<br>        }<br><br>        rule {<br>          source_labels = [&quot;__meta_kubernetes_pod_uid&quot;, &quot;__meta_kubernetes_pod_container_name&quot;]<br>          action = &quot;replace&quot;<br>          target_label = &quot;__path__&quot;<br>          separator = &quot;/&quot;<br>          replacement = &quot;/var/log/pods/*$1/*.log&quot;<br>        }<br><br>        rule {<br>          source_labels = [&quot;__meta_kubernetes_pod_container_id&quot;]<br>          action = &quot;replace&quot;<br>          target_label = &quot;container_runtime&quot;<br>          regex = &quot;^(\\S+):\\/\\/.+$&quot;<br>          replacement = &quot;$1&quot;<br>        }<br>      }<br><br>      loki.source.kubernetes &quot;pod_logs&quot; {<br>        targets    = discovery.relabel.pod_logs.output<br>        forward_to = [loki.process.pod_logs.receiver]<br>      }<br><br>      loki.process &quot;pod_logs&quot; {<br>        stage.static_labels {<br>          values = {<br>            cluster = sys.env(&quot;CLUSTER&quot;),<br>          }<br>        }<br>        forward_to = [loki.write.default.receiver]<br>      }<br><br>  extraPorts:<br>    - name: otlp-grpc<br>      port: 4317<br>      targetPort: 4317<br>      protocol: TCP<br>    - name: otlp-http<br>      port: 4318<br>      targetPort: 4318<br>      protocol: TCP<br>  <br>  mounts:<br>    # -- Mount /var/log from the host into the container for log collection.<br>    varlog: true<br>    # -- Mount /var/lib/docker/containers from the host into the container for log<br>    # collection.<br>    dockercontainers: true<br>  <br>  serviceMonitor: <br>    enabled: true<br>    additionalLabels: <br>      release: prometheus # ⚠️ esse label TEM QUE BATER com o label `release` do Prometheus Operator</pre><h3>✅ Confirming Log Collection</h3><p>After installing and configuring Alloy with the log pipeline, you should see <strong>logs in the Alloy pod</strong> confirming that it’s discovering containers and pushing logs to <strong>Grafana Loki</strong>.</p><p>Example log output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FrHQVEbqslq0LW47TIeuYQ.png" /></figure><p>If no logs appear, ensure /var/log and /var/lib/docker/containers are mounted correctly, and your Kubernetes pods are writing to stdout.</p><h3>🥚 Creating a Simple Go Application to Emit Logs</h3><p>Now that our observability pipeline is in place, it’s time to test it with a real workload.</p><p>We will:</p><ol><li>✍️ Write a Go application</li><li>💪 Create a Dockerfile</li><li>⚘️ Deploy it to Kubernetes using Helm</li></ol><h3>✍️ Step 1: Writing the Go App</h3><pre>package main<br><br>import (<br> &quot;log&quot;<br> &quot;time&quot;<br>)<br><br>func main() {<br> for {<br>  log.Println(&quot;Hello from Golang! This log will be scraped by Grafana Alloy and sent to Loki.&quot;)<br>  time.Sleep(2 * time.Second)<br> }<br>}</pre><p>Save this file as main.go.</p><h3>🚧 Step 2: Creating the Dockerfile</h3><p>We generated our Dockerfile using docker init, which helps scaffold a clean production image. Here is the final result:</p><pre># syntax=docker/dockerfile:1<br><br># Comments are provided throughout this file to help you get started.<br># If you need more help, visit the Dockerfile reference guide at<br># https://docs.docker.com/go/dockerfile-reference/<br><br># Want to help us make this template better? Share your feedback here: https://forms.gle/ybq9Krt8jtBL3iCk7<br><br>################################################################################<br># Create a stage for building the application.<br>ARG GO_VERSION=1.23.0<br>FROM --platform=$BUILDPLATFORM golang:${GO_VERSION} AS build<br>WORKDIR /src<br><br># Download dependencies as a separate step to take advantage of Docker&#39;s caching.<br># Leverage a cache mount to /go/pkg/mod/ to speed up subsequent builds.<br># Leverage bind mounts to go.sum and go.mod to avoid having to copy them into<br># the container.<br>RUN --mount=type=cache,target=/go/pkg/mod/ \<br>    --mount=type=bind,source=go.sum,target=go.sum \<br>    --mount=type=bind,source=go.mod,target=go.mod \<br>    go mod download -x<br><br># This is the architecture you&#39;re building for, which is passed in by the builder.<br># Placing it here allows the previous steps to be cached across architectures.<br>ARG TARGETARCH<br><br># Build the application.<br># Leverage a cache mount to /go/pkg/mod/ to speed up subsequent builds.<br># Leverage a bind mount to the current directory to avoid having to copy the<br># source code into the container.<br>RUN --mount=type=cache,target=/go/pkg/mod/ \<br>    --mount=type=bind,target=. \<br>    CGO_ENABLED=0 GOARCH=$TARGETARCH go build -o /bin/server .<br><br>################################################################################<br># Create a new stage for running the application that contains the minimal<br># runtime dependencies for the application. This often uses a different base<br># image from the build stage where the necessary files are copied from the build<br># stage.<br>#<br># The example below uses the alpine image as the foundation for running the app.<br># By specifying the &quot;latest&quot; tag, it will also use whatever happens to be the<br># most recent version of that image when you build your Dockerfile. If<br># reproducibility is important, consider using a versioned tag<br># (e.g., alpine:3.17.2) or SHA (e.g., alpine@sha256:c41ab5c992deb4fe7e5da09f67a8804a46bd0592bfdf0b1847dde0e0889d2bff).<br>FROM alpine:latest AS final<br><br># Install any runtime dependencies that are needed to run your application.<br># Leverage a cache mount to /var/cache/apk/ to speed up subsequent builds.<br>RUN --mount=type=cache,target=/var/cache/apk \<br>    apk --update add \<br>        ca-certificates \<br>        tzdata \<br>        &amp;&amp; \<br>        update-ca-certificates<br><br># Create a non-privileged user that the app will run under.<br># See https://docs.docker.com/go/dockerfile-user-best-practices/<br>ARG UID=10001<br>RUN adduser \<br>    --disabled-password \<br>    --gecos &quot;&quot; \<br>    --home &quot;/nonexistent&quot; \<br>    --shell &quot;/sbin/nologin&quot; \<br>    --no-create-home \<br>    --uid &quot;${UID}&quot; \<br>    appuser<br>USER appuser<br><br># Copy the executable from the &quot;build&quot; stage.<br>COPY --from=build /bin/server /bin/<br><br># Expose the port that the application listens on.<br>EXPOSE 8080<br><br># What the container should run when it is started.<br>ENTRYPOINT [ &quot;/bin/server&quot; ]</pre><p>Then build and push to Docker Hub:</p><pre>docker build -t durellirsd/goapp:latest .<br>docker push durellirsd/goapp:latest</pre><h3>⚘️ Step 3: Deploying with Helm</h3><p>We used Helm to scaffold a chart:</p><pre>helm create goapp</pre><p>Then, in values.yaml, we updated the image section:</p><pre>image:<br>  repository: durellirsd/goapp<br>  tag: latest<br>  pullPolicy: IfNotPresent</pre><p>To install the app:</p><pre>helm install goapp ./goapp - namespace golang - create-namespace</pre><h3>🔍 Exploring Logs in Grafana</h3><p>In Grafana, go to <strong>Explore</strong>, select <strong>Loki</strong> as your data source, and run:</p><pre>{app=&quot;goapp&quot;}</pre><p>You should see real-time logs from the Go application. This validates that your log pipeline from container to dashboard is working perfectly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vGXIvmv5oqaR4PWi_LTKWQ.png" /></figure><h3>✅ Conclusion</h3><p>In this third and final part of our <strong>End-to-End Observability with Kubernetes</strong> series, we:</p><ul><li>Deployed Grafana Loki for log storage</li><li>Extended Grafana Alloy to collect and forward logs</li><li>Built and deployed a Go app that emits logs</li><li>Queried the logs in Grafana using LogQL</li></ul><p>You now have a complete open-source observability stack with <strong>Metrics</strong>, <strong>Traces</strong>, and <strong>Logs</strong>.</p><blockquote><em>🌟 If you enjoyed the series, feel free to share or leave a comment!</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0338748d6fe2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[End-to-End Observability with Kubernetes — Parte 2: Tracing distribuído com Grafana Alloy + Tempo]]></title>
            <link>https://medium.com/@RafaelDurelli/end-to-end-observability-with-kubernetes-parte-2-tracing-distribu%C3%ADdo-com-grafana-alloy-tempo-87da3a467927?source=rss-ca3e83e2e680------2</link>
            <guid isPermaLink="false">https://medium.com/p/87da3a467927</guid>
            <category><![CDATA[alloy]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[grafana-tempo]]></category>
            <dc:creator><![CDATA[Rafael Durelli]]></dc:creator>
            <pubDate>Mon, 09 Jun 2025 12:41:34 GMT</pubDate>
            <atom:updated>2025-06-11T12:40:19.902Z</atom:updated>
            <content:encoded><![CDATA[<h3>End-to-End Observability with Kubernetes — Part 2: Tracing distribuído com Grafana Alloy + Tempo</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6toKWw-IWsJh7gYaVayN6A.png" /></figure><h3>✨ Introduction</h3><p>In <a href="https://medium.com/@RafaelDurelli/end-to-end-observability-with-kubernetes-prometheus-grafana-plus-a-spring-boot-bonus-6e97f220da14">Part 1</a>, we set up a robust observability foundation with Prometheus, Grafana, and Spring Boot inside a Kubernetes cluster.</p><blockquote>Now it’s time to go deeper — by adding <strong>distributed tracing</strong> using <a href="https://grafana.com/docs/alloy/latest/">Grafana Alloy</a> and <a href="https://grafana.com/oss/tempo/">Grafana Tempo</a>. This lets us follow requests as they flow through services and visualize performance bottlenecks in real-time.</blockquote><blockquote>🚀 What we’ll cover in this part<br> • Deploying Grafana Tempo in Kubernetes<br> • Installing Grafana Alloy with OTLP receiver support<br> • Configuring a Spring Boot app to emit OpenTelemetry traces<br> • Visualizing traces in Grafana</blockquote><h3>🧩 Why Tempo and Alloy?</h3><p>Before jumping into the installation, it’s important to understand what each tool does and how they work together:</p><blockquote>📦 Grafana Tempo</blockquote><blockquote>Tempo is a high-scale, cost-effective distributed tracing backend developed by Grafana Labs.<br>It stores spans (individual trace segments) and lets you analyze how requests propagate across microservices. Tempo is compatible with OpenTelemetry, Jaeger, and Zipkin protocols.</blockquote><p>You can think of Tempo as “Prometheus for traces.”</p><blockquote>🔌 Grafana Alloy</blockquote><blockquote>Grafana Alloy is a modern telemetry collector that replaces Prometheus agents, OTEL collectors, and Promtail in one unified binary.<br>In this setup, we’ll use Alloy as the OpenTelemetry receiver — collecting trace data from your applications and forwarding it to Tempo.</blockquote><p>In short:</p><p><strong>Spring Boot app → Alloy (OTLP receiver) → Tempo (trace backend) → Grafana (UI)</strong></p><h3>🚀 Let’s Get Started</h3><p>✅ Step 1 — Install Grafana Tempo</p><p>We’ll install Tempo using Helm with a custom values.yaml.</p><pre>helm install tempo grafana/tempo \<br>  --version 1.21.1 \<br>  --namespace monitoring \<br>  --values tempo-values.yaml \<br>  --wait</pre><p>Let’s break down what each part of the tempo-values.yaml file does, and why it’s important for a production-grade tracing setup:</p><pre>tempo:<br>  # Specific commit for reproducibility<br>  tag: main-814c1c6<br>    <br>  # Metrics generator - The key feature for traces → metrics integration<br>  metricsGenerator:<br>    enabled: true<br>    remoteWriteUrl: &quot;http://prometheus-kube-prometheus-prometheus:9090/api/v1/write&quot;<br>      <br>  # Enable service graphs and span metrics<br>  overrides:<br>    defaults:<br>      metrics_generator:<br>        processors: [service-graphs, span-metrics, local-blocks]<br><br># Enable persistence for realistic setup<br>persistence:<br>  enabled: true</pre><pre>tempo:<br>  tag: main-814c1c6</pre><ul><li>We explicitly set a specific image tag (based on a commit hash) to ensure reproducibility. This avoids unexpected changes if the latest tag ever updates.</li></ul><pre>  metricsGenerator:<br>    enabled: true<br>    remoteWriteUrl: &quot;http://prometheus-kube-prometheus-prometheus:9090/api/v1/write&quot;</pre><ul><li>The <strong>metrics generator</strong> is a key feature that transforms trace data into <strong>Prometheus-style metrics</strong>.</li><li>By enabling this and setting remoteWriteUrl, Tempo will automatically generate span-level metrics (like request duration, error rates, span counts) and send them directly to Prometheus.</li><li>This allows you to build dashboards <strong>without needing custom instrumentation</strong> for these metrics.</li></ul><pre>  overrides:<br>    defaults:<br>      metrics_generator:<br>        processors: [service-graphs, span-metrics, local-blocks]</pre><ul><li>We configure which <strong>metrics processors</strong> are active:</li><li>service-graphs: Builds a service dependency map based on spans.</li><li>span-metrics: Converts individual spans into aggregate metrics (e.g. duration_seconds, request_total).</li><li>local-blocks: Enables metrics generation even from locally stored traces.</li></ul><blockquote><em>These processors allow Tempo to </em><strong><em>bridge the gap between traces and metrics</em></strong><em>, offering a complete view of system performance.</em></blockquote><pre>persistence:<br>  enabled: true</pre><ul><li>This enables <strong>persistent storage</strong> of traces using a Kubernetes PersistentVolumeClaim (PVC).</li><li>Without this, Tempo would lose all trace data if the pod restarts.</li><li>This makes the setup much more realistic for staging or production environments.</li></ul><p>✅ Why this matters</p><p>With this configuration, Tempo is not just collecting and storing spans — it’s also:<br> • Generating valuable metrics from trace data<br> • Exporting them to Prometheus<br> • Persisting spans across restarts<br> • Providing powerful service maps and query options inside Grafana</p><p>This turns Tempo from a simple trace backend into a core part of your observability strategy.</p><h3>🔧 Step 2 — Installing Grafana Alloy</h3><p>Once Tempo is up and running, the next step is to deploy <strong>Grafana Alloy</strong>, which will act as our <strong>OpenTelemetry receiver</strong> — collecting traces from our application and forwarding them to Tempo.</p><p>We install it using Helm:</p><pre>helm install alloy grafana/alloy \<br>  --version 1.0.3 \<br>  --namespace monitoring \<br>  --values custom-alloy.yaml \<br>  --wait</pre><p>Let’s break down what this does:</p><ul><li>grafana/alloy: uses the official Helm chart from Grafana Labs</li><li>— version 1.0.3: installs a specific stable version</li><li>— namespace monitoring: deploys Alloy into the same namespace as Tempo and Grafana</li><li>— values custom-alloy.yaml: passes a custom configuration file</li><li>— wait: ensures Helm waits for successful deployment</li></ul><h3>📄 custom-alloy.yaml</h3><pre>alloy:<br>  configMap:<br>    content: |<br>      // OTLP receiver configuration<br>      otelcol.receiver.otlp &quot;otlp_receiver&quot; {<br>        grpc {<br>          endpoint = &quot;0.0.0.0:4317&quot;<br>        }<br><br>        http {<br>          endpoint = &quot;0.0.0.0:4318&quot;<br><br>          cors {<br>            allowed_origins = [&quot;*&quot;]<br>            allowed_headers = [&quot;*&quot;]<br>            max_age         = 600<br>          }<br>        }<br><br>        output {<br>          traces = [otelcol.exporter.otlp.tempo.input]<br>        }<br>      }<br><br>      // OTLP exporter to Tempo<br>      otelcol.exporter.otlp &quot;tempo&quot; {<br>        client {<br>          endpoint = &quot;tempo:4317&quot;<br>          tls {<br>            insecure = true<br>          }<br>        }<br>      }<br><br>  # Extra ports to expose OTLP receivers<br>  extraPorts:<br>    - name: otlp-grpc<br>      port: 4317<br>      targetPort: 4317<br>      protocol: TCP<br>    - name: otlp-http<br>      port: 4318<br>      targetPort: 4318<br>      protocol: TCP</pre><h3>🧠 What this configuration does</h3><ul><li>The otelcol.receiver.otlp block enables Alloy to listen for OTLP traces over both <strong>gRPC</strong> and <strong>HTTP</strong>.</li><li>The output block tells Alloy to forward all incoming traces to the tempo exporter.</li><li>The otelcol.exporter.otlp “tempo” block defines the connection to the <strong>Tempo distributor service</strong> within the cluster.</li><li>tls.insecure = true is used here because we’re operating inside a trusted Kubernetes network without TLS termination between services.</li><li>The extraPorts section ensures that both ports (4317 and 4318) are exposed on the Alloy service so applications can send traces to it.</li></ul><blockquote><em>This setup enables your applications to send OpenTelemetry traces directly to Alloy, which then routes them efficiently to Tempo for storage and analysis.</em></blockquote><h3>💻 Step 3 — Configuring Spring Boot for Tracing</h3><p>With both <strong>Grafana Tempo</strong> and <strong>Grafana Alloy</strong> running inside your Kubernetes cluster, it’s time to configure your <strong>Spring Boot application</strong> to emit distributed traces using <strong>OpenTelemetry</strong>.</p><p>Now that both <strong>Grafana Tempo</strong> and <strong>Grafana Alloy</strong> are running in your cluster, it’s time to enable tracing in your Spring Boot application. This allows you to generate and send OpenTelemetry spans to Alloy, which forwards them to Tempo.</p><h3>📦 Maven Dependencies</h3><p>Here’s the pom.xml snippet with the necessary dependencies:</p><pre>&lt;!-- Core Spring Boot dependencies --&gt;<br>&lt;dependency&gt;<br>  &lt;groupId&gt;org.springframework.boot&lt;/groupId&gt;<br>  &lt;artifactId&gt;spring-boot-starter-web&lt;/artifactId&gt;<br>&lt;/dependency&gt;<br><br>&lt;dependency&gt;<br>  &lt;groupId&gt;org.springframework.boot&lt;/groupId&gt;<br>  &lt;artifactId&gt;spring-boot-starter-actuator&lt;/artifactId&gt;<br>&lt;/dependency&gt;<br><br>&lt;dependency&gt;<br>  &lt;groupId&gt;org.springframework.boot&lt;/groupId&gt;<br>  &lt;artifactId&gt;spring-boot-starter-aop&lt;/artifactId&gt;<br>&lt;/dependency&gt;<br><br>&lt;!-- OpenTelemetry + Micrometer bridge --&gt;<br>&lt;dependency&gt;<br>  &lt;groupId&gt;io.micrometer&lt;/groupId&gt;<br>  &lt;artifactId&gt;micrometer-tracing-bridge-otel&lt;/artifactId&gt;<br>&lt;/dependency&gt;<br><br>&lt;!-- OTLP exporter for sending spans to Alloy --&gt;<br>&lt;dependency&gt;<br>  &lt;groupId&gt;io.opentelemetry&lt;/groupId&gt;<br>  &lt;artifactId&gt;opentelemetry-exporter-otlp&lt;/artifactId&gt;<br>&lt;/dependency&gt;<br><br>&lt;!-- Autoconfiguration support --&gt;<br>&lt;dependency&gt;<br>  &lt;groupId&gt;io.opentelemetry&lt;/groupId&gt;<br>  &lt;artifactId&gt;opentelemetry-sdk-extension-autoconfigure&lt;/artifactId&gt;<br>  &lt;version&gt;1.32.0&lt;/version&gt;<br>&lt;/dependency&gt;<br><br>&lt;!-- Required gRPC transport --&gt;<br>&lt;dependency&gt;<br>  &lt;groupId&gt;io.grpc&lt;/groupId&gt;<br>  &lt;artifactId&gt;grpc-netty-shaded&lt;/artifactId&gt;<br>  &lt;version&gt;1.63.0&lt;/version&gt;<br>&lt;/dependency&gt;</pre><h3>🛠️ application.yml Configuration</h3><p>Here’s the exact configuration we used:</p><pre># Application name that will appear in Grafana/Tempo<br>spring.application.name=meu-servico-spring-boot<br><br># Enable tracing and full sampling (100%)<br>management.tracing.enabled=true<br>management.tracing.sampling.probability=1.0<br><br># Define the OTLP endpoint exposed by Grafana Alloy<br>management.otlp.tracing.endpoint=http://alloy.monitoring.svc.cluster.local:4318/v1/traces<br>management.otlp.tracing.protocol=http/protobuf<br><br># Enable trace and span IDs in logs<br>logging.level.io.opentelemetry=DEBUG<br>logging.pattern.level=%5p [traceId=%X{traceId} spanId=%X{spanId}]</pre><h3>🔍 Step-by-Step Breakdown of application.properties</h3><p>Let’s walk through each configuration property and explain <strong>why</strong> it matters in a distributed tracing setup:</p><h4>spring.application.name=meu-servico-spring-boot</h4><p>This sets the logical name of your service.</p><p>It becomes the service.name attribute in OpenTelemetry, which is how <strong>Tempo</strong> and <strong>Grafana</strong> identify and group your traces.</p><p>✅ In Grafana Explore, you can use this name to search traces like:</p><pre>service.name = meu-servico-spring-boot</pre><h4>management.tracing.enabled=true</h4><p>This activates the <strong>Micrometer Tracing</strong> module in Spring Boot.</p><p>With this enabled, Spring Boot starts creating spans automatically for key operations like HTTP requests, scheduled tasks, and more.</p><p>Without this, no tracing data will be generated.</p><h4>management.tracing.sampling.probability=1.0</h4><p>This defines the <strong>sampling rate</strong>, which controls how many requests are traced.</p><ul><li>1.0: Trace <strong>100%</strong> of incoming requests (useful for dev/testing).</li><li>0.5: Trace 50%.</li><li>0.01: Trace only 1%.</li></ul><p>✅ For production environments, use a lower value to reduce performance overhead.</p><h4>management.otlp.tracing.endpoint=http://alloy.monitoring.svc.cluster.local:4318/v1/traces</h4><p>This sets the <strong>destination</strong> for your trace data — in this case, the <strong>HTTP OTLP receiver</strong> exposed by <strong>Grafana Alloy</strong> inside the Kubernetes cluster.</p><ul><li>4318 is the standard HTTP port for OTLP.</li><li>The /v1/traces path is defined by the OTLP protocol.</li></ul><p>All trace data will be pushed here and then <strong>forwarded to Tempo</strong> by Alloy.</p><h4>management.otlp.tracing.protocol=http/protobuf</h4><p>Defines the <strong>serialization format</strong> used when sending traces.</p><ul><li>http/protobuf is a binary format supported by Alloy and is much more efficient than JSON.</li><li>It ensures compatibility between your app and the Alloy OTLP receiver.</li></ul><h4>logging.pattern.level=%5p [traceId=%X{traceId} spanId=%X{spanId}]</h4><p>This modifies your log format to <strong>include the trace and span IDs</strong> in each log line.</p><p>✅ Why this matters:</p><p>When you look at logs and traces together, this allows you to <strong>correlate logs with traces</strong> easily — which is a key part of observability.</p><p>For example, you can search for a specific traceId in both your logs and in Grafana Tempo.</p><h3>🧪 Step 4 — Simulating Traces in Code (Java)</h3><p>To generate real tracing data from your application, we created two simple Spring Boot components that simulate a request flow and emit spans manually.</p><h3>📦 HelloController.java</h3><pre>@RestController<br>public class HelloController {<br><br>    private final SimuladorDeServico simuladorDeServico;<br><br>    public HelloController(SimuladorDeServico simuladorDeServico) {<br>        this.simuladorDeServico = simuladorDeServico;<br>    }<br><br>    @GetMapping(&quot;/hello&quot;)<br>    public String hello() {<br>        simuladorDeServico.etapa1();<br>        return &quot;Hello, OTEL!&quot;;<br>    }<br>}</pre><p>This is a basic REST controller that listens on /hello. When this endpoint is hit:<br> 1. It calls a custom service method (etapa1()).<br> 2. That method generates a custom span.<br> 3. The trace is exported to Alloy → Tempo → Grafana.</p><h3>🔧SimuladorDeServico.java</h3><pre>@Service<br>public class SimuladorDeServico {<br><br>    private static final Logger log = LoggerFactory.getLogger(SimuladorDeServico.class);<br>    private final ObservationRegistry observationRegistry;<br><br>    public SimuladorDeServico(ObservationRegistry observationRegistry) {<br>        this.observationRegistry = observationRegistry;<br>    }<br><br>    public void etapa1() {<br>        Observation.createNotStarted(&quot;processamento.cliente.validacao-dados&quot;, observationRegistry)<br>                .lowCardinalityKeyValue(&quot;tipo&quot;, &quot;validacao&quot;)<br>                .lowCardinalityKeyValue(&quot;componente&quot;, &quot;SimuladorDeServico&quot;)<br>                .lowCardinalityKeyValue(&quot;status&quot;, &quot;iniciado&quot;)<br>                .observe(() -&gt; {<br>                    log.info(&quot;Iniciando validação de dados do cliente&quot;);<br>                    try {<br>                        Thread.sleep(1000); // Simulated workload<br>                        log.info(&quot;Validação concluída&quot;);<br>                    } catch (InterruptedException e) {<br>                        log.error(&quot;Erro ao simular etapa1&quot;, e);<br>                        Thread.currentThread().interrupt();<br>                    }<br>                });<br>    }<br>}</pre><p>This service does the following:<br> • Creates a manual span called processamento.cliente.validacao-dados.<br> • Adds custom tags like tipo, componente, and status for better filtering and dashboarding.<br> • Logs the progress of a simulated workload (a 1-second delay).</p><p>🧠 Why this matters<br>Using ObservationRegistry and Micrometer Tracing, we simulate real-world business logic while maintaining full control over span metadata. This is ideal for:<br> • Teaching observability concepts<br> • Prototyping service instrumentation<br> • Demonstrating how spans enrich logs</p><h3>🚢 Step 5 — Packaging and Deploying the Java App in Kubernetes</h3><p>Now that our Spring Boot application is instrumented with OpenTelemetry and emitting trace data, it’s time to <strong>containerize and deploy it into the Kubernetes cluster</strong>.</p><p>This step is crucial because it completes our observability pipeline:</p><pre>Spring Boot → Alloy → Tempo → Grafana</pre><p>🛠 What we’ll do:<br> • Build a Docker image of our instrumented app<br> • Create Kubernetes manifests to deploy it<br> • Expose it via a Kubernetes Service so Alloy can scrape it</p><h4>📦 Dockerfile (simplified)</h4><pre>docker init            # generates a suggested Dockerfile<br>docker build -t spring-grafana:1.0.0 .<br>docker push &lt;your-registry&gt;/spring-grafana:1.0.0 </pre><h3>deployment.yaml</h3><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: spring-boot<br>  namespace: spring-boot<br>spec:<br>  selector:<br>    matchLabels:<br>      app: spring-boot<br>  template:<br>    metadata:<br>      labels:<br>        app: spring-boot<br>    spec:<br>      containers:<br>      - name: spring-boot<br>        image: durellirsd/spring-grafana-part1:v01<br>        resources:<br>          limits:<br>            memory: &quot;128Mi&quot;<br>            cpu: &quot;500m&quot;<br>          requests:<br>            memory: &quot;64Mi&quot;<br>            cpu: &quot;250m&quot;<br>        ports:<br>        - containerPort: 8080</pre><h3>service.yaml</h3><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: spring-boot<br>  namespace: spring-boot<br>  labels:<br>    app: spring-boot <br>spec:<br>  selector:<br>    app: spring-boot<br>  ports:<br>  - name: http<br>    port: 8080<br>    targetPort: 8080</pre><p>🌐 Testing the Application Locally with Port-Forward</p><p>Once the Spring Boot app is running inside the cluster, we can expose it temporarily to our local machine using kubectl port-forward. This allows us to trigger HTTP requests and generate trace data without setting up an Ingress or LoadBalancer.</p><p>Run the following command:</p><pre>kubectl port-forward svc/spring-boot 8080:8080 -n spring-boot</pre><p>Now, you can hit the /hello endpoint from your machine:</p><pre>curl http://localhost:8080/hello</pre><p>✅ This call will:<br> • Trigger a controller in the app<br> • Simulate a service layer with a span<br> • Emit trace data through Alloy<br> • Forward it to Tempo<br> • Allow you to explore it in Grafana</p><p>You can search for the trace in Grafana &gt; Explore using:</p><h3>🖼️ Visual Results</h3><p>To confirm that everything is working properly, we accessed our application, triggered a request, and inspected the resulting traces in Grafana Tempo.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/1*ZWIOpKi5e6S1t1UQcpA4Cw.png" /></figure><blockquote><em>This is the browser response after running:</em></blockquote><pre>curl http://localhost:8080/hello</pre><blockquote><em>The output Hello, OTEL! confirms that the Spring Boot app is responding correctly and internally triggering a trace.</em></blockquote><h4>Trace registered in Grafana Tempo</h4><blockquote><em>In Grafana &gt; Explore &gt; Tempo, we can see a new trace entry for the request to /hello.</em></blockquote><blockquote><em>The Service column shows meu-servico-spring-boot, and the Name indicates the endpoint traced: http get /hello.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yIyQX-U4rxWGUOOjduvDUg.png" /></figure><h4>Inspecting the full trace in Grafana</h4><p>When clicking the trace ID, we get a full breakdown of the distributed trace:<br> • The root span is the HTTP request to /hello<br> • A nested span called processamento.cliente.validacao-dados represents custom business logic<br> • Each span shows duration, attributes, and tags added in the code (tipo=validacao, componente=SimuladorDeServico)<br> • You can also see traceId and spanId, useful for correlating with logs</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K5bqAwEYvBaOu0nyhsvF7w.png" /></figure><p>This view demonstrates the power of distributed tracing with Tempo: you can track the flow of a single request across layers, monitor durations, and analyze service interactions — all in one place.</p><h3>🔄 Step 6 — Generating Synthetic Traces with k6-trace-generator</h3><p>To validate and stress-test our tracing pipeline, we deployed a <strong>trace generator</strong> based on <a href="https://github.com/grafana/xk6-client-tracing">Grafana’s xk6-client-tracing</a>. This tool simulates trace data and sends it directly to the OTLP receiver (Alloy) — without needing real user traffic.</p><p>This approach is especially useful for:<br> • Verifying that your observability setup is working<br> • Creating baseline data for dashboards<br> • Exploring Tempo’s trace querying capabilities</p><p>☸️ k6-trace-generator Deployment</p><p>Here’s the manifest we used:</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: k6-trace-generator<br>  namespace: monitoring<br>  labels:<br>    app: k6-trace-generator<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: k6-trace-generator<br>  template:<br>    metadata:<br>      labels:<br>        app: k6-trace-generator<br>    spec:<br>      containers:<br>        - name: k6-tracing<br>          image: ghcr.io/grafana/xk6-client-tracing:v0.0.7<br>          env:<br>            - name: ENDPOINT<br>              value: &quot;alloy:4317&quot;  # Send spans directly to Alloy via OTLP/gRPC<br>          resources:<br>            limits:<br>              cpu: 200m<br>              memory: 256Mi<br>            requests:<br>              cpu: 100m<br>              memory: 128Mi<br>      restartPolicy: Always</pre><h3>🌐 Optional: Exposing the Generator via a ClusterIP Service</h3><p>We also expose the pod using a Service, which can be used to monitor or control the trace generation:</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: k6-trace-generator<br>  namespace: monitoring<br>spec:<br>  selector:<br>    app: k6-trace-generator<br>  ports:<br>    - port: 8080<br>      targetPort: 8080<br>      name: http<br>  type: ClusterIP</pre><h3>🧪 Validating Synthetic Traces in Grafana Tempo</h3><p>After deploying the k6-trace-generator, we were able to generate consistent traffic that simulates interactions in a fictional e-commerce backend. Let’s walk through what we see in Grafana Tempo.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VSOoYYm65ILRXhlHU4mSkQ.png" /></figure><p>In this view, we can see several traces generated by the synthetic traffic from the k6-trace-generator.<br>The Service is labeled as shop-backend and each trace represents a simulated action such as:<br> • set-billing<br> • article-to-cart<br> • list-articles<br> • list-shipping</p><p>The Duration column gives us insight into how long each simulated request took. Some traces span over 900ms, which mimics real-world latency variations.</p><p><strong>Inspecting a Single Trace (set-billing operation)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*48S1qBPe6o0DrxSm77c3Ag.png" /></figure><p>Clicking on one of the trace IDs opens a detailed trace breakdown. In this case, the operation being traced is shop-backend: set-billing.</p><p>The Trace Timeline (top panel) shows five spans:<br> • shop-backend: set-billing<br> • authentication<br> • auth-service.authenticate<br> • cart-service.checkout<br> • billing-service.payment</p><p>Each span represents a downstream service or function involved in the request.</p><p>The Gantt-style bar chart helps visualize latency per span and the critical path of the operation.<br>This is especially helpful for spotting slow services, broken dependencies, or outliers in distributed performance.</p><h3>✅ Summary of What We See</h3><ul><li><strong>Traces are being generated continuously</strong> by the k6-trace-generator, with varied operations and durations.</li><li><strong>Each trace contains nested spans</strong>, simulating realistic service interactions (microservices).</li><li><strong>Grafana Tempo captures and displays them in real-time</strong>, allowing you to analyze them in-depth using tags, filters, or TraceQL queries.</li></ul><p>These screenshots demonstrate how powerful distributed tracing can be when paired with synthetic test data — perfect for observability setups, testing dashboards, or workshops.</p><h3>🏁 Conclusion — Observability That Goes Beyond Metrics</h3><p>In this second part of the series, we elevated our Kubernetes observability stack from simple metrics collection to <strong>true distributed tracing</strong>.By combining:<br> • Grafana Tempo for trace storage and visualization<br> • Grafana Alloy for OpenTelemetry ingestion<br> • Spring Boot with OTLP support for instrumentation<br> • And even a synthetic trace generator with k6-trace-generator</p><p>…we’ve built a fully functional and extendable tracing pipeline — 100% open source and Kubernetes-native.</p><p>Whether you’re debugging latency issues, analyzing service dependencies, or just exploring request paths, this setup gives you the visibility you need.</p><h3>🔮 What’s Next?</h3><p>In the upcoming <strong>Part 3</strong>, we’ll complete the “observability trinity” by integrating <strong>Grafana Loki</strong> for centralized logs. That will enable full <strong>trace ↔ log ↔ metric correlation</strong> — all visualized through Grafana Dashboards and Explore.</p><h3>📘 Related Reading</h3><ul><li>📄 <a href="https://medium.com/@RafaelDurelli/end-to-end-observability-with-kubernetes-prometheus-grafana-plus-a-spring-boot-bonus-6e97f220da14">Part 1: End-to-End Observability with Kubernetes, Prometheus, Grafana (plus a Spring Boot bonus)</a></li><li>🧪 <a href="https://grafana.com/docs/alloy/latest/">Grafana Alloy documentation</a></li><li>📈 <a href="https://grafana.com/docs/tempo/latest/">Grafana Tempo documentation</a></li><li>🧵 <a href="https://github.com/grafana/xk6-client-tracing">xk6-client-tracing (trace generator)</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=87da3a467927" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[End-to-End Observability with Kubernetes, Prometheus & Grafana (plus a Spring Boot bonus)]]></title>
            <link>https://medium.com/@RafaelDurelli/end-to-end-observability-with-kubernetes-prometheus-grafana-plus-a-spring-boot-bonus-6e97f220da14?source=rss-ca3e83e2e680------2</link>
            <guid isPermaLink="false">https://medium.com/p/6e97f220da14</guid>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[spring-boot]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Rafael Durelli]]></dc:creator>
            <pubDate>Fri, 06 Jun 2025 17:14:32 GMT</pubDate>
            <atom:updated>2025-06-06T17:14:32.458Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*nSt7T_wzG-i4PPV9Epg73w.png" /></figure><blockquote>TL;DR — In this hands-on guide you will<br> 1. Spin up a local Kubernetes cluster with Kind (+ Docker)<br> 2. Install the kube-prometheus-stack via Helm (Prometheus ➕ Grafana ➕ Alertmanager)<br> 3. Expose application metrics from a Spring Boot service in Prometheus Operator format<br> 4. Visualize everything in Grafana using community dashboards</blockquote><blockquote>The result is a 100 % local lab for studying observability, instrumenting apps and experimenting with alerts — without spending a cent on cloud resources.</blockquote><h3>1 · Prerequisites</h3><pre>**You’ll need a few tools before we start:**<br><br>- **Docker ≥ 24.x** — the container engine that will host Kind  <br>- **Kind ≥ 0.23** — *Kubernetes-in-Docker* (`go install sigs.k8s.io/kind@latest`)  <br>- **kubectl ≥ 1.30** — the standard Kubernetes command-line client  <br>- **Helm ≥ 3.15** — package manager for Kubernetes charts  <br>- **Java 17 or newer** — only for the Spring Boot demo; skip if you’ll test with other runtimes</pre><h3>2 · Creating the Cluster with Kind</h3><pre>cat &lt;&lt;&#39;EOF&#39; &gt; kind-cluster.yaml<br>kind: Cluster<br>apiVersion: kind.x-k8s.io/v1alpha4<br>nodes:<br>  - role: control-plane   # 1 node: control plane<br>  - role: worker          # 2 nodes: workers<br>  - role: worker<br>EOF<br><br>kind create cluster --name observability-lab --config kind-cluster.yaml<br>kubectl cluster-info --context kind-observability-lab</pre><h3>3 · Installing kube-prometheus-stack</h3><p>The community chart bundles:<br> • Prometheus Operator<br> • Prometheus (scraping &amp; rule engine)<br> • Alertmanager<br> • Grafana (pre-configured with Prometheus/Alertmanager data sources)</p><p>More info here: <a href="https://github.com/prometheus-community/helm-charts">https://github.com/prometheus-community/helm-charts</a></p><pre># Add / update repo<br>helm repo add prometheus-community https://prometheus-community.github.io/helm-charts<br>helm repo update<br><br># Full installation in a dedicated namespace<br>helm install prometheus \<br>  prometheus-community/kube-prometheus-stack \<br>  --namespace monitoring --create-namespace</pre><p>On most machines the installation takes 1–2 minutes. When it finishes:</p><pre>kubectl get pods -n monitoring<br>kubectl get svc  -n monitoring</pre><h3>3.1 · Accessing Grafana</h3><pre>kubectl port-forward svc/prometheus-grafana 3000:80 -n monitoring</pre><p>Open <a href="http://localhost:3000/">http://localhost:3000</a></p><p><em>User</em>: <strong>admin</strong> · <em>Password</em>: <strong>prom-operator</strong> (or whatever you set in values.yaml).</p><p>Under <strong>Connections → Data Sources</strong> you’ll see <em>Prometheus</em> and <em>Alertmanager</em> already configured — the Operator did that for you.</p><p>In <strong>Explore</strong> or <strong>Dashboards → Kubernetes / Nodes / etc.</strong> you can inspect metrics such as:</p><ul><li>CoreDNS, kubelet, scheduler, kube-proxy, etcd…</li><li>CPU, memory, I/O, network usage, and more.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cG4-PHLGfr_Z2S_kcIjGYQ.png" /></figure><h3>4 · Instrumenting a Spring Boot Application</h3><p>We will need to add some dependencies to export metrics to prometheus.</p><h3>4.1 · Dependencies</h3><pre>&lt;!-- pom.xml --&gt;<br>&lt;dependency&gt;<br>  &lt;groupId&gt;org.springframework.boot&lt;/groupId&gt;<br>  &lt;artifactId&gt;spring-boot-starter-actuator&lt;/artifactId&gt;<br>&lt;/dependency&gt;<br>&lt;dependency&gt;<br>  &lt;groupId&gt;io.micrometer&lt;/groupId&gt;<br>  &lt;artifactId&gt;micrometer-registry-prometheus&lt;/artifactId&gt;<br>&lt;/dependency&gt;</pre><h3>4.2 · Configuration (application.yaml)</h3><pre>spring:<br>  application:<br>    name: spring-grafana-part1<br><br>management:<br>  endpoints.web.exposure.include: health,info,prometheus<br>  prometheus.metrics.export.enabled: true<br>  endpoint:<br>    prometheus:<br>      access: unrestricted</pre><h3>4.3 · Local Test</h3><pre>./mvnw spring-boot:run<br>curl http://localhost:8080/actuator/prometheus</pre><p>You should see hundreds of lines in Prometheus format.</p><h3>4.4 · Containerizing with Docker</h3><pre>docker init            # generates a suggested Dockerfile<br>docker build -t spring-grafana:1.0.0 .<br>docker push &lt;your-registry&gt;/spring-grafana:1.0.0 </pre><h3>4.5 · Kubernetes Manifests</h3><p>Create a dedicated namespace:</p><pre>kubectl create ns spring-boot\</pre><h4>deployment.yaml</h4><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: spring-boot<br>  namespace: spring-boot<br>spec:<br>  selector:<br>    matchLabels:<br>      app: spring-boot<br>  template:<br>    metadata:<br>      labels:<br>        app: spring-boot<br>    spec:<br>      containers:<br>      - name: spring-boot<br>        image: durellirsd/spring-grafana-part1:v01<br>        resources:<br>          limits:<br>            memory: &quot;128Mi&quot;<br>            cpu: &quot;500m&quot;<br>          requests:<br>            memory: &quot;64Mi&quot;<br>            cpu: &quot;250m&quot;<br>        ports:<br>        - containerPort: 8080</pre><h4>service.yaml</h4><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: spring-boot<br>  namespace: spring-boot<br>  labels:<br>    app: spring-boot  # 🔥 need for the selector of ServiceMonitor<br>spec:<br>  selector:<br>    app: spring-boot<br>  ports:<br>  - name: http<br>    port: 8080<br>    targetPort: 8080</pre><h4>servicemonitor.yaml</h4><blockquote>ServiceMonitor — the bridge between Prometheus and your workloads</blockquote><p>A <em>ServiceMonitor</em> is a Custom Resource Definition (CRD) introduced by the <strong>Prometheus Operator</strong> that tells Prometheus <em>where</em> and <em>how</em> to scrape metrics inside the cluster. Instead of hand-editing Prometheus’ config, you simply apply a ServiceMonitor object that includes:</p><ol><li><strong>selector</strong> — label filters that pick the <em>Services</em> exposing metrics.</li><li><strong>endpoints</strong> — port, path (e.g. /metrics or /actuator/prometheus), and scrape interval.</li><li><strong>namespaceSelector</strong> — optional; restricts which namespaces Prometheus should search.</li><li><strong>discovery labels</strong> — the label release:&lt;helm-release&gt; must match the name you used when installing <em>kube-prometheus-stack</em>; this is how the Operator wires the ServiceMonitor to the correct Prometheus instance.</li></ol><blockquote>Once applied, the Operator automatically converts this declarative spec into a scrape target. Prometheus’ config stays in sync with cluster state — no restarts, no manual edits. Add, tweak, or delete ServiceMonitors and your Prometheus setup adjusts itself transparently, making GitOps workflows clean and safe.</blockquote><pre>apiVersion: monitoring.coreos.com/v1<br>kind: ServiceMonitor<br>metadata:<br>  name: spring-app-monitor<br>  namespace: spring-boot<br>  labels:<br>    release: prometheus # ⚠️ esse label TEM QUE BATER com o label `release` do Prometheus Operator<br>spec:<br>  selector:<br>    matchLabels:<br>      app: spring-boot<br>  endpoints:<br>    - port: http<br>      path: /actuator/prometheus # ou /metrics se você fez com controller customizado<br>      interval: 10s<br>  namespaceSelector:<br>    matchNames:<br>      - spring-boot # ou o namespace onde está o Service</pre><blockquote><strong><em>Pay attention to the release: prometheus label</em></strong><em> — that’s how the Prometheus Operator discovers new targets.</em></blockquote><pre>kubectl apply -f deployment.yaml -f service.yaml -f servicemonitor.yaml</pre><h3>4.6 · Validating in Prometheus</h3><p>kubectl port-forward svc/prometheus-operated 9090 -n monitoring<br># Open <a href="http://localhost:9090/">http://localhost:9090</a> → Status → Targets</p><p>You should see spring-grafana marked <strong>UP</strong>. From there:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mqwURYiWJvmu4wWsn7sUjQ.png" /></figure><h3>4.7 · Importing Spring Boot Dashboards</h3><p>With metrics flowing, the final touch is a ready-made dashboard that turns raw numbers into insight.</p><pre>kubectl port-forward svc/prometheus-grafana 3000:80 -n monitoring<br># then visit http://localhost:3000</pre><ol><li><strong>Log in</strong> (admin / prom-operator, unless you changed the creds).</li><li><strong>Import by ID</strong></li><li>Left-hand menu → Dashboards ▸ New ▸ Import</li><li>Paste <strong>14430</strong> ( <em>Spring Boot Statistics Endpoint Metrics</em> )</li><li>Click <strong>Load</strong> → choose your <strong>Prometheus</strong> datasource → <strong>Import</strong>.</li><li><strong>Explore</strong></li><li>The dashboard instantly visualises:</li><li>HTTP request rate, error ratio, latency percentiles</li><li>JVM memory &amp; GC behaviour</li><li>Thread-pool saturation, datasource (Hikari) stats</li><li>Custom counters/timers you may have added with Micrometer</li></ol><blockquote><strong><em>Tip:</em></strong><em> Need a lighter panel? Try IDs </em><strong><em>4701</em></strong><em> (Micrometer JVM) or </em><strong><em>11159</em></strong><em> (Micrometer System) and stitch them together.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BjwhEmMyIqQShAvtp9RuXw.png" /></figure><p>🚀 Final Take-Off</p><p>You’ve just built — on your laptop — a fully instrumented Kubernetes playground that rivals what many teams run in production:<br> • Kind delivers a lightweight, disposable cluster.<br> • kube-prometheus-stack turns that cluster into a metrics powerhouse.<br> • Spring Boot + Micrometer prove how effortlessly an application can surface actionable telemetry.<br> • Grafana transforms data into narratives your team can trust.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6e97f220da14" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spin Up an Elastic EKS Cluster in <15 Min with Terraform & Karpenter]]></title>
            <link>https://medium.com/@RafaelDurelli/spin-up-an-elastic-eks-cluster-in-15-min-with-terraform-karpenter-96ff54799827?source=rss-ca3e83e2e680------2</link>
            <guid isPermaLink="false">https://medium.com/p/96ff54799827</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[karpenter]]></category>
            <dc:creator><![CDATA[Rafael Durelli]]></dc:creator>
            <pubDate>Thu, 05 Jun 2025 20:00:59 GMT</pubDate>
            <atom:updated>2025-06-05T20:00:59.596Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><em>A hands-on walkthrough that turns raw AWS accounts into a cost-aware, auto-scaling Kubernetes platform — 100 % defined as code.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DoFptFgleQ61OGKrEu7e7w.png" /></figure><h3>Why bother?</h3><ul><li><strong>Elastic capacity</strong> — nodes appear in &lt; 90 s when the workload spikes and disappear when idle.</li><li><strong>No more over-provisioning</strong> — Karpenter picks the cheapest Nitro instance that satisfies the pod’s requirements.</li><li><strong>Reproducibility</strong> — one terraform apply spins the whole stack for every environment.</li></ul><h3>Project layout</h3><pre>eks-karpenter-demo/<br>├── main.tf          # provider + backend<br>├── variables.tf     # reusable knobs<br>├── vpc.tf           # network foundation (private subnets only)<br>├── eks.tf           # cluster + Karpenter Helm release<br>├── karpenter.yaml   # EC2NodeClass &amp; NodePool (applied by Helm values)<br>└── inflate.yaml     # tiny stress-test deployment</pre><h3>variables.tf</h3><p>Before we wire up any real infrastructure, we need a place to <strong>parameterise</strong> everything that might change from one environment to the next. That’s the job of variables.tf.</p><p>Think of it as the <strong>control panel</strong> for the project:</p><ul><li><strong>Portability</strong> — switching regions, CIDR blocks or even cluster names is just a matter of passing a different .tfvars file (e.g. staging.tfvars, prod.tfvars).</li><li><strong>Git-friendliness</strong> — every configuration change is tracked as code, right beside the rest of your Terraform.</li><li><strong>Safety</strong> — no hard-coded values scattered across modules; secrets can be injected at apply time instead of being committed.</li></ul><pre>variable &quot;vpc_name&quot; {<br>  description = &quot;Name of the VPC&quot;<br>  type        = string<br>  default     = &quot;vpc-learning-terraform&quot;<br>}<br><br>variable &quot;vpc_cidr&quot; {<br>  description = &quot;CIDR block for the VPC&quot;<br>  type        = string<br>  default     = &quot;10.0.0.0/16&quot;<br>}<br><br>variable &quot;vpc_azs&quot; {<br>  description = &quot;Availability Zones for the VPC&quot;<br>  type        = list(string)<br>  default     = [&quot;sa-east-1a&quot;, &quot;sa-east-1b&quot;, &quot;sa-east-1c&quot;]<br>}<br><br>variable &quot;vpc_private_subnets&quot; {<br>  description = &quot;List of private subnets in the VPC&quot;<br>  type        = list(string)<br>  default     = [&quot;10.0.1.0/24&quot;, &quot;10.0.2.0/24&quot;, &quot;10.0.3.0/24&quot;]<br>}<br>variable &quot;vpc_public_subnets&quot; {<br>  description = &quot;List of public subnets in the VPC&quot;<br>  type        = list(string)<br>  default     = [&quot;10.0.101.0/24&quot;, &quot;10.0.102.0/24&quot;, &quot;10.0.103.0/24&quot;]<br>}<br><br>variable &quot;vpc_intra_subnets&quot; {<br>  description = &quot;List of intra subnets in the VPC&quot;<br>  type        = list(string)<br>  default     = [&quot;10.0.201.0/24&quot;, &quot;10.0.202.0/24&quot;, &quot;10.0.203.0/24&quot;]<br>}<br><br>variable &quot;cluster_name&quot; {<br>  description = &quot;Name of the EKS cluster&quot;<br>  type        = string<br>  default     = &quot;eks-learning-terraform&quot;<br>}<br>variable &quot;cluster_version&quot; {<br>  description = &quot;Version of the EKS cluster&quot;<br>  type        = string<br>  default     = &quot;1.31&quot;<br>}<br><br>variable &quot;aws_region&quot; {<br>  description = &quot;AWS region to deploy resources&quot;<br>  type        = string<br>  default     = &quot;sa-east-1&quot;<br>}</pre><h3>main.tf</h3><p>With our knobs neatly parked in variables.tf, we need a file that tells Terraform <strong>where to keep state, which providers to pull, and how to talk to AWS</strong>. That maestro is main.tf.</p><pre>terraform {<br>  required_providers {<br>    aws = {<br>      source  = &quot;hashicorp/aws&quot;<br>      version = &quot;~&gt; 5.0&quot;<br>    }<br>  }<br>}<br><br># Configure the AWS Provider<br>provider &quot;aws&quot; {<br>  region = &quot;sa-east-1&quot;<br>}<br><br>provider &quot;aws&quot; {<br>  alias  = &quot;virginia&quot;<br>  region = &quot;us-east-1&quot;   # N. Virginia – exigido pelo ECR Public<br>}<br><br>provider &quot;helm&quot; {<br>  kubernetes {<br>    host                   = module.eks.cluster_endpoint<br>    cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)<br><br>    exec {<br>      api_version = &quot;client.authentication.k8s.io/v1beta1&quot;<br>      command     = &quot;aws&quot;<br>      # This requires the awscli to be installed locally where Terraform is executed<br>      args = [&quot;eks&quot;, &quot;get-token&quot;, &quot;--cluster-name&quot;, module.eks.cluster_name]<br>    }<br>  }<br>}<br><br>data &quot;aws_availability_zones&quot; &quot;available&quot; {<br>  # Exclude local zones<br>  filter {<br>    name   = &quot;opt-in-status&quot;<br>    values = [&quot;opt-in-not-required&quot;]<br>  }<br>}<br><br>data &quot;aws_ecrpublic_authorization_token&quot; &quot;token&quot; {<br>  provider = aws.virginia<br>}</pre><p>With main.tf in place, Terraform now knows <strong>which AWS regions to talk to, how to authenticate Helm against EKS, and where to fetch supporting data</strong></p><h3>vpc.tf</h3><p>VPC is where we lay the network foundation that every other resource will plug into.</p><pre>module &quot;vpc&quot; {<br>  source  = &quot;terraform-aws-modules/vpc/aws&quot;<br>  version = &quot;5.21.0&quot;<br><br>  name = var.vpc_name<br>  cidr = var.vpc_cidr<br><br>  azs             = var.vpc_azs<br>  private_subnets = var.vpc_private_subnets<br>  public_subnets  = var.vpc_public_subnets<br>  intra_subnets   = var.vpc_intra_subnets<br>  # enable_public_redshift = true # faz sentido para o SGP?<br><br>  enable_nat_gateway      = true<br>  single_nat_gateway      = true<br>  map_public_ip_on_launch = true<br><br>  private_subnet_tags = {<br>    &quot;karpenter.sh/discovery&quot; = &quot;eks-learning-terraform&quot;<br>  }<br><br>  tags = {<br>    Terraform   = &quot;true&quot;<br>    Environment = &quot;dev&quot;<br>  }<br>}</pre><p>Because we lean on the battle-tested <strong>terraform-aws-modules/vpc/aws</strong> module, the entire network blueprint fits in about 15 lines of HCL — yet we still get route tables, Internet Gateway, NAT, and subnet-to-AZ mapping out of the box.</p><h3>eks.tf</h3><p>With the network laid and the providers wired, it’s time to create the star of the show: a <strong>fully managed EKS control plane</strong> plus a <strong>Karpenter-powered data plane</strong> that scales itself in and out like magic.</p><pre><br>locals {<br>  name = var.cluster_name<br>  tags = {<br>    Example    = local.name<br>    GithubRepo = &quot;terraform-aws-eks&quot;<br>    GithubOrg  = &quot;terraform-aws-modules&quot;<br>  }<br>}<br><br><br>module &quot;eks&quot; {<br>  source  = &quot;terraform-aws-modules/eks/aws&quot;<br>  version = &quot;20.36.0&quot;<br><br>  cluster_name    = var.cluster_name<br>  cluster_version = var.cluster_version<br><br>  # Gives Terraform identity admin access to cluster which will<br>  # allow deploying resources (Karpenter) into the cluster<br>  enable_cluster_creator_admin_permissions = true<br>  cluster_endpoint_public_access           = true<br><br><br>  cluster_addons = {<br>    coredns                = {}<br>    eks-pod-identity-agent = {}<br>    kube-proxy             = {}<br>    vpc-cni                = {}<br>  }<br><br>  vpc_id                   = module.vpc.vpc_id<br>  subnet_ids               = module.vpc.private_subnets<br>  control_plane_subnet_ids = module.vpc.intra_subnets<br><br><br>  eks_managed_node_groups = {<br>    karpenter = {<br>      ami_type       = &quot;BOTTLEROCKET_x86_64&quot;<br>      instance_types = [&quot;m5.large&quot;]<br><br>      min_size     = 2<br>      max_size     = 3<br>      desired_size = 2<br><br>      labels = {<br>        # Used to ensure Karpenter runs on nodes that it does not manage<br>        &quot;karpenter.sh/controller&quot; = &quot;true&quot;<br>      }<br>    }<br>  }<br><br>  node_security_group_tags = merge(local.tags, {<br>    # NOTE - if creating multiple security groups with this module, only tag the<br>    # security group that Karpenter should utilize with the following tag<br>    # (i.e. - at most, only one security group should have this tag in your account)<br>    &quot;karpenter.sh/discovery&quot; = &quot;eks-learning-terraform&quot;<br>  })<br><br><br>  tags = {<br>    Terraform   = &quot;true&quot;<br>    Environment = &quot;dev&quot;<br>  }<br>}<br><br>module &quot;karpenter&quot; {<br>  source  = &quot;terraform-aws-modules/eks/aws//modules/karpenter&quot;<br>  version = &quot;20.36.0&quot;<br><br>  cluster_name          = module.eks.cluster_name<br>  enable_v1_permissions = true<br><br>  # Name needs to match role name passed to the EC2NodeClass<br>  node_iam_role_use_name_prefix   = false<br>  node_iam_role_name              = &quot;eks-learning-terraform&quot;<br>  create_pod_identity_association = true<br><br>  # Used to attach additional IAM policies to the Karpenter node IAM role<br>  node_iam_role_additional_policies = {<br>    AmazonSSMManagedInstanceCore = &quot;arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore&quot;<br>  }<br><br>  tags = local.tags<br>}<br><br>resource &quot;helm_release&quot; &quot;karpenter&quot; {<br>  namespace           = &quot;kube-system&quot;<br>  name                = &quot;karpenter&quot;<br>  repository          = &quot;oci://public.ecr.aws/karpenter&quot;<br>  repository_username = data.aws_ecrpublic_authorization_token.token.user_name<br>  repository_password = data.aws_ecrpublic_authorization_token.token.password<br>  chart               = &quot;karpenter&quot;<br>  version             = &quot;1.1.1&quot;<br>  wait                = false<br><br>  values = [<br>    &lt;&lt;-EOT<br>    nodeSelector:<br>      karpenter.sh/controller: &#39;true&#39;<br>    dnsPolicy: Default<br>    settings:<br>      clusterName: ${module.eks.cluster_name}<br>      clusterEndpoint: ${module.eks.cluster_endpoint}<br>      interruptionQueue: ${module.karpenter.queue_name}<br>    webhook:<br>      enabled: false<br>    EOT<br>  ]<br>}</pre><p>The <strong>NodePool</strong> &amp; <strong>EC2NodeClass</strong> in karpenter.yaml restrict instances to categories c/m/r, CPU 4–32 vCPUs, Nitro hypervisor, and generation &gt; 2</p><pre>---<br>apiVersion: karpenter.k8s.aws/v1<br>kind: EC2NodeClass<br>metadata:<br>  name: default<br>spec:<br>  amiSelectorTerms:<br>    - alias: bottlerocket@latest<br>  role: eks-learning-terraform<br>  subnetSelectorTerms:<br>    - tags:<br>        karpenter.sh/discovery: eks-learning-terraform<br>  securityGroupSelectorTerms:<br>    - tags:<br>        karpenter.sh/discovery: eks-learning-terraform<br>  tags:<br>    karpenter.sh/discovery: eks-learning-terraform<br>---<br>apiVersion: karpenter.sh/v1<br>kind: NodePool<br>metadata:<br>  name: default<br>spec:<br>  template:<br>    spec:<br>      nodeClassRef:<br>        group: karpenter.k8s.aws<br>        kind: EC2NodeClass<br>        name: default<br>      requirements:<br>        - key: &quot;karpenter.k8s.aws/instance-category&quot;<br>          operator: In<br>          values: [&quot;c&quot;, &quot;m&quot;, &quot;r&quot;]<br>        - key: &quot;karpenter.k8s.aws/instance-cpu&quot;<br>          operator: In<br>          values: [&quot;4&quot;, &quot;8&quot;, &quot;16&quot;, &quot;32&quot;]<br>        - key: &quot;karpenter.k8s.aws/instance-hypervisor&quot;<br>          operator: In<br>          values: [&quot;nitro&quot;]<br>        - key: &quot;karpenter.k8s.aws/instance-generation&quot;<br>          operator: Gt<br>          values: [&quot;2&quot;]<br>  limits:<br>    cpu: 1000<br>  disruption:<br>    consolidationPolicy: WhenEmpty<br>    consolidateAfter: 30s</pre><h3>Bootstrapping a workload to trigger scale-out</h3><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: inflate<br>spec:<br>  replicas: 5<br>  selector:<br>    matchLabels:<br>      app: inflate<br>  template:<br>    metadata:<br>      labels:<br>        app: inflate<br>    spec:<br>      terminationGracePeriodSeconds: 0<br>      containers:<br>        - name: inflate<br>          image: public.ecr.aws/eks-distro/kubernetes/pause:3.7<br>          resources:<br>            requests:<br>              cpu: 1</pre><p>Open the AWS Console and navigate to <strong>EC2 → Instances</strong> — you’ll see that Karpenter has already spun up new instances to meet the workload demand.</p><p>To watch Karpenter in action, stream its logs:</p><pre>kubectl logs -n kube-system -l app.kubernetes.io/name=karpenter --tail=100 -f</pre><p>This command tails the <strong>karpenter-controller</strong> logs in the kube-system namespace, showing every provisioning and consolidation decision in real time.</p><h3>🚦 Wrapping Up</h3><p>In roughly a coffee break we:<br> 1. Provisioned the network layer (VPC, private subnets, NAT Gateway) with a single, reusable Terraform module.<br> 2. Stood up an Amazon EKS control plane — fully managed, multi-AZ, and patched by AWS.<br> 3. Deployed Karpenter with least-privilege IAM, letting it launch the cheapest Nitro instances that fit each pod.<br> 4. Triggered an autoscale event with a tiny inflate workload and watched new nodes appear in &lt; 90 s, then disappear once idle.<br> 5. Captured everything as code, so the same terraform apply will recreate this stack tomorrow in any region or account.</p><p>What you gained<br> • Elastic capacity without over-provisioning<br> • ~30 % lower idle costs versus fixed node groups (numbers will vary — measure yours!)<br> • Git-tracked, reproducible infrastructure that a teammate or CI pipeline can spin up in minutes</p><h3>Clean-up</h3><pre>terraform destroy</pre><p><em>(Don’t forget — that NAT Gateway keeps billing while it lives.)</em></p><h3>💬 Let’s keep the conversation going</h3><p>• Drop a comment below if you hit snags or have optimisation tips — always keen to learn new tricks.<br> • If this saved you a few hours (or dollars), a clap or share helps others find it too!</p><p>Happy scaling, and may your nodes be ever right-sized. 🛠️✨</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=96ff54799827" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using KEDA, Prometheus and NGINX to scale Kubernetes resource.]]></title>
            <link>https://medium.com/@RafaelDurelli/using-keda-prometheus-and-nginx-to-scale-kubernetes-resource-8370a4bc5b30?source=rss-ca3e83e2e680------2</link>
            <guid isPermaLink="false">https://medium.com/p/8370a4bc5b30</guid>
            <dc:creator><![CDATA[Rafael Durelli]]></dc:creator>
            <pubDate>Sat, 20 Jan 2024 20:37:40 GMT</pubDate>
            <atom:updated>2024-01-20T20:51:45.079Z</atom:updated>
            <content:encoded><![CDATA[<h3>Optimizing Kubernetes: Scaling with KEDA, Prometheus, and NGINX</h3><p>In the Kubernetes universe, horizontal scalability of our workloads has become a relatively simple task thanks to Horizontal Pod Autoscalers (HPA). However, the HPAs currently available in Kubernetes, although effective, are relatively limited in their functionality, mainly allowing scalability based on CPU and memory metrics. For many applications, these metrics are adequate and meet scaling needs.</p><p>However, there are cases where scalability needs to be sized based on more specific and customized metrics. For example, it may be necessary to scale an application based on the number of requests to a service or the number of messages in a queue, such as those offered by AWS SQS or RabbitMQ services. In these situations, relying solely on Kubernetes’ “native” resources may prove to be insufficient.</p><p>Therefore, exploring additional and customized scalability solutions, which go beyond the standard CPU and memory metrics, becomes crucial to meet the specific scaling needs of certain applications.</p><p>Fortunately, we can rely on KEDA (Kubernetes Event-driven Autoscaling) to enhance our scalability capabilities in Kubernetes, allowing us to scale our resources based on a wide variety of custom metrics. KEDA broadens our horizon, going beyond the conventional CPU and memory metrics, offering a more versatile and adaptable approach to meet the specific needs of our applications.</p><blockquote><strong>KEDA</strong> is a <a href="https://kubernetes.io/">Kubernetes</a>-based Event Driven Autoscaler. With KEDA, you can drive the scaling of any container in Kubernetes based on the number of events needing to be processed. <strong>KEDA</strong> is a single-purpose and lightweight component that can be added into any Kubernetes cluster. KEDA works alongside standard Kubernetes components like the <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/">Horizontal Pod Autoscaler</a> and can extend functionality without overwriting or duplication. With KEDA you can explicitly map the apps you want to use event-driven scale, with other apps continuing to function. This makes KEDA a flexible and safe option to run alongside any number of any other Kubernetes applications or frameworks.</blockquote><p>In this article, we will explore the powerful combination of KEDA and the Prometheus Scaler to obtain metrics from Nginx and detect the moment when traffic on the Ingress reaches a specific threshold. Additionally, as an added bonus, we will introduce the use of ARGOCD in this context.</p><p>We will present how this intelligent solution, involving KEDA, Prometheus Scaler, and ARGOCD, can enhance the scalability and management of your infrastructure in Kubernetes, offering a solid approach for automatic scaling and monitoring of your Nginx-based applications.</p><p>In this example, I am using AKS (Azure Kubernetes Service), but it is important to highlight that this approach can be applied to any Kubernetes cluster.</p><h4>Installing ARGOCD</h4><p>First, let’s deploy ArgoCD in our cluster. To carry out this deployment, we will use Helm, a widely recognized package management tool in the Kubernetes community.</p><pre>kubectl create namespace argocd</pre><pre>helm repo add argo https://argoproj.github.io/argo-helm</pre><pre>helm install argo argo/argo-cd -n argocd</pre><p>To access ArgoCD in this example, we will adopt the approach of port-forwarding. This will allow us to establish a local connection with ArgoCD and access it conveniently from our development environment.</p><pre>kubectl port-forward service/argo-argocd-server -n argocd 8080:443</pre><p>You can use the administrator credentials with the username “admin”. The corresponding password can be obtained by using the following command in Kubernetes:</p><pre>kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath=&quot;{.data.password}&quot; | base64 -d\n</pre><p>The application that we will use is extremely simple and was developed using Quarkus. It has just one endpoint “/hello”. You can find the source code of this application, as well as its helm chart, in the following GitHub repository: <a href="https://github.com/rdurelli/keda-argo-nginx.git">https://github.com/rdurelli/keda-argo-nginx.git</a>.</p><h4>Installing KEDA</h4><p>The next step involves installing KEDA in our Kubernetes cluster. Once again, we will make use of the Helm tool to simplify this process and ensure efficient and reliable configuration.</p><p>For more information, you can visit the KEDA website (<a href="https://keda.sh/">https://keda.sh/</a>).</p><pre>helm repo add kedacore https://kedacore.github.io/charts</pre><pre>helm repo update</pre><pre>helm install keda kedacore/keda --namespace keda –create-namespace</pre><p>Excellent! Now that KEDA is properly installed in our cluster, we are ready to move on to the next steps.</p><h4>Installing Prometheus</h4><p>The next step is the installation of Prometheus, which will play a crucial role in collecting metrics from Nginx. This will allow KEDA to scale our Quarkus application based on the metrics we will define, providing intelligent and adaptive scalability. To install Prometheus, follow these commands:</p><pre> kubectl create namespace monitoring</pre><pre>helm upgrade --install prometheus prometheus-community/kube-prometheus-stack \ --namespace monitoring  \ --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \ --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false</pre><p>After installing Prometheus, it’s important to verify if everything went as expected. You can do this by using the following command:</p><pre>helm get values prometheus --namespace monitoring</pre><p>When you execute the command, you should see an output similar to the following:</p><pre>prometheus:<br>  prometheusSpec:<br>    podMonitorSelectorNilUsesHelmValues: false<br>    serviceMonitorSelectorNilUsesHelmValues: false</pre><p>Now, we’re almost there. The next step is to adjust the configuration of our Ingress. In order for NGINX (assuming you already have the NGINX Ingress Controller installed) to export metrics to Prometheus, we need to specify some important configurations:</p><h4>Configuring NGINX to export metrics to Prometheus.</h4><p>To enable NGINX to export metrics to Prometheus, we need to update the NGINX release via HELM. Follow this command:</p><pre>helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx --namespace ingress-nginx \             <br>--set controller.nodeSelector.&quot;beta\.kubernetes\.io/os&quot;=linux \<br>--set defaultBackend.nodeSelector.&quot;beta\.kubernetes\.io/os&quot;=linux \<br>--set controller.service.type=LoadBalancer \<br>--set controller.service.externalTrafficPolicy=Local \<br>--set controller.metrics.enabled=true \<br>--set controller.metrics.serviceMonitor.enabled=true \<br>--set controller.metrics.serviceMonitor.additionalLabels.release=&quot;prometheus&quot;</pre><p>The first four --set commands are specific to AKS (Azure Kubernetes Service). If you are using a different Kubernetes cluster, it&#39;s important to ignore these settings and proceed only with the following:</p><pre>controller.metrics.enable<br>controller.metrics.serviceMonitor.enable<br>controller.metrics.serviceMonitor.additionalLabels.release=&quot;prometheus&quot;</pre><p>After completing the process of updating the NGINX Ingress Controller, you should be able to see something similar to this:</p><pre>controller:<br>  metrics:<br>    enabled: true<br>    serviceMonitor:<br>      additionalLabels:<br>        release: prometheus<br>      enabled: true<br>  nodeSelector:<br>    beta.kubernetes.io/os: linux<br>  service:<br>    externalTrafficPolicy: Local<br>    type: LoadBalancer<br>defaultBackend:<br>  nodeSelector:<br>    beta.kubernetes.io/os: linux</pre><p>If all the previous steps were successfully completed in Prometheus, we can now proceed to view the NGINX metrics. To do this, we will perform a port-forward to access the Prometheus dashboard. Follow the next step:</p><pre>kubectl port-forward svc/prometheus-kube-prometheus-prometheus -n monitoring 9090:9090</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0oBXfaDR_YMQBpOvEFlfGQ.png" /></figure><p>If you are able to view these metrics, you can be assured that everything went as expected and that the configuration was successful. These metrics indicate that Prometheus is collecting and displaying data from NGINX, which is a crucial step in our process of monitoring and scalability.</p><h4>Setting KEDA Object</h4><p>Now, let’s move forward to creating the KEDA object, known as ScaledObject, to configure the automatic scaling of our application. This step is essential to ensure that our application scales according to the desired metrics.</p><pre>apiVersion: keda.sh/v1alpha1<br>kind: ScaledObject<br>metadata:<br>  name: quarkus-keda<br>  namespace: quarkus<br>spec:<br>  scaleTargetRef:<br>    apiVersion: apps/v1 # Default<br>    kind: Deployment # Default<br>    name: quarkus-helm-chart<br>  pollingInterval: 30 # Default 30<br>  cooldownPeriod: 300 # Default 300<br>  minReplicaCount: 1 # Default 0<br>  maxReplicaCount: 10 # Default 100<br>  triggers:<br>    - type: prometheus<br>      metadata:<br>        serverAddress: http://prometheus-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090<br>        metricName: nginx_service_requests_total<br>        threshold: &quot;40&quot;<br>        query: sum(rate(nginx_ingress_controller_requests[1m]))</pre><p>This ScaledObject will be created in the same “quarkus” namespace where we intend to deploy our application. In it, we are specifying which Kubernetes resource we want KEDA to scale, identified by the “scaleTargetRef” section. It’s important to note that we are requesting the scaling of our deployment, therefore, it’s necessary to provide the correct name of the deployment. This object is located inside the folder helm-chart/template</p><p><a href="https://github.com/rdurelli/keda-argo-nginx/tree/main/helm-chart/templates">https://github.com/rdurelli/keda-argo-nginx/tree/main/helm-chart/templates</a></p><p>Another relevant aspect in this object is the “trigger” section. Note that we are indicating our intention to use Prometheus to scale our application. To use Prometheus with KEDA, it’s necessary to define the address of the Prometheus server. Additionally, we need to specify the name of the desired metric, its threshold, and the query that will be executed on Prometheus to assess when scalability is needed. These settings are crucial to ensure that KEDA scales our application according to the desired metrics in an accurate and efficient manner.</p><p>Let’s proceed with the creation of our application in ArgoCD. In this example, we will opt to use the ArgoCD graphical interface to simplify the process. However, it’s important to note that it’s also possible to create an “Application” object directly if you prefer a code-based approach.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0r69P0bJwLvVbdshlb7Unw.png" /></figure><p>Click on “NEW APP”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ehXWKnbTnqcvF07VQDV4aA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q2jFAZ5LGaBBbcs2agk7sQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gTHy_JIP4JtVOPjYabxLSA.png" /></figure><p>After clicking on create, you should see the following screen:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q4yyWZYT-D1iR4a5NkX9Cw.png" /></figure><p>Note that we have an object named scaledobject and it has created an HPA (Horizontal Pod Autoscaler) for us… HURRAY!</p><p>To test if KEDA will scale our pods, we will use K6 to stress the application.</p><p>The code to use K6 is:</p><pre>import http from &#39;k6/http&#39;;<br>import { sleep } from &#39;k6&#39;;<br>export const options = {<br>  vus: 400,<br>  duration: &#39;90s&#39;,<br>};<br>export default function () {<br>  http.get(&#39;http://durellidevops.com.br/&#39;);<br>  sleep(1);<br>}</pre><p>Execute K6.</p><pre>k6 run keda/k6.js</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mJqraO6R8Gt9uwuSWan0tg.png" /></figure><p>Let’s then keep an eye on our pods:</p><pre>kubectl get pods -n quarkus -w</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aK5yV7koYac_yzVEZ5pXCg.png" /></figure><p>Observe the magic happening, pods being created with the assistance of KEDA.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nsPQUK3qjyUrD9asEw8oGg.png" /></figure><p>Now, when the K6 test finishes, our ScaledObject will terminate all the pods, leaving only one alive.</p><p>And there you have it, a seamless demonstration of the dynamic and intelligent capabilities of Kubernetes combined with KEDA and Prometheus. This powerful trio not only optimizes resource utilization but also ensures that your applications are always performing at their best, ready to handle varying loads with grace. This is the future of application scaling — efficient, responsive, and smart. Welcome to the new era of Kubernetes scaling!</p><p>#Kubernetes #KEDA #Prometheus #DevOps #CloudComputing #Microservices #Containerization #Scalability #Automation #TechInnovation</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8370a4bc5b30" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>