<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ramesh Yara on Medium]]></title>
        <description><![CDATA[Stories by Ramesh Yara on Medium]]></description>
        <link>https://medium.com/@yara.ramesh?source=rss-8474e1f1e204------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*c-jmYQneHpyhIMIc.</url>
            <title>Stories by Ramesh Yara on Medium</title>
            <link>https://medium.com/@yara.ramesh?source=rss-8474e1f1e204------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 22:25:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@yara.ramesh/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[I Deleted kubectl apply From My Workflow. Here’s What Replaced It.]]></title>
            <link>https://medium.com/@yara.ramesh/i-deleted-kubectl-apply-from-my-workflow-heres-what-replaced-it-b5e3090e84ec?source=rss-8474e1f1e204------2</link>
            <guid isPermaLink="false">https://medium.com/p/b5e3090e84ec</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[gitops]]></category>
            <category><![CDATA[argo-cd]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Ramesh Yara]]></dc:creator>
            <pubDate>Sun, 22 Nov 2026 20:43:30 GMT</pubDate>
            <atom:updated>2026-11-22T20:43:30.324Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MkPzFGnnz89fgZzE8J2Deg.png" /></figure><p>7 Helm releases running on AWS EKS — kube-prometheus-stack, Tempo, Loki, Promtail, Chaos Mesh, plus the microservices overlay. Each installed manually with helm install or kubectl apply -k from my laptop. No drift detection: if a resource was patched directly in the cluster, I&#39;d find out when something broke. No audit trail: the deployment history was a mix of terminal history and my own memory. Deployment ordering enforced by me running commands in the right sequence.</p><p>This is how most Kubernetes platforms look after the “it works” phase. The cluster is functional but fragile. The state of your infrastructure lives in your head, not in git.</p><p>This post documents how I changed that — using ArgoCD’s app-of-apps pattern to migrate the entire platform to GitOps in a single session.</p><h3>What ArgoCD App-of-Apps Actually Is</h3><p>The app-of-apps pattern is simpler than it sounds: you create one “root” ArgoCD Application that watches a directory in your git repository. Every YAML file in that directory is itself an ArgoCD Application manifest. When you push a new Application file to that directory, ArgoCD automatically creates the corresponding release. When you delete the file, ArgoCD prunes the release from the cluster.</p><p>One directory. One git push. ArgoCD handles the rest.</p><p>The root application for this platform is 34 lines:</p><pre>apiVersion: argoproj.io/v1alpha1<br>kind: Application<br>metadata:<br>  name: platform-root<br>  namespace: argocd<br>  finalizers:<br>  - resources-finalizer.argocd.argoproj.io<br>spec:<br>  project: default<br>  source:<br>    repoURL: https://github.com/Rummy43/ai-microservices-platform.git<br>    targetRevision: HEAD<br>    path: k8s/gitops/apps<br>  destination:<br>    server: https://kubernetes.default.svc<br>    namespace: argocd<br>  syncPolicy:<br>    automated:<br>      prune: true<br>      selfHeal: true<br>    syncOptions:<br>    - CreateNamespace=true</pre><p>That’s the entire bootstrap. One kubectl apply -f k8s/gitops/app-of-apps.yaml. Everything after that is a git push.</p><p>prune: true means ArgoCD deletes resources from the cluster when their definition is removed from git. selfHeal: true means ArgoCD automatically reverts manual cluster changes back to the git state. These two flags together are what makes this GitOps rather than just &quot;ArgoCD-managed deploys.&quot;</p><h3>The Seven Child Applications</h3><p>Inside k8s/gitops/apps/, there are seven Application manifests:</p><p>kube-prometheus-stack — monitoring — Helm multi-source<br>tempo — monitoring — Helm multi-source<br>loki — monitoring — Helm multi-source<br>observability-config — monitoring — Kustomize<br>promtail — monitoring — Helm multi-source<br>chaos-mesh — chaos-testing — Helm multi-source<br>microservices — microservices — Kustomize prod overlay</p><p>ArgoCD auto-detects the deployment type. If the target path contains a kustomization.yaml, it uses Kustomize. If it contains Chart.yaml, it uses Helm. No explicit type annotation required.</p><p>The microservices application points directly at the production overlay that was already managing the platform services before this migration:</p><pre>spec:<br>  source:<br>    path: k8s/overlays/prod<br>  syncPolicy:<br>    automated:<br>      prune: true<br>      selfHeal: true<br>    syncOptions:<br>    - ApplyOutOfSyncOnly=true</pre><p>ApplyOutOfSyncOnly=true is worth calling out: it tells ArgoCD to only apply resources that have actually drifted from git, rather than re-applying every resource on every sync cycle. For a Kustomize overlay with 20+ manifests, this matters for cluster stability.</p><h3>Sync Waves: Encoding Deployment Order in YAML</h3><p>The observability stack must be ready before microservices start. Prometheus needs to be scraping before services emit metrics. Chaos Mesh needs to be installed before you run experiments against it. This ordering exists whether you encode it or not — the question is whether it lives in a Confluence page, in your head, or in the deployment system itself.</p><p>ArgoCD sync waves encode it in the deployment system:</p><pre># Wave 0 — observability stack (must be ready first)<br>metadata:<br>  annotations:<br>    argocd.argoproj.io/sync-wave: &quot;0&quot;<br># Applied to: kube-prometheus-stack, tempo, loki, observability-config</pre><pre># Wave 1 — log shipping and chaos tooling (needs observability running)<br>metadata:<br>  annotations:<br>    argocd.argoproj.io/sync-wave: &quot;1&quot;<br># Applied to: promtail, chaos-mesh</pre><pre># Wave 2 — platform microservices (needs everything else ready)<br>metadata:<br>  annotations:<br>    argocd.argoproj.io/sync-wave: &quot;2&quot;<br># Applied to: microservices (user-service, notification-service, api-gateway)</pre><p>ArgoCD will not begin wave 1 until every application in wave 0 reports Healthy. It will not begin wave 2 until wave 1 is complete. If kube-prometheus-stack fails its health check, the microservices never start — which is correct behavior.</p><p>The practical value goes beyond ordering. It eliminates an entire category of incidents that begin with “which order did you install things?” You don’t answer that question from memory anymore. You read the wave number.</p><h3>Multi-Source Helm: Separating Chart Version from Values Config</h3><p>Standard Helm with ArgoCD gives you one source: a chart from a Helm repository with values inlined or from a file in the same application repo. Multi-source, available from ArgoCD 2.6+, lets you reference a Helm chart from one repository and your values files from another.</p><p>For kube-prometheus-stack:</p><pre>spec:<br>  sources:<br>  - repoURL: https://prometheus-community.github.io/helm-charts<br>    chart: kube-prometheus-stack<br>    targetRevision: &quot;91.8.0&quot;<br>    helm:<br>      releaseName: kube-prometheus-stack<br>      valueFiles:<br>      - $values/k8s/helm/kube-prometheus-stack/values.yaml<br>      - $values/k8s/helm/kube-prometheus-stack/values-eks.yaml<br>  - repoURL: https://github.com/Rummy43/ai-microservices-platform.git<br>    targetRevision: HEAD<br>    ref: values</pre><p>The second source defines the $values reference — a named pointer to your git repository. The first source uses $values/path/to/file.yaml to load values from that reference.</p><p>The same pattern applies to Tempo, Loki, Chaos Mesh, and Promtail — each has a base values.yaml for local kind cluster configuration and a values-eks.yaml for EKS-specific overrides (S3 backends, IRSA service accounts, gp3 storage classes).</p><p><strong>Chart version pinned in git.</strong> targetRevision: &quot;91.8.0&quot; lives in a commit in your platform repo. Upgrading the chart is a PR with a diff, not a manual helm repo update. Rollback is git revert.</p><p><strong>Environment-specific values separated cleanly.</strong> The base chart never sees your EKS configuration. Your EKS values never pollute your local cluster setup. The two layers compose at deploy time.</p><p><strong>Drift detection on Helm values.</strong> If someone runs helm upgrade with modified values directly in the cluster, ArgoCD will detect the drift and self-heal back to the git state within minutes. Without GitOps, that modification is permanently invisible.</p><h3>The 262KB Problem</h3><p>Chaos Mesh ships with large Custom Resource Definitions — the schemas that define PodChaos, NetworkChaos, StressChaos, IOChaos, and the rest of the fault injection primitives. When ArgoCD applies a large CRD using the standard kubectl apply path, it stores the entire last-applied configuration as an annotation on the resource. For Chaos Mesh&#39;s CRDs, that annotation exceeds the 262KB limit that Kubernetes imposes on annotation values.</p><p>The error on the first sync attempt:</p><pre>metadata.annotations: Too long: may not be more than 262144 bytes</pre><p>The fix is a single sync option in the ArgoCD Application manifest:</p><pre>syncOptions:<br>- ServerSideApply=true</pre><p>Server-side apply moves the apply logic from the client (ArgoCD/kubectl) to the Kubernetes API server. The last-applied-configuration annotation is no longer stored on the resource, so the annotation size limit doesn&#39;t apply.</p><p>This same fix was needed for kube-prometheus-stack — the Prometheus Operator CRDs have the same characteristic. The pattern to remember: if an ArgoCD application fails to sync with an annotation size error, ServerSideApply=true is the resolution. It&#39;s safe to enable for all Helm applications.</p><h3>What “Healthy/Synced” Actually Looks Like</h3><p>After the bootstrap and the first automated sync cycle, the ArgoCD UI showed:</p><pre>platform-root          Synced  Healthy<br>├── kube-prometheus-stack  Synced  Healthy   (wave 0)<br>├── tempo                  Synced  Healthy   (wave 0)<br>├── loki                   Synced  Healthy   (wave 0)<br>├── observability-config   Synced  Healthy   (wave 0)<br>├── promtail               Synced  Healthy   (wave 1)<br>├── chaos-mesh             Synced  Healthy   (wave 1)<br>└── microservices          Synced  Healthy   (wave 2)</pre><p>Seven applications. All Healthy. All Synced. Wave ordering verified in the event timeline.</p><p>The first real test came immediately after. I updated an image tag in a deployment manifest, committed, and pushed. ArgoCD detected the diff within 30 seconds. It applied only the changed resource (ApplyOutOfSyncOnly=true in action), triggered a rolling update, and reported Healthy within two minutes of the push.</p><p>No kubectl apply. No helm upgrade. No SSH into a bastion host. The git push was the deployment.</p><h3>Two Gaps the Game Day Found</h3><p>GitOps solved the deployment pipeline. It didn’t fix the gaps in the monitoring stack that the Phase 11 chaos engineering game day exposed.</p><p><strong>Gap 1: Kafka consumer lag alerts need a lag exporter to have a data source.</strong> The PipelineConsumptionStalled alert rule is in git, properly formatted, correctly deployed via ArgoCD. But the metric it queries — kafka_consumergroup_lag — does not exist in this cluster because no Kafka lag exporter is running. The alert rule evaluates to NaN and never fires. An alert with no metric source is a hypothesis, not a safeguard.</p><p><strong>Gap 2: SLO burn-rate alerts need real traffic to generate error budget burn.</strong> AvailabilityFastBurn fired correctly in prior testing. During the EKS game day, with only Prometheus scraping actuator endpoints as &quot;traffic,&quot; there was no request error budget to consume. The burn rate calculation worked correctly — it correctly identified no errors — which masked the actual latency degradation.</p><p>Both fixes will themselves be delivered via GitOps. The kafka-lag-exporter will be a new Application manifest in k8s/gitops/apps/. The traffic generator will be a CronJob added to the microservices overlay. When those files are pushed, ArgoCD deploys them automatically.</p><p>The irony of GitOps: it’s very good at deploying your solutions. You still have to find the gaps first.</p><h3>What Changed</h3><p><strong>Before:</strong> Deployment state lived in terminal history and my memory. Helm releases were upgraded in place with no record of what values changed. Ordering was enforced by convention and documentation. Drift was invisible until something broke. “What’s running in production?” was a question that required interrogating the cluster.</p><p><strong>After:</strong> Every resource in the cluster has a corresponding file in git. Every change is a commit — with a diff, a reason, and a timestamp. Deployment order is encoded in sync wave annotations, not in a runbook. ArgoCD detects drift within minutes and self-heals automatically. “What’s running in production?” is git log.</p><p>The bootstrap was one command. Everything since has been git push origin main.</p><p>If you’re running multiple Helm releases on Kubernetes and managing their state mentally, app-of-apps is the migration path. The pattern is straightforward once you’ve seen it once — the main decision is how to structure your sync waves to reflect the actual dependency order of your stack.</p><p><em>The platform source is at github.com/Rummy43/ai-microservices-platform. Phase 12 covers the kafka-lag-exporter and traffic generator — closing the game-day blind spots via, naturally, another git push.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b5e3090e84ec" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Resilience Feature That Wasn’t There]]></title>
            <link>https://medium.com/@yara.ramesh/the-resilience-feature-that-wasnt-there-fa12afe48186?source=rss-8474e1f1e204------2</link>
            <guid isPermaLink="false">https://medium.com/p/fa12afe48186</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[spring-boot]]></category>
            <dc:creator><![CDATA[Ramesh Yara]]></dc:creator>
            <pubDate>Thu, 22 Oct 2026 02:25:34 GMT</pubDate>
            <atom:updated>2026-10-22T02:25:34.643Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LZ4y7rctr0XuKWvkW80Xmg.png" /></figure><p>My platform’s dead-letter recovery was documented in three places. The code had never been written.</p><p>When I finished Phase 6 of my event-driven microservices platform, I updated the architecture decision record.</p><p>ADR-016 described a resilience layer: circuit breakers on outbound dependencies, rate limiting at the perimeter, and a self-healing DLT pipeline. Dead-lettered events wouldn’t sit in a table waiting for manual triage. A scheduled reprocessor would replay them with exponential backoff. A poison-message cap would prevent one bad event from blocking the rest. Dead letters would become an automatically draining queue.</p><p>The ADR status: Accepted.</p><p>The README described it. The runbook referenced it. I committed both and moved on.</p><p>Three months later, I was writing Phase 7 SLOs and noticed a Prometheus gauge I had added: dlt_unreprocessed. It was reading 32.</p><p>I went to check what the scheduled reprocessor would do with those 32 events. There was no scheduled reprocessor. The table had been growing since Phase 5 failure tests. Nothing had ever drained it. The self-healing mechanism existed in three documents and in zero Java files.</p><h4>How Documentation Gets Ahead of Implementation</h4><p>The ADR was not wrong — the design was correct. The outbox-to-DLT routing worked. When a Kafka message exhausted retries, @DltHandler caught it, persisted it to dead_letter_events, and that was it. Correct behavior. Durable failure record.</p><p>The reprocessor was the second half of ADR-016. It got tagged as “Phase 7 polish” in the backlog because the routing worked and I had moved into observability work. The backlog item never came back up. The ADR stayed at “Accepted.”</p><p>“Accepted” in an ADR means the team agreed the design is sound. It does not mean a deployment ran. It does not mean tests passed. It is not a CI gate.</p><p>This is the failure mode: the document describes the design correctly, but describes it in the present tense — “a scheduled reprocessor replays dead-lettered events” — while the code implements only part of it. The gap is invisible until someone has to operate the thing.</p><p>The gap became visible when I tried to write the DltDepthHigh runbook.</p><h4>The Runbook Forced the Issue</h4><p>The runbook question is always: what does an on-call engineer actually do when this fires?</p><p>My draft said: “The scheduled reprocessor will automatically replay pending events. Check dlt_unreprocessed to confirm it is draining.”</p><p>To validate that claim, I had to check that the reprocessor existed and would actually drain the gauge. That is when I found the 32-event backlog that had been sitting untouched since Phase 5. There was no reprocessor to reference.</p><p>The runbook is a correctness check for the alert. What I had not expected was that the runbook would become a correctness check for the architecture documentation.</p><h4>The Naive Fix and Why It Falls Short</h4><p>The obvious approach is a @Scheduled method that queries all unprocessed events, republishes them, and marks them done. This works for the happy path. It breaks in every edge case that matters in production.</p><p>No backoff. If the downstream system is unavailable, this fires every 60 seconds and hammers it. No poison-message detection. If one event has a malformed payload, it stays in the set forever and starves capacity from events that could succeed. No observability. Without tracking retry counts and failure reasons, you cannot distinguish transient failure from permanent failure.</p><h4>The Production Implementation</h4><p>The production version adds retry accounting, a poison-message cap, and metrics. Key design decisions:</p><p>Retry count cap: the query excludes events with retryCount &gt;= 5. They remain in the table — visible in the dlt_unreprocessed gauge — but the reprocessor stops attempting them. This prevents one unconvertible message from consuming retry slots indefinitely.</p><p>Synchronous send: kafkaTemplate.send(…).get() blocks until the broker acknowledges. For a low-volume scheduled job, this ensures a “sent” status actually means the broker received it.</p><p>Metrics in the loop: every success and failure increments a counter. These feed the Grafana panels and alert rules.</p><p>Configurable interval: fixedDelayString with a property reference lets you adjust reprocessing frequency without a recompile.</p><h4>Verifying the Drain</h4><p>After the first compile and restart, I watched the gauge: dlt_unreprocessed 32 → 18 → 7 → 0. notifications_dlt_reprocessed_total = 32. Clean drain — no poison events in the Phase 5 test data.</p><p>The DltDepthHigh alert — which had been firing since I had added it because the backlog was genuinely there — resolved and moved to INACTIVE.</p><p>Now the runbook was honest.</p><h4>What the ADR Gap Cost</h4><p>In this case: nothing operationally critical, because the Phase 5 DLT events were test data. But the 32-event backlog would have stayed there indefinitely. If those had been production failure events, they would have been silently unrecovered — durably captured, permanently lost.</p><p>More subtly: the DltDepthHigh alert would have kept firing. Every on-call engineer looking at it would have read a runbook that said “the reprocessor will drain it” while watching the gauge stay at 32. The runbook would have been actively misleading.</p><h4>Key Takeaways</h4><p>1. “ADR: Accepted” is not a deployment. It records that the team agreed the design is right. Someone still has to write the code, wire the beans, and verify it runs.</p><p>2. Runbooks are a correctness check for more than alerts. Writing “the scheduled reprocessor will drain it” forced me to verify the reprocessor existed. Runbooks that describe operational behavior without verifying that behavior expose exactly these gaps.</p><p>3. Poison-message caps are not optional. A scheduled reprocessor without a retry cap will starve on unconvertible messages. The cap turns a potential infinite loop into a finite retry window with a known exit state.</p><p>4. A DLT without a reprocessor is a graveyard. Events that land in a dead-letter table are durable — but durable and recoverable are not the same thing. If nothing is reading that table and replaying, you have good audit trail of things that failed permanently.</p><p>5. Verify your own documentation. It is easy to assume that what the ADR describes is what the code does. Auditing the claim — “does this mechanism actually exist?” — is as important as auditing any external dependency assumption.</p><p>The reprocessor is live. ADR-016 is now actually true. The DltDepthHigh alert resolves on its own.</p><p>The gap between documented and implemented is one of the least visible failure modes in a distributed system — because the documentation is usually correct, and the implementation is usually mostly there. The missing piece is often the second half of a compound feature, the part that seemed like “polish” when the first half worked.</p><p>Write the runbook. If you cannot verify the claim it makes, you have found the gap.</p><p>The full implementation is in the platform repository (https://github.com/Rummy43/ai-microservices-platform) under notification-service/src/main/java/…/DltReprocessorService.java. What is the most surprising gap you have found between an architecture document and the running code?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fa12afe48186" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Three Layers Deep: Getting Confluent Schema Registry to Authenticate with MSK Serverless on EKS]]></title>
            <link>https://medium.com/@yara.ramesh/three-layers-deep-getting-confluent-schema-registry-to-authenticate-with-msk-serverless-on-eks-d7793f3fc195?source=rss-8474e1f1e204------2</link>
            <guid isPermaLink="false">https://medium.com/p/d7793f3fc195</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Ramesh Yara]]></dc:creator>
            <pubDate>Mon, 12 Oct 2026 18:27:01 GMT</pubDate>
            <atom:updated>2026-10-12T18:27:01.506Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Setup</h3><p>When you deploy Confluent Schema Registry on EKS against an MSK Serverless cluster, you’re combining three things that each have their own opinions about authentication:</p><p>Confluent’s Docker image — assumes it controls its own environment<br>Kubernetes — helpfully injects environment variables from service discovery<br>MSK Serverless — requires SASL/OAUTHBEARER (IAM) with a specific JAR on the JVM classpath</p><p>In isolation, each is fine. Together, they’ll fight you in exactly three layers. Here’s what I found and how I fixed each one.</p><h3>The Environment</h3><p>EKS 1.31 (Fargate managed node group)<br>MSK Serverless (us-east-1)<br>confluentinc/cp-schema-registry:7.5.0 as the base image<br>MSK IAM auth via IRSA (IAM Roles for Service Accounts)<br>aws-msk-iam-auth-2.3.2-all.jar for the SASL callback handler</p><h3>Layer 1: Kubernetes Poisons the Schema Registry Config</h3><p>Symptom: Schema registry pod crashes immediately on startup. Logs show something like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lChIku63tmLIufTFV7pEwg.png" /></figure><p>Root cause: Kubernetes injects environment variables for every Service in the same namespace. If there’s a service named schema-registry, Kubernetes adds SCHEMA_REGISTRY_PORT=tcp://10.100.200.50:8081, SCHEMA_REGISTRY_PORT_8081_TCP, and more. Confluent’s startup scripts map any env var prefixed with SCHEMA_REGISTRY_ into the config file — so that TCP socket URL lands in the port field, which is not a valid port number.</p><p>Fix: Add enableServiceLinks: false to the pod spec. This tells Kubernetes to skip environment variable injection from service discovery for this pod. Your own explicitly-defined env vars are unaffected.</p><p>Same issue hits Kafka broker pods. Any Confluent image that uses the KAFKA_ or SCHEMA_REGISTRY_ env var convention will be crashed by K8s service link injection. Always set enableServiceLinks: false on these deployments.</p><h3>Layer 2: The MSK IAM Auth JAR Doesn’t Reach the JVM</h3><p>Symptom: Schema registry starts but immediately fails to connect to MSK Serverless with an SSL handshake failure — or silently retries PLAINTEXT/SSL when what MSK Serverless requires is SASL/OAUTHBEARER.</p><p>Root cause: MSK Serverless only supports IAM (SASL/OAUTHBEARER). The Confluent 7.5 base image doesn’t include the aws-msk-iam-auth JAR. You need to add it to the JVM classpath. But Confluent’s entrypoint scripts call java directly and ignore the CLASSPATH environment variable. Setting CLASSPATH=/path/to/aws-msk-iam-auth.jar in your deployment manifest does nothing.</p><p>Fix: Build a custom image that bakes the JAR directly into /usr/share/java/schema-registry/ — the directory Confluent’s launch scripts automatically add to the JVM classpath via wildcard. Use ADD with the Maven Central URL so Docker pulls the JAR at build time without needing it locally. The USER root / USER appuser sandwich is required because Confluent’s base image runs as appuser (uid 1000) but writing to that directory requires root.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kiTDQMwVRt8ktrqHMqvedA.png" /></figure><p>Then configure SCHEMA_REGISTRY_KAFKASTORE_SECURITY_PROTOCOL=SASL_SSL, SCHEMA_REGISTRY_KAFKASTORE_SASL_MECHANISM=AWS_MSK_IAM, SCHEMA_REGISTRY_KAFKASTORE_SASL_JAAS_CONFIG, and SCHEMA_REGISTRY_KAFKASTORE_SASL_CLIENT_CALLBACK_HANDLER_CLASS in your env vars.</p><p>Why not CLASSPATH? The Confluent Docker image uses launch scripts that construct the java command internally. The CLASSPATH env var is only used if java is invoked without explicit -cp/-classpath flags. Confluent sets its own -cp, so your CLASSPATH override is ignored.</p><h3>Layer 3: The Ensure Script Fails Pre-Flight Checks</h3><p>Symptom: Pod starts, gets past the JAR loading, but then hangs for minutes before failing with a connectivity error — even though MSK is accessible.</p><p>Root cause: The Confluent Docker image runs /etc/confluent/docker/ensure as a pre-flight check before launching Schema Registry. In the EKS environment, it runs before IRSA credentials are fully available, so the connectivity check fails before the IAM token exchange completes.</p><p>Fix: Override the ensure script with a no-op via a ConfigMap subPath mount. The subPath: ensure mount replaces only that single file in the container’s filesystem — the rest of /etc/confluent/docker/ is untouched. The defaultMode: 0755 ensures the script is executable.</p><p>Is this safe? Schema Registry itself retries connectivity on startup with proper retry/backoff logic, so skipping the pre-flight check doesn’t lose safety — it just removes a redundant check that doesn’t understand your auth setup anyway.</p><h3>The Complete Custom Image</h3><p>The full Dockerfile adds the MSK IAM auth JAR to the Confluent classpath directory. Build and push to ECR with docker build, then aws ecr get-login-password piped to docker login, then docker push.</p><h3>Why Each Layer Was Non-Obvious</h3><p>The three failures shared a pattern: the error message pointed at the symptom, not the source.</p><p>Layer 1 — enableServiceLinks: false: The error blamed SCHEMA_REGISTRY_PORT being invalid, with no mention of Kubernetes service injection at all.</p><p>Layer 2 — Custom Dockerfile with JAR: The CLASSPATH env var looks like it should work; the fact that Confluent ignores it is undocumented.</p><p>Layer 3 — No-op ensure script: The ensure failure looks like a connection error, masking that IRSA credentials were not ready yet.</p><p>The real lesson: when you combine a third-party container image with a managed cloud service and a container orchestrator, each one adds an invisible layer of convention. The debug process is peeling them back one at a time — which is exactly why the error messages never point at the right thing.</p><h3>The Smoke Test</h3><p>After all three layers are resolved, run from inside the EKS cluster:<br>kubectl exec -n microservices deploy/user-service — curl -s <a href="http://schema-registry:8081/subjects">http://schema-registry:8081/subjects</a><br>This returns [] on first boot, confirming the connection is live.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I8KsJmKKROpqIyeETyFUeg.jpeg" /></figure><p>And the full path: POST /api/v1/users through the ALB → user-service → MSK Serverless (via Schema Registry for Avro serialization) → notification-service consumer. HTTP 201.</p><p>Have you run into similar layering conflicts when integrating managed AWS services with third-party Kubernetes workloads? What was your debugging process?</p><p>Part of the AI Microservices Platform series — a production-grade event-driven system built with Java 21 + Spring Boot 4, deployed to AWS EKS with Terraform.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d7793f3fc195" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Five Things Kubernetes Quietly Added to My Containers (And How Each One Broke a Confluent Service)]]></title>
            <link>https://medium.com/@yara.ramesh/five-things-kubernetes-quietly-added-to-my-containers-and-how-each-one-broke-a-confluent-service-972d621916fd?source=rss-8474e1f1e204------2</link>
            <guid isPermaLink="false">https://medium.com/p/972d621916fd</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Ramesh Yara]]></dc:creator>
            <pubDate>Mon, 07 Sep 2026 18:31:33 GMT</pubDate>
            <atom:updated>2026-09-07T19:03:46.984Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dDRiykMCJaGO-xJ_JftG6w.png" /></figure><p>When I moved a full Confluent Kafka stack into a Kubernetes cluster, five things broke. Each one looked like a configuration mistake I had made. None of them were.</p><p>Every failure came from something Kubernetes added to my containers without being asked — a default behavior that is correct in the general case and a silent hazard for these specific images. Here is what each one did and the fix.</p><h3>1. Kubernetes injected a variable that crashed the Kafka broker</h3><p>The broker came up for a few seconds, then died. The log said the advertised listener address was pointing to a TCP socket URL — something like tcp://10.96.0.1:9092 — instead of a hostname. I had set the advertised listener explicitly. Something had overridden it.</p><p>Kubernetes had.</p><p>By default, every pod in a namespace receives environment variables for every Service in that namespace. For a Service named kafka, Kubernetes injects KAFKA_PORT=tcp://10.96.0.1:9092, KAFKA_SERVICE_HOST=10.96.0.1, and several more. This is a legacy feature from the Docker links era, on by default for backwards compatibility.</p><p>The Confluent Kafka image maps every environment variable prefixed with KAFKA_ directly to a broker configuration key. The injected KAFKA_PORT variable landed in that mapping, overrode the advertised listener I had configured, and produced an invalid broker config. The crash log blamed the configuration value — not the source of it.</p><p>The Confluent Schema Registry image has the same convention with SCHEMA_REGISTRY_* prefixed variables. A Service named schema-registry injects SCHEMA_REGISTRY_PORT, which the image reads as a configuration for its deprecated PORT key.</p><p>The fix is one field:</p><pre>spec:<br>  enableServiceLinks: false</pre><p>Both the broker and Schema Registry came up cleanly once service link injection was disabled on their pod specs.</p><h3>2. CPU limits without requests made every pod Pending</h3><p>After the broker came up, all eight pods went Pending simultaneously. kubectl describe pod showed Insufficient cpu on every one of them. The node had 2 CPU cores and nothing was running on it.</p><p>The cause was how Kubernetes handles resource specs with only limits defined. When a container specifies resources.limits.cpu but omits resources.requests.cpu, Kubernetes silently sets the request equal to the limit. With eight pods each carrying an implicit 500m CPU request, the scheduler saw 4000m of demand against a 2000m node. All eight were Unschedulable before a single one started.</p><p>The fix is an explicit minimal request:</p><pre>resources:<br>  requests:<br>    cpu: 1m<br>    memory: 512Mi<br>  limits:<br>    cpu: 500m<br>    memory: 1Gi</pre><p>1m is nominal — it tells the scheduler that the container needs one millicore of guaranteed scheduling capacity, not 500m. The scheduler placed all eight pods immediately.</p><p>The rule generalizes: omitting requests does not mean no CPU requirement. It means requirement equals limit. For a stack of JVM services alongside Kafka and a database, the implicit math almost always exhausts the node before a single pod starts.</p><h3>3. The default probe timeout was 1 second — and the JVM needed 3</h3><p>With pods scheduled, the next issue surfaced: MySQL and PostgreSQL were intermittently NotReady, causing cascading Connection refused errors in the Spring Boot services.</p><p>The default timeoutSeconds for Kubernetes probes is 1 second. Under Docker Desktop and WSL2, mysqladmin ping and pg_isready routinely take 1–3 seconds to respond even on fully initialized databases. Overlayfs plus a constrained CPU budget introduces latency that does not exist on a local host process.</p><p>When a 1-second timeout fires against a database that takes 1.5 seconds to respond, Kubernetes marks the pod NotReady and removes it from the Service endpoint list. Any connection pool — Hikari — that opens a connection during that window hits Connection refused from the cluster’s DNS resolver, not from the database itself. The error looks like a connectivity problem in the application.</p><p>Raising the timeout resolved it:</p><pre>readinessProbe:<br>  timeoutSeconds: 5<br>startupProbe:<br>  timeoutSeconds: 5</pre><p>The 1-second default is appropriate for bare-metal or cloud clusters where probe responses are in the sub-100ms range. In a local WSL2 environment it fires spurious failures that cascade through every service that depends on the probed component.</p><h3>4. Keycloak’s health endpoint wasn’t where the probe was looking</h3><p>Keycloak also had a probe problem, but a different one.</p><p>In Keycloak’s production mode (start), the health endpoints live on the management port — port 9000. In development mode (start-dev), only port 8080 is opened. Port 9000 is silent.</p><p>My probe was checking /health/ready on port 9000. It was timing out against a port that did not exist, leaving Keycloak permanently NotReady.</p><p>Fixing the port was necessary but not sufficient. /health/ready on port 8080 returns 404 in both modes. The correct readiness signal for Keycloak in this setup is the realm endpoint:</p><pre>readinessProbe:<br>  httpGet:<br>    path: /realms/ai-microservices<br>    port: 8080</pre><p>The realm endpoint returns 200 only after Keycloak’s Liquibase migration has run and the realm has been imported from the ConfigMap. It verifies that the actual downstream dependency — token issuance — is ready, not just that the process started.</p><p>The startup window also needs to account for Keycloak’s real initialization time. On WSL2, start-dev takes 8–20 minutes from pod creation to a valid realm response. The startup probe failureThreshold needs to be set accordingly:</p><pre>startupProbe:<br>  httpGet:<br>    path: /realms/ai-microservices<br>    port: 8080<br>  failureThreshold: 120<br>  periodSeconds: 15<br>  # 120 x 15s = 30-minute startup window</pre><h3>5. A rolling update killed Schema Registry — permanently</h3><p>With everything stable, I issued a kubectl apply to update the Schema Registry deployment. The new pod came up alongside the old one — standard rolling update behavior — and Schema Registry died in that window and never recovered.</p><p>Schema Registry uses Kafka as its backing store and participates in a Kafka consumer group to elect a leader that coordinates reads and writes to the _schemas topic. When two Schema Registry pods run simultaneously — even briefly during a rolling update — both attempt to join the consumer group. Kafka triggers a partition rebalance.</p><p>The KafkaGroupLeaderElector thread that manages Schema Registry’s leadership does not survive that rebalance. It exits with LEADER_NOT_AVAILABLE and does not restart. All subsequent requests to register or retrieve schemas fail until the pod is manually restarted.</p><p>With a single replica, every rolling update creates a two-pod window. The fix is to use Recreate:</p><pre>spec:<br>  strategy:<br>    type: Recreate</pre><p>Recreate terminates the old pod before starting the new one. No two-pod window, no rebalance. At single-replica scale the tradeoff is brief unavailability during the update — acceptable because the platform’s transactional outbox absorbs the gap rather than losing events.</p><h3>What these five have in common</h3><p>Each failure was documented Kubernetes behavior. None of them were bugs. Each one was a default designed for a broad set of workloads that happened to be wrong for this specific combination of images and topology.</p><p>Service link injection exists to help containers discover other services. Implicit request-equals-limit exists to prevent pods from being unschedulable when only limits are set. The 1-second probe timeout exists because most health endpoints are fast. Rolling update semantics exist because most applications can run multiple instances during a deploy.</p><p>Confluent images that interpret KAFKA_* environment variables as configuration, JVM startup latency under WSL2, Keycloak’s development-mode port behavior, and Schema Registry’s single-leader election model are each correct on their own. The failures come from the intersection.</p><p>The practical checklist that came out of this:</p><ul><li>enableServiceLinks: false on every Confluent image (Kafka, Schema Registry) — and any image that reads its own name as an env var prefix.</li><li>Explicit requests.cpu on every container, even if it’s nominal.</li><li>timeoutSeconds: 5 on readiness and startup probes in any local cluster.</li><li>Read the image’s health endpoint documentation, not the Kubernetes docs, to find the correct probe path and port for each version.</li><li>strategy: Recreate for any single-replica service that participates in leader election.</li></ul><p>All five are now set in every manifest I write for this stack.</p><p><em>This platform — Spring Boot 4, Java 21, Kafka, Keycloak, and a full in-cluster observability stack — is open-source: </em><a href="https://github.com/Rummy43/ai-microservices-platform"><em>github.com/Rummy43/ai-microservices-platform</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=972d621916fd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Outbox Pattern Is Not Enough]]></title>
            <link>https://medium.com/@yara.ramesh/the-outbox-pattern-is-not-enough-07c4fe231296?source=rss-8474e1f1e204------2</link>
            <guid isPermaLink="false">https://medium.com/p/07c4fe231296</guid>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[spring-boot]]></category>
            <dc:creator><![CDATA[Ramesh Yara]]></dc:creator>
            <pubDate>Mon, 17 Aug 2026 01:23:56 GMT</pubDate>
            <atom:updated>2026-08-17T01:23:56.250Z</atom:updated>
            <content:encoded><![CDATA[<p>The textbook version of the transactional outbox is tight. You save the domain entity and an outbox row in one local transaction. A background scheduler picks up PENDING rows and publishes them to Kafka. You never publish inside the request thread — no dual-write, no atomicity breach. The pattern closes the consistency gap.</p><p>Then you load-test it.</p><p>I ran 1,000 authenticated requests through my event-driven platform in 70 seconds. The gateway returned 201 for every one of them. The outbox absorbed every row. The consumer drained everything. By every visible metric the system looked healthy. Underneath that health, I found three production-grade problems the textbook never mentioned.</p><h3>What a correct implementation looks like</h3><p>Before the problems, the shape of the solution. The outbox publisher runs on a @Scheduled virtual-thread worker:</p><pre>@Scheduled(fixedDelay = 5000)@Transactionalpublic void publishPendingEvents() {    List&lt;OutboxEvent&gt; batch = outboxRepository        .findTop20ByStatusOrderByCreatedAtAsc(OutboxStatus.PENDING);<br>    for (OutboxEvent event : batch) {        event.setStatus(OutboxStatus.PROCESSING);        outboxRepository.save(event);<br>        try {            kafkaTemplate.send(event.getTopic(), event.getPayload()).get();            event.setStatus(OutboxStatus.PUBLISHED);        } catch (Exception e) {            event.incrementRetryCount();            if (event.getRetryCount() &gt;= MAX_RETRIES) {                event.setStatus(OutboxStatus.FAILED);            } else {                event.setStatus(OutboxStatus.PENDING);            }        }        outboxRepository.save(event);    }}</pre><p>This is correct. The PROCESSING state prevents another scheduler instance from claiming the same row. The retry cap prevents infinite cycling. The PENDING fallback on transient errors gives the event another chance. The dual-write problem is genuinely closed.</p><p>Here is what that correctness does not cover.</p><h3>Gap 1: Your throughput ceiling is a config line</h3><p>fixedDelay = 5000 means the scheduler runs every 5 seconds. findTop20 means it picks up 20 rows per cycle.</p><p>Maximum publish throughput: 20 events divided by 5 seconds = 4 events per second.</p><p>That number does not appear in your unit tests. It does not appear in your monitoring unless you specifically look for it. It is a ceiling determined by two config values chosen without measurement.</p><p>During the 1,000-event baseline run, the gateway processed ~14.3 requests per second. The publisher was draining at 4 per second. The backlog grew to 720 rows before the burst ended and the scheduler caught up. outbox_oldest_pending_age_seconds — the gauge that measures the age of the oldest PENDING row — peaked at 191 seconds.</p><pre>[outbox burst profile]t=0s    → 1,000 requests fire at 14.3 req/st=70s   → all 1,000 return HTTP 201; 720 rows queuedt=70s+  → publisher drains at 4 ev/s (~3 minutes to clear)         outbox_oldest_pending_age_seconds peak: 191st=250s  → backlog reaches 0; gauge returns to 0</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c4RPxo9OrEXWm9B6b1Vn3w.jpeg" /><figcaption>outbox_oldest_pending_age_seconds — a 200-request parallel burst on the K8s cluster. Age climbs to 110s as inbound rate (≫4 req/s) outpaces the publisher, then drops to zero as the publisher catches up. The 07–17 baseline (1,000 requests at 14.3 req/s) peaked at 191s and took 3 minutes to drain.</figcaption></figure><p>The system worked correctly. No events were lost. No data was corrupted. But the freshness SLO — events delivered within 30 seconds of creation — was structurally impossible to meet at any input rate above 4 req/s. The ceiling is not a bug. It is a design constant hiding in plain sight.</p><p>The honest version of a freshness SLO for this system is: Events are delivered within 30 seconds, given input rates below 4 req/s. That constraint belongs in your SLO catalog, not buried in two config lines.</p><pre># SLO catalog entry — what honest capacity accounting looks likefreshness_slo:  target: 99.9% of events published within 30s of creation  constraint: input rate ≤ 4 ev/s (fixedDelay=5000ms × batchSize=20)  at_higher_rates: freshness degrades proportionally; availability unaffected</pre><h3>Gap 2: The alert you write will fire for the wrong reason</h3><p>The natural monitoring instinct for the outbox is an age threshold:</p><pre>- alert: OutboxBacklogAgeHigh  expr: outbox_oldest_pending_age_seconds &gt; 60  for: 5m  labels:    severity: page</pre><p>During the 720-row burst, outbox_oldest_pending_age_seconds crossed 191 seconds. The threshold is 60 seconds. Two rules went pending.</p><p>Neither fired.</p><p>The for: 5m clause — which distinguishes a transient burst from a sustained incident — held. The burst resolved in under 5 minutes. Both rules sat pending through the whole event and silently cleared when the backlog drained.</p><p>That is the correct behavior. But it is only correct if you understand why the window exists.</p><p>The naive response to a pending alert is to shorten the window. Drop for: from 5 minutes to 30 seconds to catch problems faster. What you actually get is pages for every deployment spike, every cold-start burst, every</p><p>Schema Registry restart. The alert stops being a signal and becomes noise that engineers learn to dismiss — which is worse than no alert at all.</p><p>The for: duration is where you encode your operational definition of incident. The 720-row burst at 191 seconds is a load event: the system is processing work faster than it can publish, and it will self-resolve when the input rate drops. A broker offline for 7 minutes is an incident: the backlog grows indefinitely and the oldest-age gauge only increases.</p><p>You cannot calibrate that boundary by intuition. You need to run a burst that is definitively not an incident — a known-finite load, system otherwise healthy — and measure the oldest-age peak. Then set for: so that peak does not fire. In this system: 191s peak, 5-minute for: window, zero false positives across every deployment and cold-start since.</p><h3>Gap 3: The terminal state is invisible by design</h3><p>This is the one that costs you.</p><p>When the publisher exhausts its retry budget, the row becomes FAILED and the scheduler never touches it again. Your outbox_oldest_pending_age_seconds goes back to zero — there are no more PENDING rows to report age for. Your backlog count goes to zero. Your age alert stays silent.</p><p>The event is gone. No notification was sent. No audit trail was written downstream. Nothing alerted.</p><p>This happened on my platform. During the cluster&#39;s first night, Schema Registry took approximately four minutes to become ready after a pod restart. The outbox publisher started immediately and encountered Error registering Avro schema for every publish attempt. After five retries, five rows were marked FAILED permanently.</p><p>The Grafana dashboard showed a healthy system. The Prometheus alerts list showed no active rules. The outbox_failed gauge was plotted on the business dashboard and showing 5. No one had wired a rule to it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H-kJZmD9XS4fvwN80v3PpQ.jpeg" /><figcaption>Five terminal FAILED rows in MySQL (retry_count=5, Error serializing Avro message) alongside the Grafana Outbox Backlog panel showing failed=5 with zero firing alerts — the gauge existed, the rule did not.</figcaption></figure><p>Those rows sat there for 24 hours.</p><p>The gauge without an alert is a decoration. The outbox_failed metric existed. It had a panel. It had a y-axis label. It had no operational consequence. The fix was not adding a new gauge — it was adding the rule that should have been there from the start:</p><pre>- alert: OutboxPublishTerminalFailure  expr: outbox_failed &gt; 0  for: 2m  labels:    severity: page    slo: outbox_integrity  annotations:    runbook: https://docs.internal/runbooks/outbox-failed    summary: &quot;Terminal FAILED outbox rows detected — events permanently blocked&quot;</pre><p>The for: 2m gives the scheduler one extra cycle to confirm before paging. The severity is page because FAILED is a permanent state — unlike PENDING, it never self-heals. The runbook names the re-drive query: update rows to PENDING, identify the root cause, resolve it, let the publisher retry.</p><p>Live-fire verified against a real condition: the FAILED rows described above triggered OutboxPublishTerminalFailure after 2 minutes. Alertmanager delivered [active] OutboxPublishTerminalFailure | severity=page | slo=outbox_integrity. Rows deleted. Alert auto-resolved in 30 seconds.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XKb21I8kx2G-PtZgxfWm0Q.jpeg" /><figcaption>Alertmanager showing OutboxPublishTerminalFailure active — severity=page, slo=outbox_integrity, namespace=microservices — the pager delivery moment.</figcaption></figure><h3>The operational layer, assembled</h3><p>Three concrete additions on top of a correct outbox implementation:</p><p>​1. Name the throughput ceiling in your SLO catalog. fixedDelay and batchSize are not internal implementation details. They are your freshness SLO&#39;s capacity constraint. Make them visible, version them, and review them when input load changes.</p><p>​2. Calibrate the age alert against measured burst data. Run a known-safe burst. Record the oldest-age peak. Set for: so the burst does not page. Re-run the calibration whenever you change the batch size or schedule interval.</p><p>​3. Alert unconditionally on FAILED, with severity page. The terminal state has exactly the property that makes it most dangerous: it looks like a healthy system. Wire the rule, write the runbook, and treat every FAILED row as a lost event until the runbook says otherwise.</p><h3>The numbers</h3><p>Everything above came from a 70-second load run on a JWT-authenticated, fully containerized platform:</p><p>Requests | 1,000 POST /api/v1/users</p><p>HTTP 201 | 1,000 (100%)</p><p>Availability SLI | 1.0</p><p>Latency SLI | 0.999</p><p>p99 gateway / user-service | 186ms / 177ms</p><p>Peak backlog | 720 rows</p><p>Oldest-age peak | 191 seconds</p><p>Publisher rate | ~4 ev/s</p><p>Consumer drain rate | ~150 ev/s</p><p>False-positive alerts during burst | 0</p><p>Terminal failure was live-fired separately: synthetic FAILED row inserted directly into the outbox table → OutboxPublishTerminalFailure pending at t=0, FIRING at t=2m → Alertmanager severity=page delivered → row deleted → alert auto-resolved in 30 seconds.</p><h3>Key takeaways</h3><p>​- The outbox closes the dual-write problem; it opens an operational one. Works correctly and is observable and bounded are different properties.</p><p>​- Your throughput ceiling is determined by two config values that most implementations never name. Instrument it, measure it, put it in your SLO catalog.</p><p>​- The for: duration is where you define incident. Shortening it to catch faster is how alerts become noise. Calibrate it against real burst data, not intuition.</p><p>​- A gauge without an alert is a decoration. The terminal FAILED state has exactly the property that makes it most dangerous: it looks like a healthy system.</p><p>​- FAILED rows never self-heal. Alert on them unconditionally, with severity page, and with a runbook that names the re-drive query.</p><p>​- Honest SLOs require measurement. A freshness target that does not name its throughput constraint is either vacuous or untested.</p><p>The full implementation — outbox publisher, status lifecycle, PENDING/FAILED gauges, alert rules, and runbooks — is in the open-source repository:</p><p><a href="https://github.com/Rummy43/ai-microservices-platform">GitHub - Rummy43/ai-microservices-platform</a></p><p>What&#39;s the throughput ceiling of your outbox implementation, and have you measured it?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=07c4fe231296" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Who Did This? Identity Across Async Boundaries]]></title>
            <link>https://medium.com/@yara.ramesh/who-did-this-identity-across-async-boundaries-823c712b073f?source=rss-8474e1f1e204------2</link>
            <guid isPermaLink="false">https://medium.com/p/823c712b073f</guid>
            <category><![CDATA[backend-engineering]]></category>
            <category><![CDATA[spring-boot]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Ramesh Yara]]></dc:creator>
            <pubDate>Thu, 09 Jul 2026 05:22:26 GMT</pubDate>
            <atom:updated>2026-07-09T05:22:26.223Z</atom:updated>
            <content:encoded><![CDATA[<p>Why an authenticated request goes anonymous the moment it crosses into Kafka — and how to make identity as durable as the event itself.</p><p>You put a lot of work into authentication. A gateway validates the Keycloak JWT, maps realm roles to authorities, checks that the caller is allowed. By the time a request reaches your service, you know exactly who is calling.</p><p>Then the request crosses into async land, and all of that evaporates.</p><p>This is the story of the point where identity quietly disappears in an event-driven system, why the dead-letter queue is the worst possible place for it to disappear, and how I made the acting user as durable and replay-safe as the event itself.</p><h3>The flow everyone believes is fine</h3><p>The platform is a set of Spring Boot services: an API gateway in front, a user-service on MySQL, a notification-service on PostgreSQL, and Kafka carrying events between them. A user is created, an event is published, a notification is sent.</p><p>Authentication is handled at the edge. The gateway is an OAuth2 Resource Server; it validates the token once and propagates the caller’s identity downstream as headers:</p><pre>// api-gateway — IdentityPropagationFilter (@Order(2), after security)<br>IdentityContext identity = identityContextExtractor.extract(jwt);<br><br>// Always set all three headers (empty when absent) to mask any spoofed values.<br>enrichedRequest.putHeader(IdentityHeaders.USER_NAME,  nullToEmpty(identity.username()));<br>enrichedRequest.putHeader(IdentityHeaders.USER_EMAIL, nullToEmpty(identity.email()));<br>enrichedRequest.putHeader(IdentityHeaders.USER_ROLES, identity.rolesAsString(DELIM));</pre><p>One detail here matters more than it looks. The headers are <strong>always overwritten</strong>, even when a claim is absent. If a client tries to inject X-User-Name: admin on the inbound request, the gateway stomps it with the validated value (or empty). Downstream trust in those headers is only safe because the perimeter guarantees they cannot be forged. Miss that, and you&#39;ve built an impersonation API.</p><p>So far, so good. The synchronous hop carries identity. The problem starts one line later.</p><h3>The hidden failure: the thread boundary</h3><p>I don’t publish to Kafka inside the request. I use the transactional outbox pattern: the request persists the user <strong>and</strong> an outbox row in one local transaction, and a separate scheduled poller publishes to Kafka afterward. (Why: a direct publish inside the request can commit the DB row and then lose the event if the broker call fails — the dual-write problem. The outbox closes that gap.)</p><p>That decoupling is correct for delivery. It is also exactly where identity dies.</p><p>The outbox publisher runs on a <strong>scheduled thread</strong>, not the request thread. SecurityContextHolder, ThreadLocal, MDC — every ambient place you might have stashed &quot;who is calling&quot; — are all empty by the time the poller runs. There is no request. There is no token. There is nothing to read.</p><p>So the event goes out anonymous. The consumer logs anonymous. The notification is written anonymous. And when an event exhausts its retries and lands in the dead-letter queue — the one moment you will <em>urgently</em> want to know who triggered it — the dead-letter row is anonymous too. Provenance vanishes at precisely the point forensics begins.</p><h3>The naive fixes (and why each one fails)</h3><p>The tempting answers all fail at the same boundary:</p><p><strong>“Just log the username in the controller.”</strong> You can — but the log line you care about is the <em>publish</em>, which happens later, on another thread, after the HTTP response has already returned. The controller log tells you a request arrived; it can’t attribute the event that failed twenty seconds later.</p><p><strong>“Stash it in MDC / a ThreadLocal and read it in the publisher.”</strong> The publisher isn’t on your thread. MDC is thread-scoped; the scheduled poller starts with a clean, empty context. You’ll read null every time. Worse, under Virtual Threads with pooled carriers, a <em>stale</em> ThreadLocal is a correctness hazard — you can leak one request&#39;s identity onto another&#39;s event.</p><p><strong>“Re-read the JWT in the publisher.”</strong> There is no request in scope and no token to re-validate. The authentication event is long over.</p><p>Every naive fix assumes identity lives in ambient thread state. Across an async boundary, ambient state is exactly what you don’t have.</p><h3>The production implementation: persist identity with the event</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BsnfFGsH7c3VIKjUCeIozw.png" /><figcaption>identity propagation flow</figcaption></figure><p>If the publish is decoupled from the request in <em>time</em> and <em>thread</em>, then identity has to travel the same way the event does — <strong>as data, not as ambient context</strong>.</p><p>So I capture the actor on the request thread and persist it onto the outbox row, inside the same transaction as the entity write:</p><pre>// user-service — OutboxEventService (runs on the request thread)<br>IdentityContext actor = IdentityContextHolder.get().orElse(null);<br>String traceId = MDC.get(CorrelationConstants.TRACE_ID);<br><br>OutboxEvent outboxEvent = OutboxEvent.builder()<br>        .eventId(...).aggregateType(&quot;USER&quot;).eventType(&quot;USER_CREATED&quot;)<br>        .payload(objectMapper.writeValueAsString(payload))<br>        .status(OutboxEventStatus.PENDING)<br>        .actorUsername(actor != null ? actor.username()      : null)<br>        .actorEmail(   actor != null ? actor.email()         : null)<br>        .actorRoles(   actor != null ? actor.rolesAsString() : null)<br>        .traceId(traceId)<br>        .build();<br><br>outboxEventRepository.save(outboxEvent);   // same TX as the user write</pre><p>Now identity is as durable as the event. It survives a crash, a restart, a redeploy — because it’s a committed row, not a value on a dying thread.</p><p>The scheduled publisher later rehydrates that context from the row and rides it onto the message as Kafka headers — supplied <strong>explicitly</strong>, never read from MDC:</p><pre>// user-service — EventPublisher (runs on the scheduled outbox thread)<br>public void publishUserCreatedEvent(UserCreatedEvent event, String traceId, IdentityContext actor) {<br>    ProducerRecord&lt;String, Object&gt; record = new ProducerRecord&lt;&gt;(TOPIC, event.getId().toString(), event);<br>    addHeader(record, KAFKA_TRACE_ID_HEADER, traceId);<br>    if (actor != null) {<br>        addHeader(record, KAFKA_USER_NAME,  actor.username());<br>        addHeader(record, KAFKA_USER_EMAIL, actor.email());<br>        addHeader(record, KAFKA_USER_ROLES, actor.rolesAsString());<br>    }<br>    kafkaTemplate.send(record);<br>}</pre><p>The consumer does the mirror operation: read the headers, rebind identity to the consumer thread and MDC so every log line is attributed — and, critically, <strong>clear it in a </strong>finally<strong> block</strong> so nothing bleeds into the next message on a reused thread:</p><pre>// notification-service — KafkaConsumerService<br>IdentityContext actor = restoreIdentityContext(userNameHeader, userEmailHeader, userRolesHeader);<br>try {<br>    notificationService.sendWelcomeNotification(event, topic, partition, offset, attempt);<br>} finally {<br>    clearIdentityContext();          // MDC.remove(...) + IdentityContextHolder.clear()<br>    MDC.remove(CorrelationConstants.TRACE_ID);<br>}</pre><p>A single immutable IdentityContext record is the canonical &quot;who is acting&quot; shape at every hop — serialized to HTTP headers at the gateway, to a DB row in the outbox, to Kafka headers on publish, back to MDC on consume. It&#39;s deliberately transport-agnostic, so a sync→async hop never has to re-derive trust from the token.</p><p>Here’s the whole path, including the failure branch:</p><pre>[HTTP request thread]                    [scheduled thread]           [consumer thread]<br>Gateway (validate JWT)                    Outbox Publisher             @KafkaListener<br>   │ X-User-* headers (overwritten)          │ read actor from row        │ read actor from headers<br>   ▼                                          ▼                            ▼ bind to MDC<br>user-service ──save user + outbox row──▶ outbox_events ──Kafka headers──▶ process + log (attributed)<br>             (one transaction: entity                                     │<br>              + actor_username/email/roles)                               │ retries exhausted<br>                                                                          ▼<br>                                                             @DltHandler → dead_letter_events<br>                                                             (actor_* columns persisted)</pre><h3>The payoff: the dead-letter queue keeps its memory</h3><p>The reason all of this is worth it lives in the @DltHandler. When every retry is exhausted, the failed event is persisted for triage <strong>with the originating actor attached</strong>:</p><pre>deadLetterEventRepository.save(DeadLetterEvent.builder()<br>        .eventId(event.getEventId().toString())<br>        .lastError(errorMessage)<br>        .actorUsername(actor.username())<br>        .actorEmail(actor.email())<br>        .actorRoles(actor.roles().isEmpty() ? null : actor.rolesAsString())<br>        .build());</pre><p>A dead letter without an actor is a mystery ticket. A dead letter <em>with</em> one is an incident you can route, reproduce, and explain. The schema changes that back this are explicit migrations — Liquibase actor_* + trace_id on outbox_events (MySQL), Flyway actor_* on dead_letter_events and notification_log (PostgreSQL) — not ddl-auto guesses.</p><h3>Verification: prove it survives the failure path</h3><p>Assertion isn’t evidence. I verified the branch that matters — the failure one — by fault injection: force the notification step to fail so an event exhausts its four attempts and dead-letters, then read the audit table.</p><pre>SELECT event_id, actor_username, actor_email, last_error<br>FROM   dead_letter_events<br>ORDER  BY failed_at DESC;<br>-- actor_username / actor_email populated on the failed row ✅</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YFvFKkw-g4_jdcs4OXA6Bw.png" /><figcaption>dlt actor columns</figcaption></figure><p>The identity is there, on a <em>failed</em> event, written by a background thread that never saw the original request. That’s the whole thesis in one row.</p><p>A subtle bonus: because identity rides Kafka headers as data, it survives even where distributed <em>tracing</em> currently doesn’t. On this platform the OTel agent doesn’t yet emit Kafka spans (a gap I wrote about separately), so the async trace edge is blind — yet the actor still crosses it, because header-borne data doesn’t depend on span instrumentation. Two independent propagation mechanisms; the durable one keeps working when the automatic one has a hole.</p><h3>Key takeaways</h3><ul><li><strong>Authentication is not propagation.</strong> Validating a token proves a request is allowed <em>now</em>; it says nothing about who’s behind the event that fails an hour later on another thread.</li><li><strong>Across an async boundary, identity must be data, not ambient context.</strong> ThreadLocal/MDC/SecurityContext are all empty on the publisher and consumer threads. Persist the actor with the event.</li><li><strong>Attach identity in the same transaction as the write.</strong> That’s what makes it as durable and replay-safe as the event — surviving crashes, retries, and redeploys.</li><li><strong>The dead-letter queue is where provenance matters most and is usually lost.</strong> Persisting actor_* on the DLT row turns an anonymous mystery into a routable incident.</li><li><strong>Trust in propagated headers requires an unspoofable perimeter.</strong> Unconditionally overwrite X-User-* at the gateway, or you&#39;ve built impersonation-as-a-service.</li><li><strong>Clear context in </strong>finally<strong>.</strong> Under Virtual Threads and pooled consumers, a stale ThreadLocal is a cross-request correctness bug, not just untidy.</li></ul><p>The platform is open source — gateway, both services, the outbox, and this identity path end to end: <a href="https://github.com/Rummy43/ai-microservices-platform">https://github.com/Rummy43/ai-microservices-platform</a></p><p>When one of your events lands in the dead-letter queue tonight, can you tell who triggered it?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=823c712b073f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your Service Map Is Lying]]></title>
            <link>https://medium.com/@yara.ramesh/your-service-map-is-lying-cfc84fb38990?source=rss-8474e1f1e204------2</link>
            <guid isPermaLink="false">https://medium.com/p/cfc84fb38990</guid>
            <category><![CDATA[spring-boot]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Ramesh Yara]]></dc:creator>
            <pubDate>Sun, 21 Jun 2026 04:47:01 GMT</pubDate>
            <atom:updated>2026-06-21T04:47:01.820Z</atom:updated>
            <content:encoded><![CDATA[<p>Verifying OpenTelemetry auto-instrumentation before you trust a generated dependency graph</p><p>You attach the OpenTelemetry Java agent, point it at a collector, and within minutes Grafana is drawing a service map you never drew. A box for each service, arrows between them, latency on every edge. It feels like magic, and — more dangerously — it feels <em>complete</em>. “The agent traces everything” is the sentence repeated in every onboarding doc.</p><p>This is the story of the moment that sentence stopped being true on my platform, why I’m glad it did, and the difference between a system that is <em>working</em> and a system you can actually <em>see</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E07T6WpMn9lBh_zXI7zj0w.png" /></figure><h3>The flow everyone trusts</h3><p>The platform is an event-driven set of Spring Boot services: an API gateway in front, a user-service backed by MySQL, a notification-service backed by PostgreSQL, and Kafka carrying events between them. A user is created, an event is published, a notification is sent.</p><p>I didn’t want to <em>draw</em> that topology. A hand-drawn architecture diagram is documentation that drifts — true the day you commit it, slightly wrong a month later, actively misleading after a quarter. I wanted the dependency graph generated from live traffic, so it would always reflect what the system actually does.</p><p>Grafana Tempo does exactly this. Its service-graphs processor reads matched client/server span pairs out of trace data and emits a metric — traces_service_graph_request_total — that Grafana renders as a node graph. No edge is ever wired by hand. The topology is derived, continuously, from real spans.</p><h3>The edge that wasn’t there</h3><p>I generated the graph and the synchronous edges lit up immediately:</p><pre>api-gateway → user-service<br><br>user-service → MySQL<br><br>notification-service → PostgreSQL</pre><p>Then I looked for the one edge I actually cared about — user-service → notification-service, the asynchronous hop over Kafka.</p><p>It wasn’t there.</p><h3>The naive conclusion (and why it’s wrong)</h3><p>The tempting read is immediate and obvious: <em>the async hop is broken. The event isn’t getting across. Go debug the consumer.</em></p><p>So I checked. And the consumer was completely fine. notification-service had consumed every event and written every corresponding row to PostgreSQL. Its database edge was lit. Liveness was perfect; the feature worked end to end.</p><p>That is the trap. A missing edge looks exactly like a broken feature, and the instinct is to go “fix” something that was never broken. The defect wasn’t in the message path at all — it was in the <em>observability</em> of the message path. Those are two different failure domains that happen to render identically on a dashboard.</p><p>So I stopped trusting the picture and went to the source of truth: the trace store.</p><h3>Evidence over assertion</h3><p>Two TraceQL queries settled it.</p><pre>{ span.messaging.system = &quot;kafka&quot; }<br>→ 0 results</pre><pre>{ resource.service.name = &quot;user-service&quot; } &amp;&amp; { resource.service.name = &quot;notification-service&quot; }<br>→ 0 traces</pre><p>Zero Kafka messaging spans anywhere in the system. Zero traces spanning both services. The agent — this build, under this Spring Boot version — simply was not instrumenting the Kafka client. Nothing errored. No warning was logged. The map wasn’t wrong about the data; the data was never produced.</p><p>This is the part worth sitting with: a green dashboard would have let me believe the chain was fully traced. The absence of red is not the presence of coverage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4bQy7kzdvcJZlgPcUe5foQ.png" /></figure><h3>How the graph is actually built</h3><p>The pipeline that produces the map is worth seeing, because one of its links is <em>also</em> a silent path you have to consciously turn on:</p><pre>OTel Java Agent (spans)<br>        │<br>        ▼<br>Tempo  (service-graphs processor)<br>        │  remote_write<br>        ▼<br>Prometheus  (--web.enable-remote-write-receiver)<br>        │  query<br>        ▼<br>Grafana  (Tempo data source → serviceMap → nodeGraph)</pre><p>Three deliberate decisions shaped it:</p><p>Generated edges, not declared ones. Tempo reads client/server span pairs and emits the edge metric. The graph is a test, not a drawing — it <em>fails</em> when reality diverges from expectation, which is exactly what makes it valuable.</p><p>One processor, scoped on purpose. I enabled service-graphs only — deliberately not span-metrics. The latter generates RED/latency series and would have inflated cardinality for a deliverable that was strictly about topology. Minimal blast radius, single responsibility.</p><p>A push boundary you must open explicitly. Tempo <em>remote-writes</em> its generated metrics into Prometheus — the inverse of every other component, which Prometheus scrapes. That requires flipping Prometheus into a receiver with --web.enable-remote-write-receiver. Forget it, and the metrics silently never land. Same lesson as the Kafka gap, one layer down: the data paths that fail quietly are the ones nobody turned on.</p><h3>One missing edge, two real causes</h3><p>There’s a subtlety the graph forced me to articulate. Even once the agent emits Kafka spans, the user-service → notification-service edge would <em>still</em> not look like a normal synchronous call — because my publish doesn&#39;t happen on the request thread.</p><p>I use the transactional outbox pattern: the request persists the user and an outbox row in one local transaction, and a scheduled poller publishes to Kafka afterward. So the Kafka branch roots its <em>own</em> trace, off the poll cycle, structurally separate from the originating HTTP request. That’s the outbox decoupling working as designed — not a propagation bug.</p><p>So a single missing edge had two distinct, both-legitimate explanations: a real instrumentation gap (no Kafka spans) and a real design boundary (async decoupling). Conflating them would have been the naive read. The generated graph didn’t just find a bug — it made an architecture decision legible.</p><h3>Making the map legible, not just generated</h3><p>There was one more way the graph was technically correct but practically dishonest: both databases first rendered as a single node named localhost. The agent labels an uninstrumented peer by its host, and both databases live on the loopback interface — so MySQL and PostgreSQL collapsed into one vertex. A heterogeneous-persistence design rendered as if it shared one database.</p><p>The fix is a peer-service mapping on the agent:</p><pre>-Dotel.instrumentation.common.peer-service-mapping=\<br>  localhost:3306=MySQL,localhost:5432=PostgreSQL,\<br>  localhost:8085=SchemaRegistry,localhost:8180=Keycloak</pre><p>Now each dependency resolves to a real, named vertex, and the Schema Registry shows up as its own node. I confirmed it the same way I confirmed everything else — in the backend, not the UI:</p><pre>traces_service_graph_request_total{server=&quot;MySQL&quot;}        → present<br>traces_service_graph_request_total{server=&quot;PostgreSQL&quot;}   → present<br>traces_service_graph_request_total{server=&quot;localhost&quot;}    → 0 (gone)</pre><p>A graph that says localhost is <em>generated</em>; a graph that says MySQL and PostgreSQL is <em>legible</em>. The labels are what turn telemetry into topology.</p><h3>Key takeaways</h3><ul><li>Auto-instrumentation is a claim, not a guarantee. “The agent traces everything” holds until a version boundary quietly says otherwise. Verify spans exist in the backend; never infer coverage from the absence of errors.</li><li>A missing edge is not a missing feature. Liveness and observability are independent failure domains. A system can be correct and unobservable at the same time — the most dangerous state, because it looks fine.</li><li>A generated graph is a test, not a decoration. Its entire value is that it can fail when reality diverges. A hand-drawn diagram cannot fail; that’s precisely why it’s worthless as a safety net.</li><li>Silent data paths must be consciously enabled. Remote-write receivers, peer mappings, opt-in instrumentation — the telemetry that “just never lands” is the telemetry nobody turned on. Make the gaps loud.</li><li>Name your peers. Topology you can’t read is only half-built; labels are the difference between a generated graph and a useful one.</li><li>Verify with queries, not screenshots. Green is the absence of evidence, not evidence of absence.</li></ul><p>The platform is open source :</p><p><a href="https://github.com/Rummy43/ai-microservices-platform">GitHub - Rummy43/ai-microservices-platform</a></p><p>It includes the gateway, user-service, notification-service, Kafka, OpenTelemetry, Tempo, Loki, Prometheus, Grafana, and the service-map setup discussed in this article.</p><p>What’s the most expensive silent gap you’ve found by verifying instead of trusting the dashboard?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cfc84fb38990" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Database Transactions and Kafka Publishing Are Not Atomic]]></title>
            <link>https://medium.com/@yara.ramesh/why-database-transactions-and-kafka-publishing-are-not-atomic-45923c390dd8?source=rss-8474e1f1e204------2</link>
            <guid isPermaLink="false">https://medium.com/p/45923c390dd8</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[spring-boot]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[software-architecture]]></category>
            <dc:creator><![CDATA[Ramesh Yara]]></dc:creator>
            <pubDate>Fri, 22 May 2026 05:09:54 GMT</pubDate>
            <atom:updated>2026-05-22T05:09:54.337Z</atom:updated>
            <content:encoded><![CDATA[<h4>Building reliable event-driven microservices using the Transactional Outbox Pattern, Kafka, Prometheus, Grafana, and Loki</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LY6lPiMbbskVGOq2nfSALQ.png" /></figure><p>In event-driven microservices architectures, a common assumption is:</p><p>“Save data to the database and publish an event to Kafka.”</p><p>At first glance, the workflow seems straightforward.</p><p>But there is an important distributed systems problem hidden inside that flow:</p><p>Database transactions and Kafka publishing are not atomic.</p><p>The database transaction may succeed while the Kafka publish operation fails.</p><p>When that happens:<br>- the data already exists in the database<br>- but downstream services never receive the event</p><p>This creates distributed inconsistency across services.</p><p>To address this reliability challenge, I implemented the Transactional Outbox Pattern in my event-driven microservices platform using:<br>- Spring Boot<br>- Kafka<br>- PostgreSQL/MySQL<br>- Prometheus<br>- Grafana<br>- Loki<br>- Distributed correlation tracing</p><p><strong>The Distributed Consistency Problem</strong></p><p>Consider a simple workflow:</p><p>User API Request<br>↓<br>Save User in Database<br>↓<br>Publish UserCreatedEvent to Kafka</p><p>Now imagine this sequence:</p><p>1. Database transaction commits successfully<br>2. Kafka broker becomes unavailable<br>3. Event publishing fails</p><p>Result:<br>- the user exists in the database<br>- but downstream services never receive the event</p><p>This creates a reliability gap in distributed systems.</p><p>Asynchronous communication improves scalability and decoupling, but it also introduces consistency challenges that must be handled explicitly.</p><p><strong>Transactional Outbox Pattern</strong></p><p>To solve this problem, I implemented the Transactional Outbox Pattern.</p><p>Instead of publishing directly to Kafka inside the request transaction, the service performs:</p><p>Save User<br>Save Outbox Event<br>(COMMIT)</p><p>Both operations happen within the SAME database transaction.</p><p>Later, a dedicated outbox publisher asynchronously:<br>- reads pending outbox events<br>- publishes them to Kafka<br>- updates publishing status</p><p><strong>Event Flow</strong></p><pre>User API Request<br>      ↓<br>User Service (MySQL Transaction)<br>      ├── Save User<br>      └── Save Outbox Event<br>      ↓<br>Outbox Publisher<br>      ↓<br>Kafka Topic<br>      ↓<br>Notification Service<br>      ↓<br>Idempotent Consumer Processing</pre><p><strong>Reliability Improvements</strong></p><p>The implementation includes several production-style reliability mechanisms.</p><p>Retry-Safe Publishing<br>Failed outbox events are retried automatically before being marked permanently failed.</p><p>PROCESSING State<br>Outbox events transition through:</p><p>PENDING → PROCESSING → PUBLISHED</p><p>This reduces duplicate processing risks in multi-instance scheduler environments.</p><p>Idempotent Consumer<br>The notification service tracks processed event IDs to prevent duplicate side effects during Kafka retries or message re-delivery.</p><p>Dead Letter Topic (DLT)<br>Kafka retry failures are routed to DLT topics for operational visibility and recovery workflows.</p><p><strong>Operational Observability</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LY6lPiMbbskVGOq2nfSALQ.png" /></figure><p>Reliability without observability becomes difficult to operate at scale.</p><p>The platform exposes:<br>- JVM metrics<br>- HTTP metrics<br>- Kafka consumer metrics<br>- Transactional outbox operational metrics</p><p>using:<br>- Spring Boot Actuator<br>- Micrometer<br>- Prometheus<br>- Grafana</p><p>Custom outbox operational metrics include:<br>- pending events<br>- processing events<br>- failed events</p><p>This provides visibility into asynchronous workflow health and Kafka publishing reliability.</p><p><strong>Centralized Logging and Distributed Traceability</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rd9i_fTTdFCHRhstXPnBqg.png" /></figure><p>The platform also supports centralized logging using:<br>- Loki<br>- Promtail<br>- Grafana Explore</p><p>Structured JSON logs include:<br>- traceId<br>- service name<br>- Kafka workflow traceability</p><p>This enables cross-service debugging and distributed request tracing across asynchronous Kafka workflows.</p><p><strong>Key Takeaways</strong></p><p>Modern distributed systems require much more than asynchronous messaging.</p><p>Reliable event-driven architectures must also address:<br>- consistency<br>- retries<br>- duplicate processing<br>- operational visibility<br>- traceability<br>- centralized debugging</p><p>The Transactional Outbox Pattern helps bridge the reliability gap between database transactions and asynchronous Kafka publishing while improving operational resilience.</p><p><strong>Project Repository</strong></p><p>GitHub:<br><a href="https://github.com/Rummy43/ai-microservices-platform">https://github.com/Rummy43/ai-microservices-platform</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=45923c390dd8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Observability in Event-Driven Microservices: Metrics, Dashboards, and Traceability]]></title>
            <link>https://medium.com/@yara.ramesh/observability-in-event-driven-microservices-metrics-dashboards-and-traceability-774678de1e2c?source=rss-8474e1f1e204------2</link>
            <guid isPermaLink="false">https://medium.com/p/774678de1e2c</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[backend-engineering]]></category>
            <category><![CDATA[microservices]]></category>
            <dc:creator><![CDATA[Ramesh Yara]]></dc:creator>
            <pubDate>Sun, 17 May 2026 03:19:06 GMT</pubDate>
            <atom:updated>2026-05-17T03:19:06.762Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d4EReiJzx1BQxeop7QzOvg.png" /><figcaption>Local observability dashboard monitoring JVM, HTTP, CPU, and Kafka consumer metrics across event-driven microservices.</figcaption></figure><h3>Introduction</h3><p>Building microservices is not only about making services communicate.</p><p>In event-driven systems, a request may return successfully while the actual work continues asynchronously through Kafka consumers, retries, dead-letter topics, and downstream processing.</p><p>That means a simple question becomes harder to answer:</p><p><strong>Is the system really healthy?</strong></p><p>A successful API response does not always mean the full distributed flow completed successfully.</p><p>Recently, while building a Kafka-based event-driven microservices platform, I focused on adding observability and traceability across the system using:</p><ul><li>Spring Boot Actuator</li><li>Micrometer</li><li>Prometheus</li><li>Grafana</li><li>Correlation IDs</li><li>Kafka header propagation</li></ul><p>This post walks through what I implemented, why it matters, and what I learned while making the system observable beyond the happy path.</p><h3>Why Observability Matters in Event-Driven Systems</h3><p>In a synchronous application, debugging is usually more direct.</p><p>A request enters the system, the service processes it, and a response is returned.</p><p>But in an event-driven system, the flow can look like this:</p><pre>HTTP Request<br>      ↓<br>User Service<br>      ↓<br>Kafka Event<br>      ↓<br>Notification Service<br>      ↓<br>Retry / DLT / Processing Logs</pre><p>The original API request may succeed, but downstream processing can still fail, retry, lag behind, or produce duplicates.</p><p>Without observability, these problems remain hidden until they become production issues.</p><p>That is why metrics, dashboards, and traceability are not optional in distributed systems. They are part of the system design.</p><h3>Metrics Collection with Spring Boot Actuator, Micrometer, and Prometheus</h3><p>To expose runtime metrics from each service, I added Spring Boot Actuator and Micrometer.</p><p>Spring Boot Actuator exposes internal application metrics through endpoints like:</p><pre>/actuator/health<br>/actuator/prometheus</pre><p>Micrometer acts as the instrumentation layer. It collects metrics from the application and exports them in a format Prometheus can understand.</p><p>Prometheus then scrapes those endpoints at regular intervals and stores the data as time-series metrics.</p><p>In this setup, Prometheus collects metrics from:</p><pre>user-service<br>notification-service</pre><p>This helped monitor:</p><ul><li>JVM heap memory usage</li><li>HTTP request throughput</li><li>CPU utilization</li><li>Kafka consumer activity</li></ul><h3>Grafana Dashboard</h3><p>Prometheus stores the metrics, but Grafana makes them useful visually.</p><p>I created a Grafana dashboard to monitor the platform across multiple dimensions:</p><ul><li>JVM Heap Memory Usage</li><li>HTTP Requests Rate</li><li>CPU Usage</li><li>Kafka Records Consumption Rate</li></ul><p>The dashboard helped me quickly understand how the services behave when traffic flows through the system.</p><p>For example:</p><ul><li>HTTP request spikes showed incoming traffic</li><li>Kafka consumer metrics showed async processing activity</li><li>JVM memory helped observe runtime behavior</li><li>CPU usage showed service-level load patterns</li></ul><p>The biggest takeaway:</p><p><strong>A dashboard is not just for monitoring. It becomes a feedback loop for understanding system behavior.</strong></p><h3>Traceability with Correlation IDs</h3><p>Metrics tell us what is happening.</p><p>But when debugging distributed systems, we also need to know:</p><p><strong>Which request caused this behavior?</strong></p><p>To solve that, I added correlation ID propagation.</p><p>The flow looks like this:</p><pre>Incoming HTTP Request<br>      ↓<br>Correlation ID Filter<br>      ↓<br>User Service Logs<br>      ↓<br>Kafka Event Headers<br>      ↓<br>Notification Service Consumer<br>      ↓<br>Notification Processing Logs</pre><p>If the incoming request already has a correlation ID, the system reuses it.</p><p>If not, the user service generates one and adds it to the response header.</p><p>The same trace ID is then propagated through Kafka headers and extracted by the notification service.</p><p>Example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8_IBpJcAFMcmzBjQMZHm_w.png" /><figcaption>user-service: Traceability</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mnqtzMGzZZDJGNflsD6S2g.png" /><figcaption>notification-service: Traceability</figcaption></figure><pre>INFO [user-service,traceId:a0ad6e62-50ce-41b2-a401-6dec0d930d9e]<br>INFO [notification-service,traceId:a0ad6e62-50ce-41b2-a401-6dec0d930d9e]</pre><p>This makes it much easier to trace a single business flow across synchronous and asynchronous boundaries.</p><h3>Why This Matters in Real Systems</h3><p>In real-world systems, failures are not always immediate or obvious.</p><p>A system may look healthy from the API layer while internal async processing is struggling.</p><p>Examples:</p><ul><li>Kafka consumers may be lagging</li><li>Retry attempts may be increasing</li><li>DLT records may be accumulating</li><li>JVM memory may be growing</li><li>Downstream processing may be delayed</li></ul><p>Without observability, teams usually discover these issues too late.</p><p>With observability, teams can detect patterns earlier, debug faster, and understand how the system behaves under load and failure.</p><h3>Key Learnings</h3><p>A few important lessons stood out:</p><ul><li>A successful API response does not guarantee the full distributed flow completed</li><li>Metrics help understand system behavior beyond logs</li><li>Prometheus is useful for collecting and querying time-series metrics</li><li>Grafana makes runtime behavior visible through dashboards</li><li>Correlation IDs make async debugging significantly easier</li><li>Observability should be designed early, not added only after incidents</li></ul><p>Most importantly:</p><p><strong>In distributed systems, operational visibility is just as important as application logic.</strong></p><h3>Conclusion</h3><p>This project started as a simple event-driven microservices implementation.</p><p>Over time, it evolved into a deeper exploration of production-grade backend patterns:</p><ul><li>Kafka-based async communication</li><li>Avro schema contracts</li><li>Retry and DLT handling</li><li>Idempotent consumers</li><li>Prometheus and Grafana observability</li><li>Correlation ID based traceability</li></ul><p>The more I work on distributed systems, the more I realize that building the feature is only one part of the work.</p><p>The real engineering challenge is making the system reliable, traceable, and understandable when things go wrong.</p><p>GitHub Repository:<br> <a href="https://github.com/Rummy43/ai-microservices-platform">https://github.com/Rummy43/ai-microservices-platform</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=774678de1e2c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Idempotency in Distributed Systems: From Concept to Kafka Implementation]]></title>
            <link>https://medium.com/@yara.ramesh/idempotency-in-distributed-systems-from-concept-to-kafka-implementation-68d453a05733?source=rss-8474e1f1e204------2</link>
            <guid isPermaLink="false">https://medium.com/p/68d453a05733</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[backend-engineering]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Ramesh Yara]]></dc:creator>
            <pubDate>Sun, 26 Apr 2026 04:34:15 GMT</pubDate>
            <atom:updated>2026-04-26T04:34:15.507Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-0jTql9ac_wb6kZl399RrA.png" /><figcaption>This diagram illustrates how duplicate Kafka events are safely handled using idempotent consumer patterns and database constraints.</figcaption></figure><h3>🔹 Introduction</h3><p>In distributed systems, handling failures is expected.</p><p>But one problem that often gets underestimated is <strong>duplicate event processing</strong>.</p><p>While working on an event-driven microservices setup using Kafka, I encountered scenarios where the same event could be delivered multiple times — especially under retries, restarts, or failure recovery.</p><p>At first glance, it seems harmless.</p><p>But in real systems, duplicate processing can lead to:</p><ul><li>Sending multiple notifications</li><li>Incorrect financial transactions</li><li>Inconsistent system state</li></ul><p>This led me to implement and validate an <strong>Idempotent Consumer pattern</strong> — ensuring that even if an event is received multiple times, it is processed only once.</p><h3>🔹 Why Duplicate Events Happen</h3><p>Kafka guarantees <strong>at-least-once delivery</strong>.</p><p>This means:</p><ul><li>Messages may be delivered more than once</li><li>Consumers must be designed to handle duplicates</li></ul><p>Duplicate events can occur due to:</p><ul><li>Retries after transient failures</li><li>Consumer restarts before offset commit</li><li>Network issues or broker-level re-delivery</li></ul><p>👉 The system must be designed to handle these scenarios safely.</p><h3>🔹 What is Idempotency?</h3><p>Idempotency ensures that:</p><blockquote><em>Performing the same operation multiple times produces the same result</em></blockquote><p>In the context of event-driven systems:</p><ul><li>Processing an event once or multiple times should lead to the same final state</li></ul><p>Example:</p><ul><li>Sending a welcome email should happen only once</li><li>Even if the event is received multiple times</li></ul><h3>🔹 Naive Approach (and Why It Fails)</h3><p>A common initial implementation looks like this:</p><pre>if (!processedEventRepository.existsByEventId(eventId)) {<br>    processEvent(event);<br>    processedEventRepository.save(eventId);<br>}</pre><p>At first glance, this seems correct.</p><p>But under concurrency:</p><ul><li>Two consumers can check at the same time</li><li>Both see the event as “not processed”</li><li>Both proceed to process it</li></ul><p>👉 This leads to duplicate side effects.</p><h3>🔹 My Implementation</h3><p>To solve this, I implemented a <strong>multi-layer idempotency strategy</strong>.</p><h3>1. Unique Event Identifier</h3><p>Each event carries a unique eventId:</p><pre>{<br>  &quot;eventId&quot;: &quot;uuid&quot;,<br>  &quot;userId&quot;: &quot;123&quot;,<br>  &quot;email&quot;: &quot;test@test.com&quot;<br>}</pre><p>This becomes the <strong>idempotency key</strong>.</p><h3>2. Processed Events Table</h3><p>A dedicated table tracks processed events:</p><pre>CREATE TABLE processed_events (<br>    id UUID PRIMARY KEY,<br>    event_id VARCHAR(255) UNIQUE,<br>    event_type VARCHAR(100),<br>    processed_at TIMESTAMP,<br>    topic VARCHAR(100),<br>    partition_number INT,<br>    offset_number BIGINT<br>);</pre><h3>3. Application-Level Check</h3><p>Before processing:</p><pre>if (processedEventRepository.existsByEventId(eventId)) {<br>    log.warn(&quot;Duplicate event detected — skipping&quot;);<br>    return;<br>}</pre><h3>4. Database-Level Constraint (Critical)</h3><p>To handle race conditions:</p><pre>ALTER TABLE processed_events<br>ADD CONSTRAINT unique_event_id UNIQUE (event_id);</pre><p>Even if two consumers pass the check:</p><p>👉 The database ensures only one insert succeeds.</p><h3>5. Exception Handling</h3><pre>try {<br>    processedEventRepository.save(event);<br>} catch (DataIntegrityViolationException ex) {<br>    log.warn(&quot;Duplicate detected at DB level — skipping&quot;);<br>    return;<br>}</pre><h3>🔹 Final Processing Flow</h3><ol><li>Receive event</li><li>Check if already processed</li><li>Process event (send notification)</li><li>Save processed event</li><li>If duplicate → skip safely</li></ol><h3>🔹 Validation (How I Tested It)</h3><p>To validate idempotency:</p><ul><li>Re-published the same Kafka event with the same eventId</li><li>Simulated duplicate delivery scenarios</li><li>Verified that:<br>- Event was processed only once<br>- Duplicate attempts were skipped<br>- No duplicate notifications were triggered</li></ul><h3>🔹 Key Takeaways</h3><ul><li>Kafka provides <strong>at-least-once delivery</strong>, not exactly-once</li><li>Idempotency must be implemented at the consumer level</li><li>Application-level checks alone are not enough</li><li>Database constraints are critical for handling race conditions</li><li>Correctness is more important than execution</li></ul><h3>🔹 Conclusion</h3><p>Building distributed systems is not just about making services communicate.</p><p>It’s about ensuring that the system behaves correctly under failure, retries, and concurrency.</p><p>Idempotency is a foundational pattern that helps achieve this — especially in event-driven architectures.</p><p>If you’re working with Kafka or distributed systems, I’d be curious:</p><p>👉 How do you handle duplicate events or race conditions in your systems?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=68d453a05733" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>