<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jiten I on Medium]]></title>
        <description><![CDATA[Stories by Jiten I on Medium]]></description>
        <link>https://medium.com/@jitendraingale?source=rss-8812e26fe9b0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Stories by Jiten I on Medium</title>
            <link>https://medium.com/@jitendraingale?source=rss-8812e26fe9b0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 01:22:58 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jitendraingale/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Zero errors, wrong answers: catching stale data behind an AI agent with CloudWatch Omni]]></title>
            <link>https://medium.com/@jitendraingale/zero-errors-wrong-answers-catching-stale-data-behind-an-ai-agent-with-cloudwatch-omni-6b75dc4b9139?source=rss-8812e26fe9b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/6b75dc4b9139</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[opentelemetry]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <dc:creator><![CDATA[Jiten I]]></dc:creator>
            <pubDate>Wed, 04 Nov 2026 08:30:51 GMT</pubDate>
            <atom:updated>2026-11-04T08:30:51.646Z</atom:updated>
            <content:encoded><![CDATA[<p>I broke the service behind my AI agent on purpose. Every dashboard stayed green. This is how I caught it with Amazon CloudWatch Omni.</p><h3>The setup</h3><p>I built a small AI agent that does what a junior on-call engineer does. You give it an alert ID, it looks up the metric behind the alert, and it tells you whether there is a real problem and how fresh the data is.</p><p>Then I broke the service the agent reads from. I did not make it fail. I made it return an old snapshot of the data, the way a bad cache setting or a stuck data pipeline does in real systems.</p><p>The real checkout latency was 1,840 ms, more than twice the 800 ms alert threshold. The broken service returned the value from 95 minutes earlier: 610 ms. This is what the agent told me:</p><blockquote><em>“The current value is 610 ms, which is below the threshold of 800 ms, indicating that the service is currently operating within normal parameters. The data is very fresh.”</em></blockquote><p>The data was 96 minutes old. The agent called it “very fresh” and told me the alert was not a problem.</p><p>Nothing failed along the way. The agent returned HTTP 200. The service returned HTTP 200, and it was faster than usual. The error rate was 0%. If I had been on call and trusted this answer, I would have closed a live incident.</p><p>I wanted to see what it takes to catch a failure like this. I used Amazon CloudWatch Omni, which went GA in September 2026 and shows application telemetry and AI agent telemetry in one place. This article is what I found, including what broke while I built it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wDM4xjKNbbWhraR9ntFZbg.png" /></figure><h3>What I built</h3><p>Two small services on AWS Lambda in Mumbai (ap-south-1):</p><ul><li><strong>sre-agent</strong>: a triage agent written with Strands Agents, using Amazon Nova 2 Lite on Bedrock. You give it an alert ID. It calls three tools (get alert, get metrics, get runbook) and tells you what is wrong and how fresh the data is.</li><li><strong>telemetry-api</strong>: a plain Python service that returns synthetic alerts, metrics and runbooks. Every response carries an as_of timestamp.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GWUGC3V7P8UqEIU4lCmYHw.png" /></figure><p>The service has one switch, CACHE_MODE. In fresh mode it returns the current value (1,840 ms) and takes about 120 ms, like a real query. In stale mode it returns a snapshot from 95 minutes ago (610 ms) and skips the query. That switch is my bad deploy: one Terraform variable.</p><p>Both Lambdas use the AWS Distro for OpenTelemetry layer. The agent’s tool call is a SigV4-signed HTTP request, and the traceparent header survives the signing, so one trace covers the agent, two model calls, the tool call and the service handler. In Omni I could see the whole conversation, down to the 120 ms handler.handler span.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pihJiRQSp_l1D-UKywdb3w.png" /></figure><p>Everything is Terraform, behind a single enable_workload switch that defaults to off. The whole build cost about $0.25 over a week.</p><h3>Why the usual checks missed it</h3><p>After the bad deploy, every classic signal said the system was healthier than before:</p><ul><li><strong>HTTP status:</strong> 200 everywhere</li><li><strong>Error rate:</strong> 0%</li><li><strong>Service latency:</strong> about 120 ms lower than before</li><li><strong>Application map:</strong> green</li><li><strong>Built-in Helpfulness judge:</strong> 4 of 6 stale answers rated “Very Helpful”</li></ul><p>The first four are expected. They measure whether the system works, not whether the answer is true.</p><p>The last one surprised me. CloudWatch Omni runs evaluators from Amazon Bedrock AgentCore, including an LLM judge for helpfulness. On one trace the judge did notice that the agent “fabricated freshness information” and scored it lower. On four others it rated the same mistake “Very Helpful”. A judge that catches a problem sometimes is not something I can alarm on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_z3MbOSZvHjbf9tLv1XklQ.png" /></figure><h3>The fix in observability: one sentence, as code</h3><p>The rule I care about fits in one sentence: if the tool data is older than 15 minutes, the answer must say so.</p><p>A rule that fits in one sentence does not need a second model to judge it. I wrote it as an AgentCore code-based evaluator: a small Python Lambda with no dependencies. It reads one finished agent trace, finds the tool result, reads its as_of, and compares it with the time the agent received the result. Then it checks whether the final answer admits the data is old.</p><ul><li>Data under 15 minutes old: <strong>FRESH</strong>, score 1</li><li>Older, and the answer says so: <strong>STALE_FLAGGED</strong>, score 1</li><li>Older, and presented as current: <strong>STALE_UNFLAGGED</strong>, score 0</li></ul><p>Omni’s online evaluation runs this on every agent trace and writes the score as a CloudWatch metric. I put an alarm on it with the <strong>Minimum</strong> statistic over 5 minutes, not the Average. With Average, one wrong answer hides among nine good ones. With Minimum, one wrong answer fires the alarm.</p><p>After the bad deploy, all six stale answers scored 0. The alarm fired and the email arrived.</p><h3>What broke while I built it</h3><p>The build did not go in a straight line. These are the failures that taught me the most.</p><p><strong>No results for 11 hours.</strong> Online evaluation was enabled and scored nothing. The traces had no session.id, and AgentCore needs it to group traces into sessions before it scores them. One line in the agent fixed it: trace_attributes={&quot;session.id&quot;: session_id}.</p><p><strong>A false PASS.</strong> My first version looked for words like “not current” in the answer. One answer said the alert was “not currently firing”, and the evaluator read that as the agent admitting stale data. It scored 1 when it should have scored 0. A false PASS is worse than a false FAIL, because it hides the problem. The fix was matching whole phrases about the data only, plus a regression test for that exact sentence.</p><p><strong>A silent monitor.</strong> A truncated file reached the evaluator Lambda with a syntax error. The evaluator crashed on every trace, published no score, and the quality alarm stayed OK, because missing data was treated as not breaching. The agent was wrong and every alarm was green, again. Now a second alarm watches the evaluator’s own errors.</p><p><strong>Terraform deleted my alarms.</strong> I ran a plan from a terminal where the alert email variable was not set. My code treated an empty email as “alerting off”, so Terraform removed the topic, the subscription and the alarm. Now the plan fails with a validation error instead.</p><h3>Asking an AI to investigate</h3><p>Omni has its own AI assistant. I gave it only the symptom, the way an on-call engineer would see it: freshness score dropped to 0, the alarm fired, HTTP 200, no errors in the service. I did not mention a deploy. AWS DevOps Agent was not connected, so the Omni agent investigated on its own.</p><p>In about five minutes it came back with a clear root cause panel. Then I checked every claim against CloudTrail, which had the real deploy and rollback times.</p><p><strong>What it got right</strong></p><ul><li>The service returned old data with HTTP 200 and sub-second latency, so everything looked healthy.</li><li>Only the freshness evaluator caught it, and the data source has no freshness telemetry of its own. Good advice.</li><li>The next step should be to check CloudTrail for configuration changes. Correct direction.</li></ul><p><strong>What it got wrong</strong></p><ul><li>It said writes to a data store “stopped at 06:53Z”. In fact as_of moved with every request; a deploy made the service serve a stale view.</li><li>It said CloudTrail was “unavailable”. CloudTrail was there; the Omni agent simply has no tool to read it.</li><li>It placed the start at alarm time and said the system recovered on its own. The incident started 17 hours earlier, and the recovery was my rollback.</li></ul><p>The last mistake is on me. My question said “between 08:28 and 09:15 UTC”, which was when the alarm fired. The real onset was the evening before, when I applied the bad deploy and then stopped sending traffic. The agent anchored on my time window and built its story inside it.</p><p>Two lessons I will keep. Alarm time is detection time, not onset. And an AI investigator is very good at describing the symptom and the evidence, but its conclusion still needs a check against ground truth, here CloudTrail.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*XlkR-yIg9ktFU5dA6NkJQQ.png" /></figure><h3>Fixing the agent, and measuring the fix</h3><p>Rolling back the deploy fixes the incident. It does not fix the agent, which will do the same thing next time any data source goes stale.</p><p>The model cannot know the data is old, because it does not know what time it is. So the v2 prompt passes the current time and adds one rule: compare every as_of with now, say clearly when data is older than 15 minutes, and do not conclude that an alert is resolved from stale data.</p><p>I made the prompt version a per-request field, so v1 and v2 ran on the same deployment, against the same stale data, at the same time. Any difference in score could only come from the prompt.</p><ul><li><strong>Freshness on stale data:</strong> v1 passed 0 of 6, v2 passed 7 of 7</li><li><strong>Freshness on fresh data (control):</strong> v1 3 of 3, v2 4 of 4, so v2 raised no false alarms</li><li><strong>Helpfulness judge, average:</strong> v1 0.74, v2 0.52</li><li><strong>Tokens per run:</strong> v1 about 2,420, v2 about 2,620 (+8%)</li></ul><p>The v2 answer on the same stale data:</p><blockquote><em>“The data is 1 hour and 35 minutes old … Because the data is stale, I cannot confirm the ongoing state of this metric. Fresh data is needed before concluding whether the alert is truly resolved.”</em></blockquote><p>Look at the Helpfulness line. The general judge scored the honest answer <strong>lower</strong> than the confident wrong one. If I had used that judge as my release gate, I would have shipped v1.</p><p>One more thing I noticed on the fresh-data control run: v2 said the data was “2 minutes” old when the difference was 2 seconds. The conclusion was right and the arithmetic was wrong. That is why the freshness check lives in code, not in the model.</p><h3>Where this applies</h3><p>Any agent that answers from data it fetches at run time can be confidently wrong when that data is stale: a support agent reading order status, an assistant quoting prices or balances, a booking agent reading availability, an on-call copilot reading metrics. The pattern carries over as it is:</p><ol><li>Put an as_of time in every tool response. Without it, freshness cannot be measured by anyone.</li><li>Write the business rule as a code evaluator when it fits in one sentence. Keep LLM judges for what code cannot decide.</li><li>Alarm on the score with Minimum, and alarm on the evaluator itself.</li><li>Sample in production. Evaluation cost me 3.4 times the model cost ($0.19 against $0.06) at 100% sampling on a few dozen traces. At real volume, 5 to 10% is enough to catch a regression.</li></ol><p>The Omni console itself added almost nothing to the bill: CloudWatch came to under one cent for the week.</p><p>The code, Terraform, evaluator, test cases and the full write-up are on GitHub: <a href="https://github.com/g2ai/cloudwatch-omni-sre-agent-observability/tree/main">g2ai/cloudwatch-omni-sre-agent-observability</a></p><p>Next, I want to run the same incident through AWS DevOps Agent, which can read CloudTrail, and compare both investigators against the same ground truth.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6b75dc4b9139" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Datadog API key rotation on AWS, from census to safe revoke]]></title>
            <link>https://medium.com/@jitendraingale/datadog-api-key-rotation-on-aws-from-census-to-safe-revoke-dd94c4c03aa5?source=rss-8812e26fe9b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/dd94c4c03aa5</guid>
            <category><![CDATA[datadog]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[site-reliability-engineer]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[observability]]></category>
            <dc:creator><![CDATA[Jiten I]]></dc:creator>
            <pubDate>Fri, 23 Oct 2026 10:03:39 GMT</pubDate>
            <atom:updated>2026-10-23T10:03:39.189Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Finding, rotating and removing Datadog API keys with AWS Secrets Manager</em></p><p>A Datadog API key is a write credential for your whole Datadog org. Anyone who holds it can send metrics, logs and events into your account. In most estates it also lives in more places than anyone tracks: golden images, Lambda environment variables, ECS task definitions, CI variables, a parameter someone copied years ago.</p><p>That is why a key that has never been rotated is also a key you cannot safely revoke. On the day it leaks, nobody knows what will go dark when you pull it.</p><p>This article walks through an implementation that answers three questions for one AWS account, built so the same modules repeat per account:</p><ol><li><strong>Find.</strong> Where do copies of the key live, and which of them are still live?</li><li><strong>Rotate.</strong> Rotate with AWS Secrets Manager, and check first that nothing still uses the key that will be revoked.</li><li><strong>Remove.</strong> Show which delivery patterns survive rotation, and fix the ones that break.</li></ol><p>The code is public: <a href="https://github.com/g2ai/datadog-key-lifecycle-aws">https://github.com/g2ai/datadog-key-lifecycle-aws</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SlOrM1DNEWePuX1Q.png" /></figure><h3>Secrets Manager now rotates Datadog keys itself</h3><p>In May 2026 AWS added managed external secrets for Datadog to Secrets Manager. There are secret types for the Datadog API key, the application key and an admin key. With the API key type, Secrets Manager calls Datadog on each rotation, creates a new key and moves the AWSCURRENT label to it. No rotation Lambda to write or patch.</p><p>Two secrets are involved:</p><ul><li>The <strong>admin secret</strong> holds a service account’s API key and application key. Secrets Manager uses it to call Datadog. No consumer reads it.</li><li>The <strong>API key secret</strong> holds the key every consumer uses. This is the one that rotates.</li></ul><p>The important behaviour is the grace window. After a rotation, the previous key moves to AWSPREVIOUS and stays valid. The key from two rotations back is the one Datadog revokes. Every consumer gets one full rotation period to pick up the new key. Miss that window and you start getting 403.</p><h3>Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*l3mVwsuGoyNKaJsc.png" /></figure><p>Everything is Terraform, with Packer for the images. Key values never touch code, plan or state: they come from a local file into ephemeral Terraform variables and are written with write-only arguments.</p><h3>What I tested</h3><p>I ran 19 rotations against five ways of delivering the key to the Datadog Agent:</p><p>Consumer How it gets the key After rotation EC2, key baked into the AMI written into datadog.yaml at build time Went dark about a minute after the second rotation revoked its key. Only a rebuild fixes it. EC2, key-free AMI api_key: ENC[...], read by the Agent secrets backend through the instance profile No gap Lambda Datadog extension reads the secret ARN No gap ECS task A task definition secrets, injected once at task start 403 within seconds of the revoke, while ECS still showed the task as running ECS task B Agent secrets backend, refresh on 403 Refreshed the key in the same second as the first 403</p><p>The ECS case is the one worth remembering. ECS only knows that the container process is alive. A running task keeps the key it got at start, and after the revoke the Agent logs API Key invalid (403 response), dropping transaction while the service looks healthy.</p><h3>The fix for deploy-time consumers</h3><p>When AWSCURRENT moves, Secrets Manager emits a native EventBridge event, Secret Label Updated. It reaches the default bus without a CloudTrail trail. A rule matches it for the key secret and calls a small Lambda. The Lambda lists the services in the cluster and forces a new deployment only for those tagged DatadogKeyDelivery=deploy-time. New tasks start with the new key long before the old one is revoked. In the next two rotations ECS A had no 403 at all.</p><p>Tagging makes it opt-in, so a new service needs a tag, not a code change.</p><h3>Find the copies first</h3><p>Rotation only helps if you know where the key is. The census script scans an account region by region through a read-only role that is explicitly denied GetSecretValue. It prints only the last 4 characters of any key and sorts each finding into a class:</p><ul><li><strong>COPY</strong>: the key value itself sits in the resource (SSM parameter, Lambda env var, launch template user data). It breaks when that key is revoked.</li><li><strong>DEPLOY-TIME</strong>: the resource points at the secret but reads it once (ECS task definition secrets).</li><li><strong>REFERENCE</strong>: the consumer reads the secret at runtime. It survives.</li><li><strong>HIDDEN</strong>: AWS will not show the value (CloudFormation NoEcho). Someone checks by hand.</li></ul><p>With --datadog-status, each copy&#39;s last 4 characters are matched against the org&#39;s API key list, which never returns key values. A copy of a revoked key is cleanup work. A LIVE copy blocks rotation.</p><h3>Check before you rotate</h3><p>A rotation revokes the key from two rotations back. Before rotating, a scheduled Lambda asks Datadog when that key was last used and holds the rotation if it is still in use. It also holds if any version still carries AWSPENDING, meaning the last rotation never finished.</p><p>Three things I learned building it:</p><ul><li><strong>Never call Datadog with the old key to ask about the old key.</strong> A call made with a key can count as use of that key. The check uses the current key with an application key that has only api_keys_read.</li><li><strong>date_last_used is rounded down to the hour.</strong> A key used at 10:59 shows 10:00. The idle limit has to be well above an hour: 120 minutes in the test, a day in production.</li><li><strong>A scheduler </strong><strong>rate() without a start date fires right after it is created.</strong> One apply triggered an unplanned rotation. The schedule now takes an explicit start date.</li></ul><p>The check runs every 45 days. The AWS rotation schedule of 90 days stays as a backstop, so a hold cannot last forever.</p><h3>Detect from both sides</h3><p>Datadog cannot see data it rejected. It only notices that data stopped. So detection needs two sides:</p><ul><li><strong>AWS side:</strong> a CloudWatch Logs metric filter on API Key invalid in the Agent logs, with an alarm to SNS. It fired in about 2 minutes and sent OK about 3 minutes after recovery.</li><li><strong>Datadog side:</strong> a no-data monitor on the Agent metric. It detected the gap in about 10 minutes.</li></ul><p>The AWS side shows the cause. The Datadog side shows the effect. In the test the Datadog monitor email was never delivered, which is its own lesson: send a test notification from every new monitor.</p><h3>When rotation itself goes wrong</h3><p>Two failure cases came up, and both are now in the design:</p><ul><li>Pasting rotate-secret several times in a row left AWSPENDING stuck on the current version.</li><li>A rotation I broke on purpose, by removing the rotation role’s policy, was retried by AWS after Terraform restored the policy. It completed, and it also left AWSPENDING on the new current version.</li></ul><p>In both cases the safe-to-rotate check held the next rotation. After confirming both keys were valid, the fix was one command: aws secretsmanager update-secret-version-stage --remove-from-version-id.</p><h3>Running it across many accounts</h3><p>The repo publishes one account as a reference. The multi-account part is design, written up in docs/scale.md:</p><ul><li><strong>One Datadog key per account.</strong> A leak stays in one account, and the key name points to it.</li><li><strong>The same secret name in every account.</strong> Images and task definitions use ENC[datadog/api-key;apiKey] by name, so one golden AMI or task definition works everywhere.</li><li><strong>Admin credentials per account or central.</strong> Per account is simple but copies a powerful credential N times. Central means one copy in a security account, shared with a resource policy and a KMS key policy.</li><li><strong>A read-only census role in every account</strong>, deployed with Terraform or StackSets and trusted by one security account. A daily run on regions in use, a weekly sweep of all regions.</li><li><strong>One organization trail</strong> for the CloudTrail-based rotation alerts.</li></ul><h3>Not only AWS</h3><p>The managed rotation is an AWS feature, but the pattern is not. The Datadog Agent’s secrets backend reads keys natively from Azure Key Vault, GCP Secret Manager, HashiCorp Vault and Kubernetes secrets. On those platforms a small scheduled function calls the Datadog key API to rotate. The rest stays the same: one source of truth, Agents that read the key at runtime, a census of copies, a check before revoke, and an alert on 403.</p><h3>What I would do next</h3><ul><li>Datadog Workload Identity Federation, so Agents on EC2 prove their AWS identity and get a key that Datadog rotates. The static key stays only as a fallback.</li><li>A custom Datadog role for the rotation service account instead of the admin role.</li><li>Remote Terraform state and a configurable name prefix.</li></ul><p>The repo has the quickstart, an FAQ, the scale design and a security review: <a href="https://github.com/g2ai/datadog-key-lifecycle-aws">https://github.com/g2ai/datadog-key-lifecycle-aws</a></p><p><em>Originally published on </em><a href="https://www.linkedin.com/feed/update/urn:li:activity:7517811901763051520/"><em>LinkedIn</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dd94c4c03aa5" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>