<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sherlocks.ai on Medium]]></title>
        <description><![CDATA[Stories by Sherlocks.ai on Medium]]></description>
        <link>https://medium.com/@gauravsherlocksai?source=rss-22d28fd64678------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*4NFy3kwbS3WpC6iAethL8Q.jpeg</url>
            <title>Stories by Sherlocks.ai on Medium</title>
            <link>https://medium.com/@gauravsherlocksai?source=rss-22d28fd64678------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 10:39:36 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gauravsherlocksai/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to Detect Hallucinations in Production: An AI Monitoring Guide]]></title>
            <link>https://medium.com/breaking-prod/how-to-detect-hallucinations-in-production-an-ai-monitoring-guide-9adae8b7c102?source=rss-22d28fd64678------2</link>
            <guid isPermaLink="false">https://medium.com/p/9adae8b7c102</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[ai-hallucination]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[sre]]></category>
            <dc:creator><![CDATA[Sherlocks.ai]]></dc:creator>
            <pubDate>Wed, 08 Jul 2026 14:34:39 GMT</pubDate>
            <atom:updated>2026-07-17T11:33:29.914Z</atom:updated>
            <content:encoded><![CDATA[<h4>Your dashboards show green. Your AI is confidently wrong. Here is how to fix that blind spot.</h4><p>If your AI system answered incorrectly 2% of the time today, would you even know?</p><p>According to Gartner, 84% of CIOs have no formal process for tracking AI accuracy. Hallucinations are not a model quality problem. They are a reliability problem. A hallucinating system fails users silently, and tools like Prometheus, Datadog, and Grafana cannot see it happening.</p><p>Recent benchmarks show GPT-4.1 at a 5.6% hallucination rate and Gemini-2.5-Pro at 7.0% on standard tests. But those tests use clean, narrow questions. Real users ask messy, complex things your benchmark never covered. The number that actually matters is not on any leaderboard. It is the hallucination rate inside your specific system, on your actual user queries, against your actual retrieval index. Almost nobody is measuring that.</p><h3>What Is a Hallucination in Production?</h3><p>A hallucination is when your AI gives an answer that is wrong, made up, or not backed by any real source, but sounds completely convincing. The tricky part is that it does not look broken. There is no error. No crash. The system responds fast, sounds confident, and gets it wrong. Your users act on it anyway.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1022/1*-VWIStqYIcPtiI-Vs8YVow.png" /></figure><h3>Why Hallucinations Are a Reliability Problem</h3><p>Most teams hand this problem to the ML team and wait for the next model update. That is the wrong move. A wrong answer that looks right is a reliability failure. It is not a model bug to be fixed later. It is a system that is failing users right now, silently, with no signal showing up in your dashboards.</p><p>The field has shifted from chasing zero hallucinations to managing uncertainty in a measurable, predictable way. That is the same thing SRE has always done with infrastructure. You do not eliminate downtime. You measure it, set thresholds, and alert when things go wrong. AI output quality deserves the same treatment.</p><h3>The Hallucination Detection Stack</h3><p>Four layers, ordered from cheapest and fastest to most thorough. Run them together. Earlier layers act as quick filters. Later layers dig deeper when something looks off.</p><h4>Layer 1: Retrieval Validation</h4><ul><li>What it catches: Answers where the model had no useful context to work from.</li><li>How it works: Check whether the retrieved chunks contain anything relevant to the question. If nothing useful came back and the model answered anyway, flag it.</li><li>What it misses: Cases where the model had good context but still got the answer wrong.</li><li>Cost: Very low. Run it on every single request.</li></ul><h4>Layer 2: Semantic Consistency Checks</h4><ul><li>What it catches: Unstable responses. If you ask the same question three times and get three different answers, the model is guessing.</li><li>How it works: Run the same query a few times and measure how similar the answers are. High variance means low confidence.</li><li>What it misses: Responses that are consistently wrong. Same answer every time, all of them bad.</li><li>Cost: Moderate. Run this on a sample, not everything.</li></ul><h4>Layer 3: Groundedness Scoring (LLM-as-Judge)</h4><ul><li>What it catches: Answers that make claims the retrieved context never actually said.</li><li>How it works: A lightweight judge model reads the response alongside the source context and scores whether the claims check out.</li><li>What it misses: Hallucinations on topics your retrieval corpus does not cover at all.</li><li>Cost: Higher. Sample 10–20% of production traffic.</li></ul><h4>Layer 4: Human Feedback Loops</h4><ul><li>What it catches: Everything the automated layers missed.</li><li>How it works: Watch escalation rates, thumbs-down signals, correction rates, and direct user feedback.</li><li>What it misses: Nothing in principle, but humans are slow and expensive at scale.</li><li>Cost: High per evaluation. Use for calibration and post-incident review.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1022/1*cWHA8X8jXTfl7D9DLQXlaA.png" /></figure><h3>Five Metrics to Track in Production</h3><ul><li><strong>Hallucination Rate: </strong>The percentage of responses that fail groundedness evaluation. This is your main quality signal. Set a threshold; 3% is a reasonable place to start, and alert when it is crossed.</li><li><strong>Retrieval Miss Rate:</strong> The percentage of queries for which retrieval returned no useful results. If this is high, your model is operating without grounded context. That is the most direct path to hallucinations.</li><li><strong>Confidence Mismatch Rate: </strong>The percentage of responses where the model sounds confident but the groundedness score is low. This is the most dangerous pattern. Users trust confident answers. A confident wrong answer gets acted on.</li><li><strong>Human Escalation Rate:</strong> The percentage of AI responses that are escalated to a human or corrected. This is a lagging signal, but it is your most honest one.</li><li><strong>Unsupported Claim Rate:</strong> The percentage of responses containing at least one claim not found in the retrieved context. Track this at the claim level.</li></ul><p>Start with hallucination rate and retrieval miss rate. Two metrics, one evaluation pipeline. That alone puts you ahead of most teams.</p><h3>Why Traditional SRE Tools Miss This</h3><p>Prometheus, Datadog, Grafana. All excellent tools. All completely blind to whether your AI gave a correct answer.</p><p>Traditional observability was built for systems where something going wrong produces a visible signal. An error code. A timeout. A memory spike. AI systems break differently. A wrong answer looks exactly like a right answer at the infrastructure layer. HTTP 200. Fast response. Zero errors.</p><p>You need an evaluation pipeline running alongside your inference pipeline, scoring responses against groundedness criteria and feeding those scores into your alerting system the same way you feed in latency or error rate. Tools like OpenTelemetry can carry these evaluation scores alongside your standard infrastructure signals.</p><p>And when the alert fires, dashboards are not enough. You need to investigate. Was it a retrieval index update that removed key documents? A prompt change? A model version swap? Those are causal questions. You need to trace the chain, not just look at charts.</p><h3>Hallucination Monitoring Workflow in Production</h3><ul><li><strong>Step 1: Evaluation runs continuously.</strong> Every response passes through Layer 1 automatically. A 10% sample goes through Layer 3. The scores land in your metrics store.</li><li><strong>Step 2: Alert fires.</strong> Hallucination rate crosses 3% in the last hour. The alert includes affected query types, retrieval miss rate, and recent index updates.</li><li><strong>Step 3: Investigation starts.</strong> The AI SRE agent pulls flagged responses, retrieved context, groundedness scores, and the change log for the last 24 hours. It finds that a retrieval index update three hours ago removed a document category that most affected queries depend on.</li><li><strong>Step 4: Root cause found. </strong>The fix is either restoring the removed documents or updating the query routing.</li><li><strong>Step 5: Human review.</strong> The team reviews responses from the affected window. The post-incident review adds a new check: validate retrieval coverage before any index update goes to production.</li></ul><p>That is what treating hallucinations as an incident looks like. Not a model quality review. An actual incident.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/1*hBZWgagD2Y8HBCN71mdVmQ.png" /></figure><h3>Hallucination Monitoring for AI SRE</h3><p>For teams running AI for incident investigation, hallucination monitoring is not a nice-to-have. It is a safety requirement.</p><p>An AI SRE agent that hallucinates a root cause sends your engineers in the wrong direction during an active incident. An agent that confidently recommends the wrong fix makes things worse. On-call engineers are under pressure and time-short. They will act on a confident recommendation faster than they stop to verify it.</p><p>This is why evidence matters as much as the answer. When your AI SRE agent produces a hypothesis, you need to see the log lines, metric anomalies, and deployment events that support it. Grounded answers are the only kind worth acting on at 3 AM.</p><h3>Key Takeaways</h3><ul><li>Hallucinations are a reliability problem, not a model quality problem. Your infrastructure shows green. Your AI is confidently wrong. No error. No alert. Users act on bad answers.</li><li>Traditional monitoring tools cannot see this. Prometheus and Datadog measure infrastructure. They have no concept of whether an answer was correct. You need a separate evaluation pipeline.</li><li>Four layers beat one. Retrieval validation is fast and cheap. Semantic consistency catches unstable responses. Groundedness scoring catches subtle misrepresentations. Human feedback is your ground truth.</li><li>Five metrics to start tracking today. Hallucination rate, retrieval miss rate, confidence mismatch rate, human escalation rate, unsupported claim rate.</li><li>Confidence mismatch is the most dangerous failure mode. Uncertain answers get questioned. Confident wrong answers get acted on.</li><li>An alert without investigation is not enough. You need to trace the cause back to a system change.</li><li>For AI SRE, this is non-negotiable. Every hypothesis your AI agent produces needs evidence behind it.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9adae8b7c102" width="1" height="1" alt=""><hr><p><a href="https://medium.com/breaking-prod/how-to-detect-hallucinations-in-production-an-ai-monitoring-guide-9adae8b7c102">How to Detect Hallucinations in Production: An AI Monitoring Guide</a> was originally published in <a href="https://medium.com/breaking-prod">Breaking Prod</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Blameless Postmortems Explained: Lessons From Real Outages]]></title>
            <link>https://medium.com/@gauravsherlocksai/blameless-postmortems-explained-lessons-from-real-outages-388c08966e13?source=rss-22d28fd64678------2</link>
            <guid isPermaLink="false">https://medium.com/p/388c08966e13</guid>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[company-culture]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[blameless-post-mortem]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Sherlocks.ai]]></dc:creator>
            <pubDate>Thu, 02 Jul 2026 06:31:01 GMT</pubDate>
            <atom:updated>2026-07-02T06:31:01.026Z</atom:updated>
            <content:encoded><![CDATA[<h4>The theory fits on a slide. The practice does not. Here is what actually happened when experienced teams tried to make blameless culture real.</h4><p>Every engineering team after a major outage faces the same quiet fork in the road. One path asks: who caused this? The other asks: what made this possible?</p><p>Most teams say they take the second path. Most teams are wrong. Not out of malice, but because the first path is psychologically easier, organizationally convenient, and feels like accountability even when it produces none.</p><p>This is the gap that blameless postmortems were designed to close.</p><h3>What Is a Blameless Postmortem?</h3><p>A blameless postmortem is a structured review conducted after an incident. The goal is to understand what happened, why it happened, and how to prevent recurrence, without assigning fault to individuals.</p><p>Blameless does not mean consequence-free. It means the investigation focuses on the system that caused the failure rather than on the person closest to it when it broke. The accountability shifts from punishment to responsibility. Engineers are not off the hook. They are on the hook for helping the organization become safer.</p><p>The concept did not originate in software engineering. It came from aviation, healthcare, and nuclear safety. Industries where failures kill people. Sidney Dekker, an aviation safety professor who also flew commercially as a Boeing 737 First Officer, captured it precisely.</p><blockquote><em>“Underneath every simple, obvious story about human error, there is a deeper, more complex story about the organization.”</em></blockquote><h3>The Misconception Worth Addressing</h3><p>The most common pushback is that blameless postmortems protect bad actors. That without blame, there is no accountability. This gets the logic backward.</p><p>Blame-based cultures produce concealment, not accountability. Engineers learn that admitting involvement in a failure carries professional risk. So they minimize their role and avoid ownership. The organization gets a sanitized story and learns nothing.</p><p>You cannot fix what you cannot see. Blame makes failures invisible.</p><h3>The Postmortem Maturity Ladder</h3><p>Most organizations believe their postmortem practices are more mature than they actually are. Here are the five levels seen in practice.</p><ul><li><strong>Level 1: Punitive Review.</strong> The postmortem exists to assign fault. Engineers attend defensively. The same failure recurs.</li><li><strong>Level 2: Compliance Theater. </strong>This is the most common level. The template is followed. Action items are written. But items have no owners, no deadlines, and no follow-up.</li><li><strong>Level 3: Honest Diagnosis. </strong>The timeline is reconstructed accurately. Contributing factors are named without softening. Some action items get completed. Engineers leave feeling heard rather than scrutinized.</li><li><strong>Level 4: Systemic Learning.</strong> Architecture changes, not just behavior changes. The postmortem is shared broadly. Action items are funded, owned, and tracked to completion.</li><li><strong>Level 5: Organizational Memory.</strong> Past postmortems actively inform future system design. New engineers read incident history during onboarding. Learning compounds over time.</li></ul><p>Most teams are at Level 2. The best teams operate at Level 3. Level 5 is rare.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kh1AXdbNEGwAOACsnSR93w.png" /></figure><h3>Why Blame Happens Anyway</h3><p>If blameless postmortems are clearly better, why do organizations default to blame?</p><ul><li><strong>Hindsight bias.</strong> After an incident, the failure looks obvious. But the engineer was operating with incomplete information at the time.</li><li><strong>The organizational need for a simple story.</strong> Leadership is under pressure. A person is a simpler answer than a system. “The engineer pushed a bad config” is a clean narrative. The accurate version implies the organization has work to do.</li><li><strong>Performance review anxiety.</strong> Engineers quietly calculate how an incident will affect their standing. This shapes what they say.</li><li><strong>Executive pressure in the room.</strong> When a senior leader walks into a postmortem, the dynamic shifts. A single leading question makes the room defensive.</li><li><strong>The cognitive cost of complexity.</strong> Modern incidents are almost never caused by a single thing. “Someone made a mistake” is a relief. It ends the investigation at a point that feels satisfying rather than at a point that is actually complete.</li></ul><p>Blame survives because it is cognitively convenient and feels like accountability, right up until the same incident happens again.</p><h3>What Experienced Practitioners Actually Learned</h3><p>The theory fits on a slide. The practice does not. Here is what happened when experienced teams tried to build this for real.</p><h4>1. Sidney Dekker: The Bad Apple Theory</h4><p>Dekker studied why complex systems fail. His research across aviation, healthcare, and nuclear safety consistently showed the same result. When organizations punish individuals, engineers hide what happened, near misses go unreported, and the same failure repeats.</p><p>The alternative: instead of asking who made the mistake, ask why the system placed that person in a position where the mistake was the most likely outcome.</p><blockquote>What to do today: If your last postmortem accepted “human error” as a finding, the investigation stopped too early. Ask what about the system made that error possible, likely, and uncatchable.</blockquote><h4>2. John Allspaw: The Second Story</h4><p>In May 2012, Allspaw published “Blameless PostMortems and a Just Culture” on Etsy’s engineering blog. More than a decade later, it remains one of the most linked-to articles in SRE writing.</p><p><em>“This cycle of name, blame, shame… Engineers become silent on details about actions, situations, observations, resulting in Cover-Your-Ass engineering from fear of punishment.”</em></p><p>His framework was the “second story.” Every incident has two. The first story is what happened on the surface. The second story explains why the decision made sense to the person who made it, given what they knew at the time.</p><blockquote>What to do with this today: In your next postmortem, ask the facilitator to find the second story. Not what happened, but why it made sense to the person involved to do what they did. That is where the real learning begins.</blockquote><h4>3. Dan Milstein: One sentence that changed 250 meetings</h4><p>HubSpot discovered a pattern from their own 250 postmortem sessions. Someone would walk into the review, take personal ownership, and say: “This was my fault. I will be more careful next time.” The room would accept it. Nothing would change.</p><p>His response was a single sentence he now says at the start of every session: “We are trying to prepare for a future where we are all just as stupid as we are today.”</p><blockquote>What to do today: Say this sentence before your next postmortem. Then read your last three action items. If any require humans to behave differently without changing the system, they are wishes, not fixes.</blockquote><h4>4. Atlassian: Fixing Systems, Not People</h4><p>An engineer at Atlassian made a single syntax error in a configuration file. The company went dark for 45 minutes. The engineer was not fired.</p><p>The investigation asked why a single syntax error could take down the entire company. The answer: no automated validation existed for config files before they were loaded. The fix was an automated validation check. The engineer still works at Atlassian.</p><blockquote>What to do today: After your next postmortem, ask one question: did we change the system, or did we change our expectations of the humans inside it? Only one of those is a real fix.</blockquote><h4>5. Nora Jones: Blame-Awareness</h4><p>Jones spent years leading chaos engineering at Netflix, <a href="https://jet.com/">Jet.com</a>, and Slack. She noticed many organizations adopting the language of blamelessness without building the conditions that make honesty safe.</p><p>Her reframe is “blame-aware” culture. Blameless, taken literally, asks people to pretend blame does not exist. Blame-aware acknowledges it as a natural human response and focuses on moving past it.</p><blockquote>What to do today: In your last postmortem, did people say what actually happened, or what was safe to say? If you cannot answer that with confidence, the conditions for honest learning are not yet in place.</blockquote><h4>6. Noel Pullen: Naming is not blaming</h4><p>In 2017, an engineer at Hootsuite accidentally deleted a critical LinkedIn connection on a production account, causing a 39-minute outage. The team named their writeup: “That Time Noel Deleted LinkedIn.”</p><p>When questioned, their response was precise. Naming who was involved is not the same as blaming them. The investigation asked why deleting that connection was structurally possible, revealing that personal social accounts had excessive administrative access to production systems. “<em>Human error is a symptom, never the cause, of trouble deeper within the system.”</em></p><blockquote>What to do today: Name who was involved and what they did. Then keep asking why until you reach the system condition that made it possible.</blockquote><h4>7. Google SRE Workbook: Handling the Senior Leader Exception</h4><p>Google’s SRE team encountered the hardest case: when blame comes from the top of the room. The SRE Workbook gives facilitators exact words to use when a powerful leader asks a blaming question.</p><p>The recommended redirect: <em>“I am sure everyone had the best intent, so to keep it blameless, maybe we ask generically if there were any warning signs we could have heeded, and why we might have dismissed them.”</em></p><blockquote>What to do today: Write your own version of this redirect before you need it. The middle of a high-pressure postmortem is not the time to improvise.</blockquote><h4>8. Charity Majors: Transparency is infrastructure</h4><p>Majors is direct: managers who obscure what happened during an outage to protect their team’s image actively destroy engineering trust.</p><p>At Honeycomb, every outage gets a public postmortem. They even import actual event data from their own outages into their public demo dataset, making incident learning interactive for the community.</p><blockquote>What to do today: How your organization talks publicly about its failures is a signal to every engineer inside. Polished external communication that contradicts internal reality erodes safety.</blockquote><h3>Patterns Across the Stories</h3><p>Three patterns emerge from these stories:</p><ul><li>First, the best incident leaders reduce fear before they ask questions. The conditions for honesty must be built before honesty is requested.</li><li>Second, most root causes are organizational tradeoffs that were never resolved. A genuinely honest postmortem exposes what the organization chose not to fix for years.</li><li>Third, the postmortem document is the least important part. The document is evidence a conversation happened. The conversation and systemic changes are what matter.</li></ul><h3>Where Blameless Culture Fails</h3><p>Every organization in this article believed in blameless postmortems. Several still got it wrong at critical moments.</p><ul><li><strong>Postmortem theater.</strong> The most common failure mode. The template gets filled in. Action items are written. Everyone leaves feeling like something was accomplished. Six months later, the same failure recurs. <strong>Check what percentage of action items from your last five postmortems were actually completed. A postmortem that changes no operational behavior is a filing exercise.</strong></li><li><strong>Blameless as shield.</strong> Some organizations use blamelessness to avoid accountability entirely. <strong>A blameless postmortem should make it easier to identify what needs to change and who is responsible for changing it.</strong></li><li><strong>Performative transparency.</strong> A major incident occurs. The public postmortem is carefully written. Contributing factors are described at a level of abstraction that protects reputation. <strong>Performative transparency is hardest to avoid when the stakes are highest and it is the real test of culture.</strong></li><li><strong>The senior leader exception.</strong> Every organization has a leader who says “I know we are supposed to be blameless, but…” Under pressure, leaders revert to patterns that feel like accountability. <strong>The fix is not a training program. It is leaders who hold the line on their own behavior, in the room, when the pressure is highest.</strong></li></ul><h3>The Only Question That Matters</h3><p>Incidents are inevitable. The question was never whether failures would happen. It was what the organization would do when they did.</p><p>The practitioners in this article reached the same conclusion. Make truth easier to surface than fear. Everything else follows.</p><p>Not because it is kind. Because it is the only way complex systems actually get safer. Because engineers managing their exposure are not investigating the incident. Because the second story is only told when the people who lived through it believe it is safe to tell.</p><p>The organizations that do this well are not the ones where incidents never happen. They are the ones where the full story is told, the real contributing factors are named, and the system is changed.</p><p>That is not a process. It is a decision made by leaders repeatedly about what kind of organization they are building.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=388c08966e13" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Agent Observability for Autonomous AI SREs in 2026]]></title>
            <link>https://medium.com/@gauravsherlocksai/agent-observability-for-autonomous-ai-sres-in-2026-f9834adfc27b?source=rss-22d28fd64678------2</link>
            <guid isPermaLink="false">https://medium.com/p/f9834adfc27b</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[langfuse]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[opentelemetry]]></category>
            <dc:creator><![CDATA[Sherlocks.ai]]></dc:creator>
            <pubDate>Sun, 28 Jun 2026 18:55:36 GMT</pubDate>
            <atom:updated>2026-06-28T18:55:36.645Z</atom:updated>
            <content:encoded><![CDATA[<h4>Traditional APM was built for request-response. Agents break that model. Here is what actually works.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hcPa9RM15MBSQVSA.png" /></figure><p>You deploy an autonomous AI SRE to handle incident investigation. It works beautifully in testing. Then production happens.</p><p>The agent sees a database failover. It decides to rollback a deployment. The rollback succeeds. But you have no idea why the agent chose that action. Did it reason correctly? Did it hallucinate and get lucky? Did it follow the right playbook or invent one?</p><p>Traditional application performance monitoring cannot answer these questions. It was built for synchronous request-response. Agents break that model entirely.</p><p>This matters now because agent frameworks doubled in 2025. Operational complexity, not model capability, is the primary blocker for reliable AI at scale. When an AI deployment fails, it fails operationally, not because the LLM hallucinated. Five percent of AI model requests fail in production today. Roughly sixty percent of those are capacity-related, not model errors.</p><p>Most observability platforms are not there yet.</p><h3>The Core Problem: The Semantic Gap</h3><p>Existing tools observe an agent’s high-level intent, prompts, and tool selections, OR its low-level actions, system calls, and API latency. They do not correlate both views.</p><p>This creates a dangerous blindness. You cannot distinguish between benign operations, malicious injections, and expensive reasoning loops that waste tokens and money.</p><p>Consider an AI SRE running an infrastructure repair. You can see the LLM prompt. “Fix the database failover.” You can see the system call. A rollback command executed. But you cannot see the correlation. Did the agent intend that exact rollback? Or did it hallucinate the command and get lucky?</p><p>When failure happens, this gap becomes your investigation crater.</p><p>Recent research from UC Santa Cruz formally defines this. Using eBPF, researchers intercept TLS-encrypted LLM traffic to extract semantic intent. They correlate it with kernel events and process boundaries. They detect resource-wasting loops, prompt injections, and hidden bottlenecks in multi-agent systems. All with less than three percent overhead, framework-agnostic.</p><p>This is what real agent observability looks like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hp2VZCERyrmmR-iCakjM0Q.png" /></figure><h3>What Does the Market Actually Offer?</h3><p>Fifteen tools actively compete on agent observability in 2026. Most are built on OpenTelemetry standards.</p><p><strong>Langfuse </strong>is the consensus pick for all-in-one debugging. MIT licensed. <strong>LangSmith</strong> leads for LangChain users. <strong>Arize Phoenix </strong>is the standard for evaluation. <strong>Datadog </strong>and<strong> New Relic</strong> offer enterprise APM add-ons if you are already paying for them.</p><p>The question is not whether tools exist. It is whether the tool you pick was designed for agents or bolted onto traditional APM.</p><p>Here are some rough tests.</p><p>Does it handle reasoning loops as a first-class concern? Can you see the decision tree, prompt, tool choice, outcome, next decision, as a continuous trace? Does it distinguish between a tool failure and an agent misunderstanding? Does it alert on semantic drift, where agent behavior changed but metrics look normal?</p><p>These are the questions that separate a product built for agents from one stitching together legacy APM. Answer them before you buy.</p><h3>Where Did This Gap Come From?</h3><p>The pattern is familiar if you remember infrastructure outages.</p><p>OpenAI’s December 2024 outage illustrates it. A monitoring service deployed to Kubernetes overwhelmed the control plane. The control plane failed, preventing rollback. Blindness at exactly the moment automation was supposed to help.</p><p>When Alibaba’s OSS and IAM systems created a circular dependency, it remained invisible until the outage. Circular dependencies are architecturally toxic. When infrastructure lacks observability into its own dependencies, temporary failures become cascading failures.</p><p>Agents are similar. An autonomous SRE that cannot see its own assumptions, tool availability, API response times, and token limits will cascade failures the same way.</p><p>Observability at the automation layer is non-negotiable.</p><h3>How Do You Actually Start?</h3><p><strong>If you are building agents, start with OpenTelemetry.</strong> The standard is stable. Tool choice is wide. You avoid vendor lock-in.</p><p><strong>If you are buying, demand explicit agent loop tracking in the contract.</strong> Ask for examples. Do not accept “we can log prompts” as an answer.</p><p><strong>Expect to operate in supervised mode for the next two years. </strong>Your agent still needs human approval before running commands. The infrastructure can catch more failures every quarter. Observability is still catching up to the agent story.</p><p>Operationally, here are some things that work now.</p><p>Langfuse and LangSmith both offer multi-step debugging. Honeycomb has published its observations of its own LLMs. Datadog’s 2026 industry report confirms a pattern. Operational complexity scales faster than monitoring. Teams that invested early in agent layer instrumentation caught failures before the cascade. Teams that did not had outages instead.</p><p>The infrastructure is ready. Early movers have instrumented agent loops. Most teams are still running agents on traditional APM.</p><h3>What Does Good Look Like at Your Scale?</h3><p>Here is the bar.</p><p><strong>Can you reconstruct what the agent intended, what it actually did, why it diverged, and where in the system it failed, within five minutes?</strong> If you can answer those four questions end to end, you have observability. If you cannot, you have logging.</p><p><strong>Second bar. Can you detect that an agent’s reasoning loop is wasting tokens before the bill arrives? Prompt injection attempts? </strong>Coordination bottlenecks between agents? If your tool cannot surface these, it is not built for agents.</p><h3>The Bottom Line</h3><p>Traditional APM was built for a world of request-response. Agents live in a world of loops, reasoning, and intent. Observability needs to live there too.</p><p>The tools exist. The standards are stable. The gap is knowing what to ask for and where to start.</p><p>Start with OpenTelemetry. Demand agent loop tracking. Instrument early. The infrastructure is ready. Most teams are not.</p><p>Do not let your agent fly blind.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f9834adfc27b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[KubeCon India 2026: Key Takeaways]]></title>
            <link>https://medium.com/@gauravsherlocksai/kubecon-india-2026-key-takeaways-62cf85d265a4?source=rss-22d28fd64678------2</link>
            <guid isPermaLink="false">https://medium.com/p/62cf85d265a4</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Sherlocks.ai]]></dc:creator>
            <pubDate>Mon, 22 Jun 2026 05:58:22 GMT</pubDate>
            <atom:updated>2026-06-22T05:58:22.645Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fRqMUxS6wV_HM3HwobQK2Q.jpeg" /></figure><p>KubeCon + CloudNativeCon India 2026 wrapped up this week in Mumbai. Two days, 55 sessions- the biggest edition the India conference has had so far.</p><p>A few of us from Sherlocks.ai were on the ground for both days. We sat in on keynotes, walked the entire Solutions Showcase, spent real time in the Project Pavilion, and had more hallway conversations than we can count.</p><p>Here are the eight things we actually walked away thinking about.</p><h3>1. The gap between running AI and running it daily is bigger than anyone wants to admit</h3><p>The number that stuck with us all week comes from CNCF’s 2025 Annual Cloud Native Survey, and it framed how we read the rest of the event: 66% of organizations now host at least some of their AI inference workloads on Kubernetes, but only 7% are deploying AI models on a daily basis. A further 44% do not run AI or ML workloads on Kubernetes at all yet.</p><p>That gap matters more than it sounds like it should. It means a large part of the industry has already done the hard infrastructure work. The harder problem now is trust. Teams have the tools. They have not yet built the habit, or the confidence, to run them every single day in production. We think about this number constantly when we talk to teams evaluating AI SRE tools, because it explains a hesitation we see often that has nothing to do with the technology itself.</p><h3>2. Security has not caught up to agentic AI, and most teams know it</h3><p>Two sessions stood out for the same reason. One on giving AI agents an actual identity with scoped permissions, instead of letting them run with whatever access happens to be lying around. One on why cluster-wide policies are a hidden risk most teams have not reassessed since AI agents started touching production.</p><p>Interestingly, the survey data complicates the easy story here. Security is still cited as a top challenge by 36% of respondents, but its relative ranking has actually fallen in recent years, and for the first time cultural change within teams (47%) has overtaken every technical concern as the number one barrier to adoption. Put those two things together and a pattern emerges from the conversations we had after these talks: a lot of teams deployed an AI agent first and are only now going back to ask what it should actually be allowed to do. The tooling exists. The organizational discipline to use it well is what is lagging. To their credit, most people we spoke to already knew it.</p><h3>3. OpenTelemetry is no longer a decision. It is the starting point</h3><p>A few years ago, whether to adopt OpenTelemetry was a real debate at events like this. This year nobody pitched it and nobody defended it. It was simply assumed.</p><p>That matches the wider data. OpenTelemetry is now used by 49% of CNCF survey respondents among projects still working toward graduation, putting it well ahead of most other emerging tools in the ecosystem. Once telemetry is standardized, the real question becomes what you build on top of it, not how you collect it in the first place. We saw that shift directly reflected in the AI SRE conversations happening across the floor, where almost every serious tool now assumes OTel as the data layer it sits on.</p><h3>4. The unglamorous open source work is still doing all the heavy lifting</h3><p>We spent more time than planned in the Project Pavilion, the area right next to the Solutions Showcase where CNCF projects staff their own kiosks. The people here are maintainers and contributors, and CNCF actually goes out of its way to get them in the room, through project pass allocations and dedicated maintainer travel support like the Dan Kohn Scholarship. They are not here to sell. They are here because they care about the project.</p><p>Talks on KubeEdge, Harbor, Kyverno, and Longhorn did not draw the same crowds as the AI booths a few steps away. But every single AI demo we saw on the show floor was quietly depending on infrastructure that projects like these have been maintaining, often thanklessly, for years. It was a useful reminder of where the real foundation of this industry actually sits.</p><h3>5. The next generation showed up, and it was vocal</h3><p>The crowd itself was a signal. A noticeably large share of attendees were students and early-career engineers, and they were not passive. They asked sharp questions, crowded the newcomer sessions, and were openly, almost proudly, advocating for open source.</p><p>That matters more than it appears on the surface. The survey’s headline finding this year was that the industry’s biggest barrier is no longer technical but cultural. Adoption is largely solved; what closes the remaining gap is people who instinctively build, contribute, and operate in the open. A generation entering the field already fluent in open-source values, and willing to say so out loud, is exactly how that cultural shift happens. CNCF clearly understands this. It runs a scholarship program at every India edition specifically so cost is never the reason a student or first-time speaker misses out, paired with a track built just for newcomers.</p><p>The result is something we genuinely do not see enough of at tech events: a college student who just gave their first lightning talk, standing two feet from a principal engineer running production systems at serious scale, with neither one treated as more important than the other. That texture is hard to fake and harder to build. India already ranks among the top contributors to CNCF projects globally, and standing in that room, it was easy to see the pipeline behind that number.</p><h3>6. AI SRE has gone from a handful of startups to a real, contested category</h3><p>Three years ago this space barely existed. This year it was one of the most visible categories on the entire show floor.</p><p>We spent time talking to teams building in this space, each making a different architectural bet on the same underlying problem. What we kept coming back to in our own conversations: detection is largely solved across the industry now. The real competition has moved a layer deeper, into the gap between an alert firing and someone actually understanding why.</p><p>That gap is exactly what we work on at Sherlocks.ai. Watching that many serious, well-funded teams converge on the same problem from different angles was, honestly, validating.</p><h3>7. Cost has quietly become a platform engineering responsibility, not just a finance one</h3><p>A session on making cloud cost a platform team’s job was standing room only. A separate one on the hidden cost buried inside telemetry pipeline conversions made a near identical point from the observability side.</p><p>This is not unique to KubeCon India. Gartner projects that platform engineering adoption in large organizations will rise from 45% in 2022 to 80% by 2026, and cost ownership is increasingly part of what that role is expected to cover. Neither session was framed as exciting, and both were among the most practical conversations we had all week.</p><h3>8. This event is becoming its own thing, not a smaller version of the global conference</h3><p>Putting all of this together, the clearest takeaway for us was simple. KubeCon India is no longer a regional satellite event. It started in Delhi in 2024, grew to around 4,000 attendees in Hyderabad in 2025, and reached its largest programme yet in Mumbai this year. Between that scale, the depth of the security and platform engineering conversations, and the sheer competitiveness of the AI SRE category on display, this felt like a genuine signal of where cloud native is headed, not just in India, but more broadly.</p><p>What stayed with us most, walking out of the Jio World Convention Centre on the second evening, was not any single announcement. It was the pattern underneath all eight of these. Infrastructure is no longer the constraint anywhere we looked, whether that was Kubernetes itself, observability, or AI tooling. Trust, ownership, and the willingness to actually run these systems every day are. That is the real story of cloud native in 2026, and we suspect it will still be the real story this time next year.</p><h3>A few quick questions people have asked us since the event</h3><p><strong>What was the most talked about stat from KubeCon India 2026?</strong> The one we kept hearing referenced came from CNCF’s 2025 survey: 66% of organizations now run AI inference workloads on Kubernetes, yet only 7% deploy AI models daily. That distance between capability and daily practice became a recurring theme.</p><p><strong>Was AI SRE the biggest theme at the event?</strong> It was one of the most visible categories on the show floor, alongside platform engineering, security, and observability, but it was one part of a much bigger programme, not the whole story.</p><p><strong>Did security come up as a major concern?</strong> Yes, though with a twist. Several sessions focused specifically on identity and access control for AI agents, an area most teams have not fully addressed, even as cultural change has overtaken security as the top overall adoption barrier in CNCF’s data.</p><p><strong>Is KubeCon India growing?</strong> Significantly. It started in Delhi in 2024, grew to around 4,000 attendees in Hyderabad in 2025, and reached its largest programme yet in Mumbai in 2026.</p><p>If you were also at KubeCon India 2026 and saw something that stood out that we missed, we would genuinely love to hear it in the comments.</p><p>And if the investigation gap we talked about in point 6 sounds familiar, we would be glad to show you what Sherlocks.ai actually does about it.</p><p><em>Written by the Sherlocks.ai team</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=62cf85d265a4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Four Paths to AI-Driven Reliability: Native, OSS, Hybrid, and Agentic SRE Stacks]]></title>
            <link>https://medium.com/@gauravsherlocksai/four-paths-to-ai-driven-reliability-native-oss-hybrid-and-agentic-sre-stacks-ade6570db34a?source=rss-22d28fd64678------2</link>
            <guid isPermaLink="false">https://medium.com/p/ade6570db34a</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Sherlocks.ai]]></dc:creator>
            <pubDate>Tue, 16 Jun 2026 07:01:03 GMT</pubDate>
            <atom:updated>2026-06-16T07:01:03.096Z</atom:updated>
            <content:encoded><![CDATA[<h4>These are not competing products. They sit at different points on a maturity curve. The right one depends on where your team actually loses time.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*dhlYaqO0IC5Qj32l" /></figure><p>Most AI SRE comparisons focus on features. That misses the point. These tools are not trying to do the same thing.</p><ul><li>Native tools help you access data faster inside a cloud platform.</li><li>Open source tools help you interpret Kubernetes errors.</li><li>Hybrid tools improve observability data quality.</li><li>Agentic tools investigate incidents actively across your whole stack.</li></ul><p>None is better than the others. They just solve different problems. A team that needs faster AWS logs does not need an agentic system. A team spending three hours manually correlating signals from five tools does.</p><p>Where MTTR actually goes. For a typical 40-minute incident, investigation takes 22 minutes. That is 55 percent of the total. Pick a tool that helps the wrong part of the chain, and you will not see the results you expected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fFba4fdRWjcfmXsWxH8sKw.png" /></figure><h3>The AI SRE Maturity Curve</h3><p>AI in SRE is not one thing. It is a set of stages. Each stage takes on more of the incident workflow.</p><p>Stage 1: Assist. Native tools help engineers access data faster.</p><p>Stage 2: Analyze. Open source tools interpret signals and surface likely causes.</p><p>Stage 3: Enrich. Hybrid tools fix data quality problems.</p><p>Stage 4: Investigate. Agentic tools actively investigate across systems.</p><p>Higher stages take on more of the workflow and require more from the team to set up and tune. <a href="https://dora.dev/research/">According to DORA’s 2025 report</a>, incidents per pull request increased significantly as AI coding assistants accelerated delivery. Incident response capacity did not keep up. The investigation bottleneck is getting worse, not better.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nBkBkoexe-2YNQklgLlnCw.png" /></figure><h3>The Four Approaches</h3><h3>Native: Stage 1 (Assist)</h3><p>Example tool: <a href="https://aws.amazon.com/devops-agent/">AWS DevOps Agent</a></p><p>This approach is built directly into the AWS ecosystem. It connects to CloudWatch, deployment pipelines, and other AWS services out of the box. Setup time is low.</p><p>The strength is simplicity within AWS. You do not configure integrations because they are already there. For teams that run entirely on AWS and want AI assistance without adding another vendor, this is a reasonable starting point.</p><p>The tradeoff is scope. It works well inside AWS and less well outside it. If your system spans multiple clouds or relies on external tools, the native approach shows gaps.</p><p><strong>Fits best:</strong> Teams fully on AWS, smaller engineering orgs, situations where setup simplicity matters more than depth.</p><p><strong>Core weakness: </strong>Limited value in multi cloud environments. Does not investigate root cause autonomously.</p><h3>Open Source: Stage 2 (Analyze)</h3><p>Example tool: <a href="https://k8sgpt.ai/">K8sGPT</a></p><p>This is a lightweight open source tool focused on Kubernetes. Engineers run it against a cluster to get analysis of what is failing, with plain language explanations and suggested fixes.</p><p>The appeal is flexibility and low cost. It runs anywhere. You can extend it. No vendor relationship to manage. For Kubernetes focused teams that want a fast diagnostic layer without paying for a managed product, it fits well.</p><p>What it does not do is automate anything. The engineer runs the command, reads the output, and takes action. It does not watch your systems continuously. It is a debugging assistant, not an incident response system.</p><p>Teams that treat it as a complete solution end up disappointed. Teams that treat it as a sharp tool for Kubernetes debugging get real value.</p><p><strong>Fits best:</strong> Kubernetes focused teams, orgs that prefer open source, manual debugging assistance.</p><p><strong>Core weakness:</strong> No continuous monitoring, no automation, requires manual effort at every step.</p><h3>Hybrid: Stage 2 to 3 (Enrich)</h3><p>Example tool: Metoro</p><p>This approach improves data quality instead of improving how AI reasons about your data. It uses eBPF to collect system level telemetry without manual instrumentation.</p><p>Better data leads to better analysis. A lot of false positives come not from weak AI reasoning but from incomplete signals. Hybrid tools fix that at the source.</p><p>According to Grafana’s 2026 survey, 47 percent of teams increased OpenTelemetry usage but only 41 percent are running it in production. Incomplete instrumentation is a real problem. If your engineers say “we did not have the right signals to diagnose that quickly,” fixing the data layer is the right first move.</p><p>The limitation is that its primary focus is Kubernetes environments.</p><p><strong>Fits best: </strong>Teams where incomplete observability data causes slow root cause analysis, Kubernetes heavy environments.</p><p><strong>Core weakness:</strong> Kubernetes centric. Strong as a component rather than a complete solution.</p><p>For more on fixing observability gaps, see <a href="https://www.sherlocks.ai/blog/the-four-pillars-of-telemetry-metrics-logs-traces-and-events">The Four Pillars of Telemetry</a>.</p><h3>Agentic: Stage 4 (Investigate)</h3><p>Example tool: <a href="https://sherlocks.ai/">Sherlocks.ai</a></p><p>This approach actively investigates rather than just surfacing data. When an incident fires, it pulls signals from logs, metrics, deployment history, and past incidents. It correlates them. It tells the engineer what it found and where to look.</p><p>Teams that rely on manual signal correlation, opening four dashboards, cross referencing timestamps, digging through logs, typically spend a large chunk of incident time just getting oriented. According to Gartner, 70 percent of enterprises will deploy agentic AI to operate IT infrastructure by 2029. Manual investigation does not scale.</p><p>Agentic systems require more from you upfront. They depend on reasonable observability data. They get better as they learn your system over time.</p><p><strong>Fits best:</strong> Teams spending significant time on manual root cause analysis, multi service environments with complex failure patterns.</p><p><strong>Core weakness:</strong> Takes time to tune, requires solid observability data to work well from the start.</p><h3>Real Incident Example: Kubernetes Pod Crash</h3><p>A useful way to understand these differences is to run the same incident through each tool.</p><p>Scenario: A Kubernetes pod keeps crashing shortly after a new deployment. The on call engineer has just been paged.</p><ul><li><strong>Native approach</strong>. Pulls CloudWatch logs, recent deployment events, and error outputs from within AWS. Surfaces that a deployment happened recently and flags error patterns. The engineer reviews the output and decides what to do.</li><li><strong>Open source approach</strong>. Engineer runs a command against the cluster. Returns an explanation, like a misconfigured resource limit or failing liveness probe, with a suggested fix. Clear and fast, but only covers the Kubernetes layer.</li><li><strong>Hybrid approach.</strong> The tool has been collecting eBPF telemetry continuously. Engineer accesses deep system level data about what was happening inside the pod before it failed. Reduces guesswork around memory pressure or syscall failures.</li><li><strong>Agentic approach.</strong> The system connects the deployment event, pod crash logs, error trace, and any similar past incidents. Surfaces a summary: this pattern matches a previous deployment failure caused by an environment variable mismatch, here is where to look.</li></ul><p>None of these is the “correct” way. Which one reduces MTTR the most depends on where your investigation time is actually being lost.</p><h3>Which Approach Should You Choose?</h3><p><strong>Choose native if </strong>your infrastructure is fully on AWS, your team is small or early stage, and you want something that works without adding vendors.</p><p><strong>Choose open source if</strong> Kubernetes debugging is your primary pain point, you value flexibility and control, and your team has bandwidth to run commands during incidents.</p><p><strong>Choose hybrid if</strong> your biggest problem is incomplete or noisy observability data. If engineers regularly say “we did not have the right signals,” fixing the data layer is the right first move.</p><p><strong>Choose agentic if</strong> investigation time is where your incidents go long. If on call engineers spend 30+ minutes per incident correlating signals before they even know what they are dealing with.</p><h3>A Note on Team Size</h3><ul><li><strong>Under 50 engineers. </strong>Open source or native is usually enough. The overhead of an agentic system may not be worth it yet.</li><li><strong>50 to 500 engineers. </strong>Hybrid or agentic tools start earning their cost. Incident volume and system complexity have grown. Manual correlation starts to hurt.</li><li><strong>500+ engineers.</strong> Agentic tools are usually justified. The time cost of manual root cause analysis is significant. Institutional knowledge gaps make the knowledge retrieval aspect especially valuable.</li></ul><h3>The Bottom Line</h3><p>Most teams think they are choosing a tool. They are actually choosing which part of the incident workflow they want AI to handle.</p><p>Native tools handle data access. Open source tools handle Kubernetes interpretation. Hybrid tools handle signal quality. Agentic tools handle investigation.</p><p>The right tool for a team losing time in alerting is not the same as the right tool for a team losing time in root cause analysis. Identify where time is actually being lost before you pick a tool.</p><p>These approaches are often complementary. Many teams layer tools from multiple stages as their stack matures. For a broader view of how incident response platforms fit together, see <a href="https://www.sherlocks.ai/blog/incident-response-platforms-devops-2026">Incident Response Platforms for DevOps in 2026</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ade6570db34a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[IT Ops vs DevOps vs SRE vs Agentic Ops]]></title>
            <link>https://medium.com/@gauravsherlocksai/it-ops-vs-devops-vs-sre-vs-agentic-ops-e38d361c17fa?source=rss-22d28fd64678------2</link>
            <guid isPermaLink="false">https://medium.com/p/e38d361c17fa</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <dc:creator><![CDATA[Sherlocks.ai]]></dc:creator>
            <pubDate>Fri, 12 Jun 2026 06:31:01 GMT</pubDate>
            <atom:updated>2026-06-12T06:31:01.101Z</atom:updated>
            <content:encoded><![CDATA[<h4>They are not competitors. They are four answers to the same question, one that has gotten harder every decade.</h4><p>Here is a question that causes more confusion than it should. What is the difference between IT Ops, DevOps, and SRE? And where does this new thing called Agentic Ops fit?</p><p>The confusion makes sense. All four want the same outcome: a reliable system that serves users. Defining them by their goal blurs them together. The trick is to define each by what makes it distinct. Its posture. Its unit of work. Its relationship to toil.</p><p>Here is the simple version.</p><blockquote>IT Ops keeps the lights on. DevOps ships faster, safely, together. SRE engineers reliability. Agentic Ops lets AI agents do the investigation.They are not competitors. They are four answers to one question that has gotten harder every decade. How do you keep software running for users as systems get bigger and faster?</blockquote><p>The clearest way to tell them apart is their relationship to toil. Toil is the manual, repetitive work of running a system. IT Ops does the toil by hand. DevOps removes toil from the delivery path. SRE treats toil as a defect and engineers it away. Agentic Ops puts AI agents on the toil that remains.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ABmEkEFxqaRbb6aiIzqWiA.png" /></figure><h3>The Evolution: Four Eras That Accumulated</h3><p>Here is the key thing most explanations miss. These are not rivals you choose between. They are eras. Each one was invented to fix a pain the previous one could not handle. Newer eras did not delete the older ones. They layered on top.</p><ul><li><strong>IT Ops emerged in the 1990s and 2000s.</strong> Computers became business critical, but running them was ad hoc and unrepeatable. IT Ops formalized running and supporting production. Tickets. Runbooks. Change management. A network operations center watching dashboards.</li><li><strong>DevOps emerged around 2009.</strong> The pain was simple. Developers wrote code and threw it over the wall to a separate operations team. That loop was painfully slow. DevOps tore down the wall. Shared ownership. You build it, you run it. The culture showed up in tooling. CI/CD pipelines. Infrastructure as code.</li><li><strong>SRE emerged at Google around 2003, though the book came later.</strong> The provocation was simple. What if you asked a software engineer to design your operations function? You solve operations problems with software. You treat reliability as something you engineer. You measure it. You govern risk with error budgets. You cap toil and automate it away.</li><li><strong>Agentic Ops is emerging now.</strong> SRE is the gold standard, but great SREs are scarce and expensive. The reactive investigation work, the 2 AM digging through logs and metrics, still lands on tired humans. Agentic Ops puts AI agents on that work. They investigate the moment an alert fires. They hand a human the cause and the next best action inside guardrails the team defines.</li></ul><h3>What Each One Actually Is</h3><h3>1) IT Ops: Keep the Lights On</h3><p><strong>Posture:</strong> Reactive and manual.</p><p><strong>Unit of work:</strong> Tickets and runbooks. IT Ops keeps production running. The defining trait is that it is mostly reactive and manual. A problem fires. A human picks up a ticket. Follows a runbook. Fixes it.</p><p><strong>Enterprise example:</strong> A Network Operations Center watches dashboards. Incidents flow through ServiceNow and formal change management.</p><p><strong>Startup example: </strong>Whoever is on call gets paged. Opens the dashboard. Fixes it by hand. The runbook is a page in Notion, if it exists.</p><h3>2) DevOps: Ship Faster, Safely, Together</h3><p><strong>Posture:</strong> Proactive on delivery.</p><p><strong>Unit of work:</strong> Pipelines and infrastructure as code. DevOps is first a culture of shared ownership. You build it, you run it. The way that culture shows up in tooling is the pipeline. Continuous integration. Infrastructure as code. Continuous deployment.</p><p><strong>Enterprise example:</strong> A platform team maintains the CI/CD pipelines and golden paths every product team ships through.</p><p><strong>Startup example:</strong> The DevOps engineer wires up GitHub Actions and Terraform so the whole team can deploy many times a day.</p><h3>3) SRE: Engineer Reliability</h3><p><strong>Posture:</strong> Proactive on reliability.</p><p><strong>Unit of work:</strong> SLOs and error budgets. SRE came out of Google with a simple idea. What if you asked a software engineer to design your operations function? You solve operations problems with software. Reliability gets measured with SLIs and SLOs. Risk gets governed with error budgets. Toil gets capped and automated away.</p><p><strong>Enterprise example: </strong>A dedicated SRE org owns an error budget policy. Burn the budget and feature releases freeze until reliability returns.</p><p><strong>Startup example:</strong> The senior engineer who owns reliability sets a couple of SLOs, watches the burn, and automates the most painful manual chores first.</p><h3>4) Agentic Ops: Let AI Agents Do the Investigation</h3><p><strong>Posture:</strong> Autonomous investigation with human oversight.</p><p><strong>Unit of work:</strong> AI agents plus guardrails. Agentic Ops puts AI agents on the investigation work. They investigate the moment an alert fires autonomously. They perform root cause analysis. They hand a human the cause and the next best action. They act only inside guardrails the team has approved.</p><p><strong>Enterprise example:</strong> AI agents run first line investigation across hundreds of services. They draft the root cause analysis. They escalate to humans with full context for high stakes decisions.</p><p><strong>Startup example:</strong> A small team gets Google grade investigation without Google grade headcount. The agent does the 2 AM digging. The human approves the fix.</p><h3>The Sharpest Difference: Their Relationship to Toil</h3><p>If you remember one thing, remember this. Toil is the manual, repetitive, reactive work of running a system. It grows as the system grows. The cleanest way to separate the four eras is to ask what each one does with toil.</p><ul><li>IT Ops is the toil. Humans do the manual, repetitive work by hand.</li><li>DevOps removes delivery toil. It speeds up the path from code to production.</li><li>SRE treats toil as a defect. It caps toil and engineers it away.</li><li>Agentic Ops automates the remaining investigative toil. AI agents handle first line investigation.</li></ul><h3>Key Differences at a Glance</h3><p>Here is how the four stances compare across key dimensions.</p><p><strong>Posture</strong></p><ul><li>IT Ops: Reactive</li><li>DevOps: Proactive on delivery</li><li>SRE: Proactive on reliability</li><li>Agentic Ops: Autonomous, supervised</li></ul><p><strong>Unit of work</strong></p><ul><li>IT Ops: Tickets and runbooks</li><li>DevOps: Pipelines and infrastructure as code</li><li>SRE: SLOs and error budgets</li><li>Agentic Ops: AI agents plus guardrails</li></ul><p><strong>Who owns it</strong></p><ul><li>IT Ops: Separate operations team</li><li>DevOps: Shared development and operations</li><li>SRE: Embedded reliability engineers</li><li>Agentic Ops: Agents working for the team</li></ul><p><strong>Success metric</strong></p><ul><li>IT Ops: Tickets closed, uptime</li><li>DevOps: Deploy frequency, lead time</li><li>SRE: SLO attainment, error budget burn</li><li>Agentic Ops: Toil removed, MTTR collapsed</li></ul><h3>Key Similarities</h3><ul><li>Same north star. All four exist to keep production serving users. The disagreement is about method, not goal.</li><li>All live on observability. Metrics, logs, and traces are the shared bloodstream. You cannot run, improve, or automate what you cannot see.</li><li>All trend toward automation. Each era automates more of the previous era’s manual work. The direction is always less toil.</li><li>Modern teams blend them. A startup’s DevOps engineer often does all three older jobs at once. As Google puts it, class SRE implements DevOps.</li><li>These are overlapping disciplines and accumulated eras. They are not mutually exclusive boxes on an org chart.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RCU3J2LG9baOmhaPljKEWA.png" /></figure><h3>The Rise of Agentic Ops</h3><p>Each era solved the previous era’s bottleneck. IT Ops kept systems alive by hand. DevOps sped up and shared delivery. SRE made reliability a measured, engineered discipline. Every step pushed more toil out of human hands.</p><p>But SRE left one stubborn piece of toil behind. The discipline is brilliant, yet great SREs are rare and expensive. The reactive investigation work, the 2 AM digging through logs, metrics, and deploys to find root cause, still lands on tired humans. That is the gap Agentic Ops fills.</p><p>SRE said treat reliability as an engineering problem. Solve operations with software. Agentic Ops is the next step. Let AI agents do the investigation and the toil. Reliability engineering finally becomes within reach for every team. Your humans focus on architecture and SLOs instead of the firefight.</p><h3>The Bottom Line</h3><p>IT Ops, DevOps, SRE, and Agentic Ops are not competitors. They are four stances toward the same job: keeping software running for users.</p><p>The useful question is not which label to adopt. It is how reactive and manual your operations currently are, and how much of that toil you want to automate or hand to AI agents.</p><p>Start where you are. Layer on the next era when the pain of the current one exceeds the cost of change. That is how operations has always evolved.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e38d361c17fa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How CTOs Lose Touch With Production (And How to Fix It)]]></title>
            <link>https://medium.com/@gauravsherlocksai/how-ctos-lose-touch-with-production-and-how-to-fix-it-e767c3688da1?source=rss-22d28fd64678------2</link>
            <guid isPermaLink="false">https://medium.com/p/e767c3688da1</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cto]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[incident-response]]></category>
            <dc:creator><![CDATA[Sherlocks.ai]]></dc:creator>
            <pubDate>Tue, 09 Jun 2026 05:11:00 GMT</pubDate>
            <atom:updated>2026-06-09T05:11:00.735Z</atom:updated>
            <content:encoded><![CDATA[<h4>You did not stop caring. The job just pulled you away. Here is how to get back.</h4><p>You started as an engineer. You knew every service. You could debug a production issue in minutes.</p><p>Now you lead. Your calendar is full. Your dashboards are everywhere. But something feels off. You read incident summaries. You see the metrics. You trust your team. Yet reliability problems keep coming back. Your best engineers seem tired. And you cannot quite put your finger on why.</p><p>Here is the truth. You did not stop caring about production. The job was structurally designed to pull you away from it. This is Production Drift. It is not a personal failure. It is a predictable outcome of how the CTO role scales. And there is a fix.</p><h3>What Is Production Drift?</h3><p>Production Drift is the gradual loss of direct exposure to real system behavior by engineering leadership. It happens as organizations grow. Information flows upward through layers. Incident becomes postmortem. Postmortem becomes summary. Summary becomes leadership update. Leadership update becomes strategy.</p><p>At each step, detail is lost. Texture is compressed. Friction is rounded off. By the time operational reality reaches the CTO, it has been optimized for readability, not accuracy. Production visibility is not about having more data. It is about reducing the distance between decision makers and reality.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RgUVGKYihSRLE0py6wCTkA.png" /></figure><h3>Why CTOs Lose Touch</h3><p>The calendar fills up. Planning. Hiring. Board prep. Vendor meetings. Strategy reviews. The information flow becomes mediated. Dashboards instead of incidents. Summaries instead of timelines. Leadership updates instead of raw operational friction. That abstraction is necessary. It is also dangerous.</p><p>According to the <a href="https://leaddev.com/culture/engineering-burnout-rising-2025-layoffs-reshape-tech-industry">LeadDev Engineering Leadership Report 2025</a>, 65 percent of engineering leaders report expanded responsibilities. As scope grows, leaders become progressively disconnected from day to day operational risks. The <a href="https://dora.dev/research/2024/dora-report/">2024 DORA report</a> shows that leaders who stay actively connected to their teams significantly improve productivity and reduce burnout. The inverse is also true. Distance compounds.</p><p>It usually happens in predictable ways. CTOs read incident summaries, not incident timelines. They hear about customer pain through quarterly themes, not actual support patterns. They review architecture in slides, not in the lived experience of on call. They interact with the product as insiders, not as stressed operators. They fund feature work and migrations, but not the quality of operating the system itself.</p><p>None of these are individually unreasonable. Together, they create distance. Over time, that distance compounds into bad decisions made confidently.</p><p>The issue is not lack of visibility. Most organizations have plenty of dashboards. The issue is that dashboards are downstream artifacts. A dashboard tells you a metric crossed a threshold. It does not tell you that three engineers lost sleep last Tuesday debugging a failure that had happened before. It does not tell you that your best SRE is quietly considering leaving because on call has become untenable.</p><h3>What Not to Do</h3><p>Most engineering leaders do not ignore production deliberately. They substitute better looking inputs for the real thing. These substitutions feel productive. They are not.</p><ul><li><strong>Relying on weekly engineering updates instead of reading postmortems.</strong> Updates are written to reassure, not to surface friction. The organizational truth lives in the timeline and the Slack thread.</li><li><strong>Treating dashboards as a proxy for operational health. </strong>Dashboards show you what was instrumented. They do not show you what was missed or what your on call engineers are experiencing at 3 AM.</li><li><strong>Assuming support escalations represent the full picture. </strong>Most customer pain never escalates. It accumulates quietly in tickets triaged as noise before reaching leadership.</li><li><strong>Conducting quarterly architecture reviews as a substitute for operational context. </strong>Architecture in slides is a model of the system. The gap between the model and reality is where incidents live.</li><li><strong>Waiting for engineers to raise problems. </strong>Engineers optimize for solving problems, not for surfacing friction to leadership. If no one is raising concerns, that is rarely a sign that nothing is wrong.</li></ul><h3>The Real Cost of Being Too Far From Production</h3><p>When the CTO is too far from production, the company pays the same few ways every time. Reliability problems linger because they are legible to engineers but not salient to leadership. On call becomes a tax on your best people. Investment decisions get made from abstractions, which means the roadmap looks cleaner than reality.</p><p>The numbers make this concrete. Customer-impacting incidents increased 43 percent last year, with each incident costing nearly $800,000 on average. Operational toil rose 30 percent in 2025, the first increase in five years. For Fortune 500 companies, a single hour of downtime costs between $500,000 and $1 million.</p><p>These are not engineering problems. They are visibility problems. And visibility problems are leadership problems.</p><p><strong>A real example:</strong> At Doubtnut, a high-growth edtech platform serving millions of students across India, a payment flow degradation affected a small but real percentage of users. The issue appeared in support tickets for several days. It was categorised as intermittent noise and never escalated. By the time it surfaced to engineering leadership, it had been impacting users for nearly a week. The fix itself took a few hours. The detection lag took days. The signal existed. It just lost fidelity at each reporting layer before it reached the people who could act on it.</p><h3>The Production Reality Loop</h3><p>The Production Reality Loop is a framework of six recurring habits that keep engineering leadership grounded in operational truth. Each habit is low cost relative to the quality of signal it produces. None require the CTO to operate like a staff engineer. The goal is not to micromanage engineering. It is to stay close enough to production that strategy, staffing, and investment decisions are anchored in what is true, not what is comfortable to report.</p><h3>Habit 1: Read Incident Postmortems End to End</h3><p>Not the summary. Not the executive overview. The actual timeline, Slack threads, and root cause section. Once a week on a fixed calendar block.</p><p>Postmortems show you the engineering system as it really works under stress. Who got paged. How many people piled on. Whether the runbook helped. Whether the same failure is reappearing under different names. The summary preserves conclusions and drops texture. The texture is where the truth lives.</p><p>Do not read only high severity incidents. The most revealing patterns often live in medium severity ones. The failures that never trigger a P0 response but quietly drain engineering capacity every week. Cost: 60 to 90 minutes a week.</p><h3>Habit 2: Observe an On Call Shift</h3><p>One shift a quarter is enough. Do not take the keyboard. Do not become the loudest person in the room. Watch.</p><p>You see alert quality, dashboard usability, escalation patterns, and tooling gaps. You see whether your engineers are operating a system or wrestling one. According to the SRE Report 2025, 46 percent of SREs responded to more than five incidents in the last 30 days, and 65 percent report experiencing burnout. That number does not change if leadership is not watching.</p><p>Do not shadow a quiet shift and conclude everything is fine. Schedule observation during normal operational load. Cost: A few hours once a quarter.</p><h3>Habit 3: Build a Direct Loop With Support</h3><p>Not as an occasional escalation path. As a recurring conversation with the person who sees patterns in customer pain before engineering does.</p><p>Support often sees reliability and usability signals earlier than your internal metrics do. Slow workflows. Confusing failures. Flaky integrations. The Doubtnut scenario above is a direct example. The signal was in support. It never reached leadership through the normal reporting chain.</p><p>Do not set up a formal escalation process as a substitute. Escalation filters for severity. You want the patterns that never meet the threshold. The quiet friction that accumulates for weeks. Cost: 30 minutes every two weeks.</p><h3>Habit 4: Use the Product Like a Customer</h3><p>Create a fresh account. Follow your own onboarding. Hit the API. Complete the core workflow without shortcuts.</p><p>Production is not just uptime. It is the full operational experience. Broken docs. Confusing defaults. Missing guardrails. Awkward workflows. These are invisible to internal teams because they have learned to route around them.</p><p>Do not use an internal test account with pre configured settings. That account has been optimized for internal use. It does not represent what a new customer experiences on day one. Cost: Two hours once a quarter.</p><h3>Habit 5: Fund the On Call Experience</h3><p>Not incident response tooling in the abstract. The actual experience of being paged at 3 AM. Alert grouping. Payload context. Runbook quality. Ownership clarity.</p><p>Every improvement to on call quality reduces toil, lowers MTTR, improves retention, and keeps senior engineers willing to stay close to production. Developer toil costs roughly $9.4 million per year per 250 engineers. Most of that is recoverable with deliberate investment.</p><p>Do not conflate buying a new monitoring tool with improving the on call experience. More dashboards do not reduce toil. Fewer alerts with higher quality, better context, and clearer ownership do. Cost: Reserve engineering capacity every quarter for operational experience improvements.</p><h3>Habit 6: Review One Critical Service</h3><p>Pick a high blast radius service. Look at recent changes, incidents, alerts, and runbooks together.</p><p>Looking at change history in isolation becomes a code reading exercise. Looking at it alongside incidents and operability reveals whether the team is increasing system complexity faster than it can safely operate it.</p><p>Do not review only services that have recently had incidents. The riskiest services are often the ones that have been stable for a long time. Stability can mask accumulated complexity and undocumented dependencies. Cost: Two hours once a quarter, rotating across critical services.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fDfRc6LiTvjHC4U6EyuQDQ.png" /></figure><h3>The Habits at a Glance</h3><ul><li><strong>Read postmortems.</strong> Weekly, 60 to 90 minutes. Surfaces recurring failures and runbook gaps.</li><li><strong>Observe on call.</strong> Quarterly, 3 to 4 hours. Surfaces alert quality and engineer load.</li><li><strong>Support loop.</strong> Bi weekly, 30 minutes. Surfaces early customer pain.</li><li><strong>Use the product.</strong> Quarterly, 2 hours. Surfaces UX friction and broken defaults.</li><li><strong>Fund on call.</strong> Quarterly, planning time. Surfaces MTTR and retention risk.</li><li><strong>Operability review. </strong>Quarterly, 2 hours. Surfaces complexity growth versus operational capacity.</li></ul><p>These habits work because they are recurring. They cannot be crowded out by a busy quarter. They are low cost relative to the signal they produce. And they get you closer to raw operational reality rather than a summarized version.</p><h3>The Bottom Line</h3><p>CTOs lose touch with production structurally, not personally. Production Drift is the name for this problem. The Production Reality Loop is the fix.</p><p>You do not need to become the best incident responder in the company. You need to stay close enough to production that your strategy, staffing, and investment decisions are anchored in reality instead of abstraction.</p><p>Start with one habit this week. Read one postmortem end to end. Then do it again next week. The signal you have been missing is already there. You just need to look at it directly.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e767c3688da1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Four Pillars of Telemetry: Metrics, Logs, Traces, and Events]]></title>
            <link>https://medium.com/@gauravsherlocksai/the-four-pillars-of-telemetry-metrics-logs-traces-and-events-db199b891b9a?source=rss-22d28fd64678------2</link>
            <guid isPermaLink="false">https://medium.com/p/db199b891b9a</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[telemetry]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[observability]]></category>
            <dc:creator><![CDATA[Sherlocks.ai]]></dc:creator>
            <pubDate>Fri, 05 Jun 2026 12:27:48 GMT</pubDate>
            <atom:updated>2026-06-05T12:27:48.698Z</atom:updated>
            <content:encoded><![CDATA[<h4>Your on-call engineer does not need another dashboard. They need to know what changed. Here is how to give them that answer.</h4><p>You get paged at 3 AM. A service is unhealthy. You pull up metrics. CPU is spiking. You open logs. Error rates are climbing. You open traces. A downstream database is timing out.</p><p>Then you spend twenty minutes scrolling through Slack, checking GitHub Actions, and messaging the deploy team because none of your dashboards told you what changed thirty minutes before the spike.</p><p>That is not observability. That is three telemetry sources stitched together by a tired human.</p><p>Most SRE teams have invested heavily in metrics, logs, and traces. They are missing the fourth pillar: events. Deploys. Config changes. Feature flag flips. Infrastructure actions. This signal is scattered across Slack channels and CI/CD logs. It is not queryable. Not correlated to alerts.</p><blockquote>The fix is simple. Treat events as first-class telemetry. Structure them. Put them in a queryable event bus. Surface recent events in your alert payloads. Your on-call engineer stops asking “what changed?” and starts asking “how do we fix it?”</blockquote><h3>The Four Pillars of Telemetry</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YuJQXoTIkZEdi1N2EXRTOQ.png" /></figure><p>Four pillars. Four questions. You need all of them.</p><ul><li><strong>Pillar 1: </strong>Metrics answers “Is something wrong?” CPU, request rate, error count.</li><li><strong>Pillar 2: </strong>Logs answers “What did the system do?” Application logs, access logs.</li><li><strong>Pillar 3: </strong>Traces answer “Where did the latency come from?” Distributed traces across services.</li><li><strong>Pillar 4:</strong> Events answers “What changed in the last hour?” Deploys, config pushes, feature flag flips.</li></ul><p>The first three tell you the system is broken and where. The fourth tells you why. That asymmetry is the core of the MTTR problem.</p><h3>Pillar 1: Metrics</h3><ul><li>What metrics answer well: Is the rate of X up or down? Is latency breaching SLO?</li><li>What metrics do not answer: Why. Metrics are aggregations. The per-request context that caused the spike is already collapsed.</li><li>Where teams over-invest: Building more dashboards. The on-call engineer does not need another panel. They need a hypothesis.</li><li>Where teams under-invest: Cardinality discipline. High-cardinality labels blow up storage costs. Most databases drop them silently.</li></ul><h3>Pillar 2: Logs</h3><ul><li>What logs answer well: What did the system actually do, in order, for this request?</li><li>What logs do not answer: Aggregate behavior. “How often does this happen?” requires counting, which logs are poor at.</li><li>Where teams over-invest: Verbosity. Structured logs at five megabytes per request are a data exhaust pipe.</li><li>Where teams under-invest: Correlation IDs. Logs without a request_id or trace_id are noise. Logs with one are the spine of incident response.</li></ul><h3>Pillar 3: Traces</h3><ul><li>What traces answer well: Where did the latency come from? Which downstream service is the bottleneck?</li><li>What traces do not answer: Anything you did not instrument. Tracing is opt-in. Coverage gaps are invisible until an incident exposes them.</li><li>Where teams over-invest: Sampling configuration. The practical answer: sample tail, accept the storage cost.</li><li>Where teams under-invest: Cross-team trace context propagation. If the downstream team strips your header, your trace is half a story.</li></ul><h3>Pillar 4: Events (The One Everyone Forgets)</h3><p>Events are discrete state changes. Deploys. Feature flag flips. Config pushes. Schema migrations. Autoscaling actions.</p><ul><li>What events answer well: What changed. Almost every incident has the same first question: “what was different in the last hour?” Events are the only pillar that answers directly.</li><li>What events do not answer: Per-request behavior. Events are coarse. They tell you something changed. The other pillars tell you how the system responded.</li><li>Where teams under-invest: Treating events as first-class telemetry. Walk into a typical SRE organization. Deploy notifications in Slack. GitHub Actions logs behind auth. Terraform notifications in a separate tool. Vendor status pages are manually checked. None of it is structured. None of it is queryable. None of it is correlated to the alert.</li></ul><p>The deploy that broke production is in your CI/CD logs. It is also almost certainly not correlated to your alert. That gap is where MTTR lives.</p><h3>Why Do Most Teams Have Three Pillars But Not Four?</h3><p>Two reasons.</p><p>First, events come from many systems. CI/CD. Feature flags. Infrastructure. Vendors. No single team owns all of them.</p><p>Second, there is no natural home for events. Metrics have Prometheus. Logs have Elastic. Traces have Jaeger. Events have Slack. Slack is not a telemetry system.</p><p>The result is that the signal most directly correlated with incident causation is the least structured and least accessible.</p><h3>What Does a Well-Built Fourth Pillar Look Like?</h3><ul><li><strong>One: A single event bus.</strong> Deploys, flag changes, and config pushes all land in one place with structured payloads.</li><li><strong>Two: Structured metadata on every event.</strong> Every event carries a timestamp, a service identifier, an actor, and a reference, such as a commit SHA or a flag name.</li></ul><p>Here is what a structured event looks like:</p><pre>json<br>{<br>  &quot;timestamp&quot;: &quot;2026-05-14T10:32:21Z&quot;,<br>  &quot;service&quot;: &quot;payment-api&quot;,<br>  &quot;actor&quot;: &quot;deploy-system&quot;,<br>  &quot;action&quot;: &quot;deploy&quot;,<br>  &quot;version&quot;: &quot;v2.3.1&quot;,<br>  &quot;commit&quot;: &quot;a7f3e9b&quot;,<br>  &quot;environment&quot;: &quot;production&quot;<br>}</pre><p>If an event does not have a service identifier and an actor, it is a notification, not telemetry.</p><ul><li><strong>Three: Events queryable alongside metrics. </strong>You should overlay “deploys to service X” on the latency graph without switching tabs.</li><li><strong>Four: Events included in alert payloads.</strong> Alerts should surface recent events for the affected service in the notification itself.</li></ul><p>The most expensive minutes in any incident are between page and first hypothesis. If that time goes from “open four tabs” to “read the alert and know what changed,” MTTR drops before any new tooling is added.</p><h3>How to Start Building Your Fourth Pillar This Week</h3><p>You do not need a new vendor. You need a pattern.</p><ul><li><strong>Step 1: </strong>Identify all sources of change events. Deploys. Feature flags. Config changes. Infrastructure actions. (One day)</li><li><strong>Step 2:</strong> Pick one source. Deploys are highest leverage. Send every deploy event to a structured destination. A BigQuery table. A ClickHouse database. Even a Slack webhook that parses JSON. (Two to three days)</li><li><strong>Step 3:</strong> Add three required fields to every event. Timestamp. Service. Actor. (One day)</li><li><strong>Step 4:</strong> Surface recent events in your alert payloads. If your alerting system can pull from your event store, do it. If not, append the last five events to the alert description manually. (One to two days)</li><li><strong>Step 5: </strong>Repeat for feature flags, then config changes, then infrastructure.</li></ul><p>The goal is not perfection. The goal is to move from “events are scattered” to “events are structured and queryable.” Start small. One event type. One destination.</p><h3>How to Prioritize Telemetry Investment</h3><ul><li><strong>If your team says “we never have enough data”: </strong>The gap is metrics or logs. Add structured logging with correlation IDs.</li><li><strong>If your team says “we can see something wrong but cannot tell where”: </strong>The gap is traces. Add distributed tracing.</li><li><strong>If your team says “we know there is a problem but cannot figure out what changed”: </strong>The gap is events. Build a structured event bus. Start with deploys.</li><li><strong>If your team says “we have all the data but it takes forever to connect it”:</strong> The gap is the intelligence layer. Evaluate an AI SRE platform.</li><li><strong>Under 50 engineers: </strong>Focus on metrics, logs, and basic events. Correlation IDs and a basic deploy stream.</li><li><strong>50 to 200 engineers:</strong> Add traces and structured events. Manual context reconstruction becomes the bottleneck.</li><li><strong>200+ engineers: </strong>All four pillars plus an AI correlation layer. Missing any pillar costs engineer-hours per week.</li></ul><h3>How AI SRE Uses the Four Pillars Differently</h3><p>An AI platform that only consumes metrics and logs is a slightly faster grep. The platforms that work consume all four pillars in a specific order.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WmzxGqUgkMP6V7d62X-fGQ.png" /></figure><p>An agent that starts with logs hallucinates. Logs are unstructured and lossy at scale. An agent that starts with events finishes faster because it works causally, not symptomatically.</p><p>This is how experienced on-call engineers actually triage.</p><h3>The Bottom Line</h3><p>Your on-call engineer does not need another dashboard. They need to know what changed in the last hour.</p><p>Build the fourth pillar. Start with deploys. Add structured metadata. Surface events in your alerts.</p><p>Everything else is just noise.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=db199b891b9a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Complex Systems Fail: An SRE Perspective]]></title>
            <link>https://medium.com/@gauravsherlocksai/how-complex-systems-fail-an-sre-perspective-8da766e0f638?source=rss-22d28fd64678------2</link>
            <guid isPermaLink="false">https://medium.com/p/8da766e0f638</guid>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[site-reliability-engineer]]></category>
            <category><![CDATA[incident-response]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[complex-systems]]></category>
            <dc:creator><![CDATA[Sherlocks.ai]]></dc:creator>
            <pubDate>Wed, 27 May 2026 08:15:56 GMT</pubDate>
            <atom:updated>2026-05-27T08:15:56.669Z</atom:updated>
            <content:encoded><![CDATA[<h4>Richard Cook wrote the playbook for understanding failure in medicine and aviation. It turns out he was writing about your Kubernetes cluster too.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zANNKOjuualJSROH" /></figure><p>You are woken up at 3 AM by a page. A service is unhealthy. You open your laptop, and the dashboards are a sea of red. You start hunting: logs, metrics, traces, deployments.</p><p>You are now living inside a complex system failure.</p><p>In 1998, Dr. Richard Cook wrote 18 observations about how complex systems fail. He was looking at medicine, aviation, and nuclear power. But he could have been looking at your distributed system.</p><p>These 18 points describe your on-call life with uncomfortable accuracy. Let me translate them for SREs in 2026.</p><h3>The TL;DR Before We Start</h3><p><strong>Cook’s Version:</strong> Failure in complex systems is the natural result of the system’s complexity. Defenses contain hazards but never eliminate them. Catastrophe is a combination, not a cause.</p><p><strong>SRE Version:</strong> Production software is a complex system in Cook’s sense. The 18 points describe outages, post-mortems, and on-call life perfectly.</p><h3>Observation №1</h3><p>Complex systems are intrinsically hazardous</p><ul><li><strong>Cook (1998):</strong> The frequency of hazard exposure can sometimes be changed, but the processes involved are intrinsically and irreducibly hazardous. This drives the creation of defences.</li><li><strong>SRE (2026): </strong>Every production system ships with a latent hazard. Data loss. Cascading failure. Credential leak. Silent corruption. You did not add the hazard by deploying. It was there the moment you chose a distributed state.</li></ul><p>SRE work is not the elimination of hazards. It is the continuous containment of it.</p><h3>Observation №2</h3><p>Complex systems are heavily and successfully defended against failure</p><ul><li><strong>Cook (1998):</strong> High consequences lead to multiple layers of defence: technical, human, and organisational components. These shields normally divert operations away from accidents.</li><li><strong>SRE (2026):</strong> Your platform does not melt down daily because it is layered with defenses. Retries. Circuit breakers. Quotas. Health checks. Canaries. Multi-AZ. Multi-region. Human review. Change freezes. Alert routing.</li></ul><p>Most of these defenses are invisible when they work. You only notice them when one is missing.</p><h3>Observation №3</h3><p>Catastrophe requires multiple failures</p><ul><li><strong>Cook (1998):</strong> Overt failure occurs when small, innocuous failures join to create a systemic accident. Each is necessary, but only the combination is sufficient.</li><li><strong>SRE (2026):</strong> Nobody gets paged because of one bad commit. They get paged because a bad commit merged on a Friday during a deploy freeze exception, shipped through a canary with a broken metric, into a region whose autoscaler was already saturated, while the on-call was at dinner with their phone on silent.</li></ul><p>Single points of failure exist. An expired TLS cert. A corrupted config. But the catastrophe still required the surrounding process defenses to fail too.</p><h3>Observation №4</h3><p>Complex systems contain changing mixtures of failures latent within them</p><ul><li><strong>Cook (1998):</strong> Complexity makes it impossible for systems to operate without multiple flaws. Eradication of all latent failures is limited by cost and by the difficulty of seeing how failures might contribute to an accident before the fact.</li><li><strong>SRE (2026): </strong>Your system right now contains dozens of bugs, misconfigurations, expired-but-cached credentials, drift between your infrastructure as code and reality, and dependencies pinned to versions that have since been yanked.</li></ul><p>You do not know which ones. You will learn about some of them during the next incident.</p><h3>Observation №5</h3><p>Complex systems run in degraded mode</p><ul><li><strong>Cook (1998): </strong>A corollary to the preceding point. Complex systems run as broken systems. They continue to function because they contain redundancies and because people can make them function despite flaws.</li><li><strong>SRE (2026):</strong> The production system you imagine in your head, where every pod is healthy, every queue is drained, every replica is in sync, does not exist and never has. Real systems always have something broken.</li></ul><p>The art is running usefully despite constant partial failure.</p><h3>Observation №6</h3><p>Catastrophe is always just around the corner</p><ul><li><strong>Cook (1998): </strong>Complex systems possess potential for catastrophic failure. Practitioners are nearly always in close proximity to these potential failures. It is impossible to eliminate this potential.</li><li><strong>SRE (2026):</strong> The same system that served a billion requests today can serve zero tomorrow. Nothing about yesterday’s uptime grants you tomorrow’s.</li></ul><p>This is why leaders who say, “We haven’t had an outage in 90 days, we must be doing something right,” terrify experienced SREs.</p><h3>Observation №7</h3><p>Post-accident attribution to a ‘root cause’ is fundamentally wrong</p><ul><li><strong>Cook (1998):</strong> Because overt failure requires multiple faults, there is no isolated cause of an accident. The label ‘root cause’ reflects the social, cultural need to blame specific, localized forces or events.</li><li><strong>SRE (2026):</strong> There is no root cause. There is a tree of contributing factors. The node you pick to call “root” says more about your organizational incentives than about the system. Good post-mortems resist that choice for as long as possible.</li></ul><p>Take the 2017 S3 us-east-1 outage. The “root cause” was an engineer typing the wrong argument. But equally true: no confirmation prompt, blast radius unbounded, subsystem had not been restarted in years, dependent services had no fallback. Pick all of them and you change how the system is built.</p><h3>Observation №8</h3><p>Hindsight biases post-accident assessments</p><ul><li><strong>Cook (1998):</strong> Knowledge of the outcome makes it seem that events should have been more salient to practitioners at the time than they actually were. After an accident, actions are judged with knowledge of the outcome, biasing assessment toward the negative.</li><li><strong>SRE (2026):</strong> After the incident, the signal looks obvious. “CPU was climbing for 20 minutes, why did no one see it?” Because at the time, CPU was climbing on 400 other dashboards too, and 399 of them resolved themselves.</li></ul><p>Hindsight collapses a fan of possibilities into a single narrative arc. Evaluate decisions by what was knowable at the time, not what the timeline reveals later.</p><h3>Observation №9</h3><p>Operators have dual roles: producers and defenders</p><ul><li><strong>Cook (1998):</strong> Practitioners and organizations must simultaneously deal with production demands and the possibility of failure. Balancing these roles is a source of perpetual conflict.</li><li><strong>SRE (2026):</strong> Cook’s framing does not map cleanly onto a modern SRE org chart. You will find pure producers (product engineers), pure defenders (platform teams), and hybrids. The dual role is not a property of every individual. It is a property of the system as a whole.</li></ul><p>The faster the org ships, the more defense it owes. Organizations that pretend this tradeoff does not exist pay for it in burnout or outages. Usually both.</p><h3>Observation №10</h3><p>All practitioner actions are gambles</p><ul><li><strong>Cook (1998): </strong>After accidents, the practitioner’s gamble looks ill-considered. But all action in complex systems is gambling. It takes place amid uncertain outcomes.</li><li><strong>SRE (2026): </strong>Every deploy, every rollback, everykubectl delete pod, every &quot;let me try restarting it&quot; is a bet against a system you cannot fully observe. Most bets pay off. The ones that do not become incidents.</li></ul><p>The goal is not to stop gambling. It is to make the gambles smaller and more reversible.</p><h3>Observation №11</h3><p>Actions at the sharp end resolve all ambiguity</p><ul><li><strong>Cook (1998):</strong> Organizations are ambiguous about the relationship between production targets, costs, and acceptable risks. All ambiguity is resolved by the actions of practitioners at the sharp end of the system.</li><li><strong>SRE (2026):</strong> Dashboards, runbooks, and alerts give you signals, not answers. At 3 AM, when 429s are spiking only for authenticated traffic, no document tells you whether to roll back, drain the region, or wait it out.</li></ul><p>The on-call engineer is the one who decides. That decision is what actually changes the system. All the prep work is judged by whether it sharpens that decision or muddies it.</p><h3>Observation №12</h3><p>Human practitioners are the adaptable element</p><ul><li><strong>Cook (1998): </strong>Practitioners actively adapt the system to maximize production and minimize accidents. These adaptations occur on a moment-by-moment basis.</li><li><strong>SRE (2026):</strong> When the incident is novel, tooling cannot save you. A human decides to failover, to drain a node, to call the vendor, to accept the data loss and restore from backup. Automation handles the known. Humans handle the unknown.</li></ul><p>This is why the right framing for AI in SRE is assistive, not autonomous. AI that hands an on-call engineer better context faster makes the human decisive. AI that tries to replace human decision-making makes humans unaccountable.</p><h3>Observation №13</h3><p>Human expertise is constantly changing</p><ul><li><strong>Cook (1998):</strong> Complex systems require substantial human expertise. This expertise changes as technology changes and as experts leave. Training and refinement are part of every complex system.</li><li><strong>SRE (2026):</strong> The expert on your payments pipeline last year is not the expert this year. The pipeline has changed, and so has the expert. Expertise is perishable.</li></ul><p>Tribal knowledge is the most expensive form of knowledge. It leaves with the person. Documented, searchable, queryable system knowledge is the antidote.</p><h3>Observation №14</h3><p>Change introduces new forms of failure</p><ul><li><strong>Cook (1998): </strong>The low rate of overt accidents may encourage changes to decrease high-frequency failures. These changes may create opportunities for new, low-frequency but high-consequence failures.</li><li><strong>SRE (2026):</strong> Every deploy, every migration, every provider switch, every version bump opens a new failure surface. The stability you feel right before a major change is partly an illusion. You have simply not yet discovered the failure modes the change will introduce.</li></ul><p>Plan rollbacks accordingly.</p><h3>Observation №15</h3><p>Views of ‘cause’ limit the effectiveness of defenses</p><ul><li><strong>Cook (1998):</strong> Locating ‘the cause’ of an accident in a particular component restricts the kinds of countermeasures that can be considered.</li><li><strong>SRE (2026):</strong> If your post-mortem concludes “engineer pushed bad YAML,” your remediation is “more review.” If it concludes “our validation pipeline did not catch an invalid resource spec,” your remediation is “better validation.”</li></ul><p>Same incident, very different defenses. Where you locate the cause determines what defenses you build.</p><h3>Observation №16</h3><p>Safety is a property of systems, not components</p><ul><li><strong>Cook (1998):</strong> Safety is an emergent property of systems. It does not reside in a person, device, or department. It is not a feature separate from other components.</li><li><strong>SRE (2026): </strong>You cannot buy safety by buying safe components. A 99.99 percent service composed of 99.99 percent components is not 99.99 percent reliable. The composition matters.</li></ul><p>Reliability is an emergent property of the whole. This is why “we use AWS, so we are fine” is not a reliability strategy. Good components do not compose into a safe system on their own. You still have to design for how they fail together.</p><h3>Observation №17</h3><p>People continuously create safety</p><ul><li><strong>Cook (1998):</strong> Failure-free operations result from people who work to keep the system within tolerable performance. These activities are part of normal operations. Human adaptations create safety from moment to moment.</li><li><strong>SRE (2026):</strong> The reason your system is up right now is that, at this moment, dozens of people are quietly choosing not to do risky things. Catching near-misses in review. Noticing a weird metric. Fixing a fragile deploy.</li></ul><p>This invisible labor is the bulk of reliability work, and it never shows up in any dashboard. The loud hero breaks something on Friday and gets a Slack post. The quiet hero prevented the break on Tuesday by rewriting a fragile script and gets nothing. Most orgs reward the first and depend on the second.</p><h3>Observation №18</h3><p>Failure-free operations require experience with failure</p><ul><li><strong>Cook (1998): </strong>Recognizing hazard and keeping the system inside tolerable boundaries requires intimate contact with failure. More robust performance arises where operators can discern the “edge of the envelope.”</li><li><strong>SRE (2026):</strong> Teams that never fail never learn. Game days, chaos engineering, and controlled failure injection generate the experience that pure uptime denies you.</li></ul><p>A team that has rehearsed its failover will execute it. A team that has only read the runbook will not. You do not need real failures. You need exposure to failure. A team that has not had a real incident in two years can mistake that for resilience. The first real one will be larger and less informed precisely because nobody has practiced.</p><h3>The Spec</h3><p>An AI system useful during incidents has to reason across contributing factors, respect the on-call as the decider, stay accurate in degraded mode, and carry knowledge across engineers who come and go.</p><p>Cook’s 18 points are the spec.</p><p>At <a href="https://sherlocks.ai/">Sherlocks.ai</a>, we built our approach around observation #7. There is no root cause. There is a tree of contributing factors. Our AI surfaces that tree, not a single false answer. The rest follows.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8da766e0f638" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI SRE Explained: What Every Engineer Needs to Know in 2026]]></title>
            <link>https://medium.com/@gauravsherlocksai/ai-sre-explained-what-every-engineer-needs-to-know-in-2026-55fc5c19bdde?source=rss-22d28fd64678------2</link>
            <guid isPermaLink="false">https://medium.com/p/55fc5c19bdde</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[Sherlocks.ai]]></dc:creator>
            <pubDate>Wed, 20 May 2026 07:43:13 GMT</pubDate>
            <atom:updated>2026-05-20T07:43:13.977Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*e_f5GrUNcdMH4He6.png" /></figure><h4>The bottleneck in reliability is not detection anymore. It is investigation. Here is how AI fixes that.</h4><p>You get paged at 2 AM. Your phone buzzes. Some service is down. You open your laptop. You pull up Datadog. Then Grafana. Then your logs. You check deployments. You scroll through traces. You piece together what happened.</p><p>Thirty-five minutes later, you find the culprit. A config change that broke the database connection pool. You fix it. You go back to sleep. You lose an hour of your night.</p><p>What if that investigation took five minutes instead of thirty-five? What if someone handed you the answer the moment you opened your laptop?</p><p>That is what AI SRE does.</p><h3>What Is SRE? A Quick Refresher</h3><p><em>Site reliability engineering started at Google in 2003. </em>Ben Treynor Sloss took over an operations team and decided to run it like a software engineering problem. The idea was simple. If your system is too complex for humans to run manually, you need engineers who automate operations themselves.</p><p>SRE teams own four things:</p><ul><li><strong>Reliability targets: </strong>They define what working means using SLIs, SLOs, and error budgets. An SLO of 99.9 percent availability gives you roughly 43 minutes of downtime per month. When you burn through that budget, new features stop shipping until reliability comes back.</li><li><strong>Incident response:</strong> When something breaks, SREs are on call. They triage. They investigate. They coordinate the fix. They run the postmortem. In a distributed system with dozens of services, the investigation step eats up most of the time and most of the error budget.</li><li><strong>Toil reduction: </strong>Google capped operational work at 50 percent of an SREs time. The other half goes to engineering. Building automation. Improving systems. Killing the manual work that would otherwise consume the role.</li><li><strong>Capacity planning: </strong>SREs forecast infrastructure needs before demand spikes cause failures. Flash sales. Viral moments. Product launches. These are predictable threats you plan for in advance.</li></ul><h3>What Is AI SRE?</h3><p><em>AI SRE is the practice of using AI agents to automate incident investigation.</em></p><p>When an alert fires, an AI SRE agent ingests signals across your stack. Logs. Metrics. Traces. Deployment history. Past incidents. It correlates everything. It surfaces a ranked hypothesis about the root cause. It does this in under five minutes.</p><p><em>Traditional SRE tools</em> are good at detection. They tell you something broke. They page the right person. What they do not tell you is why it broke. That investigation has always been manual. It is the hardest and most time-consuming part of incident response.</p><p>AI SRE closes that gap. Here is the key distinction. AI SRE is the mechanism. Autonomous Reliability is the outcome. Autonomous Reliability means that investigations run without waiting for a human to start them. The system is already correlating signals and forming a hypothesis before you pick up the page. You enter the loop at the decision point. Validating a hypothesis. Approving a fix. Handling the truly weird failure that no one has seen before.</p><p>According to Gartner, <strong>70 percent of enterprises</strong> will deploy agentic AI to operate their IT infrastructure by 2029. That is up from less than 5 percent in 2025. This is not experimental anymore. It is becoming the default.</p><h3>Traditional SRE vs AI SRE</h3><p>Let me break this down:</p><ul><li><strong>Detection: </strong>Traditional SRE uses threshold-based alerts. AI SRE uses anomaly detection across multiple signals.</li><li><strong>Investigation:</strong> Traditional SRE is manual. You correlate logs and metrics by hand. AI SRE does it automatically across your entire stack.</li><li><strong>Root cause analysis:</strong> Traditional SRE takes 30 to 60 minutes of engineer time. AI SRE gives you a hypothesis in under five minutes.</li><li><strong>Incident knowledge:</strong> Traditional SRE lives in people’s heads. When a senior engineer leaves, that knowledge leaves with them. AI SRE learns from every incident. The system gets smarter over time.</li><li><strong>On-call load:</strong> Traditional SRE is high. You investigate from scratch every time. AI SRE is lower. You validate and act instead of hunting for clues.</li><li><strong>MTTR: </strong>Traditional SRE means hours in complex systems. AI SRE means minutes.</li></ul><p>The critical difference is that traditional SRE tools are optimised for the alert, and AI SRE tools are optimised for the answer.</p><h3>What Is Autonomous Reliability?</h3><p>Autonomous Reliability is the outcome of AI SRE done well. It means incident investigation runs without waiting for human initiation. The system is already working on the problem before you get out of bed.</p><p>AI SRE is the mechanism. Autonomous Reliability is the result.</p><p>This distinction matters because it changes how you evaluate tools. The question is not “does this tool use AI?” The question is “does this tool move my team closer to Autonomous Reliability?”</p><p>Most teams today sit somewhere between fully manual and partially automated. They have good observability. Datadog, Prometheus, Grafana. They have good alerting. PagerDuty, OpsGenie. What they lack is the intelligence layer between the alert and the fix. That layer is what AI SRE provides.</p><p>Humans are not removed from the loop. Humans enter the loop at the right time. When a decision needs to be made. When something truly novel happens. The routine stuff gets handled automatically.</p><h3>How AI SRE Works Technically</h3><p>Every AI SRE system runs the same five steps when an alert fires.</p><p><strong>Step 1: Ingestion</strong><br>The system pulls live signals from across your stack. Kubernetes events. Application logs. Metrics. Traces. Deployments. Error rates. All into one context. Here is a number that matters. The average on-call engineer gets roughly 50 alerts per week. Only 2–5% require human intervention. AI SRE filters the noise before you see it.</p><p><strong>Step 2: Correlation</strong><br>The AI correlates signals that fired in the same time window. It looks at historically related services. It weighs everything against known failure patterns. This step takes a human 30 to 45 minutes. It takes AI under 60 seconds.</p><p><strong>Step 3: Causal Inference</strong><br>The AI does not just say “these three things happened at the same time.” It generates a causal hypothesis. Something like: “This deployment changed a config, which caused a downstream service to exhaust its connection pool, which is why the payment API is returning 503 errors.”</p><p><strong>Step 4: Resolution</strong><br>The system presents the hypothesis with evidence. It suggests next steps. In mature implementations, it executes a known fix automatically.</p><p><strong>Step 5: Learning</strong><br>After each incident, the system updates its model. It learns how services relate. It learns what patterns precede failures. It learns what remediation worked.</p><p>This is what separates AI SRE from a stateless correlation engine. Each investigation makes the next one faster.</p><h3>What Problems Does AI SRE Solve?</h3><ul><li><strong>Alert fatigue:</strong> A 2024 study found that 70 percent of SRE teams list this as a top concern. Many alerts fire for the same root cause. AI SRE deduplicates and prioritizes. The paged engineer sees something real and actionable.</li><li><strong>Investigation time: </strong>Finding the root cause takes 30 to 60 minutes on average. That time compounds across incidents. Cutting investigation time is the highest leverage action for reducing MTTR.</li><li><strong>Knowledge loss:</strong> When a senior engineer leaves, they take incident knowledge with them. AI SRE with persistent memory preserves that knowledge. A new engineer investigating a database issue gets access to all prior investigations into that service.</li><li><strong>On-call burnout: </strong>SRE on call is one of the highest burnout roles. The primary driver is not the number of incidents. It is the cognitive load of investigating unfamiliar failures under pressure. AI SRE reduces that load by handling the investigation and handing you a starting point.</li></ul><h3>AI SRE vs DevOps vs AIOps</h3><p>These terms get mixed up. Here is the difference.</p><ul><li><strong>DevOps</strong> is a culture and a set of collaboration practices. It governs how development and operations teams work together. It does not prescribe specific reliability metrics.</li><li><strong>SRE</strong> is a specific implementation of DevOps principles. It has concrete metrics, such as SLOs and error budgets. It defines how reliability is measured and maintained.</li><li><strong>AIOps</strong> is the broad application of AI to IT operations. Alert correlation. Capacity forecasting. Anomaly detection. It predates LLMs. It typically means statistical ML applied to operational data. AIOps optimizes for noise reduction. It narrows the alert queue. It does not investigate.</li><li><strong>AI SRE</strong> is narrower than AIOps. It applies AI specifically to the SRE workflow using LLM-based reasoning.</li></ul><p>Here is the clean boundary. AIOps hands you a shorter list of alerts. AI SRE tells you which one matters, why it happened, and what to do next. AIOps tells you something is wrong. AI SRE tells you why and gets you to a resolution faster.</p><blockquote><strong><em>AI SRE does not replace your engineers. It replaces the manual investigation that burns their time and your budget. The question is not whether to adopt it. The question is how quickly your team can start investigating, in minutes rather than hours.</em></strong></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=55fc5c19bdde" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>