<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dinesh Wijekoon on Medium]]></title>
        <description><![CDATA[Stories by Dinesh Wijekoon on Medium]]></description>
        <link>https://medium.com/@dinesh707?source=rss-49a775df35b9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*iTMCECrU6FW_pALh</url>
            <title>Stories by Dinesh Wijekoon on Medium</title>
            <link>https://medium.com/@dinesh707?source=rss-49a775df35b9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 13:20:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dinesh707/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[First-Class Observability Stack for AI Agents]]></title>
            <link>https://medium.com/@dinesh707/first-class-observability-stack-for-ai-agents-278d0f6eba74?source=rss-49a775df35b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/278d0f6eba74</guid>
            <category><![CDATA[ai-observability]]></category>
            <category><![CDATA[honeycomb]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[opentelemetry]]></category>
            <category><![CDATA[victoriametrics]]></category>
            <dc:creator><![CDATA[Dinesh Wijekoon]]></dc:creator>
            <pubDate>Fri, 06 Mar 2026 11:35:12 GMT</pubDate>
            <atom:updated>2026-03-06T11:35:12.090Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A Proposal for a Dual-Layer Observability Architecture: Transforming AI Agent Telemetry into Real-Time Functional Memory. Refined and polished by Google Gemini to ensure technical clarity and structural flow.</em></p><p>For decades, observability has been treated as a post-processor — a digital autopsy performed after the fact. We build a system, it emits telemetry, and that telemetry is shipped off to a distant, centralized silo — Datadog, Dash0, HonecombIO, or a massive Prometheus cluster — to be peered at by an SRE when something breaks. This “ship and forget” model worked for static microservices because code doesn’t need to know its own latency to function; it just executes.</p><p>But as we transition into the era of Autonomous Agentic Systems, this legacy approach is becoming a catastrophic bottleneck. Agents are not just code; they are dynamic reasoning engines. For an agent, a trace isn’t just a debugging artifact — it is a memory.</p><p>I propose a fundamental shift in the deployment model for monitoring: <strong>Observability for agents is not a side effect; but it is a first-class citizen of the agent’s memory, learning architecture and operations.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J39cXyZRYlKNrx645zrIlQ.png" /></figure><h3>1. The Problem: The Observability Wall</h3><p>Traditional monitoring is designed for human consumption. We aggregate metrics to see trends and store logs to debug crashes. However, AI agents do not operate like static code. They are dynamic, stateful, and — most importantly — they need to <em>learn</em> from their own execution.</p><p>When an agent’s telemetry is stored in a general observability system three regions away, we encounter three critical failures:</p><ol><li><strong>Latency as a Learning Disability:</strong> If an agent needs to “reflect” on its previous step to improve its next action, it shouldn’t have to wait 500ms for a query to return from a central data lake.</li><li><strong>State Fragmentation:</strong> The agent’s memory is often stored in one place (a vector DB), while its execution traces are stored in another. This makes it nearly impossible for the agent to “reason” about its own performance.</li><li><strong>The Cost of “Noise”:</strong> Agent systems generate an order of magnitude more telemetry than traditional apps (LLM prompts, multi-turn reasoning steps, tool-call results). Shipping 100% of this to a global system is economically and technically unsustainable.</li></ol><h3>2. The Proposal: The Dual-Layer Monitoring Deployment Model</h3><p>I propose a <strong>Hierarchical Observability Model</strong> where telemetry is treated with the same priority as the agent’s application state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/948/1*wASZETEmfFCZCNOtCMXitg.png" /></figure><h4>Layer 1: The “Hot” Local Stack (The Agent’s Memory — Immutable Telemetry Platform)</h4><p>Every agent deployment cluster should include a high-performance, local telemetry buffer (Metrics, Traces, and Events), which is noted in above digram as “Immutable Telemetry Platform”.</p><ul><li><strong>Proximity:</strong> This stack lives in the same VPC, same AZ or near visitinity of the agent deployment.</li><li><strong>Purpose:</strong> It serves as “Working Memory”. The agent uses these metrics, traces, logs and events to perform Self-Correction, Chain-of-Thought verification, and real-time optimization.</li><li><strong>The Gateway Advantage:</strong> Agent Gateways — which handle routing, rate-limiting, and model selection — can access this telemetry in microseconds to make real-time decisions (e.g., “This model is currently returning high-latency traces for this specific task; route elsewhere”).</li></ul><h4>Layer 2: The “Cold” Global Stack (The System’s Record)</h4><p>Only aggregated, summarized, and anonymized data is pushed from the local stack to the general corporate observability system.</p><ul><li><strong>Purpose:</strong> Long-term trend analysis, billing, and high-level SLO tracking.</li><li><strong>Efficiency:</strong> Instead of pushing every raw LLM token trace, the local stack pushes a single event: Task_Success_Rate: 98% over a 5-minute window for an example.</li></ul><h3>3. Observability as a Learning Mechanism</h3><p>The core of this proposal is that <strong>monitoring is just another part of learning.</strong></p><p>In a standard system, a “Trace” is a path through a system. In an agentic system, a “Trace” is a <strong>Lesson</strong>. If an agent fails a task, the trace of that failure is the most valuable training data we have. By storing these traces in a “Fast Access Near Location,” we enable:</p><ul><li><strong>On-the-fly RAG over Telemetry:</strong> The agent can query its own recent “Hot” traces to see if it has encountered a similar obstacle in the last ten minutes.</li><li><strong>Autonomous Optimization:</strong> The system can detect its own drift. If the “Event” stream shows a pattern of tool-call failures, the local stack triggers a “Reflection” process without ever needing a human in the loop.</li></ul><h3>4. The Architecture of the “Near-Location” Stack</h3><p>To implement this, we must rethink the sidecar. The “Full Stack” of telemetry should be stored in a specialized, fast-access store — think of a localized TimescaleDB, VictoriaMetrics or an ultra-low-latency ClickHouse instance co-located with the agent.</p><h4>The Metrics (The Pulse)</h4><p>Real-time token usage, latency per reasoning step, and tool-success ratios. This data can be used by the <strong>Agent Gateway</strong> to manage cost and performance load-balancing.</p><h4>The Traces (The Thought Process)</h4><p>Detailed spans of the LLM’s internal logic. By keeping these local, we allow the agent to perform “Self-Consistency” checks by reviewing its own execution path before finalizing an output.</p><h4>The Events (The Evolution)</h4><p>Not just “errors,” but state transitions. “Agent decided to use Tool X,” “Agent pivoted from Strategy A to B.” These events become the primary source for the agent’s long-term memory updates.</p><h3>5. Why “Crossing Regions” is the Enemy of Intelligence</h3><p>Intelligence is proportional to the speed of the feedback loop. Human neurons communicate in milliseconds. If an AI agent has to cross a geographic region to “remember” why a tool call failed two seconds ago, we have introduced a cognitive delay that cripples the system.</p><p>By pushing summarized data <em>down</em> to the general system rather than pulling raw data <em>up</em>, we protect the “speed of thought” within the agent system. The general observability system becomes the “History Book,” while the local stack remains the “Pre-frontal Cortex.”</p><h3>6. Conclusion: A New Standard for Agent Ops</h3><p>We are moving away from a world where we “monitor” software to a world where we “observe” intelligence.</p><p>If we treat agent monitoring as an isolated, second-class process, we will forever be limited by the latency of our networks and the cost of our storage. If we embrace the <strong>Localized Full-Stack Observability</strong> model, we give agents the ability to perceive their own actions in real-time.</p><p>Observability isn’t just for the SRE anymore. It’s for the Agent. It’s time we built the infrastructure to reflect that.</p><h3>7. A Call to Action: The Open Standards for Agentic Memory</h3><p>The infrastructure I’ve described shouldn’t be a proprietary secret or a bespoke “hack” for elite engineering teams. If we want a future where AI agents are safe, reliable, and multi-vendor, we need an open standard for the <strong>Immutable Telemetry Platform</strong>.</p><p>I am calling on industry leaders like <strong>Dash0</strong>, <strong>Honeycomb</strong>, and the <strong>OpenTelemetry (OTel)</strong> community to spearhead three specific initiatives:</p><h4>1. Standardize “Reasoning Spans” in OTel</h4><p>Currently, OTel is excellent at capturing HTTP GET or DB Query spans. But it lacks a native understanding of a <strong>Reasoning Step</strong>. We need semantic conventions that treat &quot;Chain-of-Thought&quot; as a first-class trace attribute. This would allow any OTel-compliant tool to visualize <em>why</em> an agent made a decision, not just <em>that</em> it made a call.</p><h4>2. The “Local Buffer” Reference Architecture</h4><p>We need an open-source, lightweight “Hot Layer” implementation — a <strong>“Telemetry Sidecar”</strong> — that can be dropped into any ecosystem.</p><ul><li><strong>Dash0:</strong> Your expertise in OTel-native architectures and ClickHouse performance makes you the ideal candidate to help define how high-volume agent logs can be stored locally and queried in sub-10ms.</li><li><strong>Honeycomb:</strong> Your experience in high-cardinality debugging is exactly what agents need to “self-reflect.” We need a way to perform “BubbleUp” analysis locally within the agent’s own execution loop.</li></ul><h4>3. “Summarized Up-shipping” Protocol</h4><p>We need a standard way for a local stack to “talk” to a global stack. Instead of flooding global backends with every raw token, we need an OTLP-based protocol for <strong>Summarized Telemetry</strong>. The local stack should push “Insights” (e.g., <em>Task Alpha succeeded with 92% confidence</em>) while keeping the “Evidence” (the raw 10MB trace) local.</p><h4>Conclusion: Don’t Build a Silo, Build a Standard</h4><p>To the founders and engineers at these companies: You have already won the battle for traditional observability. But the Agentic Era requires a different kind of bravery. It requires moving the “database” closer to the “intelligence.”</p><p>Let’s not build another generation of proprietary silos. Let’s make the <strong>Immutable Telemetry Platform</strong> an open-source reality within the OpenTelemetry ecosystem.</p><p><strong>Observability is the memory of the machine. Let’s make sure that memory is accessible, fast, and open.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=278d0f6eba74" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The USB-C Moment for AI: Understanding the Model Context Protocol (MCP)]]></title>
            <link>https://medium.com/@dinesh707/the-usb-c-moment-for-ai-understanding-the-model-context-protocol-mcp-b67c914039a9?source=rss-49a775df35b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/b67c914039a9</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[mcp-server]]></category>
            <category><![CDATA[mcp-protocol]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Dinesh Wijekoon]]></dc:creator>
            <pubDate>Wed, 18 Feb 2026 14:59:56 GMT</pubDate>
            <atom:updated>2026-02-18T14:59:56.977Z</atom:updated>
            <content:encoded><![CDATA[<p><em>An Analysis of the Strengths and Scaling Challenges of the Model Context Protocol. Polished by Google Gemini to ensure content quality and flow.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qO0je8hLQAG4FHn-xBCgHQ.png" /></figure><h4>1. The Fragmentation Problem in Agentic AI</h4><p>In the current landscape of agentic AI, developers face a persistent state of architectural chaos. Historically, connecting an AI model to a database or a file system required manual, hard-coded integrations. If a database schema changed or a URL shifted, the entire agentic loop failed.</p><p>We’ve been living in the “proprietary cable” era of AI. Every tool required a bespoke adapter. The <strong>Model Context Protocol (MCP)</strong> aims to end this by moving away from hard-coded integrations toward a unified, discovery-based standard.</p><h4>2. Defining MCP: The “USB-C” of AI</h4><p>The Model Context Protocol (MCP) is an open-source standard designed to connect AI applications — like ChatGPT, Claude, or Cursor — to external data and tools. Much like how USB-C provided a standardized port that eliminated a drawer full of proprietary chargers, MCP provides a universal interface for AI models to “plug into” any external system.</p><p><strong>What MCP enables:</strong></p><ul><li><strong>Personalized Assistance:</strong> Agents can access your local files and calendars without custom API glue.</li><li><strong>Enterprise Analysis:</strong> Chatbots can connect to various organizational databases, learning the schema on the fly.</li><li><strong>Universal Tooling:</strong> A single MCP server for “Google Search” or “Slack” can work across <em>any</em> AI host that supports the protocol.</li></ul><h4>3. The Technical Bedrock: JSON-RPC 2.0</h4><p>Under the hood, MCP isn’t magic; it’s a specific implementation of <strong>JSON-RPC 2.0</strong>.</p><p>This choice is critical. Unlike REST, which is centered around resources and verbs (GET, POST), JSON-RPC is centered around <strong>notifications and method calls</strong>. This allows for a two-way conversation. When an AI “Host” (like Claude Desktop) connects to an MCP “Server” (like a GitHub integration), the following happens:</p><ol><li><strong>Capability Negotiation:</strong> The client and server agree on what they can do.</li><li><strong>Tool Listing:</strong> The server returns a <strong>JSON Schema</strong> for every tool it has.</li><li><strong>Dynamic Learning:</strong> The LLM reads that schema and “learns” the exact arguments required.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xjoVUBjVI1Tbs0aG48gREw.jpeg" /><figcaption>Figure: MCP Architecture (Original Image from : <a href="https://diamantai.substack.com/p/model-context-protocol-mcp-explained">diamantai.substack.com</a>)</figcaption></figure><p>This “loose” nature is MCP’s superpower: the server <em>teaches</em> the model how to use it in real-time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NGBy4l9fRjsLeSaP44zN2w.png" /><figcaption>Figure: Descriptive details on tools/list and tools/create works on the protocol</figcaption></figure><h4>4. The Three Primitives: Resources, Tools, and Prompts</h4><p>MCP architecture is built on three core primitives that define how the AI interacts with the outside world.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oqSm2se1DBF9Cinqs23gKQ.png" /></figure><h4>5. The “Local-First” Heritage (and its Scaling Baggage)</h4><p>The protocol currently favors two primary transport modes, revealing its roots in local developer productivity:</p><ul><li><strong>STDIO Transport:</strong> The MCP server runs as a local child process. Communication occurs through standard input/output pipes. This is incredibly secure and simple for local apps (like Cursor or Claude Desktop), but it is a “scale nightmare” for cloud environments.</li><li><strong>SSE (HTTP) Transport:</strong> The server runs as a remote web service using Server-Sent Events. This is the production-ready path, but it introduces the complexity of maintaining stateful connections.</li></ul><h4>6. The Good, the Bad, and the Production Reality</h4><p>While MCP solves “integration chaos,” deploying it at scale requires a pragmatic look at the trade-offs.</p><p><strong>The Usability Wins</strong></p><ul><li><strong>Persistent Context:</strong> Unlike the isolated events of REST, MCP supports stateful context. An AI can “hold” a resource found in step one and utilize it in step five without the developer passing data back and forth manually.</li><li><strong>Real-time Reactivity:</strong> Through subscriptions, the server can “push” updates to the AI (e.g., “The file you were reading just changed”) rather than the AI polling for updates.</li></ul><p><strong>The Scaling Challenges</strong></p><ul><li><strong>“Context Slop”:</strong> Many developers are “auto-generating” MCP servers from massive OpenAPI specs. This is a mistake. Passing internal IDs, metadata, and timestamps to an LLM leads to <strong>Context Exhaustion</strong>. The model spends its “brainpower” reading junk data instead of solving the problem.</li><li><strong>Reliability vs. Ease of Use:</strong> The STDIO-first mindset of MCP means that many early servers aren’t built for high-concurrency cloud environments. They lack the robust error handling, retries, and load balancing expected in enterprise middleware.</li><li><strong>Choice Paralysis:</strong> If an MCP server exposes 50 similar tools (e.g., get_user, fetch_user_by_id, search_user), the LLM will often hallucinate or fail to pick the correct one.</li></ul><h4>7. Bridging the Gap: Managed Gateways (e.g., Amazon Bedrock AgentCore)</h4><p>To move MCP from a “cool local hack” to an “enterprise standard,” we need a translation layer. This is where platforms like <strong>Amazon Bedrock AgentCore</strong> come in.</p><p>AgentCore acts as a <strong>Managed Gateway</strong>. It sits between the AI and the MCP servers, providing:</p><ol><li><strong>Semantic Discovery:</strong> Instead of the AI seeing 1,000 tools, the gateway uses semantic search to show only the tools relevant to the current request.</li><li><strong>Auth &amp; Identity:</strong> It handles the permissions, ensuring the Agent only “sees” the data it is allowed to access.</li><li><strong>Policy Enforcement:</strong> It can intercept a tool call in real-time, checking it against “Guardrails” before the execution happens.</li></ol><h4>8. Why “Agent-Ready” Beats “Auto-Generated”</h4><p>The biggest pitfall in the MCP ecosystem today is the “Firehose Effect.” REST APIs are <strong>Atomic</strong> (small, technical). AI Agents need tools that are <strong>Intent-based</strong>.</p><ul><li><strong>Bad (Auto-generated):</strong> Exposing add_to_cart, set_shipping, and process_payment as three separate MCP tools.</li><li><strong>Good (Agent-ready):</strong> A single process_checkout tool that handles the logic.</li></ul><p>By wrapping complex API logic into single, intent-heavy MCP tools, we reduce “slop,” save on token costs, and vastly increase the reliability of the agent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LOGbWR8yUcz5_hD_wnF0gQ.png" /><figcaption>Figure: Ideal way of bulding the MCP server with proper intent based methods/functions instead of auto generated mass scaled api exports.</figcaption></figure><h4>9. Conclusion: The Conductor’s Future</h4><p>MCP represents a fundamental shift in authority. In the old world, the developer was the conductor, hard-coding every move. In the MCP world, the <strong>Agent</strong> is the conductor, and the <strong>MCP Server</strong> is the instrument.</p><p>While we are still working through the “slop” of auto-generation and the architectural hurdles of moving from STDIO to the Cloud, the direction is clear. The age of hard-coded AI integrations is over; the age of the universal AI port has begun.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b67c914039a9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Great Regression: Why AI Agents Look Like 1980s Computing]]></title>
            <link>https://medium.com/@dinesh707/the-great-regression-why-ai-agents-look-like-1980s-computing-d5e7ec1ac4c1?source=rss-49a775df35b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/d5e7ec1ac4c1</guid>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[infrastructure]]></category>
            <dc:creator><![CDATA[Dinesh Wijekoon]]></dc:creator>
            <pubDate>Tue, 20 Jan 2026 14:04:34 GMT</pubDate>
            <atom:updated>2026-01-20T14:04:34.864Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This article looks into the patterns I have observed in the Agent Development in recent months, with a conclution of needing Engineer to stup up to make things better.</em></p><p><em>Note that this article was polished by Google Gemini to enhance its content quality and overall readability.</em></p><p>In the 1980s, computing was a Wild West of specialized hardware, rigid “expert systems,” and flat-file storage. We spent the subsequent forty years engineering our way out of that chaos. We built the OSI model to decouple networking from applications; we invented relational and NoSQL databases to move away from fragile file systems; and we developed strongly-typed, compiled languages to ensure system integrity.</p><p>Yet, as we enter the era of “Agentic AI,” we are witnessing a shocking architectural backslide. We are using Space Age intelligence to build systems that rely on Stone Age infrastructure. Welcome to the Great Regression.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*soweQYL6Y-0qXsAYH8VSNQ.png" /></figure><h3>The Protocol Trap: Coupling Brains to Pipes</h3><p>In modern software engineering, the “Separation of Concerns” is sacred. Your business logic shouldn’t care if it’s communicating via REST, gRPC, or WebSockets. However, current Agent APIs are fundamentally broken in this regard.</p><p>Take the emerging <strong>Agent2Agent (A2A)</strong> [1] and <strong>Agent Communication Protocol (ACP)</strong> [2] standards. While they aim to provide a “shared language,” they are largely built on a legacy request-response model using HTTPS and JSON-RPC or RESTful endpoints. By mandating a point-to-point, client-server architecture, these protocols tightly couple the agent’s “thought process” to the transport layer.</p><p>This creates a massive scalability wall: the <strong>N-squared connectivity problem</strong>. In a network of 50 agents, direct point-to-point coupling requires over 1,200 managed connections. Because these standards are rooted in 2010-era REST patterns, they actively block developers from adopting superior, modern architectures like <strong>Pub-Sub (Publish-Subscribe)</strong> or event-driven meshes. Instead of an agent simply broadcasting an event that others can react to, we are forcing them to maintain a “phone book” of specific endpoints — a brittle, 80s-style approach to distributed systems.</p><h3>The Return of the Flat File “Database”</h3><p>Perhaps the most nostalgic — and frustrating — trend is the resurgence of the file system as the primary state-management tool.</p><p>Visit any “state-of-the-art” agent repository on GitHub, such as early iterations of <strong>AutoGPT</strong> [3] or various “baby” AGI projects, and you’ll likely find a folder structure that looks like a time capsule from 1984. Agents are “communicating” by writing .txt or .json files to a shared folder. Another agent &quot;polls&quot; that folder, reads the text, and writes a response.</p><p>We have abandoned 40 years of database theory — ACID compliance, concurrency control, and relational integrity — in favor of “human-readable” folders. Why? Because it’s easy for a human to “see” what’s happening. It’s transparent, but it doesn’t scale. We are building global infrastructure on top of the digital equivalent of sticky notes in a shoebox.</p><h3>The Internal Thinking Gap</h3><p>Modern observability focuses on the “plumbing.” Many of the new AI SDKs provide excellent traces for API latencies, token counts, and tool-call timestamps. We can see <em>when</em> an agent called a database, but we are almost entirely blind to <em>why</em>.</p><p>In the Great Regression, the agent’s internal monologue — its “Chain of Thought” — is often treated as transient junk data. We log the final JSON output, but we discard the reasoning steps that led there. This is the ultimate black box. In a traditional system, we have stack traces that explain the logic flow. In agentic systems, we have a “vibe” and a result. When an agent makes a disastrously wrong decision, developers find themselves spelunking through unstructured text logs rather than analyzing a structured logic tree. We have external observability, but zero internal transparency.</p><h3>The Infinite Loop Fragility</h3><p>In the early days of BASIC, a single GOTO statement could trap a computer in an infinite loop until a human manually pulled the plug. We are seeing a high-tech resurgence of this failure mode.</p><p>Without deterministic state machines to govern them, agents frequently enter “reasoning loops” [6]. An agent calls a tool, gets an error, reasons that it should try again, and repeats the cycle until the API credit limit is hit. We are entrusting autonomous loops to probabilistic models without the circuit breakers that have been standard in software engineering for decades. We are essentially running “while(true)” loops where the exit condition is “trust me, I’ve got this.”</p><h3>The Unsandboxed Wild West</h3><p>Security in the 80s was an afterthought, and we are repeating that mistake with “Code Interpreter” patterns. We are giving agents the ability to write and execute Python code on local machines without proper containerization or sandboxing [7].</p><p>By allowing an LLM to generate code that interacts directly with the host’s file system or network, we are opening a door that we spent decades trying to close. We are ignoring modern Zero Trust principles in favor of the convenience of “letting the agent fix it.”</p><h3>The Path Forward: Engineering the Agentic Era</h3><p>If we want AI agents to become the “Digital FTEs” we’ve been promised, we must stop building them like 80s hobbyist projects.</p><ul><li><strong>Capture the Monologue:</strong> We need standards for “Reasoning Traces” that are just as structured and queryable as our network logs.</li><li><strong>Hardened Circuit Breakers:</strong> Autonomy requires deterministic “kill switches” that monitor for semantic loops and runaway costs at the infrastructure level.</li><li><strong>Standardize Security:</strong> No agent should execute code outside of a hardened, ephemeral sandbox. Security must be a primitive, not a plugin.</li><li><strong>Bring Back the Engineers:</strong> It’s time for software architects to take the wheel. We need to apply “Software Engineering 101” — observability, redundancy, and modularity — to the agentic world.</li></ul><h3>References</h3><p>[1] <strong>Agent2Agent (A2A) Standard</strong>: A framework focused on direct communication between AI agents, often criticized for its rigid request-response nature. <a href="https://www.google.com/search?q=https://github.com/mnot/a2a">https://github.com/mnot/a2a</a></p><p>[2] <strong>Agent Communication Protocol (ACP)</strong>: An emerging standard for agentic interaction often built on JSON-over-HTTP. <a href="https://www.google.com/search?q=https://github.com/AI-Agent-Foundation/ACP">https://github.com/AI-Agent-Foundation/ACP</a></p><p>[3] <strong>AutoGPT</strong>: An open-source interface of GPT-4 that popularized autonomous agent loops, initially relying heavily on local file-system persistence. <a href="https://github.com/Significant-Gravitas/Auto-GPT">https://github.com/Significant-Gravitas/Auto-GPT</a></p><p>[4] <strong>“English as a Coding Language”</strong>: A concept popularized by Andrej Karpathy and Jensen Huang regarding the shift toward natural language interfaces. <a href="https://twitter.com/karpathy/status/1617979122625712128">https://twitter.com/karpathy/status/1617979122625712128</a></p><p>[5] <strong>Model Context Protocol (MCP)</strong>: An open standard that enables developers to build secure, two-way connections between their data sources and AI models. <a href="https://modelcontextprotocol.io/">https://modelcontextprotocol.io/</a></p><p>[6] <strong>Agent Reasoning Loops</strong>: Research into “stalling” and “looping” behaviors in autonomous LLM agents when faced with ambiguous tasks. <a href="https://arxiv.org/abs/2402.01030">https://arxiv.org/abs/2402.01030</a></p><p>[7] <strong>OWASP Top 10 for LLMs</strong>: Documentation highlighting the risks of “Insecure Output Handling” and “Excessive Agency” in AI systems. <a href="https://owasp.org/www-project-top-10-for-large-language-model-applications/">https://owasp.org/www-project-top-10-for-large-language-model-applications/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d5e7ec1ac4c1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Observability : Stepping to 2026]]></title>
            <link>https://medium.com/@dinesh707/observability-stepping-to-2026-bbffdf0edb92?source=rss-49a775df35b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/bbffdf0edb92</guid>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Dinesh Wijekoon]]></dc:creator>
            <pubDate>Mon, 12 Jan 2026 15:43:57 GMT</pubDate>
            <atom:updated>2026-01-12T15:43:57.080Z</atom:updated>
            <content:encoded><![CDATA[<h3>Observability : Stepping to 2026</h3><p><em>This article looks into the latest trends of Observability in large cloud providers, vendors and in networking domain. Note that this article was polished by Google Gemini to enhance its content quality and overall readability.</em></p><p>The observability domain in 2026 has transitioned from a supporting technical function into the primary engine of digital business resilience. This shift is characterized by a fundamental move away from traditional monitoring, which focuses on tracking known failure modes, toward a paradigm of comprehensive <strong>system intelligence that addresses the “unknown unknowns” of hyper-complex, distributed environments</strong>. As organizations grapple with the scale of microservices, serverless functions, and the <strong>non-deterministic logic of autonomous agents</strong>, the volume and velocity of telemetry data have surpassed the capacity for manual human oversight. These agentic systems introduce a new layer of complexity — <strong>probabilistic execution paths</strong> — where the “why” behind an AI’s action becomes as critical as the system’s uptime. Consequently, the industry is now defined by the convergence of proactive site reliability engineering (SRE) practices, the rise of OpenTelemetry-native architectures capable of tracing <strong>agent-to-agent orchestrations</strong>, and a new era of network visibility that serves as the critical KPI for AI readiness.</p><h3>Cloud Provider Evolution and the Productization of Internal SRE</h3><p>The major cloud providers — Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure — have spent the years leading up to 2026 transforming their internal operational methodologies into sophisticated, customer-facing product offerings. The goal is no longer just to provide a platform for hosting applications, but to offer a self-healing infrastructure that mirrors the internal SRE capabilities of the cloud giants themselves.</p><h4>AWS: From Metric Collection to Applied Agentic AI</h4><p>AWS has consolidated its 2026 strategy around “Applied Intelligence,” moving Amazon CloudWatch beyond its origins as a basic metric and log repository. CloudWatch is now positioned as the central nervous system for AWS environments, utilizing high-frequency telemetry and generative AI to provide context-rich alerting and automated remediation. A key driver in this evolution is the AWS DevOps Agent, a tool that investigates incidents with the logic of a senior SRE. By correlating data from CloudWatch logs, X-Ray traces, and CloudTrail events, the agent identifies operational improvements and performs real-time triage, significantly reducing the burden on human engineers during peak traffic or off-hours.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZdG9NmihOrXA8ZasSIDQeQ.png" /><figcaption>Figure: <em>Screenshot of CloudWatch Logs Summary on a Dashboard with Other Widgets. (source: </em><a href="https://aws.amazon.com/blogs/mt/using-generative-ai-to-gain-insights-into-cloudwatch-logs/"><em>AWS Blog</em></a><em>)</em></figcaption></figure><p>The underlying mechanism of this shift is the<strong> transition from “reactive monitoring” to “proactive observability.”</strong> AWS has integrated machine learning models that learn a system’s normal behavior and can predict potential failures before they manifest as user-facing downtime. For example, CloudWatch Logs Insights now uses Amazon Bedrock to generate human-readable summaries from complex query results, allowing engineers to understand root causes in seconds rather than minutes.</p><h4>Google Cloud: Systems Theory and the Legacy of Monarch</h4><p>In 2026, Google’s internal monitoring platform, Monarch, continues to support planet-scale operations while serving as the architectural blueprint for the public-facing Cloud Operations Suite. Monarch’s ability to handle complex, ad-hoc queries across Google’s entire infrastructure is now mirrored in GCP’s ability to offer unified observability across multi-cloud and on-premises environments.</p><p><strong>Google’s internal SRE culture has evolved toward the adoption of systems theory, specifically the STAMP (System-Theoretic Accident Model and Processes) framework.</strong> This approach assumes that in high-complexity environments, failures are not just results of individual component breakdowns but of unintended interactions between system components. By focusing on control loops and feedback mechanisms rather than simple linear chains of causation, Google has redefined reliability for the AI era.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MmUBPAz8lFmUmSVi4mNAJw.png" /><figcaption>Figure: Diagram showing process flow from Normal operations on left through Hazard state in center to Loss Operations on Right (<a href="https://www.usenix.org/publications/loginonline/evolution-sre-google">See More infromation in USENIX source site</a>)</figcaption></figure><p>This systems-theoretic approach is integrated into the GCP Cloud Operations Suite, which provides a unified pillar for metrics, logs, traces, and profiling. The platform’s ability to reduce operational toil by up to 70% is a direct result of these deep AI and observability integrations that automate the “DevOps loop” of alerting, autoscaling, and remediation.</p><h4>Microsoft Azure: Unified Assurance and the Cisco Synergy</h4><p>Microsoft Azure’s 2026 strategy is heavily focused on end-to-end digital resilience, largely facilitated by its partnership with Cisco and the integration of ThousandEyes and Splunk into the Azure ecosystem. Microsoft recognizes that in a world of remote work and cloud-native applications, an organization’s performance depends as much on the “unowned” public internet as it does on its private cloud resources.</p><p>Azure SRE teams now utilize a <strong>“Collect, Monitor, Act” model</strong> that unifies telemetry data across the vast Azure network and external dependencies. By feeding this data into Large Language Models (LLMs), Azure enables automation to handle the vast majority of daily alerts, achieving near-zero major incidents for organizations that fully embrace the unified platform.</p><h3>The Vendor Marketplace: Open Standards versus Proprietary Depth</h3><p>The 2026 market for dedicated observability vendors is defined by a clash of philosophies between the established “all-in-one” platforms and the newer, “open-native” challengers. Organizations are increasingly wary of vendor lock-in and the unpredictable costs associated with proprietary agents and data ingestion models.</p><h4>Datadog: The Integration Leader and its Cost Challenges</h4><p>Datadog maintains its dominant position in 2026 through an unparalleled ecosystem of over 700 integrations and a highly polished user experience that developers find intuitive. Its platform is designed to correlate data across every silo — from front-end user experience to back-end database metrics and log events.</p><p>However, the primary friction point for Datadog users in 2026 remains its pricing architecture. <strong>The “per-host” and “per-GB” ingestion model, combined with high-cardinality penalties for custom metrics, often leads to “bill shock” for rapidly scaling organizations</strong>. In response, Datadog has enhanced its “Watchdog” AI to help teams identify and prioritize high-value telemetry, but the platform’s underlying proprietary nature remains a concern for teams looking for long-term flexibility.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6EQtSnY63JShPqZX-6hvKA.jpeg" /><figcaption>Figure: Datadog Watchdog Page (source: <a href="https://docs.datadoghq.com/watchdog/">here</a>)</figcaption></figure><h4>Dynatrace: Causal AI and Enterprise Automation</h4><p>Dynatrace has carved out a niche as the preferred platform for large-scale enterprise environments where manual configuration is impossible. Its “OneAgent” technology automatically discovers all running processes and containers, injecting instrumentation without requiring code changes. This “zero-touch” approach creates a comprehensive topology map of the entire environment in minutes.</p><p>The cornerstone of Dynatrace’s 2026 offering is the Davis AI engine. Unlike stochastic models that might “hallucinate” root causes based on probability, Davis AI uses causal analysis and deterministic grounding to pinpoint the exact source of a failure. This makes it particularly effective for autonomous operations, as the AI’s recommendations are based on factual dependencies rather than statistical patterns</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*ilaFEJYYWd1sAkcZ29Y_7Q.png" /></figure><h4>Dash0: The OpenTelemetry-Native Disruptor</h4><p>Dash0 represents a significant threat to the incumbents in 2026 by offering a platform built entirely on open standards like OpenTelemetry, Prometheus, and Perses. For cloud-native teams, the primary value proposition of Dash0 is “zero lock-in”. If a team decides to leave Dash0, they keep their instrumentation, their dashboards, and their alerts because they are all based on open source protocols.</p><p>Dash0’s pricing is also a major differentiator, charging a transparent flat rate per million logs, spans, or metrics. By using PromQL as the universal query language for all telemetry types, Dash0 eliminates the need for teams to learn multiple proprietary languages, effectively streamlining the SRE workflow.</p><h3>Network Observability: The Performance Foundation for the AI Era</h3><p>In 2026, network observability has transitioned from a specialized infrastructure concern to a core requirement for AI success. The massive bandwidth and low-latency demands of GPU clusters have made “AI readiness” synonymous with “network visibility”.</p><h4>Arista Networks: High-Frequency Performance and Universal Observability</h4><p>Arista has positioned itself as the performance leader for AI-centric data centers. Its CloudVision Universal Network Observability (CV UNO) platform provides an integrated application-to-network graph that fuzes network telemetry with compute and workload data. This allows operators to determine instantly whether a performance bottleneck lies in the application code, the virtualization layer, or the physical network fabric.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/351/1*HnPnAD-rk3foBMzsyevGTA.png" /><figcaption>Figure: Arista’s NetDL — EOS Network Data Lake (source: <a href="https://www.arista.com/en/solutions/cloud-networking">here</a>)</figcaption></figure><p>Arista’s approach utilizes its Network Data Lake (NetDL) to consolidate diverse datasets, enabling its Autonomous Virtual Assist (AVA) AI to perform proactive diagnostics and provide prescriptive recommendations. By leveraging merchant silicon and a single Linux-based operating system (EOS) across all its devices, Arista offers an agility and simplicity that appeals to hyperscale cloud providers and teams managing massive AI training clusters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1017/1*YWkA2qFDqgEwDtW8BGEXhg.png" /></figure><h4>Cisco Systems: The Full-Stack Resilience Ecosystem</h4><p>Cisco’s 2026 strategy is built on the concept of the “Secure AI Factory,” an integrated environment that brings together networking, security, and observability into a single pane of glass. The integration of ThousandEyes for network assurance and Splunk for data processing has created a powerhouse ecosystem capable of delivering end-to-end visibility across both internal networks and the public internet.</p><p>For Cisco, network observability is no longer just about uptime; it is about “experience intelligence”. Through the Cisco AI Assistant, teams can use natural language to query network data, map third-party dependencies, and extract insights at machine speed. This approach prioritizes enterprise-grade integration and reliability, making Cisco the preferred choice for organizations that need a comprehensive, one-stop ecosystem covering LAN, WAN, wireless, and cloud security.</p><h3>Labor Trends: The Rise of the AI-Enabled Observability Specialist</h3><p>An analysis of LinkedIn job advertisements in 2026 reveals a transformative shift in the Site Reliability Engineering and DevOps labor markets. The demand for manual “firefighting” skills has plummeted, replaced by a need for “AI fluency” and a deep understanding of open standards.</p><h4>The Skills-First Hiring Paradigm</h4><p>Approximately 41% of technical job advertisements in 2026 now explicitly require AI-related skills or focus on AI-enabled workflows. Employers are moving away from hiring based on “decades of hands-on experience” and toward “learning agility” and “digital fluency”. In this new environment, AI acts as a skill equalizer, allowing novice engineers to perform at expert levels with the help of intelligent copilots and automated troubleshooting guidance.</p><h4>The Emergence of the “AI SRE”</h4><p>The traditional SRE role is evolving into the “AI SRE,” a specialist who focuses on the reliability and performance of the AI models themselves. These roles require a unique blend of distributed systems knowledge and machine learning understanding. Job descriptions for AI SREs in 2026 frequently mention:</p><ul><li>Monitoring model drift, inference latency, and token cost efficiency.</li><li>Integrating observability tools into CI/CD pipelines for pre-merge risk analysis.</li><li>Designing “Chain of Thought” AI agents that can query and reason over production telemetry data.</li></ul><p><strong>The human SRE in 2026 is no longer the person who fixes the bug at 2 a.m.; they are the person who designs the guardrails and sets the goals for the AI agent that fixes the bug</strong>. This shift necessitates a high degree of communication skill, as SREs must now translate complex AI-driven insights into actionable business recommendations for non-technical stakeholders.</p><h3>The Strategic Path to Autonomous Operations in 2026</h3><p>The research across cloud providers, vendors, and labor markets suggests that 2026 is the year where “Autonomous IT” becomes a practical reality rather than a distant vision. Organizations are following a clearly defined maturity model as they transition away from manual monitoring.</p><h4>Step 1: Stability and Unified Telemetry</h4><p>Before any autonomous action can be taken, the underlying systems must be stable and observable in real-time. In 2026, this means adopting OpenTelemetry as the universal language for traces, metrics, and logs. By unifying data collection, organizations eliminate the silos that traditionally prevented AI from gaining the full context required to diagnose issues.</p><h4>Step 2: Tool Consolidation and Insight Generation</h4><p>With a unified data stream, organizations are pursuing “platform consolidation” as their default strategy. Having fewer platforms leads to less overhead and more unified data, which in turn allows AI to generate truly actionable insights rather than just more “noise”. IT leaders in 2026 are increasingly willing to switch platforms faster — often within 1–2 years — if a vendor fails to deliver these intelligent, automated outcomes.</p><h4>Step 3: Supervised Autonomy and Human Oversight</h4><p>The final step before full autonomy is “supervised autonomous operations”. Here, agentic AI handle repeatable and time-sensitive tasks, but humans remain in control, defining boundaries and ensuring accountability. This phase is critical for building operational trust, as it allows engineers to validate AI-driven decisions before the system is allowed to act independently.</p><h3>Conclusion: The Resilience Benchmark of 2026</h3><p>By the end of 2026, the success of a digital organization is no longer measured by simple uptime or throughput. Instead, “Resilience” has become the new benchmark for operational excellence — defined as the ability of a system to absorb disruption, recover quickly, and maintain a consistent customer experience under stress.</p><p>This resilience is built on three pillars:</p><ol><li><strong>Observability-Driven Automation:</strong> Using AI to predict and remediate issues before they impact the user.</li><li><strong>Network-Centric Performance:</strong> Ensuring the underlying fabric is visible and optimized for the unique demands of AI workloads.</li><li><strong>Human-AI Collaboration:</strong> Redesigning SRE roles so that human judgment guides the high-speed execution of intelligent agents.</li></ol><p>The 2026 landscape is one where observability is no longer a support function, but the foundation for safe, scalable, and governable digital ecosystems. As AI continues to surge in complexity, the organizations that thrive will be those that have mastered the art of seeing, predicting, and explaining every signal within their environment.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bbffdf0edb92" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Two Tails of Telemetry: Infrastructure Sanity vs. SaaS Economics]]></title>
            <link>https://medium.com/@dinesh707/the-two-tails-of-telemetry-infrastructure-sanity-vs-saas-economics-14acda13d525?source=rss-49a775df35b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/14acda13d525</guid>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[telemetry]]></category>
            <category><![CDATA[monitoring]]></category>
            <dc:creator><![CDATA[Dinesh Wijekoon]]></dc:creator>
            <pubDate>Fri, 02 Jan 2026 11:37:42 GMT</pubDate>
            <atom:updated>2026-01-02T11:37:42.688Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>This article is for anyone looking to gain a glimpse into the world of Observability (Infra Observability &amp; SaaS Observability). If you are already familiar with the basics, feel free to skip to the final section. Note that this article was polished by Google Gemini to enhance its content quality and overall readability.</blockquote><p>The evolution of observability systems has reached a critical turning point. This shift is driven by the rapid growth of distributed systems and the rising costs of cloud-native infrastructure. As global computing moves toward microservices, edge computing, and AI, the platforms used to monitor these environments have split into two distinct architectural paths.</p><p>Cloud providers face a “Physical Scale” challenge. Their monitoring tools require a deep, direct view of the hardware and software, yet they must remain physically separate from the actual data networks. This isolation ensures that a failure in the monitoring system does not cause an outage in the internet services they provide. In contrast, Software-as-a-Service (SaaS) observability vendors face a “Data Sharding and Cost” problem. Their main challenge is efficiently storing and retrieving data from trillions of signals across many different customers without making the “observability tax” too expensive.</p><p>The difference between these two approaches reflects the different roles these companies play in the technology stack. Cloud providers like AWS, Google Cloud, and Azure prioritize reliability. They ensure that even if a primary service fails, the operational teams still have the data they need to fix it. SaaS providers like Datadog and Honeycomb prioritize data economics. They build specialized storage engines that can search through petabytes of events to find rare, complex failures. This article provides a detailed technical analysis of these two models, examining the storage systems, data pipelines, and economic strategies that define modern, planet-scale observability.</p><h3>The Economics of Visibility and the Observability Tax</h3><p>The transition from legacy monitoring — characterized by reactive, threshold-based alerting on pre-aggregated metrics — to modern observability has introduced significant economic complexity. Observability is the capability to continuously generate actionable insights based on external signals, but the volume of these signals has begun to outpace the growth of the underlying infrastructure they observe. This imbalance has given rise to the “observability tax,” representing the time, human resources, and financial expenditure required to maintain operational visibility.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qi5SLUdDZLzUshUTukYRdA.png" /><figcaption>Figure: Screenshot from the Talk “SREcon25 Europe/Middle East/Africa — Taming the Cost of Telemetry: How Riot Games Reined In...”</figcaption></figure><p>Current industry analysis indicates that while 10% to 15% of a cloud bill is considered a standard expenditure for observability tooling, many organizations are seeing these costs balloon to 40%. This cost explosion is primarily driven by the “too much data” problem, where multiple engineering teams emit high-resolution metrics and voluminous logs without clear retention or sampling strategies. If you wish to learn how Riots Games reduced there 7 digit observability bill, you can follow their SRECon talk <a href="https://www.youtube.com/watch?v=wdRV-K92gEI&amp;list=PLbRoZ5Rrl5lf4XYxPv8hELkk9fnHv90pG&amp;index=33">here.</a></p><h4>Drivers of Telemetry Expenditure</h4><p>The financial burden of observability is rooted in the technical assumptions of legacy storage engines. Traditional time-series databases (TSDBs) and search indexes were designed for an era of lower cardinality, where the number of unique identifiers (e.g., hostnames, container IDs, or user IDs) was manageable. In modern environments, the “label explosion” caused by high-cardinality fields leads to memory exhaustion and glacial query performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xIVBOGkYDKaqNCQiUiuCqw.png" /></figure><p>For example, enabling detailed monitoring on a per-topic, per-broker basis in an AWS Managed Streaming for Apache Kafka (MSK) cluster can add thousands of custom metrics, each costing approximately $0.30 per month in CloudWatch. For a moderately sized cluster of 10 brokers with 100 topics, this decision can add over $1,000 per month to the monitoring bill alone, illustrating how granular visibility can quickly become an economic liability if not managed via efficient storage paradigms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CNYEHSo1bMrgqI0-ijhpEw.png" /></figure><h4>The Customer’s Strategy of Sampling and Dimensionality Reduction</h4><p>To mitigate the observability tax, organizations are increasingly moving toward sophisticated sampling and dimensionality reduction strategies. Rather than storing 100% of “regular” events (e.g., HTTP 200 status codes), teams are adopting tail-based sampling, where the system retains a representative sample of successful requests while keeping 100% of errors and outliers. This approach, combined with the transition to OpenTelemetry (OTel) for standardized instrumentation, allows for the decoupling of signal generation from the backend cost model.</p><p>Using an observability pipeline like Vector (<a href="https://vector.dev/">https://vector.dev/</a>) allows teams to filter and transform telemetry data, optimizing costs before it is ingested by expensive backend platforms.</p><h3>Cloud Provider Paradigms: Solving for Physical Infrastructure Scale</h3><p>At the scale of major cloud providers, telemetry design is driven by physical hardware and network limitations. Their central goal is the preservation of independent fault domains, creating strict isolation boundaries that prevent a single regional incident from triggering a cross-infrastructure collapse.</p><h4>[Google] The Genesis of Planet-Scale Time-Series: Google Monarch</h4><p>Google’s transition from Borgmon to Monarch is perhaps the most documented example of a physical scale pivot. Borgmon, the predecessor system, encouraged a decentralized operational model where each team managed its own instances. While this provided local autonomy, Borgmon lacked the support for distribution-typed time-series and required users to manually shard their monitored entities. As Google’s infrastructure expanded, the operational toil of maintaining thousands of Borgmon instances became unsustainable.</p><p>Monarch was designed as the next-generation, globally distributed, in-memory time-series database. It operates as a multi-tenant service, monitoring the availability and correctness of billion-user-scale applications. As of 2019, Monarch’s internal deployment consumed approximately 750TB of memory to store nearly 950 billion time-series, ingesting 4.4 terabytes of data per second and serving 6 million queries per second.</p><h4>[Google] Regional Autonomy and the Federation Model</h4><p>Monarch’s primary organizing principle is the combination of regional autonomous zones with a global query federation plane. This allows the system to keep data near its point of origin, reducing network latency and cross-regional transmission costs. Each Monarch zone is autonomous, consisting of replicated clusters in a strongly connected network region.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NUmTNLk-Wz1E5QRIkjyzQw.png" /></figure><h4>[AWS] The Architecture of Ubiquity: AWS CloudWatch</h4><p>The evolution of AWS CloudWatch represents the industry’s most significant shift toward provider-native, out-of-band telemetry. In the early days of EC2, monitoring was often a secondary consideration, forcing users to rely on traditional, agent-based tools that were blind to the underlying physical health of the cloud. As AWS scaled toward millions of concurrent instances, the limitations of these “inside-out” methods became clear: they could not survive a kernel panic, nor could they see the “noisy neighbor” effects occurring at the hardware layer.</p><p>CloudWatch was engineered to solve this by anchoring its vantage point within the hypervisor itself. By leveraging the deep integration of the Nitro System and purpose-built hardware, CloudWatch bypassed the guest operating system to provide a “black box” view of infrastructure health. This move transitioned monitoring from a guest-level luxury to an immutable part of the fabric of the cloud, creating a resilient telemetry plane that remains operational even when the customer’s own environment fails.</p><h4>[AWS] The Kinesis Outage and the Need for Physical Isolation</h4><p>The vulnerability of monitoring infrastructure to its own dependencies was starkly revealed during the 2020 Kinesis Data Streams outage in US-EAST-1. CloudWatch relies on Kinesis for the processing of metric and log data; when Kinesis hit a thread-count scaling limit in its front-end fleet, CloudWatch lost its ability to persist telemetry, causing alarms to enter an “INSUFFICIENT_DATA” state for several hours.</p><p>In response, AWS accelerated the cellularization of the CloudWatch front-end and moved the service to a separate, partitioned infrastructure. A key technical remedial action was the decision to persist three hours of metric data in a local metrics data store, creating a temporal buffer that isolates the monitoring plane from the transient failures of the real-time data plane. (you can find detailed information here: <a href="https://aws.amazon.com/message/11201/">https://aws.amazon.com/message/11201/</a>)</p><h4>[AWS] Hypervisor Telemetry and the Outside-In Vantage Point</h4><p>CloudWatch’s primary advantage as a provider-native service is its ability to collect telemetry at the hypervisor level, bypassing the guest operating system entirely. While this capability stems from the deep integration between EC2 and specialized hardware (such as the Nitro System), it is exposed through CloudWatch to provide a ‘black box’ view of infrastructure health. This perspective remains resilient even during guest OS crashes or kernel panics, where traditional agent-based monitoring would fail.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-hHMaEsnebcSyEp27LhceQ.png" /></figure><p>However, the hypervisor vantage point has inherent limitations. It cannot provide insight into container-level metrics or pod-level disk utilization in Kubernetes environments. To bridge this gap, AWS has shifted toward a hybrid model using “Container Insights” and “Application Signals,” which employ sidecar strategies to collect granular telemetry without compromising the physical isolation of the hypervisor monitoring plane.</p><h4>[Microsoft] Corporate Governance and Unified Streams: Microsoft Azure Monitor</h4><p>Microsoft’s monitoring strategy is defined by the transition from internal “Geneva” platforms to the unified Azure Monitor service, catering to the specific needs of large-scale enterprise and government clients. The “Geneva” framework was originally a claims-based access and monitoring platform that simplified user access by describing identity attributes as claims. This focus on identity-based federation was critical for multi-tenant services like Office 365, where telemetry must be isolated not just by hardware, but by jurisdictional and role-based policy mandates.</p><h4>[Microsoft] The Unified Data Platform (ADX and KQL)</h4><p>In 2018, Microsoft unified its disparate monitoring tools — Application Insights and Log Analytics — into a common data platform. This platform is built on Azure Data Explorer (ADX), providing a powerful columnar analysis engine and the Kusto Query Language (KQL). The consolidation into Azure Monitor allows for the correlation of data across the three pillars of observability (metrics, logs, traces) alongside a fourth pillar: changes.</p><p>A second-order insight from the Azure architecture is the use of “Data Collection Rules” (DCRs). DCRs allow administrators to centrally define how data is filtered, transformed, and routed from the guest OS of virtual machines. This is particularly relevant for the “Physical Scale” problem, as it allows Microsoft to enforce governance at the ingestion point, preventing “data fragmentation” that could lead to increased monitoring costs or security breaches in government clouds.</p><h4>[Microsoft] Petabyte-Scale Data Lakes on Microsoft Fabric</h4><p>The most recent evolution in Microsoft’s telemetry infrastructure is the migration toward a “Lakehouse” architecture using Microsoft Fabric. This approach partitions the data lake into a “Medallion” architecture (Bronze, Silver, Gold tiers), using Go-based OpenTelemetry collectors to append Parquet files into a landing zone with massive concurrency.</p><p>By utilizing Spark Streaming for transformations, Microsoft achieves sub-30-second latency for complex analytical queries across petabytes of data, effectively merging operational telemetry with corporate data analytics.</p><h3>SaaS Provider Paradigms: Solving for Data Sharding and Cost</h3><p>SaaS observability providers operate without direct access to the hypervisor, forcing them to solve the “Data Sharding/Cost” problem through aggressive innovations in storage engine design and multi-tenant isolation. Their systems must handle the “curse of dimensionality” — the fact that adding high-cardinality values like user IDs causes attribute costs to explode in traditional TSDBs.</p><h4>[Datadog] Deconstructing the Event Store: Datadog Husky</h4><p>Datadog’s Husky is a third-generation event store designed to process trillions of events per day while overcoming the limitations of previous clustered storage systems. In earlier generations, a single misbehaving node or a bursty tenant could disrupt the experience of all other tenants in a cluster, leading to cascading failures.</p><p>Husky solves this by “blowing apart” the storage system into a stateless, distributed model. It offloads data durability to battle-tested object storage services like Amazon S3 and uses FoundationDB as a strongly consistent metadata store.</p><h4>[Datadog] FoundationDB and Metadata-Driven Visibility</h4><p>The choice of FoundationDB is central to Husky’s ability to scale multi-tenancy. Its serializable interactive transactions allow Husky to manage a single source of truth for fragment visibility across thousands of tenants. Because FoundationDB internals are abstracted from the query path, the system can preserve atomicity even when dealing with millions of small files (fragments) uploaded by stateless writers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eucfRCzvUvs1diQIv_-45Q.png" /></figure><p>The metadata-driven approach allows for “Dynamic Resource Allocation,” where even a small tenant can “burst” into a large number of Reader nodes for a heavy query if capacity allows, as there is no fixed relationship between a customer’s data volume and specific hardware.</p><h4>[Datadog] Exactly-Once Ingestion and Deduplication</h4><p>For a SaaS provider, duplicate events are not just a technical nuisance; they skew customer billing and cause false positive alerts. Husky achieves exactly-once ingestion semantics through deterministic routing and stateless deduplication. A “Shard Router” splits the incoming stream into shards for each tenant, ensuring that an event with a given ID and timestamp is always routed to the same shard worker.</p><p>Downstream writers save processed event IDs into FoundationDB tables using optimistic concurrency control. If a worker detects a conflict — indicating a potential duplicate or a shard reassignment — it refreshes its LRU cache and resets its Kafka offset, ensuring consistency between the event data on S3 and the committed IDs in metadata.</p><h4>[Honeycomb] The Cardinality Revolution: Honeycomb Retriever</h4><p>Honeycomb’s Retriever engine is a purpose-built distributed column store optimized for the high-cardinality requirements of modern distributed tracing. Inspired by Facebook’s Scuba, Retriever differs in its choice of disk storage over memory, enabling a more sustainable cost model for a startup while supporting complex, sub-second queries.</p><h4>[Honeycomb] Segmented Retrieval and the Core Analysis Loop</h4><p>Retriever manages data through “segments” — directories created every million events or every 24 hours. Because event timestamps are naturally ordered, the system uses segment metadata to prune irrelevant data ranges instantly, enabling near-real-time querying on newly ingested data. This architectural decision supports the “Core Analysis Loop,” where engineers iteratively verify hypotheses by slicing raw, unsampled data by any dimension at query time.</p><h3>Verdict: The Balancing Act of Modern Telemetry</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t_eHti9nIT1-1UWwaYRRMw.png" /><figcaption>Figure: At end of 2025, Vendor based observability has reached almost 10x of native observability.</figcaption></figure><p>The current divide in the market has created a “telemetry paradox.” On one hand, we have <strong>Native Infrastructure Monitoring</strong> (like CloudWatch), which is reliable and low-cost but often stops at the virtual boundary. On the other, we have <strong>Vendor Observability</strong> (like Datadog), which provides deep application insights but often carries a “bill shock” that can exceed the cost of the infrastructure itself.</p><p>To move forward, the industry must shift from a competition of features to a <strong>balance of responsibilities</strong>.</p><h4>The Responsibility of Native Infra Providers</h4><p><strong>Expose the “Invisible Layers”</strong> Cloud providers (AWS, Azure, GCP) sit at a privileged position in the stack. They should stop competing with vendors on high-level application dashboards and instead focus on what only they can see: <strong>the out-of-band “Black Box” layers.</strong></p><p><strong>Hypervisor Transparency:</strong> Native tools should provide deeper insights into “noisy neighbor” effects, hardware micro-bursts, and physical network congestion that guest OS agents cannot detect.</p><p><strong>Predictive Failures:</strong> Using their global vantage point, providers should offer better telemetry on “Internet Weather” and regional infrastructure degradations before they impact the guest OS.</p><p><strong>The Baseline of Truth:</strong> Native monitoring should remain the “liveness” layer — the immutable heartbeat that stays online even when the application’s telemetry pipeline collapses.</p><h4>The Responsibility of Observability Vendors</h4><p><strong>Radical Cost Transparency and Value Alignment</strong> Third-party vendors provide the “brain” of observability — the distributed tracing and high-cardinality analysis that solves complex bugs. However, their survival depends on fixing the <strong>cost-to-value ratio.</strong></p><p><strong>Advanced Cost Estimators:</strong> Vendors must move beyond “pay-per-ingest” models and provide real-time, granular cost forecasting. Customers shouldn’t need a PhD in FinOps to predict their next bill.</p><p><strong>Intelligent Data Tiering:</strong> Vendors should focus on helping customers <em>drop</em> 90% of their useless “noise” data at the edge, rather than charging to store it. Value should be derived from the <strong>insights generated</strong>, not the <strong>gigabytes stored</strong>.</p><p><strong>Open Standards (OpenTelemetry):</strong> By embracing open standards, vendors allow customers to switch backends without re-instrumenting code. This forces vendors to compete on the quality of their <strong>analytics and UI</strong>, rather than on data lock-in.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=14acda13d525" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Observability 1.0, 2.0 & .. [Part 2]]]></title>
            <link>https://medium.com/@dinesh707/observability-1-0-2-0-part-2-b488cd118b39?source=rss-49a775df35b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/b488cd118b39</guid>
            <category><![CDATA[unified-platform]]></category>
            <category><![CDATA[active-intelligence]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[observability]]></category>
            <dc:creator><![CDATA[Dinesh Wijekoon]]></dc:creator>
            <pubDate>Wed, 24 Dec 2025 19:49:26 GMT</pubDate>
            <atom:updated>2025-12-25T22:17:25.908Z</atom:updated>
            <content:encoded><![CDATA[<h3>Observability 1.0, 2.0 &amp; .. [Part 2]</h3><h3>The Convergence of Intelligent Observability and Engineering Resilience in Modern Distributed Systems</h3><p><em>This is Part 2 of our series on the fundamentals of observability. If you missed the introduction, you can catch up on </em><a href="https://medium.com/@dinesh707/observability-1-0-2-0-part-1-d7f53e0833aa"><strong><em>Part 1 here</em></strong></a><em>. For those already familiar with the basics, feel free to jump ahead to the final section: </em><strong><em>Observability in 2026</em></strong><em>. Please note that this article was polished by Google Gemini to enhance its content quality and overall readability.</em></p><h3>Architectural Challenges and Observability 2.0</h3><p>As systems transition toward decentralized architectures, the sheer volume of telemetry data generated has led to a re-evaluation of data management strategies. This has given rise to the concept of “Observability 2.0,” which moves away from pre-aggregated metrics and siloed logs toward a unified model of arbitrarily wide events[24].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m1kxc_-qOID0m3q_5ynYpg.png" /><figcaption>Figure: A visual breakdown of “Unknown-Unknowns,” illustrating the complex failure modes that observability is designed to uncover beyond traditional monitoring. [Image source: <a href="https://boristane.com/blog/observability-wide-events-101/">Boris Tane</a>]</figcaption></figure><h3>The Mechanics of Arbitrarily Wide Events</h3><p>Observability 2.0 platforms utilize structured payloads that can carry hundreds of fields of contextual metadata within a single event[24]. These wide events capture a snapshot of system activity — including user identifiers, feature flags, request parameters, and environment tags — without being constrained by a fixed schema[24]</p><p>This model leverages “schema-on-read” capabilities, allowing engineers to perform ad hoc queries to discover bottlenecks and regressions that were not anticipated at the time of data collection[24] By capturing high-cardinality attributes — such as unique deployment IDs or A/B test flags — Observability 2.0 addresses the “cardinality explosion” that often cripples traditional time-series databases[14].</p><h3>Intelligent Operations: AI-Driven Troubleshooting and Causal RCA</h3><p>The saturation of telemetry data has made human-led analysis a bottleneck in incident response. In 2025, organizations are increasingly turning to AI-driven observability to handle the complexity of “behavioral drift” — small, incremental changes that interact in unexpected ways to cause large-scale outages[30].</p><h3>Causal AI vs. Predictive AI</h3><p>The integration of AI into observability has evolved from simple anomaly detection toward causal reasoning. Predictive AI uses machine learning to identify correlations in data and forecast potential breaches based on historical patterns[31]. However, because correlation does not equal causation, predictive models are often limited in their ability to explain <em>why</em> an output occurred [31].</p><p>Causal AI utilizes the system’s topology and dependency mapping to determine the precise underlying mechanisms driving a failure.31 By determining the exact cause-and-effect relationships, Causal AI enables automated root cause analysis (RCA) and remediation with higher accuracy than traditional statistical methods[31].</p><h4>The Role of AI in Modern Observability</h4><p><strong>Predictive Analytics<br>* Mechanism:</strong> Time-series forecasting and pattern recognition.<br><strong>* Outcome:</strong> Pre-emptive capacity planning and significant alert reduction.</p><p><strong>Causal RCA (Root Cause Analysis)<br>* Mechanism:</strong> Dependency graph analysis and topology mapping.<br><strong>* Outcome:</strong> Accelerated incident resolution and high-accuracy diagnosis.</p><p><strong>Generative AI<br>* Mechanism:</strong> Large Language Models (LLMs) and Natural Language Processing.<br><strong>* Outcome:</strong> Automated post-incident reviews and intuitive, “natural language” querying.</p><p><strong>Agentic AI<br>* Mechanism:</strong> Autonomous agents and reasoning loops.<br><strong>* Outcome:</strong> Automated remediation and truly self-healing system operations.</p><h3>The Automated Investigative Loop</h3><p>Modern RCA platforms utilize agentic AI layers that perform cross-modal reasoning, connecting signals across metrics, logs, traces, code repositories, and feature flags[30]. This investigative loop functions by forming hypotheses, collecting evidence across the production landscape, and iterating until it converges on a concrete explanation[30].</p><h3>Cultural Integration and Observability-Driven Development (ODD)</h3><p>The transition to high-functioning observability is as much a cultural shift as a technical one. The practice of Observability-Driven Development (ODD) involves weaving telemetry into every step of the software development lifecycle, from initial design to production deployment[34].</p><h3>Shifting Left: Observability as a Design Principle</h3><p>In an ODD framework, observability is not a runtime patch applied after deployment but a design-time requirement[24]. Engineers use real-time telemetry to inform design decisions, eliminating guesswork in performance testing and ensuring that every code change is measurable [24].</p><p>Key best practices for ODD in 2025 include:<br>· <strong>Observability as Code</strong>: Version-controlling dashboards, alerts, and metrics to ensure consistency and repeatability across environments [24].<br>· <strong>Pre-Merge Instrumentation Testing</strong>: Validating telemetry output in local sandboxes before code is merged, ensuring that new features are fully observable from day one [24].<br>· <strong>Universal Access</strong>: Centralizing data access to allow both technical and non-technical stakeholders to analyze how changes affect business outcomes [35].</p><p>Organizations that embrace ODD report faster delivery cycles and a reduction in technical debt, as issues are identified early in the development process before they transform into costly outages [34].</p><h3>Case Studies in Scale: Netflix and DoorDash</h3><p>The real-world application of advanced observability is best illustrated through the engineering practices of industry leaders managing hyper-scale distributed systems.</p><h3>Netflix: Title Launch Observability and “Time Travel”</h3><p>Netflix manages thousands of content launches monthly, a process fraught with “silent failures” such as missing artwork or localization gaps [38]. To ensure a seamless user experience, Netflix developed a sophisticated Title Health framework that incorporates “Time Travel” capabilities [38]. By appending a future timestamp to observability requests, engineers can simulate the state of the platform as it will appear at the moment of launch. [39]. This allows the system to detect lifecycle issues and ensure titles are correctly personalized for viewers across all devices and regions before they go live [38].</p><h3>DoorDash: Microservices Caching and Failure Mitigation</h3><p>As DoorDash transitioned to a microservices architecture, they encountered complex failure patterns such as “retry storms” and “cascading failures” [40] To mitigate these risks, they standardized microservices caching using a multi-layer strategy that includes local and distributed (Redis) tiers [41].</p><p>To protect their systems from overloading, DoorDash implemented a global failure mitigation plan using the open-source project Aperture [40]. This approach utilizes “adaptive concurrency limits” that allow individual servers to shed load based on real-time latency measurements, ensuring that degraded services do not trigger a “death spiral” across the broader RPC call graph [40].</p><h3>Tool Consolidation and Strategic Efficiency</h3><p><strong>A major trend in 2025 is the consolidation of fragmented observability tools onto unified platforms[3]. The average number of tools per organization has declined from 6 to 4.4</strong>, reflecting a recognition that tool sprawl creates data silos and slows incident response.3 Unified platforms provide a single “pane of glass” for infrastructure, application performance, security, and digital experience monitoring, accelerating root cause analysis and improving engineering efficiency [3].</p><h3>The Horizon of 2026: Toward Autonomous Resilience</h3><p>The trajectory of observability points toward a future where systems are not only observable but also autonomously resilient. The maturation of OpenTelemetry, the widespread adoption of eBPF-based instrumentation, and the rise of Agentic AI are converging to create an era where manual troubleshooting becomes the exception rather than the rule [3].</p><p>As organizations move toward 2026, the <strong>focus will shift from “collecting telemetry” toward “active intelligence”</strong> [5]. Problems will be predicted before users experience them, logs and traces will self-explain via causal models, and the democratization of data through natural language interfaces will allow every stakeholder to participate in the maintenance of system reliability [5] In this landscape, observability ceases to be a cost center and becomes a foundational pillar of digital business growth and innovation [10].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-KHKswUnV7pdh7MNT6uWPQ.png" /></figure><p>The successful implementation of these strategies requires a multi-pronged approach: adopting open standards to avoid vendor lock-in, integrating security and cost monitoring into the observability stack, and fostering a culture of shared responsibility for system health [3]. For the modern enterprise, the ability to see, understand, and act upon the internal state of its digital infrastructure is the ultimate competitive advantage in a world defined by increasing complexity and constant change [1].</p><h3>Works Cited</h3><p>1. Observability vs Monitoring: What’s the Real Difference? — Cloudchipr, accessed December 24, 2025, <a href="https://cloudchipr.com/blog/observability-vs-monitoring">https://cloudchipr.com/blog/observability-vs-monitoring</a></p><p>2. Observability in 2025: How It Works, Challenges and Best Practices — Lumigo, accessed December 24, 2025, <a href="https://lumigo.io/what-is-observability-concepts-use-cases-and-technologies/">https://lumigo.io/what-is-observability-concepts-use-cases-and-technologies/</a></p><p>3. Top Trends in Observability: The 2025 Forecast is Here | New Relic, accessed December 24, 2025, <a href="https://newrelic.com/blog/observability/top-trends-in-observability-the-2025-forecast-is-here">https://newrelic.com/blog/observability/top-trends-in-observability-the-2025-forecast-is-here</a></p><p>4. 2025 report — Observability Forecast — New Relic, accessed December 24, 2025, <a href="https://newrelic.com/sites/default/files/2025-09/new-relic-2025-observability-forecast-report.pdf">https://newrelic.com/sites/default/files/2025-09/new-relic-2025-observability-forecast-report.pdf</a></p><p>5. New Relic NOW 2025: The New Era of AI-Driven Observability, accessed December 24, 2025, <a href="https://www.bionconsulting.com/blog/new-relic-now-2025-the-new-era-of-ai-driven-observability">https://www.bionconsulting.com/blog/new-relic-now-2025-the-new-era-of-ai-driven-observability</a></p><p>6. Observability vs Monitoring — Difference Between Data-Based Processes — AWS, accessed December 24, 2025, <a href="https://aws.amazon.com/compare/the-difference-between-monitoring-and-observability/">https://aws.amazon.com/compare/the-difference-between-monitoring-and-observability/</a></p><p>7. Three Pillars of Observability: Logs, Metrics and Traces — IBM, accessed December 24, 2025, <a href="https://www.ibm.com/think/insights/observability-pillars">https://www.ibm.com/think/insights/observability-pillars</a></p><p>8. Observability vs. Monitoring: What’s the Difference? — New Relic, accessed December 24, 2025, <a href="https://newrelic.com/blog/best-practices/observability-vs-monitoring">https://newrelic.com/blog/best-practices/observability-vs-monitoring</a></p><p>9. Observability vs. monitoring: What’s the difference? — Dynatrace, accessed December 24, 2025, <a href="https://www.dynatrace.com/news/blog/observability-vs-monitoring/">https://www.dynatrace.com/news/blog/observability-vs-monitoring/</a></p><p>10. What is Observability: Benefits &amp; Use Cases | New Relic, accessed December 24, 2025, <a href="https://newrelic.com/blog/observability/what-is-observability">https://newrelic.com/blog/observability/what-is-observability</a></p><p>11. Three Pillars of Observability Explained: Metrics, Logs, Traces — StrongDM, accessed December 24, 2025, <a href="https://www.strongdm.com/blog/three-pillars-of-observability">https://www.strongdm.com/blog/three-pillars-of-observability</a></p><p>12. Tracing and Metrics Design Patterns for Monitoring Cloud-native Applications — arXiv, accessed December 24, 2025, <a href="https://arxiv.org/html/2510.02991v1">https://arxiv.org/html/2510.02991v1</a></p><p>13. Observability Trends in 2025 — What’s Driving Change? | CNCF, accessed December 24, 2025, <a href="https://www.cncf.io/blog/2025/03/05/observability-trends-in-2025-whats-driving-change/">https://www.cncf.io/blog/2025/03/05/observability-trends-in-2025-whats-driving-change/</a></p><p>14. Distributed Systems Observability Explained (2025 — EdgeDelta, accessed December 24, 2025, <a href="https://edgedelta.com/company/knowledge-center/distributed-systems-observability">https://edgedelta.com/company/knowledge-center/distributed-systems-observability</a></p><p>15. Why continuous profiling is the fourth pillar of observability — Datadog, accessed December 24, 2025, <a href="https://www.datadoghq.com/blog/continuous-profiling-fourth-pillar/">https://www.datadoghq.com/blog/continuous-profiling-fourth-pillar/</a></p><p>16. Catching up with OpenTelemetry trends in 2025 — Dynatrace, accessed December 24, 2025, <a href="https://www.dynatrace.com/news/blog/opentelemetry-trends-2025/">https://www.dynatrace.com/news/blog/opentelemetry-trends-2025/</a></p><p>17. Observability trends in 2025 — InfluxDB, accessed December 24, 2025, <a href="https://www.influxdata.com/glossary/observability-trends-2025/">https://www.influxdata.com/glossary/observability-trends-2025/</a></p><p>18. Best Observability Tools: Top 10 Software Observability Tools of 2025 — vFunction, accessed December 24, 2025, <a href="https://vfunction.com/blog/software-observability-tools/">https://vfunction.com/blog/software-observability-tools/</a></p><p>19. Observability Predictions for 2025: Trends Shaping the Future of Observability — Dash0, accessed December 24, 2025, <a href="https://www.dash0.com/blog/observability-predictions-for-2025-trends-shaping-the-future-of-observability">https://www.dash0.com/blog/observability-predictions-for-2025-trends-shaping-the-future-of-observability</a></p><p>20. KubeCon North America 2025: OpenTelemetry Recap from Atlanta — Bindplane, accessed December 24, 2025, <a href="https://bindplane.com/blog/kubecon-north-america-2025-opentelemetry-recap-from-atlanta">https://bindplane.com/blog/kubecon-north-america-2025-opentelemetry-recap-from-atlanta</a></p><p>21. OpenTelemetry eBPF Instrumentation Marks the First Release …, accessed December 24, 2025, <a href="https://opentelemetry.io/blog/2025/obi-announcing-first-release/">https://opentelemetry.io/blog/2025/obi-announcing-first-release/</a></p><p>22. OpenTelemetry eBPF Instrumentation Released — APMdigest, accessed December 24, 2025, <a href="https://www.apmdigest.com/opentelemetry-ebpf-instrumentation-released">https://www.apmdigest.com/opentelemetry-ebpf-instrumentation-released</a></p><p>23. Unveiling OpenTelemetry eBPF Instrumentation, accessed December 24, 2025, <a href="https://horovits.medium.com/unveiling-opentelemetry-ebpf-instrumentation-377cb0432bf1">https://horovits.medium.com/unveiling-opentelemetry-ebpf-instrumentation-377cb0432bf1</a></p><p>24. Observability in 2025: Key Trends Driving Log Monitoring — Hydrolix, accessed December 24, 2025, <a href="https://hydrolix.io/blog/observability-in-2025/">https://hydrolix.io/blog/observability-in-2025/</a></p><p>25. How To Effectively Monitor Kubernetes In 2025 — Logz.io, accessed December 24, 2025, <a href="https://logz.io/blog/best-kubernetes-monitoring-tools/">https://logz.io/blog/best-kubernetes-monitoring-tools/</a></p><p>26. 7 Common Kubernetes Pitfalls (and How I Learned to Avoid Them), accessed December 24, 2025, <a href="https://kubernetes.io/blog/2025/10/20/seven-kubernetes-pitfalls-and-how-to-avoid/">https://kubernetes.io/blog/2025/10/20/seven-kubernetes-pitfalls-and-how-to-avoid/</a></p><p>27. Microservices Observability: Patterns, Pillars &amp; Tools — Groundcover, accessed December 24, 2025, <a href="https://www.groundcover.com/microservices-observability">https://www.groundcover.com/microservices-observability</a></p><p>28. Serverless Observability Guide: Tools, Challenges, Best Practices — EdgeDelta, accessed December 24, 2025, <a href="https://edgedelta.com/company/blog/serverless-observability-guide">https://edgedelta.com/company/blog/serverless-observability-guide</a></p><p>29. Solving the Pitfalls of Distributed Tracing in Real-World Microservices | Odigos Blog, accessed December 24, 2025, <a href="https://www.odigos.io/blog/solving-pitfalls-of-distributed-tracing-in-real-world-microservices">https://www.odigos.io/blog/solving-pitfalls-of-distributed-tracing-in-real-world-microservices</a></p><p>30. DoorDash-Deductive: A Case Study in Broken Observability — CIO.inc, accessed December 24, 2025, <a href="https://www.cio.inc/doordash-deductive-case-study-in-broken-observability-a-30360">https://www.cio.inc/doordash-deductive-case-study-in-broken-observability-a-30360</a></p><p>31. Causal AI use cases for modern observability that can transform any business — Dynatrace, accessed December 24, 2025, <a href="https://www.dynatrace.com/news/blog/causal-ai-use-cases-for-modern-observability/">https://www.dynatrace.com/news/blog/causal-ai-use-cases-for-modern-observability/</a></p><p>32. AI Root Cause Analysis: The Ultimate Guide to Transforming …, accessed December 24, 2025, <a href="https://getcalmo.com/blog/ai-root-cause-analysis-the-ultimate-guide-to-transforming-problem-solving-2025">https://getcalmo.com/blog/ai-root-cause-analysis-the-ultimate-guide-to-transforming-problem-solving-2025</a></p><p>33. Beyond Correlation: Finding Root-Causes using a network digital twin graph and agentic AI, accessed December 24, 2025, <a href="https://aws.amazon.com/blogs/database/beyond-correlation-finding-root-causes-using-a-network-digital-twin-graph-and-agentic-ai/">https://aws.amazon.com/blogs/database/beyond-correlation-finding-root-causes-using-a-network-digital-twin-graph-and-agentic-ai/</a></p><p>34. Don’t Fear Being ODD — Embrace Observability-Driven Development For Faster, Safer Innovation — Forrester, accessed December 24, 2025, <a href="https://www.forrester.com/blogs/dont-fear-being-odd-embrace-observability-driven-development-for-faster-safer-innovation/">https://www.forrester.com/blogs/dont-fear-being-odd-embrace-observability-driven-development-for-faster-safer-innovation/</a></p><p>35. 11 Key Observability Best Practices You Should Know in 2025 — Spacelift, accessed December 24, 2025, <a href="https://spacelift.io/blog/observability-best-practices">https://spacelift.io/blog/observability-best-practices</a></p><p>36. AI for Incident Management &amp; Observability (SREs Insights) — Zenduty, accessed December 24, 2025, <a href="https://zenduty.com/blog/ai-incident-management-observability-trends/">https://zenduty.com/blog/ai-incident-management-observability-trends/</a></p><p>37. State of Observability 2024 Reveals How Leaders Outpace Their Peers | Splunk, accessed December 24, 2025, <a href="https://www.splunk.com/en_us/blog/devops/state-of-observability-2024-reveals-how-leaders-outpace-their-peers.html">https://www.splunk.com/en_us/blog/devops/state-of-observability-2024-reveals-how-leaders-outpace-their-peers.html</a></p><p>38. Inside Netflix’s Title Launch Observability System: Validating Title Availability at Global Scale — InfoQ, accessed December 24, 2025, <a href="https://www.infoq.com/news/2025/07/netflix-title-observability/">https://www.infoq.com/news/2025/07/netflix-title-observability/</a></p><p>39. Title Launch Observability at Netflix Scale | by Netflix Technology …, accessed December 24, 2025, <a href="https://netflixtechblog.com/title-launch-observability-at-netflix-scale-8efe69ebd653">https://netflixtechblog.com/title-launch-observability-at-netflix-scale-8efe69ebd653</a></p><p>40. Failure Mitigation for Microservices: An Intro to Aperture — DoorDash, accessed December 24, 2025, <a href="https://careersatdoordash.com/blog/failure-mitigation-for-microservices-an-intro-to-aperture/">https://careersatdoordash.com/blog/failure-mitigation-for-microservices-an-intro-to-aperture/</a></p><p>41. How DoorDash Standardized and Improved Microservices Caching, accessed December 24, 2025, <a href="https://careersatdoordash.com/blog/how-doordash-standardized-and-improved-microservices-caching/">https://careersatdoordash.com/blog/how-doordash-standardized-and-improved-microservices-caching/</a></p><p>42. DevOps Trends Shaping Enterprise IT Strategy in 2026 — RealVNC®, accessed December 24, 2025, <a href="https://www.realvnc.com/en/blog/devops-trends/">https://www.realvnc.com/en/blog/devops-trends/</a></p><p>43. State of Cloud Native 2025: What the most influential industry reports reveal — Cuemby, accessed December 24, 2025, <a href="https://cuemby.com/en-us/article/state-of-cloud-native-2025-what-the-most-influential-industry-reports-reveal">https://cuemby.com/en-us/article/state-of-cloud-native-2025-what-the-most-influential-industry-reports-reveal</a></p><p>44. Platform Engineering Versus SRE: 5 Differences And Working Together | — Octopus Deploy, accessed December 24, 2025, <a href="https://octopus.com/devops/platform-engineering/platform-engineering-vs-sre/">https://octopus.com/devops/platform-engineering/platform-engineering-vs-sre/</a></p><p>45. Top 6 Observability Trends Look out for 2025 — Motadata, accessed December 24, 2025, <a href="https://www.motadata.com/blog/observability-trends/">https://www.motadata.com/blog/observability-trends/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b488cd118b39" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Observability 1.0, 2.0 & .. [Part 1]]]></title>
            <link>https://medium.com/@dinesh707/observability-1-0-2-0-part-1-d7f53e0833aa?source=rss-49a775df35b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/d7f53e0833aa</guid>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[ebpf]]></category>
            <category><![CDATA[otel]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Dinesh Wijekoon]]></dc:creator>
            <pubDate>Wed, 24 Dec 2025 19:23:36 GMT</pubDate>
            <atom:updated>2025-12-24T19:51:03.718Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Observability 1.0</strong>, 2.0 &amp; .. [Part 1]</h3><h3>The Convergence of Intelligent Observability and Engineering Resilience in Modern Distributed Systems</h3><p><em>This article is for anyone looking to gain a glimpse into the world of Observability. If you are already familiar with the basics, feel free to skip to the final section: </em><strong><em>Observability in 2026 (in </em></strong><a href="https://medium.com/@dinesh707/observability-1-0-2-0-part-2-b488cd118b39"><strong><em>Part 2 of this Article</em></strong></a><strong><em>)</em></strong><em>. Please note that this article was polished by Google Gemini to enhance its content quality and overall readability.</em></p><p>The evolution of system management from the era of static server monitoring to the current landscape of intelligent observability represents a fundamental shift in the discipline of systems engineering. In the technological environment of 2025, observability is no longer viewed as a peripheral activity or a subset of monitoring; rather, it has matured into a structural property of software systems and a core engineering mindset[1]. This transformation is driven by the increasing complexity of cloud-native architectures, where the sheer number of service interactions and the ephemeral nature of infrastructure components have rendered traditional monitoring techniques insufficient[2]. Organizations are currently navigating a transition toward “Intelligent Observability,” a state where artificial intelligence, machine learning, and advanced kernel-level instrumentation coalesce to provide predictive insights and automated remediation[4].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9bg7LThZ5M-6RtZkoXraJw.png" /><figcaption>Figure: Observability Maturity Model [Image source: <a href="https://www.linkedin.com/pulse/5-stages-observability-maturity-model-logiq-ai/">LOGIQ.AI via LinkedIn</a>]</figcaption></figure><h3>Theoretical Foundations and the Monitoring-Observability Contrast</h3><p>The conceptual origins of observability are rooted in control theory, a field of mathematical engineering that defines observability as a measure of how well internal states of a system can be inferred from knowledge of its external outputs[1]. In the context of modern information technology, this translates to the ability of an engineering team to understand what is happening within a distributed system based solely on the telemetry data it emits — metrics, logs, and traces[1].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hb1ikXYzrNeZosHugosBYg.png" /><figcaption>Figure: Just as successive generations of telescopes have deepened our view of the universe, advanced instrumentation allows us to peer into the inner workings of modern computing. <strong>Without these tools, the most critical details remain invisible — if we aren’t looking, we simply don’t know what is happening within our systems.</strong></figcaption></figure><h3>Defining the Boundaries of System Awareness</h3><p>The distinction between monitoring and observability is critical for establishing high-performance site reliability engineering (SRE) practices. Monitoring is fundamentally a reactive process, focused on tracking specific, predefined metrics to ensure they remain within acceptable thresholds[8]. It addresses the “known-knowns” — those failure modes that engineers have already anticipated and for which they have established alerts[1]. When a system breaches a threshold, such as a CPU utilization spike or a storage capacity alert, monitoring provides the notification[6].</p><p>Observability, conversely, is an investigative and proactive approach. It allows engineers to query the system to understand the “why” and “how” behind unexpected behaviors[6]. This is particularly vital in microservice architectures where failure modes are often “unknown-unknowns” — emergent behaviors resulting from complex inter-service dependencies that could not have been predicted during the design phase[1].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yi6eE-FOqc3dN6aWXqsHPQ.png" /><figcaption>Figure: Charity Majors, a pioneer in transitioning the industry from traditional monitoring toward observability — a shift that fundamentally reshaped the Application Performance Monitoring (APM) landscape. [Image source: <a href="https://chariotsolutions.com/screencast/philly-ete-2022-observability-and-the-glorious-future-charity-majors/">Chariot Solutions</a>]</figcaption></figure><h3>Monitoring vs. Observability: Key Differences</h3><blockquote>P.S. I’ve just realized how limited the Medium editor is! Below is my best effort to translate a summary table into a readable text format.</blockquote><p><strong>Operational Logic<br>* Monitoring:</strong> Reactive; focuses on “when” and “what.”<br><strong>* Observability:</strong> Proactive; focuses on “why” and “how.”</p><p><strong>Failure Context<br>* Monitoring:</strong> Known failure modes (known-knowns).<br><strong>* Observability:</strong> Unforeseen failure modes (unknown-unknowns).</p><p><strong>System View<br>* Monitoring:</strong> Focuses on individual component health.<br><strong>* Observability:</strong> Focuses on distributed system interactions.</p><p>The relationship between these two concepts is often described as a continuum. Monitoring provides the necessary alerts and situational awareness, while observability provides the diagnostic tools required to resolve complex incidents and prevent their recurrence[9]. As organizations scale, the cost of downtime — reaching a median of $2 million per hour for high-impact outages in 2025 — makes the transition toward deep observability an economic imperative rather than a technical luxury[3].</p><h3>The Evolution of Telemetry: From MELT to Intelligent Context</h3><p>The foundational data types of observability are encapsulated in the acronym MELT: Metrics, Events, Logs, and Traces[8]. However, the 2025 landscape has expanded this framework to “MELTx,” where the “x” represents everything else required for complete visibility, including security signals, cost data, configuration changes, and runbooks[4].</p><p>Metrics serve as the quantitative pulse of the system, offering numerical data points collected over time[7]. They are highly efficient for storage and provide rapid insights into trends, but they lack the granular context needed to explain specific anomalies[7]. Logs, by contrast, are immutable, timestamped records of discrete events[2]. They offer the detailed history of “who, what, and where,” but their sheer volume in distributed systems can lead to significant indexing and storage costs[11].</p><p>Distributed traces provide the relational tissue between metrics and logs, tracking the end-to-end journey of a single request as it traverses multiple services[2]. Each step in the process, or “span,” captures the latency and execution details of a specific operation, allowing engineers to identify bottlenecks in the request path[11]. In 2025, continuous profiling has emerged as the “fourth pillar” of observability[15]. Continuous profiling provides always-on visibility into code execution at the function level, enabling developers to identify CPU hotspots and memory leaks without the overhead associated with traditional, manual profilers[15].</p><h3>Technological Catalysts: OpenTelemetry and the Kernel Revolution</h3><p>The standardization of telemetry data collection has been largely achieved through the maturation of the OpenTelemetry (OTel) project. As the second-largest project in the Cloud Native Computing Foundation (CNCF) after Kubernetes, OTel provides a vendor-neutral framework for generating and exporting telemetry data[16].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cM1ycYEecK00GoVZp4M3Zg.jpeg" /><figcaption>Figure: OpenTelemetry [Image source: <a href="https://middleware.io/blog/opentelementry-metrics/">Middleware.io</a>]</figcaption></figure><h3>The OpenTelemetry Ecosystem in 2025</h3><p>By 2025, OpenTelemetry has reached critical mass, with major platforms switching their default data collection methods to OTel-native protocols[19]. Key advancements in the OTel ecosystem include the stabilization of semantic conventions for HTTP metrics and the introduction of declarative configuration models, which simplify the deployment of instrumentation across diverse programming languages[16].</p><p>The OpenTelemetry Collector, reaching its 1.0 milestone in 2025, has become the central nervous system of observability pipelines[16]. It manages the ingestion, processing, and routing of telemetry data to various backends, allowing organizations to avoid vendor lock-in and optimize data flow based on cost and performance requirements[16].</p><h3>eBPF and the “Zero-Code” Instrumentation Shift</h3><p>The most transformative technological advancement in 2025 is the integration of extended Berkeley Packet Filter (eBPF) technology into observability frameworks. The release of OpenTelemetry eBPF Instrumentation (OBI) has fundamentally changed how organizations approach system visibility[21]. eBPF allows for the injection of monitoring probes directly into the Linux kernel, enabling the collection of telemetry data without modifying application code, restarting services, or adding invasive SDK dependencies[21].</p><h4>eBPF Capabilities &amp; Benefits</h4><p><strong>Automatic Capture<br>* Mechanism:</strong> Instruments at the protocol level (HTTP, gRPC, SQL).<br><strong>* Benefit:</strong> Zero manual instrumentation effort.</p><p><strong>Out-of-Process Running<br>* Mechanism:</strong> Operates within the kernel space.<br><strong>* Benefit:</strong> No impact on application performance.</p><p><strong>Universal Language Support<br>* Mechanism:</strong> Language-agnostic kernel hooks.<br><strong>* Benefit:</strong> Consistent telemetry across polyglot systems.</p><p><strong>Runtime Safety<br>* Mechanism:</strong> Kernel-verified bytecode execution.<br><strong>* Benefit:</strong> Guaranteed system stability.</p><p><strong>Security Integration<br>* Mechanism:</strong> No new app-level dependencies.<br><strong>* Benefit:</strong> Reduced attack surface and vulnerability risk.</p><p>OBI enables the automatic generation of “golden signals” — Request, Error, and Duration (RED) metrics — and the mapping of service dependencies[21]. While tracing through eBPF remains a complex technical challenge due to the difficulty of modifying payloads for context propagation in encrypted traffic, the technology has reached a level of maturity where it can provide immediate, high-fidelity visibility for legacy systems and complex microservices that were previously unobservable[21].</p><p>This concludes Part 1 of our guide to Observability. To continue exploring how the landscape is shifting toward 2026 and beyond, please head over to the second half of this deep dive.</p><p><strong>Continue to Part 2: [</strong><a href="https://medium.com/@dinesh707/observability-1-0-2-0-part-2-b488cd118b39"><strong>Link</strong></a><strong>]</strong></p><h3>Works Cited</h3><p>1. Observability vs Monitoring: What’s the Real Difference? — Cloudchipr, accessed December 24, 2025, <a href="https://cloudchipr.com/blog/observability-vs-monitoring">https://cloudchipr.com/blog/observability-vs-monitoring</a></p><p>2. Observability in 2025: How It Works, Challenges and Best Practices — Lumigo, accessed December 24, 2025, <a href="https://lumigo.io/what-is-observability-concepts-use-cases-and-technologies/">https://lumigo.io/what-is-observability-concepts-use-cases-and-technologies/</a></p><p>3. Top Trends in Observability: The 2025 Forecast is Here | New Relic, accessed December 24, 2025, <a href="https://newrelic.com/blog/observability/top-trends-in-observability-the-2025-forecast-is-here">https://newrelic.com/blog/observability/top-trends-in-observability-the-2025-forecast-is-here</a></p><p>4. 2025 report — Observability Forecast — New Relic, accessed December 24, 2025, <a href="https://newrelic.com/sites/default/files/2025-09/new-relic-2025-observability-forecast-report.pdf">https://newrelic.com/sites/default/files/2025-09/new-relic-2025-observability-forecast-report.pdf</a></p><p>5. New Relic NOW 2025: The New Era of AI-Driven Observability, accessed December 24, 2025, <a href="https://www.bionconsulting.com/blog/new-relic-now-2025-the-new-era-of-ai-driven-observability">https://www.bionconsulting.com/blog/new-relic-now-2025-the-new-era-of-ai-driven-observability</a></p><p>6. Observability vs Monitoring — Difference Between Data-Based Processes — AWS, accessed December 24, 2025, <a href="https://aws.amazon.com/compare/the-difference-between-monitoring-and-observability/">https://aws.amazon.com/compare/the-difference-between-monitoring-and-observability/</a></p><p>7. Three Pillars of Observability: Logs, Metrics and Traces — IBM, accessed December 24, 2025, <a href="https://www.ibm.com/think/insights/observability-pillars">https://www.ibm.com/think/insights/observability-pillars</a></p><p>8. Observability vs. Monitoring: What’s the Difference? — New Relic, accessed December 24, 2025, <a href="https://newrelic.com/blog/best-practices/observability-vs-monitoring">https://newrelic.com/blog/best-practices/observability-vs-monitoring</a></p><p>9. Observability vs. monitoring: What’s the difference? — Dynatrace, accessed December 24, 2025, <a href="https://www.dynatrace.com/news/blog/observability-vs-monitoring/">https://www.dynatrace.com/news/blog/observability-vs-monitoring/</a></p><p>10. What is Observability: Benefits &amp; Use Cases | New Relic, accessed December 24, 2025, <a href="https://newrelic.com/blog/observability/what-is-observability">https://newrelic.com/blog/observability/what-is-observability</a></p><p>11. Three Pillars of Observability Explained: Metrics, Logs, Traces — StrongDM, accessed December 24, 2025, <a href="https://www.strongdm.com/blog/three-pillars-of-observability">https://www.strongdm.com/blog/three-pillars-of-observability</a></p><p>12. Tracing and Metrics Design Patterns for Monitoring Cloud-native Applications — arXiv, accessed December 24, 2025, <a href="https://arxiv.org/html/2510.02991v1">https://arxiv.org/html/2510.02991v1</a></p><p>13. Observability Trends in 2025 — What’s Driving Change? | CNCF, accessed December 24, 2025, <a href="https://www.cncf.io/blog/2025/03/05/observability-trends-in-2025-whats-driving-change/">https://www.cncf.io/blog/2025/03/05/observability-trends-in-2025-whats-driving-change/</a></p><p>14. Distributed Systems Observability Explained (2025 — EdgeDelta, accessed December 24, 2025, <a href="https://edgedelta.com/company/knowledge-center/distributed-systems-observability">https://edgedelta.com/company/knowledge-center/distributed-systems-observability</a></p><p>15. Why continuous profiling is the fourth pillar of observability — Datadog, accessed December 24, 2025, <a href="https://www.datadoghq.com/blog/continuous-profiling-fourth-pillar/">https://www.datadoghq.com/blog/continuous-profiling-fourth-pillar/</a></p><p>16. Catching up with OpenTelemetry trends in 2025 — Dynatrace, accessed December 24, 2025, <a href="https://www.dynatrace.com/news/blog/opentelemetry-trends-2025/">https://www.dynatrace.com/news/blog/opentelemetry-trends-2025/</a></p><p>17. Observability trends in 2025 — InfluxDB, accessed December 24, 2025, <a href="https://www.influxdata.com/glossary/observability-trends-2025/">https://www.influxdata.com/glossary/observability-trends-2025/</a></p><p>18. Best Observability Tools: Top 10 Software Observability Tools of 2025 — vFunction, accessed December 24, 2025, <a href="https://vfunction.com/blog/software-observability-tools/">https://vfunction.com/blog/software-observability-tools/</a></p><p>19. Observability Predictions for 2025: Trends Shaping the Future of Observability — Dash0, accessed December 24, 2025, <a href="https://www.dash0.com/blog/observability-predictions-for-2025-trends-shaping-the-future-of-observability">https://www.dash0.com/blog/observability-predictions-for-2025-trends-shaping-the-future-of-observability</a></p><p>20. KubeCon North America 2025: OpenTelemetry Recap from Atlanta — Bindplane, accessed December 24, 2025, <a href="https://bindplane.com/blog/kubecon-north-america-2025-opentelemetry-recap-from-atlanta">https://bindplane.com/blog/kubecon-north-america-2025-opentelemetry-recap-from-atlanta</a></p><p>21. OpenTelemetry eBPF Instrumentation Marks the First Release …, accessed December 24, 2025, <a href="https://opentelemetry.io/blog/2025/obi-announcing-first-release/">https://opentelemetry.io/blog/2025/obi-announcing-first-release/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d7f53e0833aa" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>