SRE AI Agents in Trendyol

SRE AI Agents structure

What is the SRE AI Agents Project?

SRE AI Agents is a group of AI agents designed to automatically initiate an investigation during an incident, try to find its root cause, and assist the SRE team with day-to-day operations. The system includes ~10 AI agents that work collaboratively to achieve this goal.

Its Purpose

This system is designed for three main purposes: real-time incident root cause analysis, QA support for SRE-related topics, and on-call alert analysis across all domain teams.

Rather than waiting for manual intervention or being triggered by an SRE, the AI Investigation process is designed to automate troubleshooting and provide a faster, more efficient way to identify the root cause. It reduces operational overhead and toil for SREs and on-call engineers by performing automated static checks and investigating issues using codified knowledge and specialized skills.

While there are many tools that work exceptionally well with AI agents when queried, they almost always require constant human direction and feedback to find the right path toward the root cause or to guide SREs in analyzing problematic parts.

In complex, distributed systems, asking a standard AI agent to simply “find the exact root cause” usually backfires. The agent will typically return a flood of false-positive warnings, errors, and affected components (such as downstream applications or services) rather than the actual root cause itself. This process aims to separate background noise from real indicators, determining which parts of the system are truly affected, which are merely symptoms, and what the actual root cause is.

Most simple, single-agent setups — even when powered by advanced models like Claude 4+ Opus, GPT-5, or Gemini Pro — struggle to make this distinction. For instance, during an infrastructure incident where an application suddenly starts getting timeouts from a database, a basic agent will easily identify the root cause as “database connection failures.” However, it often fails to see that the database itself is healthy and the actual culprit is upstream connection pool starvation or a transient network partition. Our multi-agent, collaborative approach is built specifically to solve this exact challenge.

Each agent has a specific role and is responsible for a particular task. These roles include:

Gather Agent

• Responsible for gathering context (such as incident and issue details) and feeding sub-agents (specialists) with AI-generated system prompts tailored to the incident’s specific context. It tries to clarify the ongoing issue and the user’s request.

Application Metrics

• metrics-specialist. Gathers application-specific metrics like the Four Golden Signals, resource usage, etc., to find the most likely root cause of the incident. It uses Grafana and PromQL to retrieve the necessary data via SRE-MCP (our in-house MCP application). It is specialized in application performance metrics on Grafana.

Traces

• trace-specialist. Gathers trace data (OpenTelemetry Jaeger spans) to pinpoint the most likely root cause of the incident. It uses Jaeger to fetch the needed data via SRE-MCP. It specializes in Jaeger data analysis, finding the error traces that might be related to the incident and following trace relationships.

Troubleshoot

• troubleshoot-specialist. Utilizes a troubleshooting report that runs automated pre-checks (static checks) when an incident is declared on the incident platform. It executes more than 15 checks in parallel to collect evidence across various sources such as logs, Kubernetes, metrics (database health, application health, etc.), traces, P0 infrastructure health, and planned/unplanned changes (like deployments or maintenance). Based on this report, the troubleshoot-specialist tries to identify the root cause and continues the investigation using its MCP tools to verify findings when necessary.

Kubernetes

• kubernetes-specialist. Gathers Kubernetes cluster, node, application, pod, and container data to identify the root cause of the incident. It has read-only access to Kubernetes resources. It checks for anomalies such as OOMKills, pod restarts, or unhealthy clusters/nodes, and continues its investigation using MCP tools to confirm the root cause when necessary.

CUJ (Critical User Journey) / SLO

• cuj-specialist. Gathers CUJ/SLO data to identify root services that are in trouble and need attention, helping investigate the root cause of the incident. A CUJ represents a critical business flow — like “MakePayment” — that spans multiple underlying services. When any service in the CUJ flow is unhealthy, this specialist tries to find the root service that triggered the incident.

Correlate Agent

• Gathers findings from multiple sources (the specialists) and correlates them with cross-checks to isolate the root cause. It has no direct access to MCP tools; its sole role is to correlate specialists’ findings and generate an RCA hypothesis with a calculated confidence level. If a hypothesis is cross-validated (meaning the root cause is correlated across at least two independent specialists’ findings), it is proposed as the primary root cause. Its core goal is to prevent false positives by cross-referencing findings.

Evaluate Agent

• Evaluates and validates the RCA hypothesis generated by the Correlate Agent. If the hypothesis is not cross-validated, meaning confidence is low, the Deep Dive Agent is deployed to investigate further. If the hypothesis is cross-validated (confidence is high), the Synthesize Agent compiles it directly into the root cause of the incident. The Evaluate Agent decides whether to accept the hypothesis based on confidence levels or to trigger a deeper search.

Deep Dive Agent

• If the Evaluate Agent determines that the root cause hypothesis is not sufficiently validated, the Deep Dive Agent performs an active, detailed investigation using SRE-MCP tools. It uses the Correlate Agent’s initial hypothesis as a starting point to explore and verify potential causes.

Synthesize Agent

• Synthesizes the final incident report that pinpoints the root cause and directs the incident response team to the affected areas. The report includes root cause hypotheses, an evidence chain, a structured incident timeline based on findings, impact analysis, contributing factors, and immediate/short-term recommendations to mitigate the incident. We call this entire process AI Investigation. It is an automated workflow that uses collaborative AI agents to investigate and identify the most likely root cause of an incident, or at least pinpoint the exact problematic component or application. It is not a simple RAG (Retrieval-Augmented Generation) or a basic LLM conversation; no human intervention is required during the investigation phase. Once the final report is ready, SREs can continue to chat with a specialized agent based on the report. The chat agent retains the full context and has access to MCP tools to continue the investigation interactively based on human feedback.

What We Built

We built an agentic investigator system on top of SRE knowledge and skills. It starts when an incident is declared on the incident management platform, and the sub-agents work in parallel to investigate and identify the root cause, thereby reducing MTTR (Mean Time to Resolve). We used LangGraph to achieve this AI Investigation process in Python.

Each investigation generally completes in under 8 minutes. After seeing positive results, we also implemented this in our in-house QA (Question & Answer support) platform. Now, when a user asks a question, the AI investigator responds as an SRE, providing detailed answers.

The process we built is also compatible with third-party agents. For example, our container team created a well-skilled agent with dynamic knowledge to help investigate container-related issues. It is already specialized in that domain. Our process integrates seamlessly with that agent, and now it works as if that agent were a native specialist within our SRE investigation process. This capability enables us to use AI agents effectively across the company. If a platform team creates a well-skilled agent (which they can also do within our project structure), we can easily integrate it with our SRE investigation process and leverage it to help investigate issues in that specific domain.

Additionally, as this process also is used in our QA platform, it is also fully compatible with our in-house on-call system. Therefore, when high-priority (P1/P2) alerts are fired or an anomaly is detected, the on-call system automatically triggers the AI investigator to troubleshoot the issue.

All these capabilities are made possible by AI Investigation definitions/types. We created predefined investigation types for common use cases based on our needs, such as incident investigations, QA support, anomaly analysis, and cluster resource alerts. Each type has a specific system prompt and dedicated sub-agents (specialists). For example, an incident investigation uses all specialists to identify the root cause, while the QA supporter focuses on providing detailed answers to user questions and uses only specific specialists like metrics, kubernetes, and traces based on its needs. The CUJ specialist is generally not needed for QA support. This way, we reduce hallucinations, minimize the context window, and improve the overall quality of the AI investigation results.

The automated run is not the end of the story. After the final report is ready, the user can keep chatting on the same investigation: ask follow-ups about the findings, challenge a hypothesis, or assign new tasks to the agent (for example, re-check a specialist, dig into a specific service, or run a custom analysis). That chat path still has MCP tool access, and it starts from the investigation state — report, specialist findings, and correlation — so the agent does not rediscover the incident from scratch.

Users can also rate each agent output from 1 to 5 and leave a free-text comment. Those reviews feed a feedback loop we use to spot weak prompts, bad tool choices, and recurring false positives — and they are how we track quality over time (our current average is 3.32 / 5).

How It Works (Technical Details)

It’s written in Python, using LangGraph with Postgres for state/checkpoints. LangGraph’s node system was fit for our needs. When we once initiate this project we started with Agno, and it worked very well as its purpose.

Over time, we struggled with technical issues and insufficiency of its feature. We had to workaround for most features. I won’t go into depth on that here, we’ll cover that later.

Investigation pipeline (first run → final report)

When a new investigation starts, LangGraph routes through the nodes below. Specialists run in parallel via Send() fan-out. The post-investigation chat path is covered in its own section below.

AI investigation structure

Deep dive produces new findings, which are correlated again

When an investigation starts, we scan the user’s prompt with regex to detect incident IDs and troubleshooting report IDs. If a match is found, we pre-fetch that data in parallel before the Gather Agent runs — via MCP calls such as investigate_incident, get_intelligent_troubleshooting, and get_troubleshooting_detail. This reduces redundant tool calls and lowers overhead for downstream specialists.

The Gather Agent then analyzes the pre-fetched context (plus the user request) and produces a structured investigation_brief: a focused briefing that tells each specialist what to investigate and what to ignore. Specialists are launched in parallel with this brief, so they start with a clear objective instead of exploring blindly. In this way we clarify each specialist's aim and objective, focus them on what actually needs to be focused.

For example, sometimes a user wants to find an application latency suddenly occurred, then starts a troubleshooting run. Based on troubleshooting report findings, gather agent reviews the findings and point database connection issues out and this allows specialists to focus on exact point. It reduces tens of minutes for the user to find the issue, and investigation duration by redirecting agents through right direction.

gather_prompt = _get_gather_prompt(investigation_type)
try:
    context: IncidentContext | None = await invoke_structured_output(
        llm,
        IncidentContext,
        [
            SystemMessage(content=gather_prompt),
            HumanMessage(content=human_content),
        ],
    )
except Exception as exc:
    logger.warning(
        "Gather LLM structured extraction failed (%s); proceeding with empty context",
        exc,
    )
    context = None

We use different gather prompts for the same graph node depending on the investigation type (for example, incident review vs. on-call alert vs. QA support), which improved output quality by tailoring context extraction to each use case.

Instead of free-form text, the LLM returns structured IncidentContext fields and an investigation_brief. Specialists consume this as shared context, so they start aligned on scope, hypotheses, and focus areas rather than re-interpreting raw incident data on their own.

Prefetch & Gather

We do some pre-checks in that node such as finding incident id, intelligent troubleshoot id, kubernetes namespace, etc. in user context. There are some pattern for our in-house platform tools for logging, monitoring, etc. and when a user mentioned about any of them, we gets important details and pre-fetching some details and adding them to prompt that specialists will use. Therefore, our sub-agents nows where to start, what to investigate. Also it helps agents to understand the context. Sometimes they struggle to parse mentioned links, urls properly to get meaningful context such as kubernetes namespace, application name, troubleshooting report etc. When detecting these ids, if they mentioned, it helped us to reduce multiple llm and mcp tool calling for getting details for an incident or troubleshooting. Before these controls, the agent was handling all these conditions by itself and it was causing unnecessary cost and latency.

url_ctx = extract_url_context(raw_text)
url_section = format_url_context_section(url_ctx)
# ── Step 2: targeted MCP calls in parallel ────────────────────────────────
# These run before the gather LLM so their results are fed as input context.
targeted_coros = []
targeted_keys: list[str] = []
if incident_id:
  targeted_coros.append(
      _safe_call(mcp, "investigate_incident", {"incident_id": incident_id})
  )
  targeted_keys.append("investigate_incident")
  targeted_coros.append(
      _safe_call(
          mcp, "get_intelligent_troubleshooting", {"incident_id": incident_id}
      )
  )
  targeted_keys.append("get_intelligent_troubleshooting")
if troubleshoot_id:
  targeted_coros.append(
      _safe_call(mcp, "get_troubleshooting_detail", {"rca_id": troubleshoot_id})
  )
  targeted_keys.append("get_troubleshooting_detail")

Parallel specialists (tool loop + structured findings)

For a typical incident-review investigation, gather agent does not pin a fixed specialist list. Instead, route_to_specialists auto-detects which domains to invoke from the structured IncidentContext produced in Phase 1. The default set always includes four core specialists:

• metrics — application metrics (Grafana / PromQL)
• troubleshoot — automated pre-check report + follow-up verification
• traces — OpenTelemetry / Jaeger span analysis
• kubernetes — cluster, node, pod, and workload health

Depending on gather signals, we extend that list:

• cuj when has_slo_impact is true (SLO, SLI, or business-transaction impact)
• topology when has_multi_service_impact is true (cross-service blast radius) If neither flag is set, we still add cuj and topology as lightweight defaults—so a full incident review typically fans out to six native specialists, plus any external agents registered for incident-review. LangGraph dispatches them with Send("run_specialist", …) so every specialist runs in parallel against the same gathered context, rather than sequentially waiting on one another.

Each specialist has its own skills and references for its domain:

• A specialist system prompt encodes operational standards: which MCP tools to prefer, naming conventions (for example, Jaeger service names as {namespace}+{appLabel}), investigation order, and guardrails against common false positives.
• A skills library (agents/builtin/skills/) provides on-demand references via load_reference—progressive disclosure rather than stuffing everything into the initial context. Examples include promql-patterns, correlation-patterns, trace-analysis-patterns, and incident-runbooks with proven fixes from real production RCAs.
• Those references also document composite MCP tools that replace several sequential calls with one — for example, jaeger_search_service_traces (summary + fuzzy match), diagnose_pod (describe + events + usage), or get_application_overview (workload + rollout in a single call).

That reduces tool-call overhead and gives specialists richer context per step.

Shared starting context

Each specialist receives the same foundation:

• incident_context from gather (investigation_brief, affected services, clusters, routing flags, and so on)
• prefetched_data (for example, a troubleshooting report already fetched in gather—specialists are instructed not to call those tools again) This avoids duplicate MCP calls and keeps each agent focused on its domain instead of re-discovering incident metadata.

Two-phase specialist execution

Every native specialist follows the same pattern:

1. Tool-use loop — the LLM calls a scoped set of SRE-MCP tool iteratively (for incident-review, up to 15 tool calls per specialist, 10 for other investigations).
2. Structured extraction — a final LLM pass converts the accumulated analysis into a typed Pydantic model (for example, metrics_findings, trace_findings, cluster_findings). Phase 2 is deliberate: free-form tool output is useful for reasoning, but downstream phases need consistent, machine-readable findings for correlation—not another prose summary that each node interprets differently.

Example: latency incident (incident-review)

Suppose an on-call engineer opens an incident-review for sudden application latency and references an existing troubleshooting run. After gather:

• The investigation brief highlights database connection errors as the likely focus area.
• The troubleshoot specialist starts from the pre-fetched report instead of re-running static checks.
• metrics pulls latency and error-rate signals for the affected service.
• traces searches for failing or slow spans around the database dependency.
• kubernetes checks for pod restarts, OOMKills, or resource pressure on the workload.
• cuj / topology add business-flow and dependency context if multiple services are involved. When all parallel branches complete, LangGraph merges their structured findings and routes to the Correlate Agent — the next step in the pipeline.

Correlate — cross-domain evidence matching

Once every specialist finishes, LangGraph merges their structured findings into the shared state and routes to the Correlate Agent. This agent has no MCP tool access — it cannot query Prometheus, Jaeger, or Kubernetes on its own. Its only job is to reason over the evidence that the specialists already collected.

The correlator receives the typed findings from every specialist (metrics_findings, trace_findings, cluster_findings, …) and looks for convergence: do two or more independent specialists point at the same component, time window, or failure mode? If the metrics specialist flagged a latency spike at 14:32 on checkout-api, and the traces specialist found timeout spans against the same service at the same time, that convergence becomes a cross-validated signal. A finding that appears in only one specialist's output remains a hypothesis; a finding corroborated by two or more becomes the primary root cause candidate.

This separation is deliberate. Giving the correlator tool access would tempt it to run its own investigation instead of synthesizing what already exists. By keeping it evidence-only, we force the system to rely on the specialist outputs and prevent hallucinated evidence chains — the correlator can only reference data that a specialist actually retrieved.

The output is a structured CorrelationResult containing:

• A root cause with a category taxonomy (deployment / config / infrastructure / traffic / code / external)
• Contributing factors ranked by relevance
• A timeline reconstruction built from timestamped specialist evidence
• Prioritized recommendations (immediate and short-term actions)

For narrow investigation types that only involve a single specialist — such as cluster-resource-alert—the LLM correlation call is skipped entirely. A deterministic passthrough constructs the CorrelationResult directly from that specialist's findings, saving a full LLM round-trip.

Evaluate — the quality gate

The Correlate Agent produces a hypothesis, but not every hypothesis is trustworthy. The Evaluate Agent acts as a quality gate that decides whether the investigation is done or needs more work.

It checks two signals:

1. Confidence level — high, medium, or low, derived from how much evidence backs the hypothesis.
2. Cross-validation — a boolean: was the root cause corroborated by at least two independent specialists?

Only when confidence is high and cross-validation is true does the pipeline proceed to synthesis. Every other combination triggers a deep dive.

To prevent infinite loops, a hard cap of two deep-dive rounds is enforced. After two attempts, the system proceeds to synthesis regardless of confidence. This is a conscious trade-off between thoroughness and time: an investigation that takes 30 minutes is worse than a medium-confidence report delivered in 8.

A critical design choice: route_after_evaluation is a pure deterministic function, not an LLM call. The LLM produces the evaluation data, but the routing decision—synthesize or deep-dive—is made by simple if/else logic over the structured output. No LLM decides the graph's control flow.

Deep Dive — targeted re-investigation

When the evaluator decides the hypothesis needs more evidence, it doesn’t just re-run the same specialists blindly. Instead, the evaluator produces a suggested_deep_dives list: specific questions for specific specialists.

For example, after a first pass where the metrics specialist found elevated error rates but the traces specialist didn’t find matching failure spans, the evaluator might produce:

[
  {
    "specialist": "traces",
    "question": "Search for spans with 5xx status on checkout-api between 14:30–14:45"
  },
  {
    "specialist": "kubernetes",
    "question": "Check for pod restarts or OOMKills on checkout-api pods in the last 30 minutes"
  }
]

Each deep-dive task is dispatched to the named specialist with the targeted question appended as additional_instructions. This turns a generic specialist into a focused investigator—instead of exploring broadly, it answers a specific question that the correlator identified as a gap.

Get guvenaltunsoy’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Deep-dive results are written back into the same state keys as the original findings. When the pipeline loops back to correlate, the agent sees enriched data — original findings plus deep-dive additions — and can produce a stronger hypothesis without special merge logic.

One pragmatic detail: LLMs don’t reliably produce exact specialist names. The evaluator might suggest "log specialist", "k8s", or "troubleshoot/deployment specialist" instead of our canonical registry keys. A fuzzy normalization layer with an alias table and substring fallback maps these free-form names to the correct specialist, bridging the gap between natural language and hard-coded identifiers.

Synthesize — final report

Once the evaluator approves the hypothesis (or the deep-dive cap is reached), the Synthesize Agent compiles the final incident report. It receives the complete reasoning chain:

• The original incident context from gather
• All specialist findings (including deep-dive additions)
• The correlation result with root cause, contributing factors, and timeline
• The evaluator’s quality assessment

The report is produced as a structured Pydantic FinalReport, not free-form text. This guarantees a consistent, machine-parseable structure that the incident management platform (Pandora) can render directly. The report includes:

• Root cause hypotheses ranked by confidence
• Evidence chain linking each hypothesis to specific specialist findings
• Structured incident timeline reconstructed from timestamped evidence
• Impact analysis — affected services, users, and business flows
• Contributing factors — secondary issues that made the incident worse
• Immediate and short-term recommendations to mitigate the incident

The synthesis node also uses a multi-model fallback chain. It tries the primary LLM first; if that fails, it cascades to a coordinator model, then a fast model. If all LLMs fail, a deterministic _fallback_report constructs a basic report from raw specialist data. The system always produces a report— an investigation that crashes at the final step without output would be worse than a lower-quality report.

Finally, deep_dive_count is stamped on the report so readers can see how many iterative refinement loops the investigation needed—a built-in transparency signal for the SRE reviewing the results.

Post-investigation chat

The automated pipeline stops when the final report is written — but the SRE’s work often does not. The same LangGraph run can be re-entered for chat: the user continues on top of the completed investigation instead of starting a fresh agent with an empty context.

When a chat message arrives after the run is complete, an entry router sends the request to a chat path rather than back through gather. A lightweight intent router then chooses how to help:

• Answer from existing state — explain the report, findings, or correlation without new tool calls
• Tool call — call MCP tools interactively (metrics, Kubernetes, traces, and so on) to verify or extend a claim
• Re-run a specialist — give one domain agent a new task (for example, “check OOMKills on checkout-api again with this focus”) and fold the result into the conversation
• Custom analysis — fan out a focused specialist set for a user-defined follow-up
• Full re-investigation — clear findings and run the pipeline again from gather when the user wants a clean pass

Because chat starts from checkpointed investigation state — final_report, specialist findings, correlation, and history—the agent already “knows” what was found. It does not re-ask for the incident ID or re-fetch everything unless the user asks it to dig further. And unlike the Correlate / Evaluate nodes (which are intentionally tool-free), the chat agent keeps MCP access, so humans can steer live verification after the automated pass.

That split matters: the investigation phase runs unattended to produce a grounded report; the chat phase is where the on-call engineer interrogates that report, assigns manual follow-up tasks, and uses tools under human direction.

Quality is not only implicit. On each agent output (investigation messages and related surfaces), users can submit a 1–5 star rating plus an optional comment. We aggregate those reviews to compare investigation types, find low-scoring runs, and decide what to fix next in prompts, skills, or MCP tools.

Production hardening

Building the investigation pipeline is one thing; keeping it reliable in production is another. Several cross-cutting mechanisms turned this from a prototype into a system we trust during real incidents.

Cancellation and timeouts

An investigation that takes forever is worse than no investigation. Every node in the graph is wrapped in a three-way race:

1. The node’s own execution
2. A cancellation signal (user-initiated via the UI)
3. A wall-clock timeout

Whichever finishes first wins. If the user cancels, every in-flight specialist stops immediately — even mid-LLM-call. LangGraph’s built-in checkpoint cancellation doesn’t reach inside Send() fan-out boundaries, so we implemented a module-level cancellation registry that maps each run_id to an asyncio.Event. Nodes check this event at entry and at each tool-loop iteration, ensuring cancellation is responsive even during parallel execution.

Specialist guardrails

LLMs are creative — sometimes too creative. Left unchecked, a specialist will happily burn through dozens of tool calls exploring tangents. We learned this from production: early versions had specialists that wasted 8+ iterations re-calling the same discovery tool with identical arguments.

Three hard limits keep specialists focused:

• Max iterations: 15 tool-call rounds per specialist for incident reviews, 10 for lighter investigation types
• Per-result size cap: 50K characters per single tool result
• Cumulative size cap: 800K characters total across the entire tool loop

When any limit is hit, the LLM is re-invoked without tool bindings, forcing it to produce a final answer from whatever evidence it has collected.

Additionally, a duplicate tool-call detector tracks (tool_name, canonical_json_args) signatures. If the LLM re-issues an identical call, it's short-circuited with a notice ("you already called this— re-read the result above"). This alone cut wasted iterations significantly.

LLM resilience

Each LLM call has a 90-second hard timeout. After 2 failed retries, the system automatically falls back to a different LLM provider. A single provider outage mid-investigation doesn’t kill the entire pipeline.

The retry policy explicitly excludes cancellation and timeout exceptions — only genuine transient errors (network failures, rate limits) trigger retries. This prevents a cancelled investigation from burning through retry attempts before stopping.

Structured output resilience

LLMs don’t always produce clean data. A model might return "<UNKNOWN>" or "N/A" where a number is expected. Rather than letting a single malformed field crash the entire investigation's Pydantic validation, we use SafeFloat / SafeInt coercion validators that silently convert these values to 0. The investigation continues, and the correlator naturally weights low-quality findings lower.

External agent isolation

Third-party agents are registered via a JSON config and dispatched through the same run_specialist node as native specialists—the graph doesn't know the difference. But external agents can fail in ways we don't control: timeouts, malformed responses, authentication errors.

Any external agent failure is caught and converted into a low-confidence finding with an error summary. A single misbehaving agent never crashes the investigation — correlate simply weights its (empty) evidence down. Session IDs are preserved across deep-dive rounds, so an external agent that succeeded in round one can continue its conversation in round two.

Parallel state safety

When six specialists run in parallel via Send(), they all write back to the same shared state. LangGraph expects each key to be written by exactly one node per step—violating this causes an InvalidUpdateError.

We solved this with custom Annotated[..., _last_value] reducers: a "last non-None wins" merge strategy. Each specialist writes only its own findings key (e.g., metrics_findings), while shared keys like current_phase use the reducer to safely accept concurrent writes. This is a LangGraph-specific pattern that was essential for making parallel fan-out work reliably.

Results

Where we’ve failed

1. Context window

At first, we put most of our operational knowledge into agent prompts and taught the agents how to discover our systems and infrastructure in order to find an incident’s root cause. That is where we first failed. Prompt size and token usage ballooned into the millions (~8–9 million tokens). As you might expect, that confused the agents, cost hundreds of dollars, and increased hallucinations.

We then optimized the agents’ prompts and moved most of that content into the knowledge/skills library. We also added proven troubleshooting and investigation patterns so agents had clearer investigation steps without carrying everything in the system prompt.

1. MCP tool usage

We created an MCP server called sre-mcp with around 200 tools an agent might need to access information, explore, and investigate—whether to find an incident's root cause or to identify anomalies affecting service health. When we exposed that full tool set to every agent:

• Specialists started calling tools outside their domain (for example, the Kubernetes agent querying APM metrics or application traces).
• They made unnecessary calls even for tools that were in their domain.
• They used complex multi-step tool chains just to fetch basic data.

To fix this:

• We split sre-mcp into five domain-scoped MCP servers and gave each specialist only the servers relevant to its domain. As a result, each agent sees roughly 50–60 tools instead of ~200.
• We analyzed tool-calling chains and found patterns we could merge — turning three or more MCP calls into one. For example, agents were separately calling application throughput, resource usage, latency, and similar metrics. That often meant 4+ MCP calls and at least 2 LLM rounds. We introduced composite tools such as application_perf_overview, which fetches, normalizes, and returns those performance metrics in a single response. We applied the same approach to roughly 20 tools. That cut MCP traffic, investigation duration, and end-to-end latency.
• Instead of returning raw JSON from MCP to the agent, we converted MCP responses to Markdown. That reduced token size by dropping repetitive JSON field names, braces, and other structural noise.

What we achieved

We built a system that, on a typical investigation run, makes on the order of ~60–100 LLM calls and a similar number of tool calls, consumes roughly ~3M tokens, and finishes in about ~4 minutes. During a live incident that often saves at least ~30 minutes of manual triage and helps us reduce MTTR. Users rate agent outputs 1–5 (with optional comments); that feedback currently averages 3.42 / 5.

Ratings given by Investigation users (1–5)

The above charts cover approximately three months of production usage (June to mid-August 2026).

Token usage vs. run volume. Early on, average tokens per run spiked above ~7M on low-volume days (late June) — the “everything in the prompt” failure mode. After the skills/MCP optimizations, the daily average settled closer to ~2.5–3M tokens per run, even as run volume climbed (a peak near ~70 runs/day around July 9). High volume no longer meant runaway cost per investigation.

LLM and tool call intensity. Average LLM calls and tool calls per run stay in a similar band day to day (typically tens of calls each, often approaching ~100 on heavier days). Run count fluctuates independently: busy days do not automatically mean “more calls per investigation.” That stability is useful for capacity and cost planning.

Model mix. Most token cost goes to faster/cheaper models — especially gemini-3.5-flash—with smaller shares for stronger Gemini Pro variants and only thin slices for DeepSeek / Claude. We reserve heavier models for harder steps rather than paying Pro prices on every tool loop.

LLM model token cost distribution

The same period in table form: gemini-3.5-flash dominates both volume and tokens (~19.4K calls, ~911M tokens, ~5.2s avg), while Pro variants are slower (~8s) but still far cheaper than Claude Opus (~44–50s avg on a few hundred calls). Error counts stay low at the model layer (single-digit across millions of tokens)—latency and cost are the real trade-offs we manage with model profiles.

Model statistics: calls, errors, tokens, avg duration

Where the work (and cost) lands. Among specialists, kubernetes, metrics, and traces dominate both token usage and call volume. troubleshoot and topology are lighter. LLM calls and tool calls track closely (~1:1) for most specialists—the tool loop is the real workload. metrics is the exception: it often issues more tool calls than LLM rounds, which matches multi-metric fetches in a single reasoning step.

Token usage and LLM/tool calls by specialist (top 5)

The full specialist call table makes the architecture visible: domain specialists do the heavy LLM+tool loops; coordinator nodes (gather, correlate, evaluate_and_synthesize) are almost entirely LLM-only (0 tool calls)—they reason over structured findings, they do not re-query MCP. Tool errors concentrate on the hot specialists (kubernetes ~288, traces ~195, metrics ~169), which matches flaky upstreams (K8s API timeouts, Jaeger gaps) more than model failures. Isolation matters: those tool errors degrade a finding; they do not kill the whole investigation.

LLM and tool calls by specialist (with errors)

Coordinator nodes: duration vs. tokens. The next two charts focus on the graph nodes gather, correlate, and evaluate_and_synthesize (not the parallel specialists). Their durations stay in a similar band over the month (gather ~16s avg, correlate ~24s, evaluate_and_synthesize ~38s). Their token usage, however, is modest compared with specialists (tens of thousands of tokens per run on average) and trends down after the prompt/skills work—even when wall-clock time does not drop by the same amount. In other words: we made these nodes cheaper before we made them dramatically faster.

Token usage for gather / correlate / evaluate_and_synthesizeDuration for gather / correlate / evaluate_and_synthesize

Duration for gather / correlate / evaluate_and_synthesizeDuration for gather / correlate / evaluate_and_synthesize

Challenges

1. Choosing the LLM model

When we first built this system, every agent used the same LLM — mostly Claude Opus / Sonnet. In many cases Opus was capable enough to research almost anything we threw at it. Over time we realized that Opus burns a lot of tokens even on small jobs, such as converting free-form findings into structured output. We also realized that many investigation types are not hard enough to justify that cost: cheaper models (Gemini 3.5 Flash, DeepSeek in our stack) can handle them well.

After that realization, we stopped using one model for everything. We now pick models per investigation type, per graph step (gather, specialists, correlate, evaluate/synthesize), and even for structured-output conversion. That cut cost significantly. Some current profiles look like this:

• incident-review — gather: gemini-3.5-flash; specialists: gemini-3.1-pro-preview; correlate / evaluate-and-synthesize: claude-opus-4-8
• oncall-alert-analyzer — gather: gemini-3.5-flash; specialists / correlate / evaluate-and-synthesize: gemini-3.1-pro-preview
• qa-support — gather: gemini-3.5-flash; specialists / correlate / evaluate-and-synthesize: deepseek-v4-pro

Because oncall-alert-analyzer runs often and drives a large share of total cost, it uses cheaper models than full incident-review. We also started testing DeepSeek in production, but its response time is high, so we limit it to qa-support, where latency is less critical than during a live incident.

Cost was not the only reason we leaned on Gemini. Speed matters as much during an incident. With Opus on every step, a full investigation often took ~8–9 minutes — still useful, but slow. The same investigation on Gemini produced broadly similar output ~30–40% faster, saving roughly ~2–3 minutes. That difference shows up in the incident management lifecycle: on-call gets a usable report sooner.

Observations on LLM models

We compared models on real incidents with the same inputs and tools, aiming for root cause — or at least something close.

• Opus uses tools effectively, correlates evidence well, and produces the most useful executive summaries and root-cause hypotheses.
• Gemini also uses tools well, but is less reliable than Opus at pinning the true root cause in real time.
• DeepSeek performed worst on live incidents. In most cases it never got close to the root cause. It struggles to reason over a large distributed system or to correlate findings unless you point it at the exact service. Without that hint, it does not know where to start or which path to follow — and it often fails to separate false-positive noise from real failure signals. (UPDATE: We started to use new version of deepseek-v4-flash-0731. we use it for oncall-alert-analyzer and it not works very well)

1. False-positive noise

At Trendyol scale (+20K+ microservices, +2K+ databases, ~3M req/s, and more), separating noise from real issues is hard — even for humans. Manual investigation still means millions of error logs across services, thousands of traces, hundreds of Grafana dashboards. We constantly fight that noise to find the failure source. Sometimes an error log is the root cause; sometimes it is a false positive.

After reviewing AI outputs across many runs, we built a working list of common false-positive patterns. That experience is a major reason the correlate node exists: its job is to filter noise across specialist findings, refuse to trust a single similar clue, and cross-check hypotheses from multiple independent specialists.

What’s Next

Dynamic knowledge base. Today, investigation skills and runbooks are curated by hand. Next, we want a living knowledge base of incident root causes and fixes, built from closed incidents and their RCA forms — and kept fresh automatically as incidents are opened and resolved. That would let specialists start from “what and how fixed this last time” instead of rediscovering the same patterns from scratch.

Proactive agent workflows. Today the system is mostly reactive: a user, an alert, or another platform has to start an investigation. The longer-term aim is to act before a full incident — spot early signals, run a focused check, and page the right on-call engineers to prevent or shrink the blast radius.

That is harder than it sounds. Continuously asking an agent “is there an anomaly?” or “will we have an incident?” across our full telemetry is not yet practical: the data volume and variability make naive real-time scanning too expensive and too noisy. We need a narrower, cost-aware approach — high-signal triggers, scoped checks, and clear escalation paths — before proactive workflows become as trustworthy as the reactive investigation path we run today.

Join Us

Want to be a part of Trendyol? We’re hiring! Check out our open positions and other media pages from the link.

Join Us - Trendyol Careers