Dapper: Google’s Large-Scale Distributed Systems Tracing Infrastructure

This article is a review of the Dapper paper published by the Google team and all the information presented is related to the paper.

This technical report introduces Dapper, a large-scale tracing infrastructure developed by Google to monitor complex distributed systems. To achieve ubiquity and transparency, the system integrates with core software libraries, allowing it to track requests across thousands of machines without requiring manual code changes from developers. The authors emphasize that low overhead is maintained through adaptive sampling, which records only a small fraction of total requests to ensure negligible performance impact. Beyond simple monitoring, Dapper provides programmatic APIs and a web interface that enable engineers to analyze latency, debug system dependencies, and improve overall performance. Ultimately, the paper highlights how Dapper evolved from a specialized tool into a fundamental monitoring platform that supports diverse operational and security needs.

Dialogue Section:

Act I: The Spaghetti Factory (The Complexity Problem)

Prashna: Hey Uttar! I was reading about how massive modern websites like Google work. They talk about “distributed systems” . It sounds like a fancy buzzword. What does it actually mean, and why is it so hard to figure out what’s going on inside them?

Uttar: Think of a modern web service like a giant, hyper-efficient kitchen in a five-star restaurant. In a simple kitchen, you have one chef who takes an order, cooks the meal, and serves it. But in a massive “distributed system,” the work is broken up into hundreds of tiny, specialized stations.

Let’s use a Google Search query as an example . When you type a query into the search bar, a single “front-end” service receives it . But to give you a complete results page (which Google calls “universal search”), that query is instantly sent to many other systems. It goes to query servers to search the index, to another system to check your spelling, to an ad server to find relevant advertisements, and to separate sub-systems for images, videos, and news.

All in all, processing a single web search can involve thousands of different machines and dozens of different services, all working at the same time!

Prashna: Wow, thousands of machines for just one search! But if the search is slow, how does anyone know which machine or service is at fault?

Uttar: Exactly! That is the core problem. If a user experiences a slow search, an engineer looking at the overall latency knows there is a problem, but they have no easy way of knowing why or which specific station in that massive kitchen is causing the delay.

It’s incredibly tough for a few reasons:

1. Constant changes: New services are added and modified every single week for performance, security, or new features. An engineer might not even know exactly which services are currently active!
2. No single expert: Each service is built and maintained by different, specialized teams; no single person understands all of them.
3. Shared resources: Different services share the same physical machines and databases (like Google’s Bigtable). A slowdown might be caused by an entirely different program hogging the database on that same machine.

Prashna: That sounds like trying to find a needle in a haystack of moving needles! So, how did Google solve this?

Uttar: They built Dapper. Dapper is Google’s production “distributed systems tracing infrastructure”. It acts like an automated supervisor that follows a single customer’s order through every single station in the kitchen, timing exactly how long each step takes and writing it all down.

Prashna: When the engineers built Dapper, what did they have to keep in mind? What were their main design goals?

Uttar: They had three non-negotiable goals:

• Low Overhead: The tracing system must have almost zero performance impact on the running services. If the monitoring tool slows down the actual search, service teams will simply turn it off.
• Application-Level Transparency: This was arguably the most challenging goal. Programmers should not have to write extra code or even worry about tracing. If tracing relied on developers manually adding code to every program, it would be extremely fragile and break constantly in a fast-paced environment.
• Scalability: It had to be capable of handling Google’s massive size and clusters for years to come.

Additionally, they wanted the tracing data to be available very quickly — ideally within a minute — so engineers could react immediately to live production issues .

Act II: The Anatomy of a Trace (Trees, Spans, and Passports)

Prashna: Okay, so Dapper monitors this giant kitchen. But how does it keep track of a single request as it flies across thousands of machines? What does a “trace” actually look like?

Uttar: Dapper models a trace as a tree of nested actions, which are called “spans”.

Think of a trace as the complete paper trail for a single customer query. Within that trace, each individual task or step is called a span. For example, if Server A calls Server B, that action is a span. If Server B then calls Server C, that’s another span nested inside the first one.

Each span is a simple log containing:

• A human-readable name.
• A unique span ID.
• A parent ID (which tells us which previous step triggered this one, helping us reconstruct the “family tree” of the request).
• A trace ID (shared by every single span in that specific query’s journey so we can group them together). All of these IDs are unique 64-bit integers.
• Timestamp records of exactly when the action started and ended, including network transfer times.

Prashna: Wait, if different steps happen on different physical machines, how do we know their clocks are perfectly synchronized? If one machine’s clock is off by a few milliseconds, won’t the timeline look messed up?

Uttar: That is a very smart observation! It’s called clock skew. Dapper’s analysis tools handle this by applying a simple rule of physics: a client must always send a request before a server receives it, and a server must always send a response before the client receives it. This gives Dapper an upper and lower boundary to align and correct the timestamps from different machines.

Prashna: Ah! That makes perfect sense. But let’s go back to the “transparency” goal. If developers don’t have to write tracing code, how does Dapper automatically pass these trace IDs and span IDs from machine to machine?

Uttar: Dapper achieves this by hooking directly into a very small, central set of common libraries that nearly every Google developer already uses. This is where the magic happens:

1. Thread-Local Storage: When a thread on a machine handles a request, Dapper attaches a “trace context” (which contains the trace and span IDs) to the thread’s local storage. Think of it like a tiny digital passport.
2. Asynchronous Callbacks: If a program schedules a task to run later or in the background, Google developers use common control flow libraries. Dapper’s instrumentation ensures that the digital passport is automatically copied and handed off to the background thread when it starts.
3. Remote Procedure Calls (RPCs): Since almost all machine-to-machine communication at Google uses a single, unified RPC framework, Dapper hooks into that framework. When Server A sends a message to Server B, the RPC library automatically attaches the digital passport (the trace context) to the message header.

Because Dapper is built directly into these three core library types, tracing happens automatically under the hood without the developer ever knowing about it!

Prashna: That is incredibly elegant! How much code did it take to build this core engine? It must be huge.

Uttar: You’d think so, but because they focused strictly on those core libraries, the code is surprisingly compact! The core tracing code is under 1,000 lines of C++ and under 800 lines of Java.

Prashna: That is tiny! Are there any cases where this automatic tracking breaks?

Uttar: Yes, occasionally. It can fail if developers use highly non-standard threading models, or if they use uninstrumented, raw communication libraries like raw TCP sockets. In those rare cases, developers have to manually propagate the trace context. However, out of thousands of applications at Google, only a tiny fraction (40 C++ and 33 Java applications) have ever needed manual trace adjustment.

Act III: What if Developers Want to Log Extra Details? (Annotations)

Prashna: What if a developer wants to log a specific event, like “database cache hit” or “cache miss,” during a trace? Can they add custom notes?

Uttar: Yes, they can! These are called annotations. Dapper provides a simple API that lets developers add timestamped text notes or even key-value maps (like keeping counters or logging binary messages). For example, every Bigtable request is automatically annotated with the name of the database table being accessed.

Currently, 70% of all spans and 90% of all traces in Google’s production systems have at least one of these custom annotations.

Prashna: But wait! What if a developer goes crazy and logs megabytes of text, slowing down the server or filling up the hard drive?

Uttar: Dapper has a built-in safety valve for that. Each span has a strict, configurable upper-bound on the total volume of annotations it can record. No matter how much an application tries to log, it can never overwrite or push out the core structural trace data.

Act IV: Solving the Performance Crisis (The Secrets of Sampling)

Prashna: Okay, let’s talk about the elephant in the room: overhead. If Google handles billions of requests, and Dapper has to trace, log, and save data for every single one, wouldn’t that slow down Google’s servers to a crawl?

Uttar: Absolutely, it would. Writing data to local disks for every single request would severely degrade high-throughput applications like Web Search.

To solve this, Dapper uses sampling. Instead of recording every single request, Dapper only records a tiny fraction of them.

Prashna: But if you only record a tiny fraction, don’t you miss the important errors or slow requests?

Uttar: In a high-traffic system, surprisingly, no. If a performance issue occurs in a high-volume service handling tens of thousands of requests per second, that issue will happen thousands of times. Tracing just a small sample will capture it plenty of times for analysis.

In fact, Google found that a uniform sampling rate of just 1 out of every 1024 requests (about 0.1%) provides more than enough data for common analysis while keeping performance impact practically invisible! At sampling rates below 1/16, the latency and throughput impact on Web Search is so low it falls within experimental measurement error.

Prashna: What about services that don’t get much traffic? If a service only gets a few requests a day, a 1-in-1024 sampling rate means they might never get a single trace!

Uttar: Exactly! That’s why Google developed adaptive sampling [38]. Instead of a fixed probability (like 1-in-1024), adaptive sampling is parameterized by a desired rate of traces per unit of time. High-traffic systems automatically lower their sampling rate to prevent overhead, while low-traffic systems raise their rate — even tracing 100% of requests — because they can easily afford the overhead.

Prashna: That is brilliant! How does the actual data collection work? How does the trace get from the local machine to the engineer’s screen?

Uttar: Dapper uses a three-stage, out-of-band collection pipeline:

1. Asynchronous Local Logging: The running application writes trace data to local disk log files. Disk writes are expensive, but Dapper reduces the impact by grouping multiple writes together and executing them asynchronously in the background.
2. The Dapper Daemon: A background helper process (daemon) running on every machine pulls the data from those local logs. This daemon is restricted to the lowest possible kernel priority so it never competes with the actual web applications for CPU. In testing, it uses less than 0.3% of a single CPU core.
3. Bigtable Storage: The daemons write the traces into regional Dapper Bigtable repositories. Each trace is stored as a single row in Bigtable, with each span as a column.

The median time it takes for trace data to go from a local machine to the central repository is under 15 seconds!

Prashna: You mentioned “out-of-band” collection. What does that mean, and why is it important?

Uttar: “In-band” would mean attaching the tracing data to the actual response headers sent back to the user. But Google’s traces can have thousands of spans, while search responses are often tiny (less than 10 kilobytes). If they sent trace data back in-band, the monitoring data would easily dwarf the actual search results, hogging network bandwidth!

Also, in-band collection assumes all calls are perfectly nested. But many systems return a result to the caller before all background tasks are fully done. An out-of-band system collects those background tasks easily.

Prashna: What about network traffic? Doesn’t sending all these traces across the data centers slow down Google’s network?

Uttar: Not at all. Each trace span is extremely compressed, taking up only 426 bytes on average. Dapper’s collection pipeline is responsible for less than 0.01% of the total network traffic in Google’s production environment!

Prashna: Even with sampling, Google must generate a mountain of trace data every day. How do they handle the storage costs?

Uttar: Google’s production clusters generate more than 1 terabyte of trace data every single day! To keep costs under control, they keep trace data for only two weeks before deleting it.

They also use a secondary round of sampling during collection. The collectors hash the unique Trace ID of each span to a number between 0 and 1. If that number is below a certain coefficient, they keep the entire trace; otherwise, they discard it completely. By using the Trace ID, they ensure they discard entire traces rather than accidental fragments. This single configuration knob allows the Dapper team to easily adjust the global storage and write rate.

Act V: Security, Privacy, and Keeping Secrets

Prashna: If Dapper is tracing everything, does it log users’ private data, like passwords, search queries, or emails?

Uttar: This was a non-negotiable issue during design. To guarantee security and privacy, Dapper never logs any request payload data. It only records the names of the internal functions (the RPC methods). If developers need to log specific debugging data, they must explicitly opt-in using the annotation API after confirming the data is safe to log.

Prashna: That’s highly responsible. Have there been any unexpected benefits of Dapper’s security model?

Uttar: Yes, actually! Security teams use Dapper to audit systems. Since Dapper tracks how services communicate, it can automatically monitor whether applications are satisfying security policies — like making sure they are using correct levels of authentication or encryption, and ensuring that systems with sensitive data are isolated from unauthorized components. This is far more reliable than manual code audits!

Act VI: The Tools Engineers Use (The Dapper UI)

Prashna: Let’s say I’m a Google engineer and my service is acting slow. How do I actually use Dapper to find the bug? Do I have to write a complex database query?

Uttar: Most engineers use the interactive web-based Dapper User Interface (UI). It’s extremely popular, with about 200 engineers using it daily and 750 to 1,000 distinct users every week.

A typical debugging session goes like this:

1. Define the Search: You select the service, the time window, and a cost metric you care about (usually how long the service took — latency).
2. View Execution Patterns: The UI shows a big table of performance summaries for all the different ways your service executes. You can sort these to find the slowest paths.
3. Inspect the Call Tree: You select a pattern and see a graphical diagram of how the services call each other.
4. Analyze the Latency Histogram: The UI shows a bar chart of latencies, showing you the distribution (like a curve of fast vs. slow requests) and providing a list of specific example traces in each range.
5. Examine the Timeline: You click a specific slow trace and see a detailed, color-coded timeline. Successive layers of communication are drawn as nested rectangles. The UI even splits the bars to show exactly how much time was spent doing actual computation on the server (colored green) versus how much time was spent waiting on the network (colored blue)!

Prashna: What if an engineer wants to build their own custom tool using Dapper data?

Get Sandeep Chakravartty’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Uttar: They can use the Dapper Depot API (DAPI). It allows developers to load traces on demand by trace ID, perform massive parallel analysis on billions of traces using MapReduce, or use an index to find traces by service name, host machine, and timestamp.

Currently, there are 3 persistent web apps, 8 on-demand tools, and dozens of one-off scripts built using DAPI.

Act VII: Real-World Success Stories (How It Help Google)

Prashna: Can you share some real examples of Dapper helping Google engineers solve massive problems?

Uttar: There are several incredible success stories in the paper:

1. The AdWords Ads Review Service

The team rebuilding this critical ad-checking system used Dapper from day one. It helped them in four ways:

• Performance: They identified slow, unnecessary “serial” requests (requests waiting for each other instead of running at the same time) and eliminated them.
• Correctness: They caught a bug where read-only database queries were being mistakenly sent to the expensive, high-load “write master” database instead of the cheap “read replica” databases.
• Understanding: They redesign their queries to minimize stress on database dependencies.
• Testing: They built a QA process where new code releases had to pass Dapper analysis to prove they didn’t introduce slowdowns. Ultimately, the team improved their service speed by two orders of magnitude (100x faster) using Dapper!

2. Hunting “Long Tail” Latency in Universal Search

“Long tail” latency refers to the rare cases where a search is extremely slow. Google engineers wanted to know why. Using Dapper, they discovered that momentary drops in network performance on the critical path were the primary cause of these outlier latencies, even though overall network throughput looked completely healthy!

3. Shared Storage Systems (GFSand Bigtable)

Google shares storage systems across many services. If Bigtable or GFS starts running slowly, it’s hard to know who is hogging the resources. Dapper allows storage owners to group trace performance by client, making it easy to see exactly which services are putting the most read/write load on the shared filesystems.

4. Live Firefighting

When a service is experiencing an outage or severe slowdown, developers can put Dapper into real-time mode. In this mode, the UI talks directly to the local Dapper daemons, getting fresh traces within seconds of real time so firefighters can instantly pinpoint where the bottleneck is.

Act VIII: Every Hero Has a Weakness (Dapper’s Limitations)

Prashna: This sounds like the perfect tool! But surely it must have some weaknesses or limitations?

Uttar: It does indeed! The report lists four major limitations that the Dapper team has identified:

1. The “Coalescing” (Request Bundling) Effect: Sometimes, to be efficient, a server will wait and bundle multiple user requests together before doing a batch action (like writing a block of data to disk) Because Dapper expects one request at a time, it might blame a single user’s request for the massive work of the entire batch, or only trace one request and completely ignore the others in that batch.
2. Batch Workloads: Dapper was designed for interactive, online systems triggered by a user query. It doesn’t map easily to off-line, data-heavy batch jobs like MapReduce, where the unit of work isn’t a single web request but rather a chunk of data.
3. Finding the True “Root Cause”: Dapper is incredible at telling you where a slowdown is, but it can’t always tell you why. For example, a request might be slow because it was stuck waiting in a long queue behind other requests, not because its own code is slow.
4. The Kernel-Space Gap: Dapper operates at the “user level” (the application software layer). Sometimes, performance issues are caused by the operating system’s kernel, but linking user-level trace contexts to kernel events in a low-overhead way is extremely difficult.

Act IX: Looking Back and Looking Forward

Prashna: This has been an eye-opening conversation, Uttar! If you had to summarize the ultimate lesson Google learned from building and running Dapper for over two years, what would it be?

Uttar: The biggest lesson is that combining a very simple, automatic, and invisible tracing core with an open, accessible API is incredibly powerful.

By making tracing automatic and transparent, they achieved ubiquitous coverage with zero effort from developers. And by opening up the trace database with a simple API, they allowed engineers across Google to build incredible tools — like network utilization dashboards, service dependency mappers, and security audit consoles — that the Dapper designers themselves never even imagined!

Prashna: Fantastic! I feel like I completely understand how Google keeps its massive, complex network of machines running so fast. Thank you, Uttar!

Uttar: You’re very welcome, Prashna! It’s amazing how a few clever design choices can bring order to a universe of thousands of machines.

I hope that the above conversation has given you a good overview of Dapper and what the journey was in terms of building it and using it to create new services on top of it.

Build Section:

The Dapper paper provides an excellent foundation for building modern developer tools around distributed tracing. Rather than simply recreating Dapper, these projects extend its core ideas with modern visualization, automation, and AI capabilities.

• Distributed Trace Visualizer
• What it is: A web-based tool that visualizes distributed traces as interactive timelines and service trees.
• Key features: Trace ID search, span timelines, service-level latency, expandable traces, error highlighting, and critical-path visualization.
• Technologies: OpenTelemetry, Jaeger, React, Node.js, and Docker.
• Why build it: It provides the foundation for the rest of the projects and demonstrates core concepts in distributed systems, observability, APIs, and data visualization.
• Portfolio value: ⭐⭐⭐⭐☆
• Agentic IDE potential: High — an agentic IDE can rapidly scaffold the frontend, backend, telemetry instrumentation, Docker environment, and visualization components.
• Service Dependency Graph Generator
• What it is: A tool that automatically discovers relationships between microservices by analyzing distributed traces and generates a live architecture diagram.
• Key features: Automatic service discovery, dependency mapping, request-flow visualization, dependency depth, high-traffic services, and bottleneck identification.
• Technologies: OpenTelemetry, Jaeger, graph algorithms, React, and a graph visualization library.
• Why build it: Dapper demonstrated that trace data can be used to infer dynamically changing service dependencies rather than relying solely on configuration information.
• Portfolio value: ⭐⭐⭐⭐⭐
• Agentic IDE potential: Very high — the IDE can build the ingestion pipeline, graph-generation logic, API layer, and interactive architecture visualization.
• AI Root Cause Analyzer
• What it is: An AI-powered debugging assistant that examines distributed traces and identifies the most likely source of performance problems.
• Key features: Automatic bottleneck detection, slow-span identification, critical-path analysis, anomaly detection, natural-language explanations, and suggested fixes.
• Example: Instead of showing that a checkout request took 2.4 seconds, the tool could explain that 1.7 seconds were spent waiting for a database query and recommend investigating the query or adding an appropriate index.
• Technologies: OpenTelemetry, Jaeger, Python/Node.js, an LLM, and React.
• Why build it: Dapper was particularly useful for determining where slowdowns occurred and for helping engineers investigate performance problems. An AI layer can turn that raw diagnostic information into an accessible engineering assistant.
• Portfolio value: ⭐⭐⭐⭐⭐
• Agentic IDE potential: Extremely high — this project naturally involves multiple agents or development stages for trace analysis, reasoning, backend services, prompt engineering, testing, and UI development.
• AI Incident Investigator
• What it is: An intelligent incident-response assistant that correlates traces, application logs, and metrics to investigate production incidents.
• Key features: Incident timeline generation, affected-service identification, anomalous trace detection, root-cause hypotheses, evidence collection, impact analysis, and recommended next steps.
• Example workflow: An engineer reports that the checkout API is experiencing high latency. The system examines recent metrics, finds the affected traces, identifies the slow service, correlates relevant errors and logs, and produces an incident summary with supporting evidence.
• Technologies: OpenTelemetry, Prometheus, Grafana, Jaeger, a log store such as Loki, an LLM, and a React-based interface.
• Why build it: Dapper was used during real production troubleshooting to isolate latency bottlenecks and inspect fresh high-latency traces. This project takes that concept further by automating parts of the investigation.
• Portfolio value: ⭐⭐⭐⭐⭐
• Agentic IDE potential: Extremely high — it is an ideal demonstration of agentic development because the application itself performs an investigative workflow involving multiple data sources and reasoning steps.
• AI Performance Consultant
• What it is: A developer-facing AI assistant that analyzes application traces and produces actionable performance-optimization recommendations.
• Key features: Performance scoring, slow-operation detection, excessive database-call detection, sequential-request detection, caching recommendations, dependency analysis, and optimization reports.
• Example output: “The checkout endpoint makes four sequential calls to Inventory, Pricing, Promotions, and Shipping. Three calls are independent and could potentially be executed concurrently.”
• Technologies: OpenTelemetry, Jaeger, Python, an LLM, and React.
• Why build it: Dapper was used to identify unnecessary serial requests, inefficient database access, and expensive interactions between services during development. The AI Performance Consultant turns these types of observations into an automated developer workflow.
• Portfolio value: ⭐⭐⭐⭐⭐
• Agentic IDE potential: Very high — it combines telemetry analysis, rule-based engineering heuristics, LLM reasoning, report generation, and a polished developer experience.

The Bigger Opportunity

These projects don’t have to remain five independent applications. They can be developed as one evolving observability platform:

Trace Visualizer → Dependency Graph → AI Root Cause Analyzer → AI Incident Investigator → AI Performance Consultant

Each stage builds upon the previous one. The trace visualizer provides the telemetry foundation, the dependency graph adds system-level understanding, the root-cause analyzer introduces AI-assisted debugging, the incident investigator brings together multiple observability signals, and the performance consultant turns those insights into proactive recommendations.

This progression also creates a compelling builder portfolio because it demonstrates more than the ability to generate applications. It shows an understanding of distributed systems, observability, graph analysis, AI agents, developer tooling, and product design — all around a single coherent technical theme.

Research Section:

Following are the open research questions related to the paper:

1. Fine-Grained Tracing for Offline/Batch Workloads (e.g., Spark, Data Pipelines)

• Status: Open / Active Research
• Why it remains open: While distributed batch engines (like Apache Spark and Flink) track job-, stage-, and partition-level metrics, fine-grained distributed tracing (tracking individual data records or key ranges through complex transformations) remains impractically expensive. Propagating trace context per record creates massive memory overhead and serialization penalties. Modern architectures use high-level metadata lineage tools (like OpenLineage) rather than full distributed trace trees.

2. Real-Time “Firefighting” During Catastrophic Outages

• Status: Partially Open
• Why it remains open: During severe production outages, trace volume surges dramatically while system resources become constrained. Tail-sampling collectors can experience memory backpressure, forcing them to drop traces precisely when visibility is needed most. Research continues into zero-loss streaming aggregation and low-overhead fallback mechanisms that degrade gracefully during system collapse.

Infographic Section:

Dapper Infographic