Distributed Tracing with Spring Boot 4.1 and Grafana: A Practical Guide

Learn how to follow a single request across multiple microservices using OpenTelemetry, Micrometer, Grafana Tempo, and Spring Boot.

In a microservices architecture, a single user action can trigger a chain of operations across multiple services. An order request might enter through an order service, call an inventory service, execute a database query, and publish an event to a message broker.

Everything may appear to work correctly until a customer reports that placing an order takes five seconds instead of the expected few hundred milliseconds.

You check the logs. The order service looks fine. The inventory service has no obvious errors. The database is responding, and the metrics dashboard shows that the application is running.

So, where did the time go?

The problem is that each service sees only a part of the story. Logs tell you what happened within individual services, while metrics show aggregated behaviour over time. Neither necessarily tells you how one particular request travelled through your entire system.

This is where distributed tracing becomes essential.

With distributed tracing, you can follow a request from the moment it enters your application until it finishes, even when it passes through several services. You can identify which service consumed the most time, where an error occurred, and how different operations relate to one another.

In this article, we will explore the fundamentals of distributed tracing and build a practical example using Spring Boot 4.1, OpenTelemetry, and Grafana.

1. What Is Distributed Tracing?

Distributed tracing is an observability technique that records the journey of a request as it moves through multiple services.

Imagine an e-commerce application with the following architecture:

• A client sends a request.
• The order service receives the request.
• The order service calls the inventory service.
• The inventory service queries a database.
• The order service publishes an order-created event.

Without distributed tracing, each component might record its own logs independently. Connecting those logs to understand the complete request lifecycle can require manual investigation.

Distributed tracing connects these operations into a single trace, allowing developers to see the entire request journey.

With a trace, you can answer three important questions:

Where did the time go? Identify the operations responsible for slow requests.

Which service failed? Locate the service or operation where an error occurred.

Who called whom? Understand the relationships between services and their dependencies.

These capabilities become increasingly valuable as an application grows from a monolith into a distributed system.

2. Logs, Metrics, and Traces: Understanding the Three Signals

Before implementing distributed tracing, it is important to understand how it complements other observability signals.

Logs: What happened?

Logs are timestamped records of events produced by an application.

For example:

Payment failed for order 42

Logs are useful when investigating specific events, exceptions, and application behaviour. However, when several services generate independent logs, reconstructing the complete journey of one request can be difficult without additional correlation information.

Metrics: How much and how often?

Metrics are numerical measurements collected over time.

Examples include request counts, error rates, CPU utilization, and response latency.

A metric might tell you that the 95th-percentile response time, or p95 latency, is 800 milliseconds. It does not necessarily reveal which operation within a particular request consumed most of that time.

Traces: Where did this request go?

Traces describe the path and timing of an individual request across services.

For example, a trace might reveal that an order request spent most of its time waiting for an inventory service, even though the order service itself performed very little work.

The three signals work best together:

• Logs explain individual events.
• Metrics reveal patterns and trends.
• Traces show the path and timing of individual requests.

Together, they provide a much more complete picture of application behaviour.

3. Understanding Traces and Spans

Two concepts form the foundation of distributed tracing: traces and spans.

What is a trace?

A trace represents the complete journey of a single request across the system.

It connects the work performed by different services into one logical execution history.

For example:

POST /orders
    |
    +-- GET /stock/42
    |       |
    |       +-- SELECT stock
    |
    +-- Publish OrderCreated

The trace captures the relationships between these operations and records their timing.

What is a span?

A span represents one unit of work within a trace.

A span might describe an incoming HTTP request, an outgoing HTTP call, a database query, or a business operation that you explicitly instrument.

Each span contains information about the work it represents, including its name, timing, identifiers, attributes, and status.

A typical span includes the following fields:

FieldPurposeTrace IDIdentifies the complete request journeySpan IDUniquely identifies an individual spanParent Span IDIdentifies the span that initiated the operationNameDescribes the operation being performedTimingRecords when the operation started and how long it tookAttributesProvides searchable key-value detailsStatusIndicates whether the operation succeeded or failed

Every span in the same trace shares the same trace ID, while each span has its own span ID.

The parent span ID establishes the relationship between operations.

For example, the order service might create the root span for POST /orders. When it calls the inventory service, the outgoing operation becomes a child of the order request. The inventory service then creates a child span for its database query.

This hierarchy allows the tracing backend to reconstruct the complete request journey.

Reading a trace waterfall

Tracing tools commonly display spans in a waterfall view.

The waterfall makes it easier to understand execution timing and service relationships.

The width of a span represents its duration, while indentation shows how operations relate to their parent spans.

When investigating a slow request, start by examining the longest spans and their children. A long-running operation is often a useful starting point for identifying the bottleneck.

However, overlapping operations, waiting time, and parallel execution can affect interpretation, so the waterfall should be read in the context of the request’s actual execution flow.

4. Span Attributes, Events, and Errors

A trace becomes much more useful when spans contain meaningful information.

Attributes

Attributes are key-value pairs that describe an operation.

For example:

http.route = /orders

An inventory operation might include an attribute describing the type of item being checked.

These attributes help developers filter and search for traces that match particular conditions.

For example, you might want to investigate slow requests associated with a specific route.

Events

Events are timestamped records attached to a span.

They can describe important moments during an operation, such as a cache miss or a retry.

An event might look conceptually like this:

cache.miss at +12 ms

This helps developers understand what happened during the lifetime of a span, not just how long it lasted.

Status and exceptions

A span can record whether an operation succeeded or encountered an error.

When exceptions occur, recording the relevant error information makes it easier to identify failures within the complete request journey.

Meaningful span names, useful attributes, and accurate error information make traces considerably more valuable during production incidents.

5. How Context Propagation Connects Services

Creating spans inside individual services is not enough. The tracing system must also understand that operations in different services belong to the same request.

This is achieved through context propagation.

When the order service calls the inventory service, tracing information is carried along with the outgoing request. The receiving service uses that information to continue the existing trace rather than creating an unrelated one.

A common mechanism is the W3C traceparent HTTP header.

For example:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

The header contains four components:

• Version: identifies the header format version.
• Trace ID: identifies the overall trace.
• Parent ID: identifies the caller’s span.
• Trace flags: include sampling information.

The propagation process works as follows:

1. The calling service creates a span and attaches tracing context to its outgoing request.
2. The receiving service extracts the context and creates a child span.
3. Both services export their spans to the tracing backend.
4. The backend groups the spans using the trace ID and reconstructs the request journey.

If the tracing header is lost at a client, proxy, or service boundary, the trace may appear fragmented.

This is why context propagation is one of the most important parts of a distributed tracing implementation.

6. Sampling: Do You Need to Trace Every Request?

Recording every request can generate a significant amount of telemetry, especially in high-traffic applications.

Sampling allows you to control which requests are recorded.

Two common strategies are head sampling and tail sampling.

Head sampling

Head sampling makes the sampling decision at the beginning of a request.

It is relatively simple and inexpensive, but the system cannot yet know whether the request will eventually fail or become unusually slow.

For example, if a request is not sampled at the beginning, its trace may not be available later when you discover that the request encountered an error.

Tail sampling

Tail sampling makes the decision after enough of the trace has been collected to evaluate its characteristics.

This makes it possible to retain traces based on conditions such as errors or unusually long durations.

The trade-off is that the tracing infrastructure needs to buffer and process traces before deciding which ones to keep.

Configure sampling in Spring Boot

The workshop demonstrates the following property:

management.tracing.sampling.probability=1.0

A probability of 1.0 requests sampling of every eligible trace under this configuration. This is useful while learning because it makes the demonstration easier to follow.

The workshop notes that Spring Boot samples approximately 10% of requests by default.

In production, sampling should be chosen according to traffic volume, storage capacity, cost, and troubleshooting requirements. A smaller sampling percentage may be appropriate for routine traffic, while collector-based tail sampling can help retain important slow or failed traces.

7. The Tools: Spring Boot, Micrometer, OpenTelemetry, and Grafana

The practical implementation combines several technologies, each with a distinct responsibility.

OpenTelemetry

OpenTelemetry is a vendor-neutral observability framework that provides APIs, instrumentation, and protocols for generating and exporting telemetry.

In this example, it provides the tracing foundation, including trace identifiers, spans, and the OpenTelemetry Protocol (OTLP).

Micrometer

Micrometer provides Spring applications with an observability API.

Spring Boot integrates with Micrometer to instrument supported operations and connect application activity to tracing infrastructure.

It also provides the Observation API, which can be used to instrument custom business operations.

OTLP

OTLP stands for OpenTelemetry Protocol.

It defines how telemetry data is transmitted to compatible backends and collectors.

The workshop uses OTLP over HTTP on port 4318. OTLP over gRPC is available on port 4317.

Grafana Tempo

Tempo is the tracing backend used to store and retrieve traces.

It receives exported trace data and makes it available for querying.

Grafana

Grafana provides the interface for searching and exploring traces.

The workshop uses Grafana’s Explore interface to select Tempo as the data source, search for traces, and inspect individual spans.

The demonstration uses the grafana/otel-lgtm Docker image, which bundles Grafana, an OpenTelemetry Collector, Tempo, and other observability components for a convenient local setup.

LGTM stands for Loki, Grafana, Tempo, and Mimir. In the workshop’s demonstration environment, Prometheus is used for metrics.

The focus here is distributed tracing, but the broader stack can support a unified observability workflow across logs, metrics, and traces.

8. Building a Distributed Tracing Demo with Spring Boot 4.1

Let’s put these concepts into practice.

We will create two Spring Boot services:

• order-service, running on port 8080.
• inventory-service, running on port 8081.

The order service will receive a request and call the inventory service. Both services will export tracing information to the local Grafana LGTM environment.

Step 1: Add the dependencies

The workshop uses Maven dependencies for Spring MVC and OpenTelemetry.

Get Naijajug Info’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The relevant entries are:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-webmvc</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-opentelemetry</artifactId>
</dependency>

Use the dependency versions managed by the Spring Boot version selected for your project.

The OpenTelemetry starter supplies the tracing integration and the components required by the workshop’s setup.

The important objective is to enable supported automatic instrumentation without having to create every HTTP span manually.

Step 2: Start Grafana with Docker Compose

Create a compose.yaml file:

services:
  lgtm:
    image: grafana/otel-lgtm
    ports:
      - "3000:3000"
      - "4317:4317"
      - "4318:4318"

The ports serve different purposes:

• 3000: Grafana web interface.
• 4317: OTLP over gRPC.
• 4318: OTLP over HTTP.

Start the environment:

docker compose up -d

This launches the local tracing infrastructure needed by the demonstration.

Step 3: Configure the order service

In the order service’s application.properties, add:

spring.application.name=order-service
server.port=8080
management.tracing.sampling.probability=1.0
management.opentelemetry.tracing.export.otlp.endpoint=http://localhost:4318/v1/traces

The application name identifies the service in Grafana.

The sampling probability is set to 1.0 so that every eligible request is sampled during the workshop.

The OTLP endpoint specifies where the application sends its traces.

For the inventory service, use the same tracing configuration, but set the service name to inventory-service and the application port to 8081.

This distinction is important because the service names help you identify each application’s contribution to a trace.

Step 4: Make an HTTP call between services

The order service will use Spring’s RestClient to call the inventory service.

The workshop demonstrates the following controller:

@RestController
class OrderController {
private final RestClient inventory;
    OrderController(RestClient.Builder builder) {
        inventory = builder
                .baseUrl("http://localhost:8081")
                .build();
    }
    @PostMapping("/orders")
    String create() {
        return inventory.get()
                .uri("/stock/42")
                .retrieve()
                .body(String.class);
    }
}

When a request reaches POST /orders, the controller makes an HTTP request to /stock/42 on the inventory service.

Notice that the controller receives an injected RestClient.Builder.

This is important for the workshop’s tracing setup because Spring’s configured client builder supports the relevant automatic instrumentation.

Creating an independent client using RestClient.create() can bypass that instrumentation.

With the tracing integration correctly configured, the outgoing request can carry the existing trace context into the inventory service.

Step 5: Add a custom business span

Automatic instrumentation captures many useful operations, including supported incoming and outgoing HTTP requests.

However, it may not describe every business operation in enough detail.

Suppose the inventory service has a method that checks stock availability. We can instrument that operation using Spring’s Observation API.

The workshop demonstrates this pattern:

@Service
class StockService {
private final ObservationRegistry registry;
    StockService(ObservationRegistry registry) {
        this.registry = registry;
    }
    String check(String item) {
        return Observation.createNotStarted(
                    "stock.check", registry)
                .lowCardinalityKeyValue(
                    "item.type", "book")
                .observe(() -> lookup(item));
    }
}

Here, stock.check identifies the business operation.

The item.type attribute provides additional context that can help when searching for traces.

The observe() method executes the operation within the observation's lifecycle and records the result, including relevant error information.

In a real implementation, the lookup(item) method must be supplied by the application.

The broader lesson is to begin with automatic instrumentation and introduce custom spans where important business operations are missing from the trace.

Step 6: Run the services and generate requests

Start Grafana if it is not already running:

docker compose up -d

Then start each Spring Boot service in its own terminal, using the workshop’s Maven command:

./mvnw spring-boot:run

Run this command from each service’s project directory.

Send a request to the order service:

curl -X POST http://localhost:8080/orders

The order service should call the inventory service, creating a request journey that can be explored in Grafana.

Send several requests if necessary so that you have trace data to inspect.

Step 7: Find the trace in Grafana

Open the Grafana interface:

http://localhost:3000

If prompted for credentials in the workshop’s local setup, use admin / admin. These are demonstration credentials, not a recommended production configuration.

In Grafana:

1. Open Explore from the navigation menu.
2. Select Tempo as the data source.
3. Search by service name, using order-service.
4. Run the query and select a trace from the results.
5. Examine the waterfall and select individual spans to inspect their attributes.

You should now be able to follow the request from the order service to the inventory service and inspect the timing of the individual operations.

This is the point at which distributed tracing becomes especially useful: rather than guessing where a problem might be, you can investigate the actual path and duration of the request.

9. Searching Traces with TraceQL

As your application generates more traces, manually browsing every request becomes inefficient.

Grafana Tempo provides TraceQL, a query language for searching traces based on their properties.

The workshop introduces several useful examples.

Find traces from a particular service

{ resource.service.name = "order-service" }

This query matches spans associated with the order service.

Find slow spans

{ duration > 500ms }

This helps identify spans that took longer than 500 milliseconds.

Find spans with errors

{ status = error }

This filters for spans marked with an error status.

Combine multiple conditions

{
  resource.service.name = "order-service"
  && duration > 300ms
}

This narrows the search to spans associated with the order service that exceed 300 milliseconds.

These examples demonstrate how trace queries can turn a large collection of telemetry into a more focused investigation.

Exact query behaviour depends on the attributes recorded and the TraceQL capabilities of the Tempo version in use.

10. Common Distributed Tracing Problems

Even when the basic configuration looks correct, several issues can prevent traces from appearing as expected.

Outgoing HTTP calls have no spans

One common problem is using an HTTP client that bypasses Spring’s configured instrumentation.

In the workshop’s setup, use the injected RestClient.Builder rather than constructing an independent client with RestClient.create().

The same principle applies when using other HTTP clients: verify that the selected client and its configuration support the instrumentation you expect.

Only some requests appear in Grafana

This may be caused by sampling.

If the application samples only a percentage of requests, some requests will not produce recorded traces.

During the workshop, set:

management.tracing.sampling.probability=1.0

For production systems, select a sampling strategy appropriate to the workload and observability requirements.

Nothing arrives in Grafana

Check the configured OTLP endpoint and confirm that the collector is reachable.

For the workshop’s local environment, OTLP over HTTP uses port 4318 and the endpoint path /v1/traces. OTLP over gRPC uses port 4317.

Also confirm that the Grafana LGTM container is running and that the application is using the intended configuration.

The trace appears as separate, unrelated traces

This often indicates that the tracing context was not propagated correctly.

Check whether the traceparent header is preserved across the outgoing request, service boundary, and any intervening proxies.

A receiving service that cannot recover the original context may start a new trace instead of continuing the existing one.

Duplicate spans appear

The workshop warns against combining the OpenTelemetry Java agent with the Spring Boot OpenTelemetry starter in the same setup without understanding how their instrumentation overlaps.

Choose an instrumentation strategy deliberately and verify that it does not produce duplicate spans.

Configuration from an older tutorial does not work

Configuration properties can differ between Spring Boot versions.

A property copied from a Spring Boot 3 tutorial may not match the Spring Boot 4.1 setup demonstrated here.

Use documentation appropriate to your selected version and verify which properties your application actually recognizes.

11. Taking Distributed Tracing Beyond Your Laptop

A local demonstration is a good starting point, but production environments introduce different concerns.

Use an OpenTelemetry Collector

Instead of sending all telemetry directly to a backend, applications can export spans to an OpenTelemetry Collector.

The collector can receive, process, filter, and route telemetry to supported backends.

It also provides a useful place to implement sampling strategies, including tail sampling, and to centralize telemetry pipeline configuration.

Protect sensitive information

Span attributes can contain valuable debugging information, but they can also expose confidential data if used carelessly.

Avoid placing secrets, authentication tokens, or unnecessary personal information in span attributes.

Prefer meaningful operation names and carefully selected attributes that help with debugging without exposing sensitive values.

Choose meaningful span names

Names such as stock.check communicate the purpose of an operation.

Generic names such as process or handler are less useful because they do not clearly identify the work being performed.

Good naming conventions make trace searches and waterfall analysis much easier.

Review sampling and storage costs

Recording every request can be useful during development, but production systems may generate large volumes of trace data.

Choose sampling policies that balance cost, performance, and diagnostic value.

Where appropriate, collector-based tail sampling can help retain errors and slow traces while reducing the volume of routine telemetry.

12. Practical Exercises to Strengthen Your Understanding

Once the basic demonstration is working, try extending it.

Exercise 1: Add a third service.

Introduce another service into the request flow and observe how the trace gains another level of spans.

Exercise 2: Simulate a slow operation.

Add a 300-millisecond delay to the inventory service and use TraceQL to find the slow span.

Exercise 3: Introduce an error.

Make an operation throw an exception and search for the resulting error span.

Exercise 4: Instrument a business operation.

Create a custom stock.check observation, attach a useful attribute, and search for it in Grafana.

These exercises help move distributed tracing from theory into practical debugging skills.

Conclusion

Distributed tracing gives developers visibility into how requests move through distributed systems.

By connecting operations across services, it helps teams identify slow dependencies, investigate errors, and understand the relationships between different parts of an application.

With Spring Boot 4.1, Micrometer, OpenTelemetry, and Grafana Tempo, you can instrument an application, propagate tracing context between services, export spans using OTLP, and inspect the results in Grafana.

The most important concepts to remember are:

1. A trace represents the complete journey of a request.
2. A span represents one unit of work within that journey.
3. Trace IDs connect spans, while parent span IDs establish their relationships.
4. The traceparent header helps propagate tracing context between services.
5. Automatic instrumentation captures common operations, while custom observations add visibility into business logic.
6. Sampling controls the volume of recorded traces.
7. Grafana Tempo and TraceQL make it possible to search and investigate distributed request behaviour.

Ultimately, observability is not just about collecting more data. It is about collecting the right data and making it useful when something goes wrong.

When a request becomes slow or fails unexpectedly, distributed tracing helps you stop guessing and start following the evidence.

If you are building microservices with Spring Boot, distributed tracing is a valuable skill to add to your engineering toolkit.

Further Reading

• Spring Blog: OpenTelemetry with Spring Boot — https://spring.io/blog/2025/11/18/opentelemetry-with-spring-boot
• OpenTelemetry documentation — https://opentelemetry.io/docs/what-is-opentelemetry/
• Grafana OpenTelemetry LGTM Docker image — https://github.com/grafana/docker-otel-lgtm

#SpringBoot #DistributedTracing #OpenTelemetry #Grafana #Microservices #Observability #BackendDevelopment