Implementation OpenTelemetry with .NET -Chapter 3

Module 2: Implementation Chapter : 3 Sending Telemetry to Azure Monitor

What this chapter covers

So far the telemetry has been printed to the console. That is useful for learning, but you cannot search it, chart it, or alert on it.

In this chapter we send the same telemetry to Azure Monitor and view it in the Azure portal.

You will learn:

• How to create an Application Insights resource
• How to replace the console exporter with the Azure Monitor exporter
• Where your spans are stored in Azure
• How to write your first query

You need an Azure subscription for this chapter. We are still working with traces only.

Step 1: Create an Application Insights resource

Application Insights is the part of Azure Monitor that stores application telemetry.

Using the Azure CLI:

# Create resource group
az group create \
  --name stayflow-rg \
  --location centralindia

# Create Application Insights
az monitor app-insights component create \
  --app stayflow-insights \
  --location centralindia \
  --resource-group stayflow-rg \
  --application-type web

Or in the Azure portal: search for Application Insights, select Create, and fill in the resource group, name, and region.

This series produces a small amount of data. If cost is a concern, check the current Application Insights pricing page before you begin and delete resource after finish the exercise.

Step 2: Copy the connection string

Open your Application Insights resource in the portal. The connection string is on the Overview page.

It looks like this:

text

InstrumentationKey=12345678-abcd-1234-abcd-123456789abc;IngestionEndpoint=https://eastus-1.in.applicationinsights.azure.com/;LiveEndpoint=https://eastus.livediagnostics.monitor.azure.com/

The connection string tells the exporter two things: which Application Insights resource to send to, and which regional endpoint to use.

Treat it as a secret. Do not commit it to source control.

Step 3: Add the exporter package

dotnet add package Azure.Monitor.OpenTelemetry.Exporter
dotnet add package OpenTelemetry.Exporter.Console

Step 4: Change the exporter

using Azure.Monitor.OpenTelemetry.Exporter;
builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource
        .AddService(
            serviceName: "stayflow-booking-api",
            serviceVersion: "1.0.0",
            serviceNamespace: "stayflow",
            serviceInstanceId: Environment.MachineName))
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation()
        .AddAzureMonitorTraceExporter());

AddAzureMonitorTraceExporter() sends spans to Application Insights.

Only one line changed. The instrumentation and the resource stayed exactly the same. This is the point of the exporter model: instrumentation decides what is collected, and the exporter decides where it goes. Changing the destination does not change how your application is instrumented.

Step 5: Set the Application Insights connection string

.

.AddAzureMonitorTraceExporter(options =>
{
    options.ConnectionString =
        Environment.GetEnvironmentVariable("APPLICATIONINSIGHTS_CONNECTION_STRING");
})

Step 6: Run the project

Call the endpoint a few times:

curl http://localhost:5000/bookings/abc123

Your console is now quiet. The telemetry is being sent to Azure instead.

Two delays are normal here:

• The exporter sends spans in batches, not one at a time. A single span does not trigger a network call.
• Azure needs a short time to make ingested data queryable.

Together, expect your first spans to appear within a few minutes. This is normal and does not mean something is wrong.

Step 7: Find your telemetry

In the Azure portal, open your Application Insights resource and select Transaction search.

Set the time range to the last 30 minutes .

You should see entries named GET /bookings/{id}. Select one. The details panel shows the same information you saw in the console: the duration, the URL, the response status code, and the application name.

The trace ID is there too. When more services are added, that ID is what connects their spans into one view.

Where spans are stored

Application Insights stores spans in different tables depending on their kind:

• requests — spans for work your application did in response to a caller
• dependencies — spans for calls your application made to something else

Your GET /bookings/{id} span has kind Server, so it is stored in requests.

Get Compile & Conquer’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

There is one naming detail worth knowing. Application Insights also has a table called traces. That table holds log messages, not distributed traces. Distributed traces live in requests and dependencies. The name predates OpenTelemetry.

Your first query

Select Logs in your Application Insights resource and run:

requests
| where timestamp > ago(30m)
| project timestamp, name, duration, resultCode, cloud_RoleName
| order by timestamp desc

This shows your recent requests with their duration and status code.

cloud_RoleName is where Application Insights stores service.name. It shows stayflow-booking-api.

To see how long the endpoint takes:

requests
| where timestamp > ago(1h)
| summarize count(), avg(duration), percentile(duration, 95) by name

This is the same latency data you saw in the console, now aggregated across every request.

Recommended: keep the console exporter in development

You can send telemetry to more than one destination at the same time:

.WithTracing(tracing =>
{

tracing.AddAspNetCoreInstrumentation();

if (builder.Environment.IsDevelopment())
    {
        tracing.AddConsoleExporter();
    }
    tracing.AddAzureMonitorTraceExporter();
})

In development you get immediate console output while the data also goes to Azure. This makes it easy to confirm that telemetry is being produced, without waiting for Azure to show it.

In other environments, only the Azure Monitor exporter runs.

Program.cs for StayFlow App

Gotchas:
No data appears in the portal

Cause: Not enough time has passed. Spans are sent in batches and Azure takes a short time to index them.

Some requests are missing

Cause: The Azure Monitor exporter records up to 5 traces per second by default. If you send many requests quickly, some are not recorded.

check at below sampling section:

You cannot find your spans in the traces table

Cause: That table holds log messages. Spans are in requests and dependencies.

Fix: Query requests instead.

Sampling: how much is actually recorded (Don’t keep every trace. Keep only some of them.)

The Azure Monitor exporter does not record every request. By default it records up to 5 traces per second. This is called sampling, and it exists to control cost.

You can change the rate:

“Keep approximately 20 traces per second.”

.AddAzureMonitorTraceExporter(options =>
{
    options.TracesPerSecond = 20.0;
})

Or use a percentage instead:

“Keep 10% of the traces.”

.AddAzureMonitorTraceExporter(options =>
{
    options.SamplingRatio = 0.1f;   // record 10% of traces
    options.TracesPerSecond = null;
})

Before choosing a value for production, it helps to know exactly what sampling does and does not affect.

Metrics are never sampled. Performance metrics and custom metrics are always kept, whatever the sampling rate. Your request count, error rate, and response time percentiles stay accurate.

Traces are kept whole. The sampling decision is made once for the whole operation (end to end), not separately for each span. If a trace is recorded, every span in it is recorded. You never see half a trace.

Counts remain correct. Each stored record carries an itemCount value. At 10% sampling, each stored record has an itemCount of 10, because it represents ten original requests.

Then how can we keep all errors?

Recommended settings:

For low traffic — record everything.

.AddAzureMonitorTraceExporter(options =>
{
    options.SamplingRatio = 1.0f;
    options.TracesPerSecond = null;
})

Below roughly ten requests per second, the cost of full recording is small, and complete traces are far more useful. Microsoft’s own guidance is that most small and medium applications do not need sampling at all.

Sampling a low-traffic service is counterproductive. You end up with too few traces to find anything.

Reduce noise before reducing the sampling rate.

Health probes, readiness checks, and static file requests often make up a large share of traffic. Excluding them reduces volume without losing anything you would have looked at. This is a better first step than lowering the sampling rate.

For high traffic — sample, and accept the trade-off.

At high request volumes, full recording becomes expensive. Set a rate you are comfortable with, and rely on metrics for detection.

Checking what is being sampled

This query shows the percentage of each telemetry type that was kept:

union requests, dependencies, exceptions, traces
| where timestamp > ago(1d)
| summarize RetainedPercentage = 100 / avg(itemCount) by bin(timestamp, 1h), itemType

If a value is below 100, that telemetry type is being sampled. It is worth running this once after your first production deployment, particularly to confirm whether your log messages are being sampled as well as your traces.

<< Chapter : 2