Elastic APM processor

Serverless Observability Stack EDOT Collector Planned

The Elastic APM processor enriches OpenTelemetry trace data with Elastic-specific attributes and metadata, ensuring optimal compatibility with Elastic APM UIs and functionality. It bridges the gap between OpenTelemetry's trace format and Elastic's expectations, enabling features like service maps, transaction groups, and enhanced trace visualization.

The processor works together with the Elastic APM connector, which generates pre-aggregated APM metrics from trace data.

The elasticapmprocessor is included by default in EDOT Collector deployments that ingest trace data directly into Elasticsearch. It's not needed when using the Elastic Cloud Managed OTLP Endpoint, as the enrichment happens server-side.

In standalone deployments, the Elastic APM processor is used in both agent and gateway modes:

Agent mode: The processor is part of the default application and traces collection pipeline. It processes trace data received from OpenTelemetry SDKs before exporting to Elasticsearch.

Gateway mode: The processor is part of the Gateway mode pipeline, where it enriches traces received from other collectors running in agent mode before ingesting them into Elasticsearch.

In Kubernetes, the Elastic APM processor runs in the Gateway collectors pipeline when using direct ingestion to Elasticsearch. The Gateway receives traces from DaemonSet collectors and enriches them before writing to Elasticsearch.

For more details about the Kubernetes configuration, refer to Default configuration (Kubernetes).

The Elastic APM processor typically requires minimal configuration. Usually, an empty configuration block is sufficient:

processors:
  elasticapm: {}
		

When combined with the elasticapm connector in a complete pipeline:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

connectors:
  elasticapm: {}

processors:
  batch:
    send_batch_size: 1000
    timeout: 1s
  elasticapm: {}

exporters:
  elasticsearch/otel:
    endpoints:
      - ${ELASTIC_ENDPOINT}
    api_key: ${ELASTIC_API_KEY}
    mapping:
      mode: otel

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, elasticapm]
      exporters: [elasticapm, elasticsearch/otel]

    metrics/aggregated-otel-metrics:
      receivers: [elasticapm]
      processors: []
      exporters: [elasticsearch/otel]
		

The elasticapm connector appears as both an exporter in the traces pipeline (to generate APM metrics) and as a receiver in the metrics pipeline (to forward those metrics to Elasticsearch).

The Elastic APM processor enhances trace data with the following capabilities:

For detailed information about specific attributes and transformations, refer to the contrib elasticapmprocessor documentation.

Follow these recommendations when using the Elastic APM processor:

• Always pair with the elasticapm connector: The processor and connector work together to provide the full Elastic APM experience. The processor enriches traces while the connector generates APM metrics. Include both in your pipeline configuration for complete functionality.

• Place after batching in the pipeline: Configure the Elastic APM processor after the batch processor to ensure optimal throughput. Batching first reduces the number of processing operations.

• Use only for direct Elasticsearch ingestion: If you're using the Elastic Cloud Managed OTLP Endpoint, you don't need the Elastic APM processor, because the endpoint handles enrichment automatically. Using both can cause conflicts or duplicate processing.

• Keep the processor updated: The Elastic APM processor evolves with new Elastic APM features. Keep your EDOT Collector version current to benefit from the latest enhancements and compatibility improvements.

• Configure OTel SDKs with semantic conventions: The processor relies on OpenTelemetry semantic conventions to identify and enrich trace data correctly. Ensure your SDKs follow standard conventions for service name, span attributes, and resource attributes.

Be aware of these constraints and behaviors when using the Elastic APM processor:

• Required for Elastic APM UIs: Without the Elastic APM processor, OpenTelemetry traces will be stored in Elasticsearch but may not render correctly in Elastic APM UIs. Service maps, transaction groups, and other APM-specific visualizations depend on the enrichments this processor provides.

• Not available in contrib OTel Collector: The Elastic APM processor is an Elastic-specific component not included in the standard OpenTelemetry Collector or Collector Contrib distributions. To use it, you must either use EDOT Collector or build a custom collector that includes Elastic's components.

• Minimal configuration options: Unlike some processors, the Elastic APM processor operates with fixed behavior and offers few configuration parameters. While this simplifies setup, it also means you can't customize the enrichment logic.

• Replaces elastictrace processor: If you're upgrading from versions prior to 9.2, be aware that elastictrace is deprecated. Update your configurations to use elasticapm for continued support and new features.

• Contrib component: elasticapmprocessor
• Elastic APM connector
• Default configuration (Standalone)
• Default configuration (Kubernetes)
• Build a custom collector with Elastic components