Deprecation Notes

• Dynamic Instrumentation:
  • The DD_DYNAMIC_INSTRUMENTATION_UPLOAD_FLUSH_INTERVAL environment variable has been deprecated in favor of DD_DYNAMIC_INSTRUMENTATION_UPLOAD_INTERVAL_SECONDS. The old environment variable will be removed in a future major release.
• Tracing
  • The escaped and timestamp arguments in the record_exception method are deprecated and will be removed in version 4.0.0.

New Features

• DSM:
  • Add flag in set_consume_checkpoint() to indicate if DSM checkpoint was manually set.
• Tracing:
  • Adds the environment variable DD_RUNTIME_METRICS_RUNTIME_ID_ENABLED to enable runtime metrics for tagging runtime metrics with the current runtime ID. This is useful for tracking runtime metrics across multiple processes. Previously, this was DD_TRACE_EXPERIMENTAL_RUNTIME_ID_ENABLED.
  • azure_servicebus: Add support for Azure Service Bus producers.
  • azure_functions: Adds messaging span attributes for service bus triggers
  • azure_functions: Add distributed tracing support for Service Bus triggers.
  • ddtrace-api: Adds patching of ddtrace_api.tracer.set_tags to the ddtrace_api integration
  • loguru,structlog,logbook:
    • Enable trace-log correlation for structured loggers by default.
    • Adds support for trace-log correlation via remote configuration. Previously, this functionality was only available for Python’s built-in logging library.
• Dynamic Instrumentation:
  • Code Origins for Spans is now automatically enabled when Dynamic Instrumentation is turned on.
• LLM Observability:
  • Introduces tracing support for bedrock-agent-runtime invoke_agent calls. If bedrock agents tracing is enabled, the internal bedrock traces will be converted and submitted as LLM Observability spans.
  • Adds support for configuring proxy URLs for LLM Observability using the DD_LLMOBS_INSTRUMENTED_PROXY_URLS environment variable or by enabling LLM Observability with the instrumented_proxy_urls argument. Spans sent to a proxy URL will now show up as workflow spans instead of LLM spans.
  • Adds LLM Observability tracing support for the OpenAI Responses endpoint.
  • google_genai: Introduces tracing support for Google's Generative AI SDK for Python's generate_content and generate_content_stream methods.
    See the docs for more information.
  • pydantic_ai: Introduces tracing support for PydanticAI's Agent.iter and Tool.run methods.
    See the docs for more information.
• CI Visibility:
  • This introduces preliminary support to link children pytest-xdist tests (and test suites and test modules) to their parent sessions, instead of being sent as independent sessions.
• Exception Replay:
  • Added in-product enablement support.
• Code Security (IAST):
  • Handle IAST security controls custom validation and sanitization methods. See the Security Controls documentation for more information about this feature.
• Profiling:
  • Add gevent support to the new stack sampling mechanism (stack v2).
• AAP:
  • This introduces the WAF trace tagging feature. This feature enables Datadog’s security research team to efficiently gather API security findings without generating appsec events, which bypass tracer sampling mechanisms. As an example, trace-tagging rules can be used to add attributes to traces with details about the signing algorithm and expiration of a JWT token with the goal of providing authentication-related findings.

Bug Fixes

• Tracing:
  • algoliasearch: Fix for algoliasearch dangling reference.
  • This resolves a TypeError in encoding when truncating a large bytes object.
  • Resolves a sampling issue where agent-based sampling rates were not correctly applied after a process forked or the tracer was reconfigured.
  • Resolves a bug where os.system or subprocess.Popen could return the wrong exception type.
  • This fix resolves an issue in which traced nested generator functions had their execution order subtly changed in a way that affected the stack unwinding sequence during exception handling. The issue was caused by the tracer's use of simple iteration via for v in g: yield v during the wrapping of generator functions where full bidrectional communication with the sub-generator via yield from g was appropriate. See PEP380 for an explanation of how these two generator uses differ.
  • This fix resolves an issue where the @tracer.wrap() decorator failed to preserve return values from generator functions, causing StopIteration.value to be None instead of the actual returned value.
  • rq: Enable parsing distributed tracing metadata in perform job
• AAP:
  • This fix resolves an issue where track_user was generating additional unexpected security activity for customers.
  • This fix resolves an issue where the new ATO SDK track_user was reporting differently email, name, scope and role of the tracked user.
• CI Visibility:
  • This fix resolves an issue where test spans would be left unfinished if the pytest_runtest_protocol hook was overridden in conftest.py, causing the corresponding module and suite to be unfinished as well.
  • This fix resolves an issue where code coverage would not be enabled if ddtrace was enabled via the PYTEST_ADDOPTS environment variable.
• azure_functions:
  • This fix resolves an issue where functions throw an error on loading when the function_name decorator follows another decorator.
• LLM Observability:
  • This fix resolves an issue where modifying bedrock converse streamed chunks caused traced spans to show modified content.
  • Resolved an issue where manual instrumentation would raise DD_LLMOBS_ML_APP missing errors when LLM Observability was disabled.
  • litellm: This fix resolves an out of bounds error when handling streamed responses. This error occurred when the number of choices in a streamed response was not set as a keyword argument.
  • Fixes an issue where the trace ID exported from export_span was incorrect.
  • langchain: Resolved an AttributeError that could occur when async tasks are cancelled during agenerate calls.
• Dynamic Instrumentation:
  • Fixed an incompatibility issue with code origin that caused line probes on the entry point functions to fail to instrument.
  • Fixed an issue with the instrumentation of the first line of an iteration block (e.g. for loops) that could have caused undefined behavior.
  • Fixed an issue that prevented line probes from being instrumented on a line containing just the code try: for CPython 3.11 and later.
  • Fixes an issue with the instrumentation of generators with Python 3.10.
• Code Origin:
  • Fixed a potential memory leak when collecting entry span location information.
• Logging:
  • Ensured that ddtrace.tracer.get_log_correlation_context() returns the expected log correlation attributes (e.g., dd.trace_id, dd.span_id, dd.service instead of trace_id, span_id, service). This change aligns the method's output with the attributes in ddtrace log-correlation docs.
  • Fixed an issue where ddtrace.tracer.get_log_correlation_context() would return the service name of the current span instead of the global service name.

Other Changes

• Tracing:

  • Adds explicit support ranges for all integrations. These support ranges can be used in conjunction with DD_TRACE_SAFE_INSTRUMENTATION_ENABLED=true to enable safer patching of integrations, by ensuring that only compatible versions of an integration are patched.

• Profiling:

  • The native profile exporter is now the default profile exporter, and the legacy Python exporter is removed. The DD_PROFILING_EXPORT_LIBDD_ENABLED configuration variable is removed. As a result of this change, profiling for 32-bit Linux is not supported. Please file an issue or open a support ticket if you need profiling for 32-bit Linux.

• Single Step Instrumentation:

  • Updates library injection guardrails to use safe instrumentation patching feature DD_TRACE_SAFE_INSTRUMENTATION_ENABLED. This change ensures that instrumentation patching is only applied to for supported versions of packages, leaving unsupported package versions unpatched.
  • Additional fields have been added to the telemetry forwarder used during Single Step to surface troubleshooting insights in the Datadog UI.

New Features

• LLM Observability

  • pydantic_ai: Introduces tracing support for PydanticAI's Agent.iter and Tool.run methods.
    See the docs for more information.

• Tracing

  • azure_functions: Adds messaging span attributes for service bus triggers

Bug Fixes

• Dynamic Instrumentation
  • Fixes an issue with the instrumentation of generators with Python 3.10.

Deprecation Notes

• Tracing
  • The escaped and timestamp arguments in the record_exception method are deprecated and will be removed in version 4.0.0.

New Features

• LLMObs:

  • google_genai: Introduces tracing support for Google's Generative AI SDK for Python's generate_content and generate_content_stream methods.
    See the docs for more information.

• Tracing:

  • azure_servicebus: Add support for Azure Service Bus producers.
  • azure_functions: Add distributed tracing support for Service Bus triggers.
  • ddtrace-api: Adds patching of ddtrace_api.tracer.set_tags to the ddtrace_api integration

Bug Fixes

• CI Visibility:

  • This fix resolves an issue where code coverage would not be enabled if ddtrace was enabled via the PYTEST_ADDOPTS environment variable.

• Dynamic Instrumentation:

  • Fixed an issue with the instrumentation of the first line of an iteration block (e.g. for loops) that could have caused undefined behavior.
  • Fixed an issue that prevented line probes from being instrumented on a line containing just the code try: for CPython 3.11 and later.

• Tracing:

  • This fix resolves an issue where the @tracer.wrap() decorator failed to preserve return values from generator functions, causing StopIteration.value to be None instead of the actual returned value.
  • rq: Enable parsing distributed tracing metadata in perform job

• LLMObs:

  • langchain: Resolved an AttributeError that could occur when async tasks are cancelled during agenerate calls.

• Logging:

  • Ensured that ddtrace.tracer.get_log_correlation_context() returns the expected log correlation attributes (e.g., dd.trace_id, dd.span_id, dd.service instead of trace_id, span_id, service). This change aligns the method's output with the attributes in ddtrace log-correlation docs.
  • Fixed an issue where ddtrace.tracer.get_log_correlation_context() would return the service name of the current span instead of the global service name.

Other Changes

• Library Injection:
  • Additional fields have been added to the telemetry forwarder used during Single Step to surface troubleshooting insights in the Datadog UI.

Deprecation Notes

• Dynamic Instrumentation:
  • The DD_DYNAMIC_INSTRUMENTATION_UPLOAD_FLUSH_INTERVAL environment variable has been deprecated in favor of DD_DYNAMIC_INSTRUMENTATION_UPLOAD_INTERVAL_SECONDS. The old environment variable will be removed in a future major release.

New Features

• DSM: Add flag in set_consume_checkpoint() to indicate if DSM
  checkpoint was manually set.
• Tracing:
  • Adds the environment variable DD_RUNTIME_METRICS_RUNTIME_ID_ENABLED to enable runtime metrics for tagging runtime metrics with the current runtime ID. This is useful for tracking runtime metrics across multiple processes. Previously, this was DD_TRACE_EXPERIMENTAL_RUNTIME_ID_ENABLED.
• loguru,structlog,logbook:
  • Enable trace-log correlation for structured loggers by default.
  • Adds support for trace-log correlation via remote configuration. Previously, this functionality was only available for Python’s built-in logging library.
• Dynamic Instrumentation:
  • Code Origins for Spans is now automatically enabled when Dynamic Instrumentation is turned on.
• LLM Observability:
  • Introduces tracing support for bedrock-agent-runtime invoke_agent calls. If bedrock agents tracing is enabled, the internal bedrock traces will be converted and submitted as LLM Observability spans.
  • Adds support for configuring proxy URLs for LLM Observability using the DD_LLMOBS_INSTRUMENTED_PROXY_URLS environment variable or by enabling LLM Observability with the instrumented_proxy_urls argument. Spans sent to a proxy URL will now show up as workflow spans instead of LLM spans.
  • Adds LLM Observability tracing support for the OpenAI Responses endpoint.
• CI Visibility:
  • This introduces preliminary support to link children pytest-xdist tests (and test suites and test modules) to their parent sessions, instead of being sent as independent sessions.
• Exception Replay:
  • Added in-product enablement support.
• Code Security (IAST):
  • Handle IAST security controls custom validation and sanitization methods. See the Security Controls documentation for more information about this feature.
• Profiling:
  • Add gevent support to the new stack sampling mechanism (stack v2).
• AAP:
  • This introduces the WAF trace tagging feature. This feature enables Datadog’s security research team to efficiently gather API security findings without generating appsec events, which bypass tracer sampling mechanisms. As an example, trace-tagging rules can be used to add attributes to traces with details about the signing algorithm and expiration of a JWT token with the goal of providing authentication-related findings.

Bug Fixes

• Tracing:
  • algoliasearch: Fix for algoliasearch dangling reference.
  • This resolves a TypeError in encoding when truncating a large bytes object.
  • Resolves a sampling issue where agent-based sampling rates were not correctly applied after a process forked or the tracer was reconfigured.
  • Resolves a bug where os.system or subprocess.Popen could return the wrong exception type.
  • This fix resolves an issue in which traced nested generator functions had their execution order subtly changed in a way that affected the stack unwinding sequence during exception handling. The issue was caused by the tracer's use of simple iteration via for v in g: yield v during the wrapping of generator functions where full bidrectional communication with the sub-generator via yield from g was appropriate. See PEP380 for an explanation of how these two generator uses differ.
• AAP:
  • This fix resolves an issue where track_user was generating additional unexpected security activity for customers.
  • This fix resolves an issue where the new ATO SDK track_user was reporting differently email, name, scope and role of the tracked user.
• CI Visibility:
  • This fix resolves an issue where test spans would be left unfinished if the pytest_runtest_protocol hook was overridden in conftest.py, causing the corresponding module and suite to be unfinished as well.
• azure_functions:
  • This fix resolves an issue where functions throw an error on loading when the function_name decorator follows another decorator.
• LLM Observability:
  • This fix resolves an issue where modifying bedrock converse streamed chunks caused traced spans to show modified content.
  • Resolved an issue where manual instrumentation would raise DD_LLMOBS_ML_APP missing errors when LLM Observability was disabled.
  • litellm: This fix resolves an out of bounds error when handling streamed responses. This error occurred when the number of choices in a streamed response was not set as a keyword argument.
  • Fixes an issue where the trace ID exported from export_span was incorrect.
• Dynamic Instrumentation:
  • Fixed an incompatibility issue with code origin that caused line probes on the entry point functions to fail to instrument.
• Code Origin:
  • Fixed a potential memory leak when collecting entry span location information.

Other Changes

• Tracing:

  • Adds explicit support ranges for all integrations. These support ranges can be used in conjunction with DD_TRACE_SAFE_INSTRUMENTATION_ENABLED=true to enable safer patching of integrations, by ensuring that only compatible versions of an integration are patched.

• Profiling:

  • The native profile exporter is now the default profile exporter, and the legacy Python exporter is removed. The DD_PROFILING_EXPORT_LIBDD_ENABLED configuration variable is removed. As a result of this change, profiling for 32-bit Linux is not supported. Please file an issue or open a support ticket if you need profiling for 32-bit Linux.

• Single Step Instrumentation:

  • Updates library injection guardrails to use safe instrumentation patching feature DD_TRACE_SAFE_INSTRUMENTATION_ENABLED. This change ensures that instrumentation patching is only applied to for supported versions of packages, leaving unsupported package versions unpatched.

Bug Fixes

• AAP
  • This fix resolves an issue where track_user was generating additional unexpected security activity for customers.
• Tracing
  • This resolves a TypeError in encoding when truncating a large bytes object.
• LLM Observability
  • Resolved an issue where manual instrumentation would raise DD_LLMOBS_ML_APP missing errors when LLM Observability was disabled.
  • litellm: This fix resolves an out of bounds error when handling streamed responses. This error occurred when the number of choices in a streamed response was not set as a keyword argument.
• Dynamic Instrumentation
  • Fixed an incompatibility issue with code origin that caused line probes on the entry point functions to fail to instrument.

Release Notes

Bug Fixes

• Algoliasearch: Fix for potential dangling reference that could trhow an exception. -
• Tracing: Resolves a bug where os.system or subprocess.Popen could return the wrong exception type.
• AAP: This fix resolves an issue where the new ATO SDK track_user was reporting differently email, name, scope and role of the tracked user.

Bug Fixes

• This fix resolves an issue in which traced nested generator functions
  had their execution order subtly changed in a way that affected the
  stack unwinding sequence during exception handling. The issue was
  caused by the tracer's use of simple iteration via
  for v in g: yield v during the wrapping of generator functions where
  full bidrectional communication with the sub-generator via
  yield from g was appropriate. See PEP380 for an explanation of how
  these two generator uses differ.

Bug Fixes

• LLM Observability:
  • Fix an issue where the trace ID exported from export_span was incorrect.

New Features

• DSM
  • Add support for context extraction for SQS -> Lambda messsage events.
• Add support for Python 3.13 on Windows.
• azure_functions
  • Add distributed tracing support for http triggers.
• Code Security (IAST)
  • Unvalidated Redirect detection for Django, Flask and FastAPI applications, which will be displayed on your DataDog Vulnerability Explorer dashboard. See the Application Vulnerability Management documentation for more information about this feature.
• CI Visibility
  • This introduces report links to the pytest plugin. At the end of a test session, ddtrace shows links to the Datadog Test Optimization pages with the test results for the current commit and for the current CI job (provided that the CI environment variables with the current job and pipeline ID are available).
• AAP
  • This introduces the capability for the waf to decide of the sampling priority of the trace in case of a security event.
• litellm
  • Adds APM and LLM Observability tracing support for LiteLLM's synchronous and asynchronous completion and text_completion router methods.
• LLM Observability
  • add processor capability to process span inputs and outputs. See usage documentation [here](https://docs.datadoghq.com/llm_observability/setup/sdk/python/#span-processing).
  • Propagate the ml_app of the most recent LLM Observability span (or the global ml_app) when injecting distributed headers. In distributed services, uses the ml_app from the distributed trace headers.
  • This introduces tracing for system prompts in the OpenAI Agents SDK.
  • This introduces tracing for the content of tool call outputs passed to LLM spans for the OpenAI Agents integration.
• dynamic instrumentation
  • Add support for excluding identifiers from redaction with DD_DYNAMIC_INSTRUMENTATION_REDACTION_EXCLUDED_IDENTIFIERS

Upgrade Notes

• AAP
  • Upgrade the WAF value regex obfuscator.

Deprecation Notes

• tracing
  • Deprecates support for DD_TRACE_GLOBAL_TAGS in favor of DD_TAGS.
  • Removes support for APM Legacy App Analytics. There are still some remnants of App Analytics in the codebase, but they are not functional and output a deprecation warning.
• Deprecates starting the serverless mini agent from the tracer. Use the datadog-serverless-compat package instead.

Bug Fixes

• LLM Observability

  • This fix resolves an issue where error type was being set to the full error message for OpenAI Agents SDK errors, resulting in long error types.
  • This fix resolves an issue where LLM interactions were not being traced when a non-default base URL was provided for the Anthropic, Bedrock, LangChain, Open AI, and Azure Open AI integrations.
  • This fix resolves an issue where parsing token usage from langchain AI message types causes an attribute error.
  • Fixes an issue where using astream_events on a compiled graph would raise a KeyError.

• CI Visibility

  • This fix resolves an issue where the DD_CIVISIBILITY_ITR_ENABLED was not honored properly.
  • This fix resolves an issue where running from a GitHub action triggered on a tag push would cause the branch name to be null, causing errors when fetching Test Optimization settings from the backend.

• crewai

  • This fix resolves an issue where using crewai>=1.120.0 with ddtrace caused a TypeError to be thrown due to empty task contexts.

• tracing

  • Fixes a bug in distributed tracing where pickling ddtrace.trace.Context fails in coroutines. This regression was introduced in v3.7.0.
  • fix issue where Trace Agent sampling rates were not being applied when using a TraceFilter and tracer.configure.
  • Resolves the "sample_before_fork was unregistered without first being registered" warning by removing tracer at_exit hooks from the product protocol, ensuring hooks are registered and unregistered exactly once.
  • Fixes an issue where truncation of span attributes longer than 25000 characters would not consistently count the size of UTF-8 multibyte characters, leading to a unicode string is too large error.

• openai

  • Resolves an issue where streamed completions and chat completions immediately returning None would result in unfinished spans.

• dynamic instrumentation

  • fixes an issue where only module scopes were being uploaded, preventing method probes from being created.

• langgraph

  • Fixes an issue where using astream_events on a compiled graph would cause missing spans.

Bug Fixes

• LLM Observability
  • This fix resolves an issue where LLM interactions
    were not being traced when a non-default base URL was provided for the
    Anthropic, Bedrock, LangChain, Open AI, and Azure Open AI
    integrations.