<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Adharsh H on Medium]]></title>
        <description><![CDATA[Stories by Adharsh H on Medium]]></description>
        <link>https://medium.com/@adharsh.h?source=rss-629e727ebdc0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Y49ig4zTv-05cptL_kJY8g.jpeg</url>
            <title>Stories by Adharsh H on Medium</title>
            <link>https://medium.com/@adharsh.h?source=rss-629e727ebdc0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 18:07:04 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@adharsh.h/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Enabling Gen AI Observability in IBM Instana with Manual OpenTelemetry Instrumentation]]></title>
            <link>https://medium.com/ibm-cloud/enabling-gen-ai-observability-in-ibm-instana-with-manual-opentelemetry-instrumentation-5823920f82d6?source=rss-629e727ebdc0------2</link>
            <guid isPermaLink="false">https://medium.com/p/5823920f82d6</guid>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[instana]]></category>
            <category><![CDATA[genai-observability]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Adharsh H]]></dc:creator>
            <pubDate>Thu, 03 Sep 2026 08:37:54 GMT</pubDate>
            <atom:updated>2026-09-03T08:37:54.548Z</atom:updated>
            <content:encoded><![CDATA[<p>As organizations adopt agentic AI applications, observability becomes a critical challenge. Agentic workflows involve dynamic interactions between multiple agents, LLMs, tools, and decision steps — often with iterative execution paths. Without observability, you cannot determine which agents and tools ran, which model was used, how many tokens were consumed, where latency originated, or why a particular execution path was taken.</p><p>You must capture telemetry across each component involved in the execution to gain visibility into these workflows. OpenTelemetry instrumentation provides a standardized approach for capturing traces, spans, attributes, and metrics from agent activities, LLM calls, and tool invocations. This telemetry gives you the context to trace the complete workflow, monitor performance, troubleshoot issues, and understand agent behavior.</p><p>This article demonstrates how to manually instrument an agentic AI application using OpenTelemetry and export the resulting telemetry to IBM Instana. The instrumentation covers agent activities, LLM interactions, tool calls, token usage, and operation latency, enabling end-to-end GenAI observability and making the agentic workflow traceable, measurable, and easier to optimize.</p><h3><strong>Application overview</strong></h3><p>This section describes an AI-powered research assistant that demonstrates agentic observability. For every user question, the application performs three steps:</p><ol><li>Research the topic using Tavily Search.</li><li>Generate an answer using a Groq-hosted LLM.</li><li>Evaluate the generated answer and decide whether to approve or revise it.</li></ol><p>LangGraph orchestrates the workflow through three specialized agents:</p><ul><li><strong>Research Agent</strong> gathers relevant information using Tavily Search.</li><li><strong>Answer Agent</strong> uses the research results and a Groq-hosted LLM to generate an answer.</li><li><strong>Critic Agent </strong>evaluates the answer for accuracy, completeness, and unsupported claims.</li></ul><p>If the Critic Agent identifies issues, the workflow routes back to the Answer Agent with the critic’s feedback. The process continues until the answer is approved or the maximum revision limit is reached.</p><p>The instrumentation covers agent execution, LLM calls, tool invocations, and interactions between workflow steps. This lets you capture traces, inputs and outputs, execution latency, token usage, and other telemetry needed to understand and monitor end-to-end agent behavior.</p><p>The following diagram illustrates the conceptual workflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/561/1*CJN4ZRGWdhRDxg_stflsvA.png" /></figure><h3><strong>Prerequisites</strong></h3><p>Before you begin, make sure that you have met the following conditions:</p><ul><li>Python 3.11 or later installed</li><li>A Tavily Search API key — create one from the <a href="https://app.tavily.com/home">Tavily dashboard</a></li><li>A Groq API key — create and manage keys from the <a href="https://console.groq.com/">Groq Console</a></li></ul><h3><strong>Set up manual OpenTelemetry instrumentation</strong></h3><p>Rather than relying on automatic instrumentation, this example uses manual OpenTelemetry instrumentation to capture telemetry from the agentic workflow.</p><p>Manual instrumentation gives you greater control over how the workflow appears in telemetry. You can explicitly instrument agent activities, tool invocations, LLM interactions, and GenAI-specific metrics, and you can define the spans and attributes that provide context for each operation.</p><p>The instrumentation in this example covers three main areas:</p><ol><li>Create the tracer and meter to generate traces and record metrics</li><li>Define the OpenTelemetry resource to identify the application generating the telemetry</li><li>Configure trace and metric providers and OTLP exporters to process and export telemetry</li></ol><p><strong>Create the tracer and meter</strong></p><p>The first step creates the OpenTelemetry <strong>Tracer</strong> and <strong>Meter</strong>, which you use throughout the application to generate traces and record metrics.</p><pre>TRACER_NAME = &quot;&lt;tracer-name&gt;&quot;  # For example: genai-agent-observability<br>METER_NAME = &quot;&lt;meter-name&gt;&quot;    # For example: genai-agent-observability<br><br>tracer = trace.get_tracer(TRACER_NAME)<br>meter = metrics.get_meter(METER_NAME)</pre><p>Use the <strong>Tracer</strong> to create spans that represent operations within the agentic workflow:</p><ul><li>Overall workflow execution</li><li>Agent execution</li><li>Tool invocations</li><li>LLM calls</li></ul><p>Use the <strong>Meter</strong> to create metric instruments that record measurements such as:</p><ul><li>LLM token usage</li><li>LLM operation duration</li></ul><p>These instrumentation interfaces are the foundation for capturing telemetry in the subsequent steps.</p><p><strong>Define the OpenTelemetry resource</strong></p><p>Define an OpenTelemetry <strong>Resource</strong> to provide contextual information about the application generating the telemetry.</p><pre>def build_resource() -&gt; Resource:<br>    return Resource.create(<br>        {<br>            &quot;service.name&quot;: &lt;service name&gt;,<br>            &quot;INSTANA_PLUGIN&quot;: &quot;genai&quot;,<br>        }<br>    )</pre><p>The <strong>service.name</strong> attribute identifies the application and associates its telemetry with the corresponding service in IBM Instana. Set <strong>INSTANA_PLUGIN</strong> to <strong>genai</strong> to identify the telemetry as GenAI-related for Instana. Associate the same resource with both the <strong>TracerProvider</strong> and <strong>MeterProvider</strong> to ensure that traces and metrics share a consistent application context throughout the observability pipeline.</p><p><strong>Configure trace providers, metric providers, and OTLP exporters</strong></p><p>After you define the OpenTelemetry resource, configure the <strong>TracerProvider</strong> and <strong>MeterProvider</strong> and connect them to OTLP exporters.</p><p>The <strong>TracerProvider</strong> processes and exports spans. The <strong>MeterProvider</strong> manages metric instruments and exports their measurements. The application supports both OTLP gRPC and OTLP HTTP protocols.</p><h4><strong>Configure the trace provider</strong></h4><p>OpenTelemetry supports multiple OTLP transport protocols, allowing you to configure the exporter according to your use case.</p><pre># For HTTP<br>exporter = OTLPHttpSpanExporter(<br>    endpoint=endpoint,<br>    headers=headers,<br>)<br><br><br># For gRPC<br>exporter = OTLPGrpcSpanExporter(<br>    endpoint=endpoint,<br>    headers=headers,<br>    insecure=not (<br>        endpoint.startswith(&quot;https://&quot;)<br>        or endpoint.startswith(&quot;grpcs://&quot;)<br>    ),<br>)</pre><p>Attach the selected exporter to the <strong>TracerProvider</strong> using a <strong>BatchSpanProcessor</strong>:</p><pre>trace_provider = TracerProvider(<br>    resource=resource,<br>)<br><br>trace_provider.add_span_processor(<br>    BatchSpanProcessor(exporter)<br>)<br><br>trace.set_tracer_provider(trace_provider)</pre><p>The <strong>BatchSpanProcessor</strong> collects spans and exports them in batches rather than sending each span individually.</p><h4><strong>Configure the metrics provider</strong></h4><p>Metrics use a separate OTLP exporter and <strong>MeterProvider</strong>. Configure the exporter based on your selected OTLP transport protocol:</p><pre># For HTTP<br>exporter = OTLPHttpMetricExporter(<br>    endpoint=endpoint,<br>    headers=headers,<br>)<br><br><br># For gRPC<br>exporter = OTLPGrpcMetricExporter(<br>    endpoint=endpoint,<br>    headers=headers,<br>    insecure=not (<br>        endpoint.startswith(&quot;https://&quot;)<br>        or endpoint.startswith(&quot;grpcs://&quot;)<br>    ),<br>)</pre><p>Attach the configured metric exporter to a <strong>PeriodicExportingMetricReader</strong> which periodically collects and exports the recorded metric data through the <strong>MeterProvider</strong>:</p><pre>reader = PeriodicExportingMetricReader(exporter)<br><br>provider = MeterProvider(<br>    resource=resource,<br>    metric_readers=[reader],<br>)<br><br>metrics.set_meter_provider(provider)</pre><p>This configuration separates the trace pipeline from the metric pipeline. Both pipelines share the same OpenTelemetry resource and export telemetry to Instana using OTLP.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*RSc-ECSI8yOBiNgVmLpg7g.png" /></figure><h3><strong>Trace view in Instana</strong></h3><p>Trace View gives you a complete view of an application’s execution path, showing how individual operations connect and how the application progresses from one step to the next. It captures interactions such as LLM calls, database queries, HTTP requests, internal service calls, and other application operations within a single trace. Use Trace View to understand the end-to-end flow and identify where time is spent or where issues occur.</p><p>The following example shows how to manually instrument a research agent using OpenTelemetry. The agent is represented by a parent span, while the tool invocation and LLM call are represented by child spans. Because these spans are created within the agent’s span context, OpenTelemetry automatically maintains the parent-child relationship in the trace.</p><p>The example instruments a research agent that uses a Tavily search tool and a Groq LLM:</p><pre>search_tool = TavilySearchResults(<br>    max_results=TAVILY_MAX_RESULTS,<br>)<br><br><br>def research_agent(question):<br><br>    # Research Agent span<br>    with tracer.start_as_current_span(<br>        &quot;research_agent.task&quot;<br>    ) as agent_span:<br><br>        agent_span.set_attribute(<br>            &quot;gen_ai.operation.name&quot;,<br>            &quot;execute_task&quot;,<br>        )<br><br>        agent_span.set_attribute(<br>            &quot;gen_ai.task.name&quot;,<br>            &quot;research_agent&quot;,<br>        )<br><br>        # Tool invocation<br>        with tracer.start_as_current_span(<br>            &quot;tavily.search.tool&quot;<br>        ) as tool_span:<br><br>            tool_span.set_attribute(<br>                &quot;gen_ai.tool.name&quot;,<br>                &quot;tavily_search&quot;,<br>            )<br><br>            tool_span.set_attribute(<br>                &quot;gen_ai.tool.type&quot;,<br>                &quot;search&quot;,<br>            )<br><br>            tool_span.set_attribute(<br>                &quot;gen_ai.tool.call.arguments&quot;,<br>                question,<br>            )<br><br>            results = search_tool.invoke(question)<br><br>            tool_span.set_attribute(<br>                &quot;gen_ai.tool.call.result&quot;,<br>                str(results),<br>            )<br><br>        agent_span.set_attribute(<br>            &quot;gen_ai.task.status&quot;,<br>            &quot;completed&quot;,<br>        )<br><br>    return results<br><br><br>def answer_agent(question, research_results):<br><br>    # Answer Agent span<br>    with tracer.start_as_current_span(<br>        &quot;answer_agent.task&quot;<br>    ) as agent_span:<br><br>        agent_span.set_attribute(<br>            &quot;gen_ai.operation.name&quot;,<br>            &quot;execute_task&quot;,<br>        )<br><br>        agent_span.set_attribute(<br>            &quot;gen_ai.task.name&quot;,<br>            &quot;answer_agent&quot;,<br>        )<br><br>        # LLM invocation<br>        with tracer.start_as_current_span(<br>            &quot;ChatGroq.chat&quot;<br>        ) as llm_span:<br><br>            llm_span.set_attribute(<br>                &quot;gen_ai.operation.name&quot;,<br>                &quot;chat&quot;,<br>            )<br><br>            llm_span.set_attribute(<br>                &quot;gen_ai.provider.name&quot;,<br>                &quot;groq&quot;,<br>            )<br><br>            llm_span.set_attribute(<br>                &quot;gen_ai.request.model&quot;,<br>                GROQ_MODEL,<br>            )<br><br>            llm_span.set_attribute(<br>                &quot;gen_ai.input.messages&quot;,<br>                str(research_results),<br>            )<br><br>            response = llm.invoke(<br>                f&quot;&quot;&quot;<br>                Question: {question}<br><br>                Research results:<br>                {research_results}<br>                &quot;&quot;&quot;<br>            )<br><br>            llm_span.set_attribute(<br>                &quot;gen_ai.response.model&quot;,<br>                response.response_metadata.get(<br>                    &quot;model_name&quot;,<br>                    GROQ_MODEL,<br>                ),<br>            )<br><br>            llm_span.set_attribute(<br>                &quot;gen_ai.output.messages&quot;,<br>                str(response.content),<br>            )<br><br>        agent_span.set_attribute(<br>            &quot;gen_ai.task.status&quot;,<br>            &quot;completed&quot;,<br>        )<br><br>        return response</pre><p><strong>tracer.start_as_current_span()</strong> manually instruments each operation, and <strong>set_attribute()</strong> adds the required span attributes. The <strong>research_agent.task</strong> span represents the Research Agent execution, with the <strong>tavily.search.tool</strong> child span capturing the tool invocation — including its name, type, input, and result. The <strong>answer_agent.task</strong> span represents the Answer Agent execution, with the <strong>ChatGroq.chat</strong> child span capturing the LLM operation and relevant GenAI attributes such as the operation, provider, and model.</p><p>The following screenshot shows the trace view of the instrumented application, displaying the captured agent task, tool calls, and LLM interactions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rpl_vAAVXatVecB-oJR-rA.png" /></figure><h3><strong>Task hierarchy view in Instana</strong></h3><p>When a trace contains GenAI operations, Instana Task View provides a focused view of the GenAI and agentic interactions within that trace. Use Task View to understand:</p><ul><li>LLM calls</li><li>Agent tasks</li><li>Tool interactions</li></ul><p>For an agentic application, Task View answers the question: “What happened during the GenAI and agentic interactions?”</p><p><strong>Note: </strong>Task View currently supports agentic applications built with LangChain, LangGraph, and CrewAI.</p><p>Task View supports the following commonly used LLM operation patterns:</p><ul><li><strong>chat.completions.create</strong></li><li><strong>WatsonxLLM.completion</strong></li><li><strong>watsonx.generate</strong></li><li>Methods that end with <strong>.chat</strong>, such as <strong>ChatWatsonx.chat</strong>, <strong>openai.chat</strong>, and <strong>ChatOpenAI.chat</strong></li></ul><p>The following screenshot shows an example of Instana Task View displaying GenAI and agentic interactions within a trace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sw5OdaGrbSo5XBvQ6Ra19w.png" /></figure><p><strong>Capturing LLM input and output in Task View</strong></p><p>To capture LLM input and output messages in Task View, include the following attributes in the corresponding span:</p><ul><li><strong>gen_ai.input.messages</strong></li><li><strong>gen_ai.output.messages</strong></li></ul><p>These attributes allow Task View to display the messages exchanged during the LLM operation, giving you visibility into the prompt sent to the model and the response the model generates.</p><p>Additionally, include the <strong>gen_ai.provider.name</strong> attribute in the span. This attribute identifies the span as a GenAI operation and allows Instana to recognize and process it as a GenAI trace.</p><p><strong>Capturing tool calls in Task View</strong></p><p>To capture tool-related operations in Task View, end the span name with <strong>.tool</strong>. For example: <strong>search.tool</strong>.</p><p>To display the tool input arguments and output result in Task View, add the following attributes to the span:</p><ul><li><strong>gen_ai.tool.call.arguments</strong></li><li><strong>gen_ai.tool.call.result.</strong></li></ul><p><strong>Capturing agent task spans</strong></p><p>For agentic applications, you can represent individual agent executions as task spans. These spans allow Task View to identify and display the work each agent performs within the overall workflow.</p><p>Include the following GenAI attributes in each agent task span:</p><ul><li><strong>gen_ai.operation.name — </strong>set to <strong>execute_task</strong> to identify the operation as an agent task</li><li><strong>gen_ai.provider.name</strong> — identifies the framework or provider executing the task</li><li><strong>gen_ai.task.name</strong> — identifies the specific agent executing the task</li></ul><h3><strong>LLM metrics in the GenAI dashboard</strong></h3><p>Tracing shows what happened and in what order. Metrics show the performance and consumption characteristics of LLM operations.</p><p>Instana presents the following LLM metrics in the GenAI dashboard:</p><ul><li><strong>Token usage</strong> — the number of input and output tokens consumed</li><li><strong>Cost</strong> — the calculated input, output, and total LLM cost</li><li><strong>LLM calls</strong> — the number of LLM operations</li><li><strong>LLM latency</strong> — the time taken to complete LLM operations</li></ul><p>You can analyze metrics by model or GenAI service to understand LLM consumption and performance.</p><p>For cost calculation, Instana uses the recorded input and output token usage together with model-specific prices you configure in the <strong>Pricing configuration</strong> tab. This produces input cost, output cost, and total cost.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xAzWqBLRhrpaaqwn3fxFcg.png" /></figure><p>To populate these metrics in the GenAI dashboard, your application must record the appropriate OpenTelemetry GenAI metrics and attributes.</p><p><strong>Define GenAI metrics</strong></p><p>Instana uses the following OpenTelemetry metrics to monitor LLM consumption and performance:</p><ul><li><strong>gen_ai.client.token.usage</strong> — measures the number of input and output tokens consumed during an LLM operation</li><li><strong>gen_ai.client.operation.duration</strong> — measures the time taken to complete an LLM operation</li></ul><p>Define both metrics as OpenTelemetry histograms:</p><pre>TOKEN_USAGE = meter.create_histogram(<br>    name=&quot;gen_ai.client.token.usage&quot;,<br>    description=&quot;Number of tokens used in a GenAI operation&quot;,<br>    unit=&quot;{token}&quot;,<br>)<br><br>OPERATION_DURATION = meter.create_histogram(<br>    name=&quot;gen_ai.client.operation.duration&quot;,<br>    description=&quot;Duration of a GenAI operation&quot;,<br>    unit=&quot;s&quot;,<br>)</pre><p><strong>Record GenAI metrics with attributes</strong></p><p>Recording only the metric value provides limited context. For example, a token usage value shows how many tokens were consumed, but not which provider or model was responsible.</p><p>Record GenAI attributes alongside each metric measurement to provide this context:</p><pre>attributes = {<br>    &quot;gen_ai.provider.name&quot;: provider,<br>    &quot;gen_ai.response.model&quot;: model,<br>}</pre><p>These attributes allow you to analyze metrics by GenAI provider and response model. For example, you can distinguish token usage from two different models even when they use the same metric.</p><p>For token usage, use the <strong>gen_ai.token.type</strong> attribute to to distinguish between input and output tokens.</p><pre>TOKEN_USAGE.record(<br>    input_tokens,<br>    attributes={<br>        **attributes,<br>        &quot;gen_ai.token.type&quot;: &quot;input&quot;,<br>    },<br>)<br><br>TOKEN_USAGE.record(<br>    output_tokens,<br>    attributes={<br>        **attributes,<br>        &quot;gen_ai.token.type&quot;: &quot;output&quot;,<br>    },<br>)</pre><p>This lets Instana differentiate between the tokens sent to the model and the tokens the model generates.</p><p>For LLM operation duration, apply the same provider and response model attributes:</p><pre>start_time = perf_counter()<br>response = llm.invoke(messages)<br><br>duration = perf_counter() - start_time<br><br>OPERATION_DURATION.record(<br>    duration,<br>    attributes=attributes,<br>)</pre><h3><strong>Conclusion</strong></h3><p>Manual OpenTelemetry instrumentation gives you a flexible and precise way to enable GenAI observability in agentic AI applications. By capturing agent tasks, LLM calls, tool interactions, token usage, and latency, and exporting this telemetry to IBM Instana, you gain end-to-end visibility through Trace View and Task View. This helps you understand agent execution flows, monitor LLM performance, troubleshoot issues, and optimize GenAI applications.</p><h3><strong>References</strong></h3><p>The complete instrumented application is available in the <a href="https://github.com/adharshctr/Manual-instrumentation/tree/main/python-langraph-app">LangGraph Research Agent repository</a>. Follow the README for instructions on configuring the environment, installing the required dependencies, and running the application.</p><p>For more information about Instana backend OTLP endpoints for different SaaS environments, see <a href="https://www.ibm.com/docs/en/SSV8Z4_1.0.319/src/pages/ecosystem/opentelemetry/instana-backend.html#endpoints-of-the-instana-backend-otlp-acceptor">Endpoints of the Instana backend OTLP acceptor</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5823920f82d6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ibm-cloud/enabling-gen-ai-observability-in-ibm-instana-with-manual-opentelemetry-instrumentation-5823920f82d6">Enabling Gen AI Observability in IBM Instana with Manual OpenTelemetry Instrumentation</a> was originally published in <a href="https://medium.com/ibm-cloud">IBM Cloud</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Elevating AI observability: Monitoring OpenAI agents with Instana]]></title>
            <link>https://medium.com/ibm-cloud/elevating-ai-observability-monitoring-openai-agents-with-instana-2f704d022c30?source=rss-629e727ebdc0------2</link>
            <guid isPermaLink="false">https://medium.com/p/2f704d022c30</guid>
            <category><![CDATA[opentelemetry]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[instana]]></category>
            <dc:creator><![CDATA[Adharsh H]]></dc:creator>
            <pubDate>Tue, 05 Aug 2025 15:09:33 GMT</pubDate>
            <atom:updated>2025-08-05T15:09:33.759Z</atom:updated>
            <content:encoded><![CDATA[<p>In today’s fast-paced world of AI-powered applications, maintaining the performance, reliability, and cost-effectiveness of your OpenAI agents is critical. As these intelligent agents independently execute tasks, make decisions, and engage with users or systems, robust observability becomes essential. This is where an advanced observability platform, such as Instana proves invaluable. Instana delivers in-depth monitoring for both your OpenAI agents and the large language models (LLMs) they rely on.</p><p><strong>The challenge of observing AI agents</strong></p><p>Observing AI agents presents unique challenges that traditional monitoring tools are not equipped to handle effectively:</p><ul><li><strong>Dynamic and non-deterministic behavior: </strong>Unlike conventional applications, AI agents often make autonomous decisions that can vary with the same input. This non-determinism renders static monitoring approaches ineffective, as they fail to capture the fluid and adaptive nature of agent behavior.</li><li><strong>The black box nature of LLMs: </strong>Large language models (LLMs) are inherently opaque, making it difficult to understand the reasoning behind a specific response. In agent workflows, this lack of transparency poses a major challenge for debugging, auditing, and ensuring accountability.</li><li><strong>Complex chains of reasoning and tool use: </strong>AI agents typically perform multi-step tasks that involve reasoning, calling external tools or APIs, and sometimes coordinating with other agents. Monitoring and tracing these sequences end-to-end are essential for understanding how the agent reaches conclusions and where failures may occur.</li><li><strong>Cost visibility and management: </strong>Each interaction with an LLM — especially through APIs like OpenAI — incurs a cost. Without fine-grained visibility into usage patterns and token consumption, it becomes difficult to control spending and optimize for cost-efficiency.</li><li><strong>Ensuring quality, safety, and compliance: </strong>Beyond technical performance, it’s critical to ensure that the AI agent delivers accurate, safe, and unbiased results. This is crucial in regulated or high-stakes domains where errors can have significant consequences.</li></ul><p>These challenges demonstrate the need for an automated observability tool tailored to the specific demands of LLM-powered systems.</p><p><strong>Instana for OpenAI agent monitoring</strong></p><p>Instana solves the challenges of monitoring AI agents by offering a comprehensive, automated observability platform tailored to the unique demands of LLM-powered systems. Instana is an enterprise-grade, AI-powered observability platform designed to deliver real-time insights into complex, cloud-native environments. It automatically discovers services, traces transactions, and monitors performance across the entire technology stack with minimal manual intervention.</p><p>Here is how it enables effective monitoring and optimization of your OpenAI agents:</p><ol><li><strong>Seamless OpenTelemetry integration with Traceloop: </strong>Instana natively supports OpenTelemetry (OTel), the open standard for telemetry data. By using Traceloop — an OTel-native SDK purpose-built for OpenAI agent instrumentation — you can generate standardized traces, metrics, and logs directly from your AI applications. Instana ingests this data effortlessly, providing a unified, real-time view of your agent workflows and the underlying LLM interactions.</li><li><strong>LLM-specific metrics:</strong> Beyond generic application metrics, Instana focuses on the unique performance indicators crucial for AI agents and LLMs:<br><strong>a. Token usage:</strong> Monitor prompt, completion, and total token counts to directly track consumption and manage costs.<br>b. <strong>Latency:</strong> Measure the end-to-end response time of your agents, including LLM inference time and tool execution.<br>c. <strong>Error rates:</strong> Quickly identify and diagnose failures in agent runs, function calls, and LLM interactions.</li><li><strong>End-to-end distributed tracing:</strong> Instana’s powerful distributed tracing capabilities provide complete visibility into the execution of OpenAI agents. You can trace the journey from the user’s initial request, through the agent’s internal decision-making and tool usage, to the final response. This end-to-end view is crucial for identifying “soft failures,” where an agent appears to function correctly but returns inaccurate or less-than-ideal outcomes.</li><li><strong>Customizable alerts and events:</strong> Instana allows you to define events and alerting conditions based on the behavior of your OpenAI agents. Rather than relying solely on automatic anomaly detection, you can set specific thresholds — for example, sudden spikes in latency, increased token usage, or higher error rates. When these thresholds are crossed, Instana triggers alerts, helping teams respond quickly and maintain reliable AI performance.</li><li><strong>Custom dashboards for focused insights:</strong> Instana allows you to build custom dashboards to visualize the metrics most important to your specific OpenAI agent use cases. By visualizing these metrics in a customizable and interactive interface, teams can quickly detect anomalies, optimize model usage, and make informed decisions to improve reliability and efficiency across AI-driven workflows.</li></ol><p>Together, these capabilities position Instana as a purpose-built solution for monitoring the complex and dynamic behavior of OpenAI agents. It converts raw observability data into actionable insights, closing visibility gaps and removing the manual effort usually needed to monitor LLM-powered systems.</p><p><strong>Get started with Instana and OpenAI agents</strong></p><p>To get started with Instana and OpenAI agents, follow these steps:</p><ol><li><strong>Install OpenAI agents</strong>: To get started with building and monitoring OpenAI agents, install the official openai-agents package. You can install it using pip:</li></ol><pre>pip install openai-agents==0.0.19</pre><p>2. <strong>Deploy the Instana agent: </strong>Deploy the Instana agent to your host to collect telemetry data. This enables real-time monitoring and observability for your OpenAI agent applications. To deploy the Instana agent, see <a href="https://www.ibm.com/docs/en/instana-observability/1.0.299?topic=installing-instana-agents">Deploy Instana agent</a></p><p>3. <strong>Installation of OTel LLM data collector:</strong> Install and set up the OTel LLM Data Collector to enable metrics collection. For installation and configuration instructions, refer to the official guide on <a href="https://github.com/instana/otel-dc/blob/main/llm/README.md">OTel Data Collector for LLM</a>.</p><p>4. <strong>Instrument your application with Traceloop:</strong> Leverage the Traceloop SDK to seamlessly instrument your OpenAI agent application. With OpenTelemetry embedded, Traceloop simplifies the process of capturing traces and metrics from OpenAI API calls and agent frameworks. To install Traceloop, use the following command:</p><pre>pip install traceloop-sdk==0.41.0</pre><p>5. <strong>Configure environment variables for Instana export: </strong>Configure the environment variables where your OpenAI agents application runs to enable the export of traces, logs, and metrics to Instana. To view the required environment variables, see<a href="https://www.ibm.com/docs/en/instana-observability/1.0.299?topic=started-traceloop-instrumentation#configuring-the-environment"> Configuring the environment</a>.</p><p><strong>Monitoring OpenAI agents with Instana</strong></p><p>In this section, we’ll walk through a simple example app — the Travel Itinerary Assistant — to show how you can use OpenAI Agents to build a smart, multi-step workflow for planning a trip. Once you’ve created the app, you can see how metrics and traces collected during execution are visualized in Instana, offering deep insight into agent behavior and LLM-driven workflows.</p><p>The Travel Itinerary Assistant is an OpenAI agent-powered application that generates personalized travel plans from natural language input. It uses a modular, multi-agent workflow where each agent handles a specific task — from extracting destinations and planning routes to estimating travel times and summarizing the itinerary. This step-by-step design makes the planning process both accurate and conversational. The application is integrated with the Traceloop SDK for observability, enabling real-time tracing and debugging.</p><pre>import asyncio<br>from agents import Agent, Runner, function_tool<br>from traceloop.sdk import Traceloop<br>from traceloop.sdk.decorators import workflow<br><br># Initialize Traceloop for observability<br>Traceloop.init(app_name=&quot;Travel_Itinerary_Assistant&quot;, disable_batch=True)<br><br>@function_tool(<br>    name_override=&quot;calculate_travel_time&quot;,<br>    description_override=&quot;Estimate travel time based on cities and transport type.&quot;<br>)<br>def calculate_travel_time(source: str, destination: str, mode: str) -&gt; str:<br>    if mode.lower() == &quot;flight&quot;:<br>        return f&quot;Approx. 2 hours from {source} to {destination} by air.&quot;<br>    elif mode.lower() == &quot;train&quot;:<br>        return f&quot;Approx. 6 hours from {source} to {destination} by train.&quot;<br>    else:<br>        return f&quot;Approx. 10 hours from {source} to {destination} by road.&quot;<br><br>useModel = &quot;gpt-4.1&quot;<br><br># 1. Destination Extractor<br>Destination_Extractor_Agent = Agent(<br>    name=&quot;Destination Extractor&quot;,<br>    instructions=&quot;&quot;&quot;<br>    You are a helpful assistant that extracts destination cities from the user&#39;s travel request.<br>    Only return a comma-separated list of places they want to visit.<br>    &quot;&quot;&quot;,<br>    model=useModel,<br>)<br><br># 2. Planner Agent<br>Planner_Agent = Agent(<br>    name=&quot;Planner Agent&quot;,<br>    instructions=&quot;&quot;&quot;<br>    You are a smart travel planner. Given a list of destinations, return an ideal travel order and suggest how many days to spend in each place.<br>    Keep it realistic and concise.<br>    &quot;&quot;&quot;,<br>    model=useModel,<br>)<br><br># 3. Travel Time Estimator<br>Time_Estimator_Agent = Agent(<br>    name=&quot;Travel Time Estimator&quot;,<br>    instructions=&quot;&quot;&quot;<br>    You are a travel time estimator. Use the &#39;calculate_travel_time&#39; tool to estimate how long it takes to travel between the destinations.<br>    Assume travel is by flight unless stated otherwise.<br>    &quot;&quot;&quot;,<br>    tools=[calculate_travel_time],<br>    model=useModel,<br>)<br><br># 4. Final Recommender<br>Recommendation_Agent = Agent(<br>    name=&quot;Recommendation Agent&quot;,<br>    instructions=&quot;&quot;&quot;<br>    You are a friendly travel assistant. Based on the itinerary and travel times, summarize the full travel plan for the user.<br>    Keep it organized and engaging.<br>    &quot;&quot;&quot;,<br>    model=useModel,<br>)<br><br><br><br>@workflow(name=&quot;travel_itinerary_builder&quot;)<br>async def main():<br>    user_input = &quot;&quot;&quot;I&#39;m planning a trip next month. I want to visit New York, San Francisco, and Las Vegas.<br>    I&#39;d like to spend more time in San Francisco and travel by flights where possible.&quot;&quot;&quot;<br><br>    # Step 1: Extract destinations<br>    result = await Runner.run(<br>        starting_agent=Destination_Extractor_Agent,<br>        input=f&quot;Extract all cities mentioned: {user_input}&quot;,<br>    )<br>    destinations = result.final_output<br>    print(&quot;\n Destinations:\n&quot;, destinations)<br><br>    # Step 2: Plan order and duration<br>    result = await Runner.run(<br>        starting_agent=Planner_Agent,<br>        input=f&quot;Plan the travel route and suggest stay durations for: {destinations}&quot;,<br>    )<br>    itinerary = result.final_output<br>    print(&quot;\n Itinerary Plan:\n&quot;, itinerary)<br><br>    # Step 3: Estimate travel time<br>    result = await Runner.run(<br>        starting_agent=Time_Estimator_Agent,<br>        input=f&quot;Estimate time to travel between: {destinations}&quot;,<br>    )<br>    travel_times = result.final_output<br>    print(&quot;\n Travel Times:\n&quot;, travel_times)<br><br>    # Step 4: Final recommendation<br>    result = await Runner.run(<br>        starting_agent=Recommendation_Agent,<br>        input=f&quot;User wants to travel to {destinations}. Here is the plan: {itinerary}. Travel time info: {travel_times}. Summarize everything.&quot;,<br>    )<br>    print(&quot;\nFinal Recommendation:\n&quot;, result.final_output)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    asyncio.run(main())</pre><p>The following image displays the outputs generated by the Travel Itinerary Assistant application after processing the user’s input. It visually represents the results from each stage of the agent-driven workflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CvDOZASSao6m3RLNPXcAXw.png" /><figcaption>Output generated by OpenAI agents</figcaption></figure><p>Here is an in-depth view of the traces collected during the execution of the user’s input. These traces offer a sequential breakdown of how each agent handled the input, triggered relevant tools, and produced corresponding outputs, providing clear insights into the internal flow and behavior of the system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s2N9r6HpUy0P5kdaaUYf-g.png" /><figcaption>Detailed traces collected from the application</figcaption></figure><p>The following image shows metrics collected from the large language model (LLM) used by agents in the Travel Itinerary Assistant application. These metrics provide valuable insights into the model’s performance, including token usage, response time, number of requests, and overall latency. Monitoring these metrics helps evaluate the efficiency and cost-effectiveness of the application, identify performance bottlenecks, and ensure optimal resource utilization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-LaE_M73UCHc7ICZ-jxsjQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zp7H9xpelLRuFQQaUtXvSA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RVtcOurfsk-_aGaB6WgJOw.png" /><figcaption>The metrics dashboard showing the metrics collected from LLMs used in the OpenAI agent powered application</figcaption></figure><p><strong>Conclusion</strong></p><p>Thus, monitoring OpenAI agents with Instana delivers real-time, in-depth observability required to navigate the complexity of AI-driven systems. With its full-stack monitoring capabilities, seamless OpenTelemetry integration, and actionable insights, Instana offers a clear window into the performance of your AI workloads. This enables teams to swiftly detect and address issues, fine-tune performance, manage costs, and enhance the overall effectiveness of your intelligent agents. Leveraging Instana’s observability ensures you get the most out of your OpenAI agent deployments.</p><p><strong>References</strong></p><ul><li><a href="https://medium.com/ibm-cloud/enhancing-ai-agent-observability-monitoring-crewai-using-watsonx-with-instana-bffadb968180"><strong>Enhancing AI agent observability: Monitoring CrewAI using WatsonX with Instana</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2f704d022c30" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ibm-cloud/elevating-ai-observability-monitoring-openai-agents-with-instana-2f704d022c30">Elevating AI observability: Monitoring OpenAI agents with Instana</a> was originally published in <a href="https://medium.com/ibm-cloud">IBM Cloud</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Enhancing AI Agent Observability: Monitoring CrewAI using WatsonX with Instana]]></title>
            <link>https://medium.com/ibm-cloud/enhancing-ai-agent-observability-monitoring-crewai-using-watsonx-with-instana-bffadb968180?source=rss-629e727ebdc0------2</link>
            <guid isPermaLink="false">https://medium.com/p/bffadb968180</guid>
            <category><![CDATA[watsonx]]></category>
            <category><![CDATA[crew-ai]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[instana]]></category>
            <dc:creator><![CDATA[Adharsh H]]></dc:creator>
            <pubDate>Tue, 25 Feb 2025 17:44:02 GMT</pubDate>
            <atom:updated>2025-03-12T06:55:56.413Z</atom:updated>
            <content:encoded><![CDATA[<p>In the rapidly evolving landscape of AI-driven automation, monitoring and observability have become critical to ensure the efficiency, reliability, and performance of AI agents. CrewAI, a powerful framework for orchestrating AI agents, offers seamless integrations with various LLMs, including IBM WatsonX. However, to gain deep insights into the behavior, performance, and cost-effectiveness of these agents, robust observability solutions are required.</p><p>This blog explores how Instana, an enterprise-grade observability platform, can be leveraged to monitor CrewAI agents. Traceloop is integrated with Instana for AI agent tracing, capturing essential execution flows, and WatsonX<strong> </strong>models for intelligent decision-making. By combining these technologies, you can track real-time token usage, latency, and costs while ensuring smooth AI-driven workflows.</p><p>With this blog you will learn how to:</p><ul><li><strong>Instrument CrewAI agents with Traceloop</strong> for detailed execution traces</li><li><strong>Monitor LLM token usage and cost metrics</strong> from WatsonX by using Instana</li><li><strong>Gain actionable insights</strong> into AI agent workflows, bottlenecks, and optimizations</li><li><strong>Generate Events and Alerts</strong> for the metrics collected from the CrewAI application</li></ul><h3><strong>Architecture</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BkpvLFVjVs2AAjVHN_FoUw.png" /></figure><p>This architecture enables comprehensive monitoring of CrewAI applications. The CrewAI application is instrumented with Traceloop, which captures traces, logs, and metrics. These telemetry data points are then sent either directly to Instana’s OTLP acceptor (agentless mode) or through an Instana agent (agent mode). Additionally, LLM-related metrics, such as token usage, cost, and request counts, are collected by LLM Otel DC for detailed AI observability.</p><p>Instana processes this data and presents real-time insights in its UI, allowing teams to analyze the performance, behavior, and cost of AI agents effectively. This approach ensures scalability and flexibility, supporting both <strong>a</strong>gent-based and agentless monitoring while providing a clear breakdown of CrewAI execution and LLM interactions. By integrating Instana with Traceloop, developers can achieve end-to-end observability for AI-driven workflows, making it easier to optimize performance and detect anomalies.</p><p><strong>Prerequisites</strong></p><p>a. Ensure that the required packages are installed. If not, install them by running the following command:</p><pre>pip3 install crewai==0.100.0 crewai-tools==0.33.0 traceloop.sdk==0.38.0</pre><p>b. Install the LLM Data Collector, complete the necessary configuration, and keep it running. For installation instructions, see <a href="https://github.com/instana/otel-dc/blob/main/llm/README.md#odcl-otel-data-collector-for-llm">ODCL (OTel Data Collector for LLM)</a>.</p><p>c. Export the following credentials to access the WatsonX models and SerperDevTool:</p><pre>export WATSONX_URL=&quot;&lt;watsonx-url&gt;&quot;<br>export WATSONX_PROJECT_ID=&quot;&lt;watsonx-project-id&gt;&quot;<br>export WATSONX_API_KEY=&quot;&lt;watsonx-api-key&gt;&quot;<br>export SERPER_API_KEY=&quot;&lt;serper-api-key&gt;&quot;</pre><p>d. Export traces and metrics to Instana by using the following export configuration:</p><pre>#For Agent mode<br>export TRACELOOP_BASE_URL=&lt;instana-agent-host&gt;:4317<br>export TRACELOOP_LOGGING_ENABLED=true<br>export TRACELOOP_LOGGING_ENDPOINT=$TRACELOOP_BASE_URL<br>export TRACELOOP_METRICS_ENABLED=true<br>export TRACELOOP_METRICS_ENDPOINT=&lt;otel-dc-llm-host&gt;:8000<br>export OTEL_EXPORTER_OTLP_METRICS_INSECURE=true<br>export OTEL_METRIC_EXPORT_INTERVAL=10000<br>export OTEL_EXPORTER_OTLP_INSECURE=true<br><br>#For Agentless mode<br>export TRACELOOP_BASE_URL=&lt;instana-otlp-endpoint&gt;:4317<br>export TRACELOOP_HEADERS=&quot;x-instana-key=&lt;agent-key&gt;,x-instana-host=&lt;instana-host&gt;&quot;<br>export TRACELOOP_LOGGING_ENABLED=true<br>export TRACELOOP_LOGGING_ENDPOINT=$TRACELOOP_BASE_URL<br>export TRACELOOP_METRICS_ENABLED=true<br>export TRACELOOP_METRICS_ENDPOINT=&lt;otel-dc-llm-host&gt;:8000<br>export OTEL_EXPORTER_OTLP_METRICS_INSECURE=true<br>export OTEL_METRIC_EXPORT_INTERVAL=10000<br>export OTEL_EXPORTER_OTLP_INSECURE=false</pre><p>The following example shows a sample CrewAI application code. You can try it out and verify that traces and metrics are collected:</p><pre>from crewai import Agent, Task, Crew, Process<br>from crewai.llm import LLM<br>from crewai_tools import SerperDevTool<br><br>from pydantic import BaseModel<br><br>from traceloop.sdk import Traceloop<br>from traceloop.sdk.decorators import workflow<br><br>Traceloop.init(app_name=&quot;WatsonX_Crew_App&quot;)<br><br>search_tool = SerperDevTool()<br><br>llm1 = LLM(model=&quot;watsonx/meta-llama/llama-3-1-70b-instruct&quot;)<br>llm2 = LLM(model=&quot;watsonx/ibm/granite-34b-code-instruct&quot;)<br><br>class JsonOutput(BaseModel):<br>    agent: str<br>    expected_output: str<br>    total_tokens: int<br>    prompt_tokens: int<br>    completion_tokens: int<br>    successful_requests: int<br><br><br># Define the Agents<br>data_analyst = Agent(<br>    role=&quot;Data Analyst&quot;,<br>    goal=&quot;Analyze market data and extract key insights.&quot;,<br>    backstory=&quot;An AI-powered expert in data processing, trend analysis, and forecasting.&quot;,<br>    verbose=True,<br>    allow_delegation=False,<br>    llm=llm1,<br>    tools=[search_tool],<br>)<br><br>strategy_advisor = Agent(<br>    role=&quot;Strategy Advisor&quot;,<br>    goal=&quot;Develop strategic business recommendations based on data insights.&quot;,<br>    backstory=&quot;An AI-driven consultant that translates market trends into actionable strategies.&quot;,<br>    verbose=True,<br>    allow_delegation=False,<br>    llm=llm2,<br>    tools=[search_tool],<br>)<br><br># Define the Tasks<br>task1 = Task(<br>    description=&quot;Analyze the following market data and summarize key insights:\t{market_data}&quot;,<br>    expected_output=&quot;summarized insights in a structured format.&quot;,<br>    agent=data_analyst,<br>)<br><br>task2 = Task(<br>    description=&quot;Develop a business strategy based on the analyzed market data.&quot;,<br>    expected_output=&quot;Generate actionable business strategies and an implementation plan based on the analyzed trends.&quot;,<br>    agent=strategy_advisor,<br>    output_json=JsonOutput,<br>)<br># Create the crew<br>crew = Crew(<br>    agents=[data_analyst, strategy_advisor],<br>    tasks=[task1, task2],<br>    verbose=True,<br>    process=Process.sequential,<br>)<br><br><br>@workflow(name=&quot;watsonx_crew_workflow&quot;)<br>def watsonx_crew_kickoff(data):<br>    return crew.kickoff(inputs={&quot;market_data&quot;: data})<br><br><br>market_data = [<br>    &quot;Cloud computing revenue is projected to reach $600 billion by 2026.&quot;,<br>    &quot;Companies are investing more in AI-driven automation for efficiency.&quot;,<br>]<br><br>for data in market_data:<br>    result = watsonx_crew_kickoff(data)</pre><h3><strong>Traces and Metrics</strong></h3><p>You can create an application perspective to view trace information that is gathered from the LLM application runtime, complete the following steps:</p><p>a. Open the <strong>New Application Perspective</strong> wizard in one of the following ways:</p><ul><li>On the Instana dashboard, go to the <strong>Applications</strong> section and click <strong>Add application</strong>.</li><li>From the navigation menu, click <strong>Applications</strong> to open the <strong>Applications</strong> dashboard. Then, click <strong>Add</strong> and select <strong>New Application Perspective</strong>.</li></ul><p>b. Select <strong>Services or Endpoints</strong> and click <strong>Next</strong>.</p><p>c. Click <strong>Add filter</strong> and select a service name. You can select multiple services and endpoints by using OR conditions. The service name is specified by the app_name parameter in Traceloop.init(). For example, <strong>WatsonX_Crew_App</strong>.</p><p>d. In the <strong>Application Perspective Name</strong> field, enter a name for the LLM application perspective. Then, click <strong>Create</strong>.</p><p>To view trace information, go to the navigation menu and click <strong>Analytics</strong> to open the <strong>Analytics</strong> dashboard. Here, you can analyze calls by application, service, and endpoint, by breaking down the data that Instana presents by service, endpoint, and call names. You can filter and group traces or calls by using arbitrary tags, such as filtered by ‘Trace-&gt;Service Name’ equals <strong>WatsonX_Crew_App</strong>. You can view log information along with the trace information.</p><p>The following image shows Instana<strong> trace view</strong> for a CrewAI application. By analyzing this trace, you can identify performance inefficiencies, optimize agent workflows, and ensure smooth AI-driven operations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qb_nlYIjpKh9K3_2l6Ezmw.png" /></figure><p>To view the metrics, complete following steps:</p><p>a. Open the Instana UI, and click <strong>Infrastructure</strong>. Then, click <strong>Analyze Infrastructure</strong>.</p><p>b. Select <strong>OTEL LLMonitor</strong> from the list of types of the entities.</p><p>c. Click the entity instance of <strong>OTEL LLMonitor</strong> entity type to open the associated dashboard.</p><p>The following image shows the Instana dashboard that visualizes key metrics collected from the LLM used in the CrewAI application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2WTTimbbl2AlXGzspqk48g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A1xZVHLn7d-fjalh2zz7BA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n5WRQHxjjrjvxxDoZBerQg.png" /></figure><h3><strong>Events and Alerts</strong></h3><p>You can also trigger events and alerts based on above metrics when specific conditions are met. To set events based on any metrics, perform the following steps:</p><p>a. Go to <strong>Settings</strong> from the Instana UI, Select <strong>Events</strong>.</p><p>b. Click <strong>New Event.</strong></p><p>c. Add the event details and conditions. For example, if you want to set an event when Tokens by Model is greater than 100. Set the values as shown in the following example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*82vA8bR9rl1X4KaEYf8ugg.png" /></figure><p>d. Click <strong>Create</strong> to create the event</p><p>For creating alerts based on events. From the <strong>settings </strong>tab, select <strong>Alerts</strong> and create new alert by clicking <strong>New Alert</strong>. Add your event, application perspective and alerting channel. Then click <strong>Create</strong> to create the alert</p><p>I hope this blog helps you understand how to integrate Instana, Traceloop, and WatsonX to monitor CrewAI applications, providing insights into AI agent execution, LLM interactions, latency, and token usage. By capturing traces and metrics, you can track workflows, detect bottlenecks, and optimize costs. With Instana’s observability capabilities, developers can enhance the reliability, efficiency, and scalability of AI-driven applications, ensuring better performance through data-driven optimizations. Additionally, by setting events and alerts based on CrewAI metrics such as token usage, cost, and request rates, teams can proactively address anomalies, prevent overages, and maintain seamless AI operations.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bffadb968180" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ibm-cloud/enhancing-ai-agent-observability-monitoring-crewai-using-watsonx-with-instana-bffadb968180">Enhancing AI Agent Observability: Monitoring CrewAI using WatsonX with Instana</a> was originally published in <a href="https://medium.com/ibm-cloud">IBM Cloud</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monitoring Etcd database using Instana]]></title>
            <link>https://medium.com/ibm-cloud/monitoring-etcd-database-using-instana-793a89099601?source=rss-629e727ebdc0------2</link>
            <guid isPermaLink="false">https://medium.com/p/793a89099601</guid>
            <dc:creator><![CDATA[Adharsh H]]></dc:creator>
            <pubDate>Fri, 03 Nov 2023 13:33:18 GMT</pubDate>
            <atom:updated>2024-05-02T09:59:21.761Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Co-author: Shivangi Mehra</strong></p><p>IBM Instana is an application performance monitoring (APM) tool that offers real-time visibility and insights into the performance of apps and microservices in contemporary IT systems. It is intended to support businesses in managing and monitoring the operation of their infrastructure and applications in complex and dynamic on-premises, hybrid cloud, and cloud-native environments.</p><p><strong>Etcd</strong></p><p>Etcd is a distributed, reliable, and highly available key-value store that’s frequently used in distributed systems as a configuration and coordination service. It is open source and created by the CoreOS team, which is currently a part of Red Hat and is owned by IBM. The word “etcd” was derived from “etc” and “d”istributed system, the configuration directory for Unix. The purpose of etcd is to offer a dependable method for managing configuration data, service discovery data, and other crucial data in a distributed setting.</p><p>Etcd is made to be deployed across several cluster nodes in order to provide high availability and fault tolerance. For the purpose of ensuring data consistency and replication throughout the cluster, a consensus algorithm (often Raft) is used. Different applications and services operating on various nodes within the cluster or even across network borders can remotely access data that is stored in etcd. It is uncommon to use etcd as a local database within an application or on a single machine, although it can be used as a remote database for distributed applications and services. It works well in situations when you need to communicate and coordinate data across several distributed system services or between multiple instances of an application.</p><p><strong>Architecture</strong></p><p>The etcd cluster has shared-nothing nodes. The cluster has a leader and followers, with one node acting as the cluster’s leader. The Raft algorithm determines the leader node at run-time. The leader node casts votes against followers in response to requests. The leader commits the request and asks followers to commit if the majority of nodes concur. Any node in the cluster may receive a request from an etcd client. If the client sends a request to a follower, the follower relaysthe request to a leader node.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/781/1*bKylhR194zAG7pXYCBiNrA.png" /></figure><p><strong>How to interact with etcd ?</strong></p><p>You can communicate with etcd through the etcd command-line client (etcdctl), the HTTP/gRPC API, or one of the numerous client libraries for various programming languages. To use etcdctl, you can follow the instructions below:</p><ol><li>Since etcdctl is frequently included in the etcd distribution, installing etcd is usually required before installing etcdctl. Use the following command to install etcd for Ubuntu/Debian:</li></ol><pre>sudo apt-get update<br>sudo apt-get install etcd</pre><p>2. After the installation is complete, you can verify that etcdctl is installed. by running the following command:</p><pre>etcdctl version</pre><p>3. To connect to your etcd cluster, etcdctl must be configured. The endpoint of your etcd cluster is typically specified using environment variables like <strong>ETCDCTL_ENDPOINTS</strong>.</p><pre>export ETCDCTL_ENDPOINTS=&quot;http://localhost:2379&quot;</pre><p>4. To set the <strong>ETCDCTL_API</strong> environment variable to specify a particular API version for the etcdctl command-line client, you can use the following syntax:</p><pre>export ETCDCTL_API=&lt;API version&gt;</pre><p>Setting the <strong>ETCDCTL_API</strong> environment variable to the desired API version will instruct etcdctl to use that version when communicating with the etcd cluster. The etcdctl command-line client supports two major API versions: version 2 and version 3. In newer versions of etcd, version 3 is the default API version.</p><p>5. Once configured, etcdctl can be used to communicate with your etcd cluster by executing commands like etcdctl put, etcdctl get, etc.</p><pre>etcdctl set &lt;key&gt; &lt;value&gt;  #Setting a Key-Value Pair for version 2 API<br>etcdctl put &lt;key&gt; &lt;value&gt;  #Setting a Key-Value Pair for version 3 API<br>etcdctl get &lt;key&gt;  #Getting a Value by Key<br>etcdctl del &lt;key&gt;  #Deleting a Key<br>etcdctl watch &lt;key&gt;  #Watching for Changes</pre><p><strong>Configuration</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m9q6CuQhyqKhzYV76FiiJA.png" /></figure><p>Here is the configuration for monitoring etcd that uses the Instana plug-in. This configuration specifies the paths to the required certificate and key files for establishing a secure connection to the etcd cluster. The <strong>CA certificate</strong> is used to verify the authenticity of the etcd server’s certificate.</p><p>The <strong>client certificate</strong> is used by Instana to authenticate itself to the etcd server. It should be a valid certificate that has been signed by the CA certificate. <strong>Clientkey</strong> is the private key used for cryptographic operations to establish a secure connection.</p><p><strong>Metrics and dashboards</strong></p><p>To ensure the dependability and stability of your distributed systems, it’s crucial to monitor etcd performance, especially when it’s used as a crucial component in platforms like Kubernetes. A range of performance indicators are offered by etcd to track the functions and overall health of the etcd cluster.</p><p>The details provided by the Etcd dashboard includes the number of requests that are sent and received, the amount of incoming and outgoing data in bytes, the number of keys that expire each second, the number of watchers, and the number of operations such as sets, gets, create, update, delete, compare and swap, and compare and delete etc.</p><p>There are currently two dashboards for etcd, one for metrics gathered from /v2 endpoints and the other for metrics gathered from /v3 endpoints.</p><p><strong>V2 dashboard</strong></p><p>The v2 dashboard provides information like the cluster version, server version, and the number of unsuccessful and successful database operations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4EubzAGozp_3Ri4B9WEpIQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xNsIP_KSxUxgJ1DkukT-QA.png" /></figure><p><strong>V3 dashboard</strong></p><p>The v3 dashboard includes Additional information like the Api version and counts of various database operations as compared to v2 dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ANR_s_7fWIfwg4pr6XQDNQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LWXTRy-Z1BkRS3j9HBO7YQ.png" /></figure><p><strong>Conclusion</strong></p><p>Monitoring etcd using Instana offers a robust and comprehensive solution to ensure the reliability, performance, and health of your etcd clusters. It provides a strong solution for assuring the efficiency and dependability of your distributed etcd clusters. You can proactively manage your etcd infrastructure with its real-time visibility, security, and actionable analytics, and it will provide a solid basis for your distributed applications.</p><p><strong>References</strong></p><p>Instana: <a href="https://www.ibm.com/docs/en/instana-observability/current">https://www.ibm.com/docs/en/instana-observability/current</a></p><p>Etcd: <a href="https://www.ibm.com/docs/en/instana-observability/current?topic=technologies-monitoring-etcd">https://www.ibm.com/docs/en/instana-observability/current?topic=technologies-monitoring-etcd</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=793a89099601" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ibm-cloud/monitoring-etcd-database-using-instana-793a89099601">Monitoring Etcd database using Instana</a> was originally published in <a href="https://medium.com/ibm-cloud">IBM Cloud</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monitoring VIOS through Instana]]></title>
            <link>https://medium.com/ibm-cloud/monitoring-vios-through-instana-328c3776db28?source=rss-629e727ebdc0------2</link>
            <guid isPermaLink="false">https://medium.com/p/328c3776db28</guid>
            <dc:creator><![CDATA[Adharsh H]]></dc:creator>
            <pubDate>Fri, 26 May 2023 12:15:28 GMT</pubDate>
            <atom:updated>2023-05-26T12:15:28.526Z</atom:updated>
            <content:encoded><![CDATA[<p>Instana is a comprehensive application performance monitoring (APM) solution that provides real-time visibility into the performance of your applications and infrastructure. It also allows you to monitor the virtual I/O server (VIOS).</p><h4><strong>VIOS</strong></h4><p>VIOS is a virtualization technology in IBM Power Systems that allows multiple virtual servers to share access to the physical input/output (I/O) resources of the server, such as storage and network adapters. In simple terms, VIOS acts as a mediator between virtual machines (VMs) and the underlying physical hardware resources of the server. It provides a way for VMs to access and use physical resources efficiently, without direct access to the underlying hardware.</p><p>VIOS allows multiple virtual servers to share access to the physical storage area network (SAN) and network adapters, providing a more efficient and cost-effective way to manage resources in the data center. It works by creating virtual adapters for each virtual machine and mapping these virtual adapters to physical adapters on the server. The virtual adapters can be configured with different types of network and storage protocols, depending on the needs of the virtual machine.</p><p>By utilising Instana to monitor your VIOS, you can ensure that your infrastructure and apps are functioning properly and swiftly, and also identify and fix any performance issues that might occur.</p><h4><strong>Installation and configuration of Instana agent on VIOS</strong></h4><p>Usually, if you log in to the VIOS system, it works in a restricted Korn shell environment. A restricted Korn shell limits some of the capabilities to a shell script operating inside of an interactive user session. So you can’t execute most of the AIX commands. To perform any operations in VIOS, you have to be in a Korn shell environment. For that, you have to use the <strong>oem-setup-env</strong> command. The original equipment manufacturer (OEM) software installation and set up environment is made available to you by the <strong>oem_setup_env</strong> command.</p><ol><li>To initiate the OEM setup and installation environment, run the following command:</li></ol><pre>$ oem_setup_env</pre><p>2. Download the Instana agent for AIX from any available Instana server. Select OS as Unix, and search for AIX agent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*URzBW5-6XLJn9kWC1WHfdg.png" /></figure><p>3.Extract the Instana agent file by using the following command:</p><pre>$ gunzip -c &lt;instana agent tar file&gt; | tar -xf -</pre><p><strong>Note:</strong> If there is any issue in unzipping the file, then download the required AIX fileset “ <strong>tar-1.33- 11.aix6.1.ppc.rpm” </strong>from the <a href="https://www.ibm.com/support/pages/node/883796#T">AIX toolbox</a>. Install the downloaded tar package from the AIX toolbox by using the following command:</p><pre>$ rpm -ihv &lt;.rpm file for tar&gt;</pre><p>4. Navigate to <strong>&lt;home&gt;/instana-agent/etc/</strong> and open the <strong>mvn-settings.xml</strong> file. Update the password as Instana Agent key in two places, as shown in the following screenshot:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1015/1*uEMzzWi2WNacrM6EaopAnA.png" /></figure><p>5. Navigate to <strong>&lt;home&gt;/instana-agent/etc/instana</strong> and update <strong>com.instana.agent.main.sender. Backend.cfg</strong>configuration file with server details.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1019/1*ziKAtArksSh9Ok6PjLI8GQ.png" /></figure><p>6. Go to <strong>&lt;home&gt;/instana-agent/bin</strong> path and open <strong>setenv</strong> file. Update the path for <strong>JAVA_HOME</strong> as shown in the following screenshot:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KVQCDAiqY04Q9LkupMXHng.png" /></figure><p>7. Start the agent by executing <strong>./start</strong> command and check the status by using .<strong>/status </strong>command.<strong> </strong>Stop the agent by using <strong>./stop</strong> command.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n8N9I8M73r0C5Bst1y0d1w.png" /></figure><p>8. Log in to the server using the credentials. Go to <strong>More-&gt;Agents</strong>, and find your agent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NrCss4fbmUHwzL2jmG5_rw.png" /></figure><p>9. Click the agent and check the metrics that are available in the dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2JBYl2jKHZ5S3SpdUegREQ.png" /></figure><p>Also, you can explore the AIX dashboard. The various metrics available in the dashboard include CPU usage, Memory usage, CPU load, Memory, File systems, Network interfaces, TCP activity, and Process toplist.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HcqEKPSnas7oOViLhDG5Sw.png" /></figure><h4><strong>Conclusion</strong></h4><p>Using Instana for Virtual I/O Server monitoring provides several benefits. Instana offers a comprehensive and real-time view of the VIOS environment, allowing administrators to quickly identify and troubleshoot any issues that might arise. Overall, Instana is a powerful tool for VIOS monitoring that can help to maintain the health and performance of your IT infrastructure.</p><h4><strong>References</strong></h4><ul><li>Instana: <a href="https://www.ibm.com/docs/en/instana-observability/current">https://www.ibm.com/docs/en/instana-observability/current</a></li><li>VIOS: <a href="https://www.ibm.com/docs/en/power9?topic=environment-virtual-io-server">https://www.ibm.com/docs/en/power9?topic=environment-virtual-io-server</a></li><li>OEM Environment: <a href="https://www.ibm.com/docs/en/power7?topic=commands-oem-setup-env-command">https://www.ibm.com/docs/en/power7?topic=commands-oem-setup-env-command</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=328c3776db28" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ibm-cloud/monitoring-vios-through-instana-328c3776db28">Monitoring VIOS through Instana</a> was originally published in <a href="https://medium.com/ibm-cloud">IBM Cloud</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>