<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Chethan B S on Medium]]></title>
        <description><![CDATA[Stories by Chethan B S on Medium]]></description>
        <link>https://medium.com/@chethanbs21?source=rss-f60dbb05200d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*kY6CBeMr54gWKSlRtzSJwA.jpeg</url>
            <title>Stories by Chethan B S on Medium</title>
            <link>https://medium.com/@chethanbs21?source=rss-f60dbb05200d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 21:08:33 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@chethanbs21/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Solving the Infamous “Deployment Pending” Issue in WSO2 API Manager on Kubernetes]]></title>
            <link>https://medium.com/@chethanbs21/solving-the-infamous-deployment-pending-issue-in-wso2-api-manager-on-kubernetes-670ee8f5e58b?source=rss-f60dbb05200d------2</link>
            <guid isPermaLink="false">https://medium.com/p/670ee8f5e58b</guid>
            <category><![CDATA[api-gateway]]></category>
            <category><![CDATA[api-development]]></category>
            <category><![CDATA[wso2-api-manager]]></category>
            <category><![CDATA[api-integration]]></category>
            <category><![CDATA[deployment]]></category>
            <dc:creator><![CDATA[Chethan B S]]></dc:creator>
            <pubDate>Wed, 05 Aug 2026 14:16:12 GMT</pubDate>
            <atom:updated>2026-08-05T14:20:26.056Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5o_rD-GmZHY1tMsDvIlA1Q.png" /></figure><h3>A deep dive into how an unresolved environment variable in your JMS configuration can break API Gateway deployments — and how to fix it.</h3><p>Setting up WSO2 API Manager (APIM) in a Kubernetes environment should ideally be smooth sailing — especially in an all-in-one deployment. You deploy the pod, bring up the services, and voila! You can access the <strong>Publisher</strong>, <strong>DevPortal</strong>, and <strong>Gateway</strong> endpoints without a hitch.</p><p>However, a frustrating issue can arise when you attempt to publish your first API: <strong>the API status gets stuck on “Deployment Pending.”</strong></p><p>When invoking the endpoint, instead of a successful response, the Gateway throws a 404 Not Found error.</p><p>In this article, we’ll walk through the exact symptoms of this issue, uncover the root cause under the hood of WSO2’s internal architecture, and show you step-by-step how to resolve it.</p><h3>🚩 The Symptoms</h3><p>In our setup, the APIM all-in-one pod was up and running fine on Kubernetes. However, during testing, we observed three critical anomalies:</p><ol><li><strong>Stuck UI State:</strong> Publishing an API via the Publisher Portal changed its status to <strong>“Deployment Pending”</strong> indefinitely, rather than switching to <strong>“Deployed”</strong>.</li><li><strong>Missing Deployment Logs:</strong> Comparing the pod logs against a healthy local APIM instance revealed that several expected deployment-related log events were completely missing.</li><li><strong>404 Errors on Gateway:</strong> Invoking the published API URL routed directly to the main execution sequence, returning a 404 Resource Not Found error. This confirmed the Gateway never actually received or deployed the API artifact.</li></ol><p>Additionally, this behavior was observed in environments where email domains as usernames were enabled in deployment.toml:</p><pre>[tenant_mgt]<br>enable_email_domain=true</pre><h3><strong>🔍 The Deep Dive: How WSO2 Deploys APIs to the Gateway</strong></h3><p>To understand why this happens, we need to look at how WSO2 APIM handles deployment internally.</p><p>When you click <strong>Deploy/Publish</strong> in the Publisher Portal, APIM doesn’t instantly write files to the Gateway directly. Instead, it relies on an internal <strong>JMS (Java Message Service) Event Broker</strong> to publish deployment events. The Gateway component listens to these JMS topics and dynamically loads the API artifacts into memory upon receiving the event.</p><pre>+------------------+         JMS Topic         +------------------+<br>| Publisher Portal |  =======================&gt; |   API Gateway    |<br>| (Publishes Event)|   (Fails if Auth Fails)   | (Deploys API)    |<br>+------------------+                           +------------------+</pre><p>If the JMS client on either end fails to authenticate or lacks the necessary permissions, <strong>the deployment event is silently dropped or unhandled</strong>, leaving the UI waiting indefinitely in the &quot;Deployment Pending&quot; state.</p><h3>🎯 Finding the Root Cause</h3><p>By dissecting the carbon.log file and analyzing the deployment.toml configuration, we pinpointed the exact point of failure: <strong>JMS Authentication</strong>.</p><p>Here is a snippet of the configuration used in the deployment:</p><pre>[apim.throttling.policy_deploy]<br>username = &quot;$ref{super_admin.username}@carbon.super&quot;</pre><pre>[apim.throttling.jms]<br>username = &quot;$env{JMS_ADMIN_USERNAME}!carbon.super&quot;<br>password = &quot;$env{ADMIN_PASSWORD}&quot;<br>start_delay = &quot;2m&quot;</pre><p>Notice how username under [apim.throttling.jms] dynamically extracts an environment variable: $env{JMS_ADMIN_USERNAME}.</p><h3>What went wrong?</h3><ol><li><strong>Unresolved Environment Variable:</strong> In the Kubernetes deployment, JMS_ADMIN_USERNAME was either missing from the ConfigMap/Secret, misnamed, or failing to resolve properly at container runtime.</li><li><strong>Invalid JMS Credentials:</strong> Because $env{JMS_ADMIN_USERNAME} resolved to an empty string or an invalid username format (especially when combined with !carbon.super or email domain formatting), the JMS client failed to establish an authorized connection to the internal broker.</li><li><strong>Privilege Misconfiguration:</strong> In cases where the variable <em>did</em> resolve, the user associated with it lacked the necessary <strong>Super Admin</strong> privileges required to publish events across internal system topics.</li></ol><p>Without valid credentials, the event notification chain broke. The Publisher recorded the intent to deploy, but the event never reached the Gateway listener — leaving the API forever stuck in <strong>“Deployment Pending.”</strong></p><h3>🛠️ The Solution</h3><p>To fix this issue, you must ensure that your JMS configuration resolves to a valid user with <strong>Super Admin permissions</strong> at runtime.</p><h3>Step 1: Verify K8s Environment Variables</h3><p>Check your Kubernetes deployment manifest (deployment.yaml), ConfigMap, or Secret to ensure JMS_ADMIN_USERNAME and ADMIN_PASSWORD are explicitly defined and accessible to the pod environment.</p><pre>env:<br>  - name: JMS_ADMIN_USERNAME<br>    value: &quot;admin&quot; # or your designated admin user<br>  - name: ADMIN_PASSWORD<br>    valueFrom:<br>      secretKeyRef:<br>        name: apim-secret<br>        key: password</pre><h3>Step 2: Handle Email Domains &amp; Tenant Formatting</h3><p>If enable_email_domain=true is enabled in your deployment.toml, make sure your JMS username format aligns with WSO2&#39;s tenant naming rules.</p><p>If using the default super tenant admin, you can test with hardcoded values first to isolate the issue:</p><pre>[apim.throttling.jms]<br>username = &quot;admin!carbon.super&quot;  # Or &quot;admin@carbon.super&quot; depending on tenant configuration<br>password = &quot;$env{ADMIN_PASSWORD}&quot;<br>start_delay = &quot;2m&quot;</pre><h3>Step 3: Restart and Validate</h3><ul><li>Apply the updated configuration/environment variables to your Kubernetes cluster.</li><li>Restart the WSO2 APIM pod.</li><li>Log in to the Publisher Portal, create a test API, and click <strong>Deploy</strong>.</li><li>Check the carbon.log. You should now see full deployment logs, and the API status will instantly transition to <strong>&quot;Deployed&quot;</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*leJu0lItFtZYTm-tuioqzQ.png" /></figure><h3>💡 Key Takeaways for WSO2 on Kubernetes</h3><ul><li><strong>Check Internal Event Brokers:</strong> If WSO2 APIM UI states get stuck (Deployment, Key Generation, or Synchronization), suspect the internal JMS/Event publishing mechanism first.</li><li><strong>Audit Environment Variables carefully:</strong> WSO2’s $env{VAR_NAME} syntax in deployment.toml will fail silently or resolve to empty strings if the environment variable isn&#39;t injected correctly into the pod context.</li><li><strong>Pay Attention to Email Domains:</strong> Enabling enable_email_domain=true alters how usernames and tenant domains (@carbon.super vs !carbon.super) are parsed in internal configuration blocks.</li></ul><p>‼️ Feel free connect with me on <a href="https://www.linkedin.com/in/chethan-b-s-769920175/">LinkedIn</a> 👨‍💻 if you need any assistance.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=670ee8f5e58b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering WSO2 API Manager: The Ultimate Guide to Configuring -Xms and -Xmx in Production]]></title>
            <link>https://medium.com/@chethanbs21/mastering-wso2-api-manager-the-ultimate-guide-to-configuring-xms-and-xmx-in-production-f3c93f59ac37?source=rss-f60dbb05200d------2</link>
            <guid isPermaLink="false">https://medium.com/p/f3c93f59ac37</guid>
            <category><![CDATA[heap-memory]]></category>
            <category><![CDATA[jstack]]></category>
            <category><![CDATA[heap]]></category>
            <category><![CDATA[outofmemoryerror]]></category>
            <dc:creator><![CDATA[Chethan B S]]></dc:creator>
            <pubDate>Mon, 13 Jul 2026 12:03:45 GMT</pubDate>
            <atom:updated>2026-07-13T12:03:45.876Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yDYwWvjbtDhiENBx.png" /></figure><p>If you are running WSO2 API Manager (APIM) in a production environment, you already know that performance and stability are non-negotiable. An API Gateway is the front door to your enterprise data; if it goes down or suffers from latency spikes, your entire ecosystem feels the pain.</p><p>One of the most critical — yet frequently misunderstood — aspects of tuning WSO2 APIM is configuring the Java Virtual Machine (JVM) heap memory. Specifically, the <strong><em>-Xms</em></strong> and <strong><em>-Xmx</em></strong> arguments.</p><p>In this post, we will demystify what these arguments do, why they are so important, and how exactly to configure them inside your api-manager.sh script for a rock-solid production deployment.</p><h3>What are -Xms and -Xmx Anyway?</h3><p>WSO2 API Manager is built on top of the Java runtime. Like all Java applications, it uses a dedicated block of system RAM known as the Heap to store the objects, payloads, and variables it needs to process API requests.</p><p>The size of this heap is controlled by two specific JVM flags:</p><ol><li><strong>-Xms (Initial Heap Size)</strong>: This tells the JVM how much memory to allocate to the heap the very second the WSO2 server starts up.</li><li><strong>-Xmx (Maximum Heap Size)</strong>: This is the hard ceiling. It tells the JVM the absolute maximum amount of memory it is allowed to consume. If WSO2 needs more memory than this limit, the JVM will throw a fatal <em>java.lang.OutOfMemoryError</em> and the server will crash.</li></ol><p>(Tip: The letters “m” or “g” are appended to the numbers to represent Megabytes or Gigabytes. For example, -Xmx4g means 4 Gigabytes).</p><h3>Why Does This Matter for WSO2 APIM?</h3><p>When thousands of API requests are flowing through your WSO2 Gateway per second, the JVM is constantly creating and destroying objects (headers, JSON payloads, authentication tokens, etc.).</p><p>If your heap is under-provisioned (the -Xmx is too low), WSO2 will constantly pause to run Garbage Collection (GC) to free up space, resulting in severe API latency. If it runs out entirely, the node crashes.</p><p>If your heap is over-provisioned (the -Xmx is too high), Garbage Collection pauses might happen less frequently, but when they do happen, they will take much longer to scan the massive memory space, again causing sudden spikes in API latency.</p><h3>The Golden Rule for Production Environments</h3><p>When you open the default WSO2 configuration, you might see a discrepancy between the initial and maximum heap sizes <em>(e.g., -Xms256m -Xmx1024m)</em>. This is fine for a developer’s laptop, but it is bad practice for production.</p><p><strong>The Golden Rule</strong>: In a WSO2 production environment, you should always set -Xms and -Xmx to the exact same value.</p><h3>Why?</h3><p>If <em>-</em><strong><em>Xms</em> </strong>is lower than <strong><em>-Xmx</em></strong>, the JVM starts small and has to continually ask the operating system for more memory as API traffic ramps up. This resizing process is computationally expensive and causes the JVM to pause. By setting them to the same value (<strong><em>e.g., -Xms4g -Xmx4g</em></strong>), the JVM claims the full chunk of memory from the OS at startup. It never has to pause to resize the heap, granting your APIs a smoother, more predictable latency profile.</p><h3>How to Determine the Right Values</h3><p>There is no one-size-fits-all number, but here are WSO2’s general guidelines based on typical traffic profiles:</p><ol><li><strong>Standard Production Load (Minimum)</strong>: 2GB (-Xms2048m -Xmx2048m). This is the absolute minimum WSO2 recommends for a production Gateway handling moderate traffic.</li><li><strong>High Traffic / Heavy Payloads:</strong> 4GB (-Xms4096m -Xmx4096m). If you are processing a high volume of transactions per second (TPS) or dealing with very large XML/JSON payloads, 4GB is the sweet spot for most enterprise deployments.</li><li><strong>Extreme Loads:</strong> 8GB+. Rarely needed unless you have massive bursts of traffic, heavily complex mediation sequences, or you are running the APIM Analytics worker profiles.</li></ol><blockquote><strong>Warning</strong>: Never allocate 100% of your server’s RAM to the JVM heap. The OS needs memory to run, and the JVM itself uses non-heap memory (Metaspace, thread stacks) for operations. A safe rule of thumb is to allocate no more than 60–70% of total physical RAM to the -Xmx flag.</blockquote><h3>Step-by-Step: Configuring api-manager.sh</h3><p>Ready to optimize your WSO2 APIM instance? Here is how to apply these changes.</p><p><strong>Step 1:</strong> SSH into your WSO2 server and navigate to the bin directory of your WSO2 installation.</p><pre>cd &lt;API-M_HOME&gt;/bin</pre><p><strong>Step 2:</strong> Open the startup script in your preferred text editor (like vi or nano).</p><p>Note: If you are on Linux/macOS, edit api-manager.sh. If you are on Windows, edit api-manager.bat.</p><pre>vi api-manager.sh</pre><p><strong>Step 3:</strong> Locate the JVM_MEM_OPTS variable.</p><p>Scroll through the file until you find a block of code defining the memory options. It usually looks something like this by default:</p><pre># default memory settings<br>if [ -z &quot;$JVM_MEM_OPTS&quot; ]; then<br> JVM_MEM_OPTS=&quot;-Xms256m -Xmx1024m&quot;<br>fi</pre><p><strong>Step 4:</strong> Update the values to your production requirements, ensuring both flags match. For a standard 4GB setup, change it to:</p><pre># Optimized production memory settings<br>if [ -z &quot;$JVM_MEM_OPTS&quot; ]; then<br> JVM_MEM_OPTS=&quot;-Xms4096m -Xmx4096m&quot;<br>fi</pre><p><strong>Step 5:</strong> Save the file and restart the WSO2 server.</p><pre>sh api-manager.sh restart</pre><h3>Pro-Tips for the Road</h3><ol><li><strong>Monitor your Garbage Collection:</strong> Just setting the heap is not enough. Enable GC logging in WSO2 and use tools like GCeasy to analyze your logs. If your GC pauses are consistently taking longer than a few milliseconds, you may need to adjust your heap size or tune your GC algorithm (like switching to G1GC).</li><li><strong>Keep environments consistent:</strong> If your production Gateway has 4GB of heap, ensure your staging/load-testing environment is configured identically. You cannot accurately load test an API if the memory profiles don’t match.</li><li><strong>Don’t forget the OS:</strong> Remember that if you give WSO2 a 4GB heap, the actual Java process will consume slightly more than 4GB of RAM at the OS level due to native memory overhead. Always leave breathing room for the operating system!</li></ol><p>By simply equalizing your <strong><em>-Xms</em></strong> and <strong><em>-Xmx</em></strong> values and sizing them appropriately for your traffic, you instantly eliminate a massive source of JVM overhead, ensuring your APIs remain lightning-fast and highly available.</p><p>‼️ Feel free connect with me on <a href="https://www.linkedin.com/in/chethan-b-s-769920175/">LinkedIn</a> 👨‍💻 if you need any assistance.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f3c93f59ac37" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering OpenTelemetry: Integrating WSO2 API Manager 4.5.0 with Dynatrace]]></title>
            <link>https://medium.com/@chethanbs21/mastering-opentelemetry-integrating-wso2-api-manager-4-5-0-with-dynatrace-570e0cba2e92?source=rss-f60dbb05200d------2</link>
            <guid isPermaLink="false">https://medium.com/p/570e0cba2e92</guid>
            <category><![CDATA[wso2-api-manager]]></category>
            <category><![CDATA[dynatrace]]></category>
            <category><![CDATA[opentelemetry]]></category>
            <category><![CDATA[otlp]]></category>
            <dc:creator><![CDATA[Chethan B S]]></dc:creator>
            <pubDate>Fri, 13 Mar 2026 15:34:25 GMT</pubDate>
            <atom:updated>2026-03-13T15:34:25.003Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aiGUxHQz2rPd2Fo5.png" /></figure><p>In modern distributed architectures, APIs are the central nervous system of your business. But when things go wrong — or when you simply need to understand the performance bottlenecks of your microservices — you need a way to track requests from the edge all the way down to the database. This is where OpenTelemetry (OTel) and robust observability backends like Dynatrace come into play.</p><p>Starting from version 4.2.0, WSO2 API Manager introduced native support for OpenTelemetry (OTLP), eliminating the need for custom, drop-in tracers.</p><p>In this post, we’ll walk through the complete setup of configuring WSO2 API Manager 4.5.0 to send traces to Dynatrace using the OpenTelemetry Collector. We’ll also cover a common challenge you might encounter — default “Internal” span kinds — and how to fix it using the OpenTelemetry Transformation Language (OTTL).</p><h3>Step 1: Prepare Dynatrace</h3><p>Before configuring your gateway, you need a destination for your traces. Dynatrace natively ingests OTLP data, making this process highly streamlined.</p><ol><li><strong>Sign up</strong> for a free trial of Dynatrace (if you don’t already have an environment). URL —<em> </em><a href="https://www.dynatrace.com/signup/"><em>https://www.dynatrace.com/signup/</em></a></li><li><strong>Generate an Access Token:</strong> Navigate to the Dynatrace Access Tokens app within your tenant. Create a new token and ensure it has the <strong>Ingest OpenTelemetry traces</strong> (openTelemetryTrace.ingest) scope enabled.</li><li>Note down your SaaS environment URL (e.g., <a href="https://{your-environment-id}.live.dynatrace.com)./">https://{your-environment-id}.live.dynatrace.com).</a> You can get the environment-id from the URL after logging into the dynatrace environment.</li></ol><h3>Step 2: Deploy the OpenTelemetry Collector</h3><p>While WSO2 can export OTLP data, it uses gRPC by default. Dynatrace’s native ingestion endpoint expects HTTP/protobuf. To bridge this gap — and to add powerful batching and processing capabilities — we use the standard OpenTelemetry Collector.</p><p>Deploy the OTel Collector using Docker with the following <strong><em>otel-collector-config.yaml</em></strong> :</p><pre>receivers:<br>  otlp:<br>    protocols:<br>      grpc:<br>        endpoint: 0.0.0.0:4317<br><br>processors:<br>  batch:<br>    timeout: 1s<br>    send_batch_size: 1024<br><br>exporters:<br>  otlphttp/dynatrace:<br>    endpoint: &quot;https://{environment-id}.live.dynatrace.com/api/v2/otlp&quot;<br>    headers:<br>      Authorization: &quot;Api-Token &lt;valid-token&gt;&quot;<br><br>service:<br>  pipelines:<br>    traces:<br>      receivers: [otlp]<br>      processors: [transform/span_kind, batch]<br>      exporters: [otlphttp/dynatrace]</pre><p>(Note: We use otlp_http as the exporter type, which is the modern standard over the deprecated otlphttp alias)</p><p>Run the below commad to spin up the instance of OTel collector:</p><pre>docker run -d \            <br>  --name otel-collector \<br>  -p 4317:4317 \<br>  -v $(pwd)/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml \<br>  otel/opentelemetry-collector-contrib:latest</pre><h3>Step 3: Configure WSO2 API Manager 4.5.0</h3><p>With the collector running and listening on port 4317, we simply need to point our WSO2 API Gateway to it.</p><p>Open your &lt;WSO2_HOME&gt;/repository/conf/deployment.toml file and append the following configuration:</p><pre>[apim.open_telemetry]<br>remote_tracer.enable = true<br>remote_tracer.name = &quot;otlp&quot;<br>remote_tracer.url = &quot;http://localhost:4317&quot;<br><br>[[apim.open_telemetry.resource_attributes]]<br>name = &quot;service.name&quot;<br>value = &quot;WSO2-API-Gateway&quot;<br><br>[[apim.open_telemetry.resource_attributes]]<br>name = &quot;deployment.environment&quot;<br>value = &quot;Production&quot;</pre><p>Restart WSO2 API Manager, invoke a few APIs, and navigate to the Dynatrace Distributed Tracing dashboard. You should see your traces flowing in!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_mzq5Ry9Ke90t48BAeIPSg.png" /></figure><h3>The Challenge: The “Internal” Span Trap</h3><p>If you stop at Step 3, you will notice a glaring issue in the Dynatrace dashboard. Following successful invocations, all traces grouped under WSO2 show up with the Span Kind labeled as <strong>“Internal”</strong>.</p><p>By default, WSO2 tags its internal mediation steps and message processors as Internal spans. However, Dynatrace relies on <strong>Server</strong> spans (incoming requests) and <strong>Client</strong> spans (outgoing requests to backends) to accurately map your topology and calculate Service-Level Indicators (SLIs). If everything is Internal, Dynatrace cannot stitch the distributed trace together properly.</p><h3>The Solution: Enter OTTL and the Transform Processor</h3><p>To solve this, we don’t need to rewrite any WSO2 code. Instead, we can leverage the <strong>Transform Processor</strong> in our OTel Collector. Using the OpenTelemetry Transformation Language (OTTL), we can intercept the traces on the fly and intelligently rewrite the span kinds based on regex matching.</p><p>Here is the updated OTel Collector configuration that fixes the issue:</p><pre>receivers:<br>  otlp:<br>    protocols:<br>      grpc:<br>        endpoint: 0.0.0.0:4317<br><br>processors:<br>  batch:<br>    timeout: 1s<br>    send_batch_size: 1024<br><br>  # Manipulate the spans on the fly<br>  transform/span_kind:<br>    error_mode: ignore<br>    trace_statements:<br>      - context: span<br>        statements:<br>          # 1. Change specific API entry points to SERVER spans (Span Kind 2)<br>          - set(kind, 2) where kind == 1 and IsMatch(name, &quot;^PizzaShack.*&quot;)<br>          <br>          # 2. Change HTTP resource invocations to SERVER spans<br>          - set(kind, 2) where kind == 1 and IsMatch(name, &quot;^(GET|POST|PUT|DELETE|PATCH|OPTIONS)--.*&quot;)<br>          <br>          # 3. Change outbound backend calls to CLIENT spans (Span Kind 3)<br>          - set(kind, 3) where kind == 1 and IsMatch(name, &quot;.*Endpoint.*&quot;)<br><br>exporters:<br>  otlp_http/dynatrace:<br>    endpoint: &quot;https://{your-environment-id}.live.dynatrace.com/api/v2/otlp&quot;<br>    headers:<br>      Authorization: &quot;Api-Token &lt;valid-token&gt;&quot;<br><br>service:<br>  pipelines:<br>    traces:<br>      receivers: [otlp]<br>      # Add the transform processor BEFORE the batch processor<br>      processors: [transform/span_kind, batch]<br>      exporters: [otlp_http/dynatrace]</pre><p>Deploy a new instance of the OTel collector using the above configuration and invoke few APIs to see the traces in dynatrace with different kinds of spans as shown in the screenshot below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WB-lKPVryJWbUBIqo-3JPg.png" /></figure><h3>Why this works:</h3><p>In the OTLP specification, span kinds are represented as integers (1 for Internal, 2 for Server, 3 for Client). Using the IsMatch OTTL function, we look at the dynamic name of the span WSO2 generated (e.g., GET--/menu or PizzaShackAPI). If it matches an entry point, we use set(kind, 2) to promote it to a Server span. If it matches an outbound call to a backend (.*Endpoint.*), we promote it to a Client span.</p><h3>Conclusion</h3><p>By combining WSO2 API Manager’s native OpenTelemetry support with an OTel Collector, you create a highly performant, vendor-neutral observability pipeline. Furthermore, by tapping into the power of the Transform Processor and OTTL, you retain total control over your telemetry data, ensuring that robust backends like Dynatrace can ingest, map, and analyze your API traffic flawlessly.</p><p><em>Have you implemented OpenTelemetry in your WSO2 environments? Let me know your thoughts and architecture patterns in the comments!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=570e0cba2e92" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Troubleshooting High CPU in WSO2 products: A Proactive Script for Thread Dump Analysis]]></title>
            <link>https://medium.com/@chethanbs21/troubleshooting-high-cpu-in-wso2-products-a-proactive-script-for-thread-dump-analysis-c031d5ee5aa3?source=rss-f60dbb05200d------2</link>
            <guid isPermaLink="false">https://medium.com/p/c031d5ee5aa3</guid>
            <category><![CDATA[analysis]]></category>
            <category><![CDATA[wso2]]></category>
            <category><![CDATA[threaddump]]></category>
            <dc:creator><![CDATA[Chethan B S]]></dc:creator>
            <pubDate>Wed, 17 Dec 2025 07:57:05 GMT</pubDate>
            <atom:updated>2025-12-17T07:57:05.343Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/450/0*j9h6qiSRIkA8tLGN.png" /></figure><p>In the world of enterprise integration, few things are as stressful as a sudden CPU spike in your production environment. If you’ve ever used WSO2 Micro Integrator or API Manager, you know that by the time you realize there’s a performance issue and manually run jstack, the &quot;evidence&quot; is often gone.</p><p>To solve this, I’ve developed a bash script that provides a “Black Box” recorder for your Java process. It continuously captures thread dumps and OS-level thread usage, keeping a rolling window of data so you always have the last hour of history when a crash occurs.</p><h3>The Problem with Manual Troubleshooting</h3><p>Standard troubleshooting usually involves:</p><ol><li>Identifying the high CPU.</li><li>Running top -H to find the Thread ID.</li><li>Running jstack to find the code.</li></ol><p>The problem? These two data points must be captured at the <strong>exact same millisecond</strong> to be useful. If you capture them 10 seconds apart, the thread IDs might have already changed tasks.</p><h3>The Solution: The Indefinite Rotation Script</h3><p>This script automates the capture of both jstack and ps thread usage in parallel. It includes a rotation logic to ensure your disk doesn&#39;t fill up, making it safe to run in the background for days.</p><p><strong><em>capture_threads_forever.sh</em></strong></p><pre>#!/bin/bash<br><br># --- Argument Validation ---<br>if [ -z &quot;$1&quot; ]; then<br>    echo &quot;Error: No PID provided.&quot;<br>    echo &quot;Usage: sh capture_threads_forever.sh &lt;PID&gt; [INTERVAL] [MAX_FILES_TO_KEEP]&quot;<br>    echo &quot;Example: sh capture_threads_forever.sh 1234 5 100&quot;<br>    exit 1<br>fi<br><br>PID=$1<br>INTERVAL=${2:-5}<br>MAX_FILES=${3:-100} <br><br># --- Process Validation ---<br>if ! ps -p &quot;$PID&quot; &gt; /dev/null; then<br>    echo &quot;Error: Process ID &#39;$PID&#39; is not running.&quot;<br>    exit 1<br>fi<br><br>OUT_DIR=&quot;./infinite_analysis_${PID}&quot;<br>mkdir -p &quot;$OUT_DIR&quot;<br><br>echo &quot;------------------------------------------------&quot;<br>echo &quot;RUNNING INDEFINITELY - Press Ctrl+C to stop&quot;<br>echo &quot;Target PID      : $PID&quot;<br>echo &quot;Interval (sec)  : $INTERVAL&quot;<br>echo &quot;Rotation Limit  : $MAX_FILES file sets&quot;<br>echo &quot;Output Directory: $OUT_DIR&quot;<br>echo &quot;------------------------------------------------&quot;<br><br>i=1<br>while true; do<br>    TS=$(date +%Y%m%d_%H%M%S)<br>    DUMP_FILE=&quot;$OUT_DIR/threaddump_${TS}.txt&quot;<br>    USAGE_FILE=&quot;$OUT_DIR/threadusage_${TS}.txt&quot;<br><br>    echo &quot;[$i] Capturing data at $TS...&quot;<br><br>    # Capture OS Thread Usage and Java Dump in parallel<br>    ps -p $PID -Lo pid,tid,%cpu,time,comm &gt; &quot;$USAGE_FILE&quot; &amp;<br>    jstack -l $PID &gt; &quot;$DUMP_FILE&quot; &amp;<br><br>    wait # Wait for both to finish writing<br><br>    # --- ROTATION LOGIC ---<br>    # Prevents disk pressure by deleting the oldest files<br>    file_count=$(ls -1 $OUT_DIR/threaddump_*.txt | wc -l)<br>    if [ &quot;$file_count&quot; -gt &quot;$MAX_FILES&quot; ]; then<br>        oldest_dump=$(ls -1t $OUT_DIR/threaddump_*.txt | tail -n 1)<br>        oldest_usage=$(ls -1t $OUT_DIR/threadusage_*.txt | tail -n 1)<br>        rm -f &quot;$oldest_dump&quot; &quot;$oldest_usage&quot;<br>    fi<br><br>    i=$((i+1))<br>    sleep $INTERVAL<br>done</pre><h3>How to Use It for “Last 1 Hour” History</h3><p>If you want to keep the last hour of data (capturing every 5 seconds), you simply calculate the rotation count (3600s/ 5s = 720 files) (no_of_seconds_per_hour / interval).</p><p>Run command:</p><pre>sh capture_threads_forever.sh &lt;PID&gt; 5 720</pre><p><em>&lt;PID&gt;</em> of wso2 process can be found in <em>&lt;WSO2_Product_Home&gt;/wso2carbon.pid</em> file.</p><h3>Analyzing the Data: The “Decimal to Hex” Secret</h3><p>The real power comes after you’ve captured the data. Here is how you map an OS CPU spike to WSO2 code:</p><ol><li><strong>Find the High CPU Thread:</strong> Open a threadusage file and find the TID with the highest %cpu (e.g., 14253).</li><li><strong>Convert to Hex:</strong> Convert that TID to Hexadecimal (e.g., 14253 becomes 0x37ad).</li><li><strong>Search the Dump:</strong> Open the corresponding threaddump file and search for nid=0x37ad.</li></ol><p>This will take you to the exact line of Java code (or WSO2 mediator) that is hogging your CPU.</p><h3>Conclusion</h3><p>Monitoring production environments requires more than just reactive tools. By running this script during peak loads or intermittent performance drops, you ensure that you never miss the “smoking gun” again. It’s lightweight, disk-safe, and provides the granular data needed to solve the toughest performance bottlenecks.</p><p>‼️ Feel free connect with me on <a href="https://www.linkedin.com/in/chethan-b-s-769920175/">LinkedIn</a> 👨‍💻.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c031d5ee5aa3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering Mutual TLS in WSO2 APIM: Dynamic SSL Profiles for Backend Security]]></title>
            <link>https://medium.com/@chethanbs21/mastering-mutual-tls-in-wso2-apim-dynamic-ssl-profiles-for-backend-security-2db9b2139f4a?source=rss-f60dbb05200d------2</link>
            <guid isPermaLink="false">https://medium.com/p/2db9b2139f4a</guid>
            <category><![CDATA[tlm]]></category>
            <category><![CDATA[ssl]]></category>
            <category><![CDATA[wso2-api-manager]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[wso2]]></category>
            <dc:creator><![CDATA[Chethan B S]]></dc:creator>
            <pubDate>Fri, 12 Dec 2025 09:16:46 GMT</pubDate>
            <atom:updated>2025-12-12T09:16:46.790Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ppJfocthd5EOTaO_.png" /></figure><p>When securing APIs, standard one-way TLS is often not enough. For true defense-in-depth, especially when connecting your WSO2 API Gateway to sensitive backend services, <strong>Mutual TLS (mTLS)</strong> is the gold standard. mTLS requires both the client (your WSO2 Gateway) and the server (your backend service) to present and validate certificates, ensuring both parties are who they claim to be.</p><p>Manually configuring keystores for every backend service in an integration platform can quickly become unmanageable. Fortunately, WSO2 API Manager offers a powerful solution: <strong>Dynamic SSL Profiles</strong>.</p><p>This guide walks you through configuring WSO2 APIM to use dynamic SSL profiles, enabling secure, scalable mTLS authentication for your HTTPS backend transport.</p><h3>🔑 The Core Concept: Dynamic SSL Profiles</h3><p>WSO2 APIM uses the <strong>Passthrough HTTPS Sender</strong> to establish connections to backend endpoints. Instead of relying on a single global keystore, Dynamic SSL Profiles allow the Gateway to choose the correct keystore and truststore <strong>based on the hostname and port</strong> of the target backend.</p><p>This configuration is loaded from an external XML file, enabling updates without a full server restart.</p><h3>Step 1: Create the Custom SSL Profile Configuration File</h3><p>You need to create a new XML file containing the specific SSL profiles tailored for your backend connections. The recommended location for this file is:</p><p>&lt;APIM_HOME&gt;/repository/deployment/server/mutual_ssl_profiles.xml</p><p>Copy the following configuration snippet into this file. This defines two separate profiles: one for standard SSL and one for Mutual SSL (mTLS).</p><pre>&lt;parameter name=&quot;customSSLProfiles&quot;&gt;<br>&lt;!-- For SSL Handshake configure only trust store--&gt; <br>    &lt;profile&gt;<br>        &lt;servers&gt;localhost:port&lt;/servers&gt;<br>        &lt;TrustStore&gt;<br>            &lt;Location&gt;repository/resources/security/client-truststore.jks&lt;/Location&gt;<br>            &lt;Type&gt;JKS&lt;/Type&gt;<br>            &lt;Password&gt;wso2carbon&lt;/Password&gt;<br>        &lt;/TrustStore&gt;<br>    &lt;/profile&gt;<br>&lt;!-- For Mutual SSL Handshake configure both trust store and key store--&gt;  <br>    &lt;profile&gt;<br>        &lt;servers&gt;developers.google.com:443&lt;/servers&gt;<br>        &lt;TrustStore&gt;<br>            &lt;Location&gt;repository/resources/security/client-truststore.jks&lt;/Location&gt;<br>            &lt;Type&gt;JKS&lt;/Type&gt;<br>            &lt;Password&gt;wso2carbon&lt;/Password&gt;<br>        &lt;/TrustStore&gt;<br>        &lt;KeyStore&gt;<br>            &lt;Location&gt;repository/resources/security/google-developers.jks&lt;/Location&gt;<br>            &lt;Type&gt;JKS&lt;/Type&gt;<br>            &lt;Password&gt;xxxxxx&lt;/Password&gt;<br>            &lt;KeyPassword&gt;xxxxxx&lt;/KeyPassword&gt;<br>        &lt;/KeyStore&gt;<br>    &lt;/profile&gt;<br>&lt;/parameter&gt;</pre><h3>Step 2: Enable Dynamic Loading in deployment.toml</h3><p>Next, you must instruct the API Manager’s Passthrough HTTPS Sender to load and periodically check this custom configuration file.</p><p>Add the following configurations to your primary configuration file: &lt;APIM_HOME&gt;/repository/conf/deployment.toml:</p><pre>[transport.passthru_https.sender.ssl_profile]<br># Set the path to the XML file created in Step 1<br>file_path = &quot;repository/deployment/server/multi_ssl_profiles.xml&quot;<br># Interval (in milliseconds) to check the file for updates (1 hour default)<br>interval = 3600000 <br><br>[transport.passthru_https.sender.parameters]<br># WARNING: AllowAll bypasses hostname verification. <br># Use this only for initial testing or specific IP-based configurations.<br>HostnameVerifier = &quot;AllowAll&quot;</pre><h3>❗ Important Best Practices for Configuration</h3><p>For stability and security, keep the following rules in mind when defining your profiles:</p><ol><li><strong>Always Specify the Port:</strong> When defining the &lt;servers&gt; tag (e.g., localhost:port), <strong>never omit the port</strong>. Omitting the port can lead to unexpected behavior or reliance on default port settings that may not be correct.</li><li><strong>Use Hostnames, Not IPs (Recommended):</strong></li></ol><ul><li>For custom SSL profiles, it is highly recommended to use the <strong>hostname</strong> as the server definition (&lt;servers&gt;my-backend.com:9444&lt;/servers&gt;).</li><li>If you <em>must</em> use an IP address (like 10.100.5.130), you should map that IP to a hostname in the host file of your Gateway server and then use the <strong>hostname</strong> in the SSL profile. This ensures proper security validation during the TLS handshake.</li></ul><h3>Step 3: Deployment and Verification</h3><p>After creating the XML configuration and updating the deployment.toml, you must <strong>restart the WSO2 API Manager server</strong>.</p><p>Your API Gateway is now configured to:</p><ol><li>Use the google-developers.jks keystore to present its client certificate to the backend server developers.google.com:443.</li><li>Use the client-truststore.jks to verify the certificate presented by the backend server.</li></ol><p>This dynamic configuration ensures that only the traffic destined for the specific mTLS protected backend will use the client certificate, while all other HTTPS traffic will continue to use the standard TrustStore for server verification. This modularity is key to building a resilient and secure API ecosystem.</p><p>Congratulations! You now possess the blueprint to build a more resilient and securely integrated system, significantly reducing your operational risk.</p><p>‼️ Feel free connect with me on <a href="https://www.linkedin.com/in/chethan-b-s-769920175/">LinkedIn</a> 👨‍💻 if you need any assistance.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2db9b2139f4a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Simulating Network Latency with ToxiProxy]]></title>
            <link>https://medium.com/@chethanbs21/simulating-network-latency-with-toxiproxy-bc4aaafb3c93?source=rss-f60dbb05200d------2</link>
            <guid isPermaLink="false">https://medium.com/p/bc4aaafb3c93</guid>
            <category><![CDATA[wso2]]></category>
            <category><![CDATA[network]]></category>
            <category><![CDATA[toxiproxy]]></category>
            <category><![CDATA[delay]]></category>
            <category><![CDATA[networking]]></category>
            <dc:creator><![CDATA[Chethan B S]]></dc:creator>
            <pubDate>Thu, 11 Dec 2025 13:45:37 GMT</pubDate>
            <atom:updated>2025-12-11T13:45:37.928Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1JZxbECSdlTTuLab.png" /></figure><h3>Introduction</h3><p>In the world of microservices and integration platforms like WSO2 Micro Integrator (MI), applications constantly interact with external dependencies — most crucially, the database. But what happens when that database connection becomes slow? Does your application retry gracefully? Does the transaction timeout correctly?</p><p>Manually simulating network failures is a pain. This is where <strong>ToxiProxy</strong>, an open-source tool by Shopify, becomes indispensable. ToxiProxy acts as a reliable TCP proxy that allows you to inject controlled network conditions — or “toxics” — like latency, bandwidth limits, and connection failure into your tests.</p><p>This guide walks you through setting up a dedicated proxy using ToxiProxy to inject a massive, controlled network delay between your WSO2 MI server and its database, allowing you to thoroughly test your resilience configurations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HQkdagvrzRHOFKNoaU3zjg.png" /></figure><h3>Step 1: Set Up the ToxiProxy Server</h3><p>We will run ToxiProxy inside a Docker container. <br>We expose two crucial ports: <br> — the control API port (8474) and <br> — the proxy port (15433) we will use for our database connection.</p><p>Run the following command to start the ToxiProxy container:</p><pre>docker run -d --name toxiproxy \<br>  -p 8474:8474 -p 15433:15433 \<br>  shopify/toxiproxy</pre><ul><li><strong>Port 8474:</strong> This is the <strong>ToxiProxy API control port</strong>. You use this port to dynamically create, modify, and delete proxies and toxics.</li><li><strong>Port 15433:</strong> This is the <strong>proxy port</strong>. Your WSO2 MI will connect to this port instead of the database’s default port (3306).</li></ul><h3>Step 2: Configure WSO2 MI to use the Proxy</h3><p>You must reconfigure your WSO2 MI’s datasource configuration in the deployment.toml file to point to the new proxy port.</p><p>Change the url for your database (e.g., WSO2_COORDINATION_DB) from the direct database port to the ToxiProxy listening port (15433).</p><pre>[[datasource]]<br>id = &quot;WSO2_COORDINATION_DB&quot;<br># Original URL (Direct DB connection):<br># url = &quot;jdbc:mysql://localhost:3306/clusterdb&quot; <br># New URL (Via ToxiProxy):<br>url = &quot;jdbc:mysql://localhost:15433/clusterdb&quot;</pre><h3>Step 3: Create the Database Proxy</h3><p>Now, we use the ToxiProxy API to tell it to listen on the proxy port (15433) and forward all traffic to the actual database port (3306).</p><p>Run the following curl command:</p><pre>curl -X POST http://localhost:8474/proxies \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{<br>    &quot;name&quot;: &quot;db_proxy&quot;,<br>    &quot;listen&quot;: &quot;0.0.0.0:15433&quot;,<br>    &quot;upstream&quot;: &quot;host.docker.internal:3306&quot;<br>}&#39;</pre><p><strong>Note on </strong><strong>host.docker.internal:</strong> If your database is running directly on your host machine (not in another Docker container), you must use host.docker.internal as the upstream address. If your database is in another container on the same Docker network, replace this with the database container&#39;s service name.</p><h3>Step 4: Inject the Latency Toxic</h3><p>With the proxy configured, we can now inject a specific toxic. We will add a 20,000-millisecond (20-second) delay to the <strong>downstream</strong> connection (data flowing from the database, through the proxy, back to WSO2 MI).</p><p>Run the following curl command:</p><pre>curl -X POST http://localhost:8474/proxies/db_proxy/toxics \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{<br>    &quot;name&quot;: &quot;latency_downstream&quot;,<br>    &quot;type&quot;: &quot;latency&quot;,<br>    &quot;stream&quot;: &quot;downstream&quot;,<br>    &quot;toxicity&quot;: 1.0,<br>    &quot;attributes&quot;: {<br>      &quot;latency&quot;: 20000,<br>      &quot;jitter&quot;: 0<br>    }<br>  }&#39;</pre><p>name: latency_downstream- A unique name for this specific fault injection.<br>type: latency - Specifies the type of fault (network delay).<br>stream: downstream - Applies the delay to the response traffic (DB -&gt; MI).<br>toxicity: 1.0 - A probability of 1.0 means the toxic is applied to 100% of packets.<br>latency: 20000 - The fixed delay in milliseconds (20 seconds).<br>jitter: 0 - Specifies the random variance in delay (set to 0 for a consistent 20s delay).</p><p>Your WSO2 MI is now connected to a database that will consistently take <strong>20 seconds longer</strong> to respond, allowing you to test how MI handles various endpoint timeouts and retry mechanisms.</p><h3>Step 5: Clean Up</h3><p>Once your testing is complete, it is crucial to remove the toxic so that the proxy returns to normal pass-through mode.</p><p>Run the following curl command:</p><pre>curl -X DELETE http://localhost:8474/proxies/db_proxy/toxics/latency_downstream</pre><p>The latency is instantly removed, and traffic through port 15433 will flow immediately to port 3306 without delay. You can also shut down the ToxiProxy container when finished.</p><p>This technique is a cornerstone of modern Chaos Engineering, ensuring all critical applications and services are truly resilient against real-world network fluctuations and external dependency degradation.</p><p>After this, you realize the only thing more satisfying than fixing a network delay is knowing exactly how to confidently create the next one 😉</p><p>‼️ Feel free connect with me on <a href="https://www.linkedin.com/in/chethan-b-s-769920175/">LinkedIn</a> 👨‍💻.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bc4aaafb3c93" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Configure WSO2 Identity Server 7.1.0 as an External IDP for WSO2 API Manager 4.5.0 Using OIDC (SSO)]]></title>
            <link>https://medium.com/@chethanbs21/configure-wso2-identity-server-7-1-0-as-an-external-idp-for-wso2-api-manager-4-5-0-using-oidc-sso-8952dd7bda3d?source=rss-f60dbb05200d------2</link>
            <guid isPermaLink="false">https://medium.com/p/8952dd7bda3d</guid>
            <category><![CDATA[oidc]]></category>
            <category><![CDATA[wso2-identity-server]]></category>
            <category><![CDATA[sso]]></category>
            <category><![CDATA[wso2-api-manager]]></category>
            <dc:creator><![CDATA[Chethan B S]]></dc:creator>
            <pubDate>Wed, 17 Sep 2025 13:24:10 GMT</pubDate>
            <atom:updated>2025-09-17T13:24:10.066Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u1z6qpIA27lxzZCI_w4ruA.png" /></figure><p>Managing user access across multiple applications and APIs often leads to login fatigue and complex security challenges. The WSO2 ecosystem offers a powerful solution by enabling Single Sign-On (SSO) between WSO2 Identity Server (IS) and WSO2 API Manager (APIM) using the OpenID Connect (OIDC) standard.</p><p>This integration allows users to authenticate once with the Identity Server and gain secure, seamless access to all APIs protected by the API Manager. In this guide, we’ll walk you through the essential steps to configure this powerful SSO flow, streamlining your security architecture and improving the end-user experience.</p><h3>Prerequisites</h3><p>Download and extract the below product zip files:</p><ol><li>WSO2 Identity Server 7.1.0</li><li>WSO2 API Manager 4.5.0</li></ol><h3>Step 1: Configure the Offset in Identity Server</h3><p>Open the deployment.toml file located in the &lt;IS_Home&gt;/repository/conf directory. If you are running both Identity Server (IS) and API Manager (APIM) on the same host, set the offset value to 1</p><pre>[server]<br>hostname = &quot;localhost&quot;<br>node_ip = &quot;127.0.0.1&quot;<br>offset = &quot;1&quot;<br>base_path = &quot;https://$ref{server.hostname}:${carbon.management.port}&quot;</pre><p>Save the file and start the Identity Server.</p><h3>Step 2: Create an Application (Service Provider) in Identity Server</h3><p>1. Login to the IS carbon console using the below URL.</p><pre>https://localhost:9444/console</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YSJu5UulJqEdAO_k6oz1Sw.png" /></figure><p>2. Navigate to Applications section and click on New Application.</p><p>3. On the next step, select “<strong>Traditional Web Application</strong>” and fill in the details by using the below details.</p><pre>Name: IS71IDCSSO<br>Protocol: OpenID Connect<br>Authorized redirect URLs: https://localhost:9443/commonauth</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zfTutbAuI3fK8mPRC2rs6A.png" /></figure><p>4. Click on create button to create a new service provider.</p><p>5. After creating a new application, go to <strong>“User Attributes”</strong> section and select the <strong>Groups</strong> claim under the User Attribute Selection section as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4XW2YKmk-q6Jv8w5xZsDlA.png" /></figure><p>6. Scroll down to the <strong>Subject</strong> section, select the <strong>Assign alternate subject identifier</strong> option and select the <strong>Username</strong> claim from the dropdown. (This claim will be used to create the username when performing the JIT provision at the API Manager level).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s6P2NgqjG06vxy3bmEQqBw.png" /></figure><p>7. Scroll down and click on update button.</p><p>8. Navigate to <strong>Protocol</strong> section in the same application and get the clientId and clientSecret which are required at the time of configuring the IdP in API Manager.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LEttyJn4oSKRb6_lIonXWA.png" /></figure><h3>Step 3: Adding the groups claims to the OpenID scopes in Identity Server</h3><p>To send the user’s groups to the API Manager for the JIT provisioning process based on their claims, we need to include the groups claim in the OpenID scopes.</p><p>This can be achieved by following the steps outlined below:</p><ol><li>In the left navigation menu, expand the <strong>User Attributes &amp; Stores</strong> section.</li><li>Click on the <strong>Attributes</strong> option.</li><li>On the following page, under the <strong>Manage Attribute Mappings</strong> section, select <strong>OpenID Connect</strong>. Then click on <strong>Scopes.</strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4rHC7WVBHq_yn5LHEaAF_Q.png" /></figure><p>4. You will then be presented with a page similar to the one shown below. Click on <strong>OpenId </strong>from the list.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pc-R53YguextiS9gOlFwMQ.png" /></figure><p>5. On the next page, you will be presented with a page similar to the one shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-KYAW9opLFAwOQm4kjOeng.png" /></figure><p>6. Click on “<strong>New Attribute</strong>” under <strong>Manage Attributes</strong> section and select “Groups”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RCHopDhgxH9RPAKPdjXcGg.png" /></figure><p>7. Finally, click on save changes.</p><h3>Step 4: Create User Profiles and groups in Identity Server</h3><p>Since we are configuring SSO with WSO2 Identity Server for both the Publisher and Developer Portals in API Manager, we will create the following three groups:</p><pre>TEST_CREATOR_GROUP<br>TEST_PUBLISHER_GROUP<br>TEST_SUBSCRIBER_GROUP</pre><ol><li>Navigate to the User Management section on the left panel and click on <strong>Groups</strong>.</li><li>Click on <strong>+ New Group</strong> on the next page and enter “<strong>TEST_CREATOR_GROUP</strong>” in the name field.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/781/1*xokR3lAXi1q_dHBaEBXFrQ.png" /></figure><p>3. Click on Next and Finish.</p><p>4. Repeat the above steps to add other 2 groups — <strong>TEST_PUBLISHER_GROUP</strong> &amp; <strong>TEST_SUBSCRIBER_GROUP</strong></p><p>5. Finally, you should be able to see all the created groups as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hsr9DwQX5MCcLCoKQReIog.png" /></figure><p>As a final step, let’s create below 3 users and assign the above created groups.</p><pre>creatoruser -&gt; TEST_CREATOR_GROUP<br>publisheruser -&gt; TEST_PUBLISHER_GROUP<br>subscriberuser -&gt; TEST_SUBSCRIBER_GROUP</pre><p>1. In the Carbon Console of the Identity Server, expand the <strong>User Management</strong> section from the left navigation menu.</p><p>2. Select the <strong>Users</strong> option to view the existing users.</p><p>3. Click on the <strong>+ Add User</strong> button to begin creating a new user.</p><p>4. Choose the <strong>Single User</strong> option to add one user at a time.</p><p>5. Fill in the required user details on the form, including:</p><p><strong>Username</strong>: Provide a unique identifier for the user (ex. creatoruser).<br><strong>Email</strong>: Enter the user’s email address.<br><strong>First Name</strong>: Input the user’s first name.<br><strong>Last Name</strong>: Input the user’s last name.<br><strong>Password</strong>: Set a secure password for the user account.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/783/1*6EUiWCtAlf9qHAKDjP2uOg.png" /></figure><p>6. Click on Next and assign the relevant group. (creatoruser → TEST_CREATOR_GROUP).</p><p>7. Repeat the above steps and add other 2 users (publisheruser and subscriberuser). You should be able to see all the created users as shown in the below screenshot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NYVI8MRI2Q2bO0oo6KNLtA.png" /></figure><p>With the Identity Server configuration finalized, we can now move forward to configure the API Manager.</p><h3>Step 5: Configure an IDP in API Manager</h3><p>1. Log in to the Carbon Management Console of APIM at <a href="https://localhost:9443/carbon">https://localhost:9443/carbon</a></p><p>2. In the left navigation menu, click on <strong>Add</strong> under the <strong>Identity Providers</strong> section.</p><p>3. Enter a name for the Identity Provider (e.g., WSO2_IS7).</p><p>4. Expand the Claim Configuration section.</p><p>5. Open the Basic Claim Configuration subsection.</p><p>6. Choose <strong>Define Custom Claim Dialect</strong> under the <strong>Select Claim Mapping Dialect field</strong>.</p><p>7. Click the <strong>Add Claim Mapping</strong> button to add a new claim mapping row.</p><p>8. In the <strong>Identity Provider Claim URI</strong> column, enter <strong>groups</strong>.</p><p>9. In the <strong>Local Claim URI</strong> column, select <a href="http://wso2.org/claims/role"><strong>http://wso2.org/claims/role</strong></a>.</p><p>10. In the <strong>Role Claim URI</strong> dropdown, select <strong>groups</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JU617fp4v-TVlZm0B3d6sQ.png" /></figure><p>11. Expand the Role Configuration section.</p><p>12. Click on the Add Role Mapping button to add a new role mapping.</p><p>13. In the Identity Provider Role column, enter the group name created in the Identity Server (e.g., TEST_CREATOR_GROUP).</p><p>14. In the Local Role column, select the corresponding role on the API Manager side (e.g., Internal/creator).</p><p>15. Map other 2 groups to the internal roles of APIM as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JPIp2YGVYMThx3n0QVldfA.png" /></figure><p>16. Expand the <strong>Federated Authenticators</strong> section.</p><p>17. Expand the <strong>OAuth2/OpenID</strong> Connect Configuration subsection.</p><p>18. Enable the Enable <strong>OAuth2/OpenIDConnect</strong> option by ticking the checkbox.</p><p>19. Enter the <strong>Client ID</strong> and <strong>Client Secret</strong> fields using the consumer key and secret obtained from the service provider configured in the WSO2 Identity Server.</p><p>20. Fill in the remaining fields with the following values:</p><pre>Authorization Endpoint URL: https://&lt;IS_HOST&gt;:&lt;IS_PORT&gt;/oauth2/authorize<br>Token Endpoint URL: https://&lt;IS_HOST&gt;:&lt;IS_PORT&gt;/oauth2/token<br>Callback Url: https://&lt;APIM_HOST&gt;:&lt;APIM_PORT&gt;/commonauth<br>Userinfo Endpoint URL: https://&lt;IS_HOST&gt;:&lt;IS_PORT&gt;/oauth2/userinfo?schema=openid<br>Logout Endpoint URL: https://&lt;IS_HOST&gt;:&lt;IS_PORT&gt;/oidc/logout<br>Scopes: openid</pre><p>21. Expand the <strong>Just-in-Time Provisioning</strong> section.</p><p>22. Select <strong>Always provision to User Store Domain PRIMARY</strong> with the <strong>Provision silently</strong> option enabled.</p><p>23. Click the <strong>Update</strong> button at the bottom of the page to save all the configurations.</p><h3>Step 6: Configure the IDP for the publisher and devportal service providers in APIM</h3><p>1. Click on List under the Service Providers section in the carbon console of APIM.</p><p>2. Locate the <strong>apim_devportal</strong> and <strong>apim_publisher</strong> service providers in the list.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-K76J4o-eDFeePKlCuBgxw.png" /></figure><p>3. Click the <strong>Edit</strong> button for one of the service providers (<strong>apim_publisher</strong>).</p><p>4. Expand the <strong>Local &amp; Outbound Authentication</strong> Configuration section.</p><p>5. Under the <strong>Authentication Type</strong> section, select <strong>Federated Authentication</strong>.</p><p>6. Choose the created Identity Provider (e.g., <strong>WSO2_IS7</strong>) from the dropdown.</p><p>7. Check the option <strong>Assert identity using mapped local subject identifier</strong> — this is mandatory for proper role mapping.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kcMmYlo6G8o2mlFKWfnE0A.png" /></figure><p>8. Then click on upate button and close.</p><p>9. Repeat the above steps to configure WSO2_IS7 as an IDP for <strong>apim_devportal.</strong></p><h3><strong>Step 7: Exchange the certificates</strong></h3><p>To establish a secure connection between WSO2 Identity Server (IS) and API Manager (APIM), you need to export the TLS public certificate from the Identity Server and import it into the API Manager truststore.</p><p>Follow these steps:</p><ol><li>Export the Public Certificate from Identity Server.<br>Navigate to the security directory inside the Identity Server home folder:</li></ol><pre>&lt;IS_HOME&gt;/repository/resources/security</pre><p>2. Run the following keytool command to export the public certificate from the TLS keystore:</p><pre>keytool -export -alias &lt;ALIAS_NAME&gt; -file PUB_CERT.crt -keystore &lt;TLS_KEYSTORE&gt; -storepass &lt;STORE_PASSWORD&gt; -noprompt</pre><p>Example:</p><pre>keytool -export -alias wso2carbon -file PUB_CERT.crt -keystore wso2carbon.jks -storepass wso2carbon -noprompt</pre><p>3. Import the Public Certificate into API Manager Truststore.<br>Navigate to the security directory inside the APIM Server home folder:</p><pre>&lt;APIM_HOME&gt;/repository/resources/security</pre><p>4. Run the following command to import the exported public certificate (PUB_CERT.crt) into the API Manager’s truststore:</p><pre>keytool -import -trustcacerts -alias &lt;ALIAS_NAME&gt; -file PUB_CERT.crt -keystore &lt;APIM_TRUSTSTORE&gt; -storepass &lt;TRUSTSTORE_PASSWORD&gt; -noprompt</pre><p>Example:</p><pre>keytool -import -trustcacerts -alias IS_CERT -file PUB_CERT.crt -keystore client-truststore.jks -storepass wso2carbon -noprompt</pre><p>5. After importing the certificate, restart the API Manager server to apply the changes.</p><blockquote>Hurray! We’ve conquered the setup beast on both the Identity Server and API Manager. Now, it’s time to put on our detective hats and dive into the testing phase — let’s see if everything plays nice!</blockquote><h3>Final Step is Testing</h3><ol><li>Open a devportal of APIM in your browser using the URL <a href="https://localhost:9443/devportal.">https://localhost:9443/devportal.</a></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9qVKt_XmidGZOSLHkuTT1w.png" /></figure><p>2. Click on <strong>SIGN-IN</strong> button on the top-right corner. It should redirect you to the login page of Identity Server as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QHJtRbRjs7sQ3XZTblw80Q.png" /></figure><p>3. Login with the correct credentials of an user <strong>subscriberuser</strong> who has access to devportal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*IpXt9N7hKtNdCeHVEuNGzg.png" /></figure><p>4. Once the user logs into the portal, their profile will be provisioned over to the API Manager side and will be redirected back to devporal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PhOuhlva7RxvRwtu5kf1DQ.png" /></figure><p>5. Login to the publisher portal using the credentials of the user publisheruser and test if the login and logout flow is working as expected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EJIjbhS45A64BKIOILVZZA.png" /></figure><p>Yes! Finally, our SSO configuration decided to play nice and is working as expected — no more login drama, just smooth single sign-on magic! 🎉✨.</p><p><strong>Optionally</strong>, you can verify that both the publisher user and subscriber user have been provisioned on the APIM side by checking the carbon console, as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z0nH70yYlbjC_STdZAL6ug.png" /></figure><p>Let me know in comments if you encounter any issues.</p><p>‼️ Feel free to connect with me on LinkedIn 👨‍💻.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8952dd7bda3d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build and Deploy a Custom Docker Image for Apache ActiveMQ 5.16.1]]></title>
            <link>https://medium.com/@chethanbs21/build-and-deploy-a-custom-docker-image-for-apache-activemq-5-16-1-be44223972c8?source=rss-f60dbb05200d------2</link>
            <guid isPermaLink="false">https://medium.com/p/be44223972c8</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[activemq]]></category>
            <category><![CDATA[customdockerimages]]></category>
            <dc:creator><![CDATA[Chethan B S]]></dc:creator>
            <pubDate>Fri, 01 Aug 2025 12:52:15 GMT</pubDate>
            <atom:updated>2025-08-01T12:52:15.939Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*Zynz4pMdReBIUbG9.png" /></figure><p>A step-by-step guide to building a custom image when the version you need isn’t on Docker Hub.</p><p>Apache ActiveMQ is a powerful and popular open-source message broker. When you’re working on projects, you often need to pin a specific version for stability and compatibility. But what happens when the exact version you need, like 5.16.1, isn&#39;t readily available on Docker Hub?</p><p>Don’t worry! You can easily build your own custom Docker image. In this guide, we’ll walk you through the entire process, from creating the Dockerfile to troubleshooting a common connectivity issue. Let’s get started. 🚀</p><h3>Step 1: Crafting the Dockerfile</h3><p>The Dockerfile is our blueprint. It contains all the instructions Docker needs to assemble your image. Create a new file named Dockerfile and add the following content.</p><pre># Start with a lightweight Java 8 base image<br>FROM openjdk:8-jre-alpine<br><br># Install necessary tools<br>RUN apk add --no-cache wget tar<br><br># Download and extract the specific ActiveMQ version<br>RUN wget -O activemq.tar.gz https://archive.apache.org/dist/activemq/5.16.1/apache-activemq-5.16.1-bin.tar.gz<br>RUN tar -xzf activemq.tar.gz -C /opt<br>RUN rm activemq.tar.gz<br><br># Set the environment variable for the ActiveMQ home directory<br>ENV ACTIVEMQ_HOME /opt/apache-activemq-5.16.1<br><br># Set the working directory<br>WORKDIR $ACTIVEMQ_HOME<br><br># Expose the necessary ports: 61616 for broker communication and 8161 for the web console<br>EXPOSE 61616 8161</pre><p>You may update the version in the URL https://archive.apache.org/dist/activemq/<strong>5.16.1</strong>/apache-activemq-<strong>5.16.1</strong>-bin.tar.gz to match the specific version you wish to download.</p><h3>Step 2: Building Your Custom Image</h3><p>With the Dockerfile in place, navigate to the same directory in your terminal and run the following command. This will build your Docker image and tag it as activemq:5.16.1</p><pre>docker build -t activemq:5.16.1 .</pre><p>Run the following command to list the available Docker images — you should see the newly built image in the output list.</p><pre>docker image ls</pre><h3><strong>Step 3: Launching the ActiveMQ Container</strong></h3><p>Now that you have your custom image, you can spin up a container. This command runs the container in detached mode (-d) and maps the ports to your local machine.</p><pre>docker run -d -p 61616:61616 -p 8161:8161 --name activemq activemq:5.16.1</pre><h3>Step 4: Accessing the Web Console</h3><p>You should now be able to access the ActiveMQ web console. Open your browser and navigate to <a href="http://localhost:8161/">http://localhost:8161</a></p><p>Use the default credentials to log in:</p><ul><li><strong>Username:</strong> admin</li><li><strong>Password:</strong> admin</li></ul><h4>Troubleshooting: Can’t Access the Console?</h4><p>If the page doesn’t load, you’ve likely hit a common issue. By default, ActiveMQ’s web server (Jetty) is configured to only listen on 127.0.0.1 (localhost) <em>inside</em> the container, which prevents you from accessing it from your host machine.</p><p>Here’s how to fix it:</p><p>1. Get a shell inside the running container.</p><pre>docker exec -it activemq /bin/sh</pre><p>2. Edit the jetty.xml configuration file.</p><pre>vi /opt/apache-activemq-5.16.1/conf/jetty.xml</pre><p>3. Find the jettyPort bean and change the host property from <strong>127.0.0.1</strong> to <strong>0.0.0.0</strong>. This tells ActiveMQ to accept connections from any IP address.</p><p>Look for this section:</p><pre>&lt;bean id=&quot;jettyPort&quot; class=&quot;org.apache.activemq.web.WebConsolePort&quot; init-method=&quot;start&quot;&gt;<br>         &lt;property name=&quot;host&quot; value=&quot;127.0.0.1&quot;/&gt;<br>         &lt;property name=&quot;port&quot; value=&quot;8161&quot;/&gt;<br>&lt;/bean&gt;</pre><p>Change the host value:</p><pre>&lt;property name=&quot;host&quot; value=&quot;0.0.0.0&quot;/&gt;</pre><p>4. Save the file (Escp, type :wq and enter) and exit the editor. Then, type exit to leave the container&#39;s shell.</p><h3>Step 5: Restart and Verify</h3><p>For the configuration change to take effect, you need to restart the container.</p><pre>docker container restart activemq</pre><p>Now, try accessing http://localhost:8161 again. Voila! You should be greeted by the ActiveMQ console, ready to go.</p><p>By building your own image, you gain full control over the specific versions and configurations of your services. Happy coding!</p><p>‼️ Feel free connect with me on <a href="https://www.linkedin.com/in/chethan-b-s-769920175/">LinkedIn</a> 👨‍💻.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=be44223972c8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monitoring WSO2 Micro Integrator with OpenTelemetry and Jaeger]]></title>
            <link>https://medium.com/@chethanbs21/monitoring-wso2-micro-integrator-with-opentelemetry-and-jaeger-5e258713c662?source=rss-f60dbb05200d------2</link>
            <guid isPermaLink="false">https://medium.com/p/5e258713c662</guid>
            <category><![CDATA[jaeger]]></category>
            <category><![CDATA[wso2]]></category>
            <category><![CDATA[opentelemetry]]></category>
            <category><![CDATA[wso2-mi]]></category>
            <category><![CDATA[jaegertracing]]></category>
            <dc:creator><![CDATA[Chethan B S]]></dc:creator>
            <pubDate>Sat, 28 Jun 2025 13:13:33 GMT</pubDate>
            <atom:updated>2025-06-28T13:13:33.201Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Z7xcp725OOFfmZ0A" /></figure><h3>Introduction</h3><p>Observability is crucial in any modern integration stack. With the increasing complexity of distributed systems, being able to trace how requests flow through your services is key to identifying issues, measuring performance, and ensuring reliability.</p><p>In this tutorial, we’ll walk through how to enable OpenTelemetry tracing in WSO2 Micro Integrator (MI) and visualize those traces using Jaeger, a popular distributed tracing system.</p><h3>Prerequisites</h3><ol><li>WSO2 MI 4.x (4.3)</li><li>docker setup to install jaegar tool</li></ol><h3>Step 1: Install Jaeger Using Docker</h3><p>Jaeger provides an all-in-one Docker image that includes all necessary components like the collector, query service, and UI.</p><p>Run the following command to spin up Jaeger:</p><pre>docker run -d --name jaegar \<br>  -p 16686:16686 \<br>  -p 6831:6831/udp \<br>  -p 6832:6832/udp \<br>  -p 14250:14250 \<br>  jaegertracing/all-in-one:1.70.0</pre><p>Check if the container is running:</p><pre>docker ps</pre><p>Access the Jaeger UI at: <a href="http://localhost:16686/search">http://localhost:16686/search</a></p><h3>Step 2: Set Up WSO2 Micro Integrator</h3><p>Download WSO2 MI 4.3.0 from the official WSO2 website and extract it to your preferred directory.</p><p>Next, enable OpenTelemetry tracing by editing the <em>deployment.toml</em> file located at <em>&lt;MI_HOME&gt;/conf/deployment.toml</em></p><p>Add the following configuration:</p><pre>[mediation]<br>flow.statistics.capture_all= true<br>stat.tracer.collect_payloads= true<br>stat.tracer.collect_mediation_properties= true<br><br>[opentelemetry]<br>enable = true<br>logs = true<br>type = &quot;jaeger&quot;<br>host = &quot;localhost&quot;<br>port = 14250</pre><p>Save the file and start the Micro Integrator. Make sure there are no errors in the logs indicating connection issues with Jaeger.</p><h3>Step 3: Create and Deploy Traceable APIs</h3><p>Let’s create a sample API that makes an HTTP call to a backend and logs tracing information using the below configurations:</p><pre>&lt;api context=&quot;/age/predict&quot; name=&quot;Agify_API&quot; statistics=&quot;enable&quot; trace=&quot;enable&quot; xmlns=&quot;http://ws.apache.org/ns/synapse&quot;&gt;<br>    &lt;resource methods=&quot;POST GET&quot;&gt;<br>        &lt;inSequence&gt;<br>            &lt;log&gt;<br>                &lt;property name=&quot;Flow Status&quot; value=&quot;Age Predict API is invoked&quot;/&gt;<br>            &lt;/log&gt;<br>            &lt;call&gt;<br>                &lt;endpoint&gt;<br>                    &lt;http method=&quot;get&quot; uri-template=&quot;http://localhost:8290/predict/age?name=michael&quot;&gt;<br>                        &lt;suspendOnFailure&gt;<br>                            &lt;initialDuration&gt;-1&lt;/initialDuration&gt;<br>                            &lt;progressionFactor&gt;-1&lt;/progressionFactor&gt;<br>                            &lt;maximumDuration&gt;0&lt;/maximumDuration&gt;<br>                        &lt;/suspendOnFailure&gt;<br>                        &lt;markForSuspension&gt;<br>                            &lt;retriesBeforeSuspension&gt;0&lt;/retriesBeforeSuspension&gt;<br>                        &lt;/markForSuspension&gt;<br>                    &lt;/http&gt;<br>                &lt;/endpoint&gt;<br>            &lt;/call&gt;<br>            &lt;log&gt;<br>                &lt;property name=&quot;Flow Status&quot; value=&quot;Api invocation is completed&quot;/&gt;<br>            &lt;/log&gt;<br>            &lt;respond/&gt;<br>        &lt;/inSequence&gt;<br>        &lt;outSequence/&gt;<br>        &lt;faultSequence/&gt;<br>    &lt;/resource&gt;<br>&lt;/api&gt;</pre><p>Note: Make sure <em>trace=”enable”</em> is set on the &lt;api&gt; element.</p><p>The below is the backend endpoint that will be called by the first API.</p><pre>&lt;api context=&quot;/predict&quot; name=&quot;Agify_Backend&quot; xmlns=&quot;http://ws.apache.org/ns/synapse&quot;&gt;<br>    &lt;resource methods=&quot;GET&quot; uri-template=&quot;/age?name={name}&quot;&gt;<br>        &lt;inSequence&gt;<br>            &lt;payloadFactory media-type=&quot;json&quot;&gt;<br>                &lt;format&gt;{<br>&quot;name&quot;: &quot;$1&quot;,<br>&quot;age&quot;: $2<br>}&lt;/format&gt;<br>                &lt;args&gt;<br>                    &lt;arg evaluator=&quot;xml&quot; expression=&quot;get-property(&#39;uri.var.name&#39;)&quot;/&gt;<br>                    &lt;arg value=&quot;25&quot;/&gt;<br>                &lt;/args&gt;<br>            &lt;/payloadFactory&gt;<br>            &lt;respond/&gt;<br>        &lt;/inSequence&gt;<br>        &lt;outSequence/&gt;<br>        &lt;faultSequence/&gt;<br>    &lt;/resource&gt;<br>&lt;/api&gt;</pre><h3>Step 4: Deploy and Invoke</h3><p>Bundle both APIs into a .car file and deploy it to your MI server. Once deployed, invoke the API using curl or Postman:</p><pre>curl http://localhost:8290/age/predict</pre><h3>Step 5: View Traces in Jaeger</h3><p>Now, head over to the Jaeger UI:</p><pre>http://localhost:16686/search</pre><p>In the Service dropdown, select <em>WSO2_SYNAPSE </em>and<em> </em>Click Find Traces.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lYYERMANeBxURCOPKfONGw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CRxROjBOGf89w4wPMmOSNg.png" /></figure><p>You should see your API invocation listed as a trace. Click on a trace to explore detailed spans and execution flow of mediators.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w_45hZMPMcgtEOgVaHzI8Q.png" /></figure><p>This provides visibility into each mediator executed, latency of each call and more.</p><h3>Wrapping up</h3><p>With OpenTelemetry and Jaeger, WSO2 MI gains powerful observability features that help track, debug, and optimize integration flows. This is especially helpful in production environments where understanding end-to-end latency and dependencies is crucial.</p><p>‼️ Feel free connect with me on <a href="https://www.linkedin.com/in/chethan-b-s-769920175/">LinkedIn</a> 👨‍💻.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e258713c662" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Implementing a head_message_timestamp Feature with JDBC Message Store in WSO2]]></title>
            <link>https://medium.com/@chethanbs21/implementing-a-head-message-timestamp-feature-with-jdbc-message-store-in-wso2-e9539192db74?source=rss-f60dbb05200d------2</link>
            <guid isPermaLink="false">https://medium.com/p/e9539192db74</guid>
            <category><![CDATA[wso2]]></category>
            <category><![CDATA[message-queue]]></category>
            <category><![CDATA[wso2-mi]]></category>
            <category><![CDATA[head-time-stamp]]></category>
            <category><![CDATA[message-store]]></category>
            <dc:creator><![CDATA[Chethan B S]]></dc:creator>
            <pubDate>Sat, 14 Jun 2025 11:50:04 GMT</pubDate>
            <atom:updated>2025-06-14T11:50:04.262Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/643/0*SbMDz4XIjcRdObw6.gif" /></figure><p>In message-driven architectures, tracking when messages are added to a store is crucial for monitoring and troubleshooting. While RabbitMQ provides a convenient head_message_timestamp feature out of the box, WSO2’s JDBC message store lacks native support for this functionality.</p><p>To overcome this limitation, a practical workaround is to extend the JDBC message store table by adding a timestamp column that records the exact time each message is inserted. This enables you to monitor message delays and identify how long messages have been pending by sorting entries according to their insertion timestamps.</p><p>In this blog, we’ll walk through how to implement this solution step-by-step, helping you enhance visibility into your message processing with WSO2’s JDBC message store.</p><h3>Steps to Implement Timestamp Tracking in JDBC Message Store</h3><p><strong>Step 1: Create the JDBC Message Store Table</strong></p><p>Start by creating the message store table with an additional inserted_at column to capture the timestamp when each message is inserted. Use the following SQL script:</p><pre>CREATE TABLE jdbc_message_store (<br>    indexId BIGINT(20) NOT NULL AUTO_INCREMENT,<br>    msg_id VARCHAR(200) NOT NULL,<br>    message BLOB NOT NULL,<br>    inserted_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,<br>    PRIMARY KEY (indexId)<br>);</pre><p><strong>Step 2: Set Up WSO2 Artifacts</strong></p><ol><li><strong>Message Store:</strong> Create a sample message store by referring to the below sample.</li></ol><pre>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;<br>&lt;messageStore class=&quot;org.apache.synapse.message.store.impl.jdbc.JDBCMessageStore&quot; name=&quot;SampleStore&quot; xmlns=&quot;http://ws.apache.org/ns/synapse&quot;&gt;<br>    &lt;parameter name=&quot;store.jdbc.driver&quot;&gt;com.mysql.cj.jdbc.Driver&lt;/parameter&gt;<br>    &lt;parameter name=&quot;store.producer.guaranteed.delivery.enable&quot;&gt;false&lt;/parameter&gt;<br>    &lt;parameter name=&quot;store.jdbc.username&quot;&gt;db_username&lt;/parameter&gt;<br>    &lt;parameter name=&quot;store.jdbc.connection.url&quot;&gt;jdbc:mysql://localhost:3306/db_name?useSSL=false&lt;/parameter&gt;<br>    &lt;parameter name=&quot;store.jdbc.password&quot;&gt;db_password&lt;/parameter&gt;<br>    &lt;parameter name=&quot;store.jdbc.table&quot;&gt;jdbc_message_store&lt;/parameter&gt;<br>&lt;/messageStore&gt;</pre><p>Replace the username, password and url with the appropriate values specific to your environment.</p><p>2. <strong>Endpoint:</strong> Create a target endpoint to use with Message Processor.</p><pre>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;<br>&lt;endpoint name=&quot;SampleEndpoint&quot; xmlns=&quot;http://ws.apache.org/ns/synapse&quot;&gt;<br>    &lt;http method=&quot;post&quot; uri-template=&quot;http://localhost:8290/services/MessageProcessingProxy&quot;&gt;<br>        &lt;suspendOnFailure&gt;<br>            &lt;initialDuration&gt;-1&lt;/initialDuration&gt;<br>            &lt;progressionFactor&gt;1.0&lt;/progressionFactor&gt;<br>        &lt;/suspendOnFailure&gt;<br>        &lt;markForSuspension&gt;<br>            &lt;retriesBeforeSuspension&gt;0&lt;/retriesBeforeSuspension&gt;<br>        &lt;/markForSuspension&gt;<br>    &lt;/http&gt;<br>&lt;/endpoint&gt;</pre><p><strong>3. Message Processor:</strong> Create a message processor which consumes the message from the store with an interval of 20 seconds by referring to the below sample.</p><pre>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;<br>&lt;messageProcessor class=&quot;org.apache.synapse.message.processor.impl.forwarder.ScheduledMessageForwardingProcessor&quot; messageStore=&quot;SampleStore&quot; name=&quot;SampleMessageProcessor&quot; targetEndpoint=&quot;SampleEndpoint&quot; xmlns=&quot;http://ws.apache.org/ns/synapse&quot;&gt;<br>    &lt;parameter name=&quot;client.retry.interval&quot;&gt;10000&lt;/parameter&gt;<br>    &lt;parameter name=&quot;max.delivery.attempts&quot;&gt;4&lt;/parameter&gt;<br>    &lt;parameter name=&quot;member.count&quot;&gt;1&lt;/parameter&gt;<br>    &lt;parameter name=&quot;store.connection.retry.interval&quot;&gt;1000&lt;/parameter&gt;<br>    &lt;parameter name=&quot;max.store.connection.attempts&quot;&gt;-1&lt;/parameter&gt;<br>    &lt;parameter name=&quot;max.delivery.drop&quot;&gt;Disabled&lt;/parameter&gt;<br>    &lt;parameter name=&quot;interval&quot;&gt;20000&lt;/parameter&gt;<br>    &lt;parameter name=&quot;is.active&quot;&gt;true&lt;/parameter&gt;<br>&lt;/messageProcessor&gt;</pre><p>4. <strong>ProxyService 1:</strong> Create a proxy service which uses the JDBC message store created above by referring to the below sample.</p><pre>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;<br>&lt;proxy name=&quot;MessageStoreProxy&quot; startOnLoad=&quot;true&quot; transports=&quot;http https&quot; xmlns=&quot;http://ws.apache.org/ns/synapse&quot;&gt;<br>    &lt;target&gt;<br>        &lt;inSequence&gt;<br>            &lt;property name=&quot;FORCE_SC_ACCEPTED&quot; scope=&quot;axis2&quot; type=&quot;STRING&quot; value=&quot;true&quot;/&gt;<br>            &lt;property name=&quot;OUT_ONLY&quot; scope=&quot;default&quot; type=&quot;STRING&quot; value=&quot;true&quot;/&gt;<br>            &lt;store messageStore=&quot;SampleStore&quot;/&gt;<br>        &lt;/inSequence&gt;<br>        &lt;outSequence/&gt;<br>        &lt;faultSequence/&gt;<br>    &lt;/target&gt;<br>&lt;/proxy&gt;</pre><p>5. <strong>Proxy service 2</strong>: Create a proxy service to log the message processed by MessageProcessor.</p><pre>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;<br>&lt;proxy name=&quot;MessageProcessingProxy&quot; startOnLoad=&quot;true&quot; transports=&quot;http https&quot; xmlns=&quot;http://ws.apache.org/ns/synapse&quot;&gt;<br>    &lt;target&gt;<br>        &lt;inSequence&gt;<br>            &lt;log&gt;<br>                &lt;property expression=&quot;json-eval($)&quot; name=&quot;Message Received&quot;/&gt;<br>            &lt;/log&gt;<br>            &lt;drop/&gt;<br>        &lt;/inSequence&gt;<br>        &lt;outSequence/&gt;<br>        &lt;faultSequence/&gt;<br>    &lt;/target&gt;<br>&lt;/proxy&gt;</pre><p>Export a carbon application by including all the above created artifacts and deploy in WSO2 MicroIntegrator.</p><p><strong>Step 3: Invoke the proxy service</strong></p><p>Invoke the proxy service multiple times with a sample json payload to store several messages in the jdbc_message_store.</p><pre>curl --location &#39;http://localhost:8290/services/MessageStoreProxy&#39; \<br>--header &#39;Content-Type: application/json&#39; \<br>--data &#39;{<br>    &quot;greetings&quot;:&quot;Hi! Good Morning&quot;<br>}&#39;</pre><p><strong>Step 4: Check the timestamp field in the database</strong></p><p>Check if the timestamps are attached to the messages inserted in the <em>jdbc_message_store</em> table.</p><pre>select * from jdbc_message_store;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/1*Ymp3K3UU4Ch1gsgVpjiGaw.png" /></figure><p><strong>Step 5: Retrieve the head_message_timestamp</strong></p><p>In order to check the <em>head_message_timestamp</em>, sort the records stored in the table in ascending order and check the timestamp field of the first record. This gives you the timestamp of head message in the JDBC message store.</p><pre>select inserted_at from jdbc_message_store order by inserted_at asc LIMIT 1;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/177/1*4jxgnsKYChZU6lhPbTPJ3A.png" /></figure><blockquote>“With great message storage comes great extensibility — go forth and extend responsibly!”</blockquote><p>‼️ Feel free connect with me on <a href="https://www.linkedin.com/in/chethan-b-s-769920175/">LinkedIn</a> 👨‍💻.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e9539192db74" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>