<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Awynimantha on Medium]]></title>
        <description><![CDATA[Stories by Awynimantha on Medium]]></description>
        <link>https://medium.com/@awynimantha?source=rss-7d6567f2a533------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*_yPEqIIOLSWdj9CM</url>
            <title>Stories by Awynimantha on Medium</title>
            <link>https://medium.com/@awynimantha?source=rss-7d6567f2a533------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 18:36:25 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@awynimantha/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Lightweight Correlation Logs for WSO2 API Manager Versions That Don’t Have Them: A Custom Synapse…]]></title>
            <link>https://medium.com/@awynimantha/lightweight-correlation-logs-for-wso2-api-manager-versions-that-dont-have-them-a-custom-synapse-31991192c842?source=rss-7d6567f2a533------2</link>
            <guid isPermaLink="false">https://medium.com/p/31991192c842</guid>
            <dc:creator><![CDATA[Awynimantha]]></dc:creator>
            <pubDate>Sat, 14 Nov 2026 03:49:45 GMT</pubDate>
            <atom:updated>2026-11-14T04:15:05.380Z</atom:updated>
            <content:encoded><![CDATA[<h3>Lightweight Correlation Logs for WSO2 API Manager Versions That Don’t Have Them: A Custom Synapse Handler</h3><h3>What are correlation logs in WSO2 API Manager?</h3><p>When an API call is slow, the first question is always: <strong>where did the time go?</strong> It could be the gateway’s own processing (authentication, throttling, mediation policies), a database or LDAP lookup, or the backend service. A single access log line gives you only the total round trip, so it can’t answer that. WSO2 API Manager’s answer is <strong>correlation logs</strong>. Each incoming request gets a <strong>correlation ID</strong>. The client can send its own ID in the activityid header, otherwise APIM generates a random one. Every timed operation APIM performs for that request is then written to &lt;APIM_HOME&gt;/repository/logs/correlation.log under that ID:</p><ul><li><strong>Method calls:</strong> how long selected internal methods took</li><li><strong>External calls:</strong> every HTTP/HTTPS call (for example, to the backend or the Key Manager), with its duration</li><li><strong>Database calls:</strong> every JDBC query and LDAP operation, with its duration Grep one correlation ID and you can follow the full journey, client → gateway → backend → gateway → client, with the cost of each step. You enable correlation logs with a system property at startup:</li></ul><p>From APIM 4.2.0, you can also switch them on and off at runtime through the DevOps REST API. APIM 3.2.0, from the <strong>29 October 2021</strong> update onwards, also has a much lighter mode called <strong>Lightweight Observability</strong> (message tracking logs, -DenableMessageTrackingLogs=true). It logs only four entries per request: when the request arrives, when it is sent to the backend, when the backend responds, and when the response goes back to the client.</p><h3>The catch: correlation logs can change what you are measuring</h3><p>Full correlation logging is one of the most useful debugging tools in APIM, but it is heavy. WSO2’s own documentation says it is disabled by default because it impacts performance. From 4.1.0 onwards, the docs introduce a separate “API Logs” feature specifically because correlation logs “cause lower performance on the API gateway”. In practice, enabling it on a busy gateway can <strong>hide or distort the very behaviour you are trying to observe</strong>:</p><ul><li><strong>It adds its own latency.</strong> Every instrumented method, JDBC statement, LDAP call and HTTP call produces a timed log entry, and the default appender writes synchronously to disk. The latencies you read include time spent producing the logs.</li><li><strong>The observer effect.</strong> If the problem is latency or throughput under load, the extra I/O and CPU change the system’s timing. A contention problem can get worse, move somewhere else, or disappear. Thread-pool saturation looks different. You can end up diagnosing the instrumentation rather than the original issue.</li><li><strong>Log volume.</strong> Many entries per request on a high-traffic gateway fill correlation.log quickly. Rotation can delete the incident window before anyone looks at it.</li><li><strong>Restart required (before 4.2.0).</strong> Turning it on means restarting the gateway, and a restart can clear a transient problem such as a leaked connection pool or stuck threads. So full correlation logs are best used for <strong>short, targeted sessions</strong>, ideally on one node or in a reproduction environment. For the everyday question “is it the gateway or the backend?”, you want something much lighter.</li></ul><h3>Solution: a lightweight correlation handler</h3><p>To fill that gap, we wrote <strong>LightweightObservabilityHandler</strong>, a small custom <strong>global Synapse handler</strong> (about 150 lines). It gives any APIM gateway the lightweight four-point trace, with latency figures already worked out. The API Gateway runs on Apache Synapse. A class that extends org.apache.synapse.AbstractSynapseHandler gets four callbacks for every message:</p><pre>Client ──► [ handleRequestInFlow ] ──► mediation ──► [ handleRequestOutFlow ] ──► Backend<br> Client ◄── [ handleResponseOutFlow ] ◄── mediation ◄── [ handleResponseInFlow ] ◄── Backend</pre><p>At each point, the handler writes one line to correlation.log. It reuses the pack&#39;s existing correlation logger, so no log4j2 changes are needed. All four lines share one correlation ID (the activityid header, or a generated UUID), and each line carries the latency known at that moment:</p><pre>CorrelationId: b157e3fb-…, Direction: RequestIn, HTTPMethod: GET, Destination: /orders/1.0.0/items<br>CorrelationId: b157e3fb-…, Direction: RequestOut, HTTPMethod: GET, Destination: http://backend:9001/items, MediationInLatencyMs: 5<br>CorrelationId: b157e3fb-…, Direction: ResponseIn, HTTPStatusCode: 200, Origin: http://backend:9001/, BackendLatencyMs: 1532<br>CorrelationId: b157e3fb-…, Direction: ResponseOut, HTTPStatusCode: 200, MediationOutLatencyMs: 4, RoundTripLatencyMs: 1541</pre><p>Here, 1532 ms of the 1541 ms round trip was spent on the backend, and the gateway added under 10 ms. The three segments are:</p><ul><li><strong>MediationInLatencyMs</strong> (RequestIn → RequestOut): gateway processing on the way in</li><li><strong>BackendLatencyMs</strong> (RequestOut → ResponseIn): network plus backend processing</li><li><strong>MediationOutLatencyMs</strong> (ResponseIn → ResponseOut): gateway processing on the way out Compared with full correlation logging, the cost is <strong>four log lines per request, whatever the request does</strong>. There are no per-method, per-query or per-call entries. Being registered globally, it covers every API on the gateway with no per-API changes.</li></ul><h3>The code</h3><pre>package org.wso2.cre.handler;<br>import org.apache.commons.logging.Log;<br>import org.apache.commons.logging.LogFactory;<br>import org.apache.synapse.AbstractSynapseHandler;<br>import org.apache.synapse.MessageContext;<br>import org.apache.synapse.core.axis2.Axis2MessageContext;<br>import java.util.Map;<br>import java.util.UUID;<br><br>public class LightweightObservabilityHandler extends AbstractSynapseHandler {<br>    private static final Log correlationLog = LogFactory.getLog(&quot;correlation&quot;);<br>    private static final String CORRELATION_ID_MC_PROPERTY = &quot;correlation_id&quot;;<br>    private static final String ACTIVITY_ID_HEADER = &quot;activityid&quot;;<br>    private static final String T_REQUEST_IN = &quot;lwo_t_request_in&quot;;<br>    private static final String T_REQUEST_OUT = &quot;lwo_t_request_out&quot;;<br>    private static final String T_RESPONSE_IN = &quot;lwo_t_response_in&quot;;<br>    private static final String NOT_AVAILABLE = &quot;N/A&quot;;<br>    public boolean handleRequestInFlow(MessageContext mc) {<br>        try {<br>            mc.setProperty(T_REQUEST_IN, System.currentTimeMillis());<br>            String correlationId = resolveCorrelationId(mc);<br>            mc.setProperty(CORRELATION_ID_MC_PROPERTY, correlationId);<br>            correlationLog.info(&quot;CorrelationId: &quot; + correlationId +<br>                &quot;, Direction: RequestIn&quot; +<br>                &quot;, HTTPMethod: &quot; + getAxis2Property(mc, &quot;HTTP_METHOD&quot;) +<br>                &quot;, Destination: &quot; + (mc.getTo() != null ? mc.getTo().getAddress() : null));<br>        } catch (Exception e) {<br>            correlationLog.error(&quot;Error logging RequestIn: &quot; + e.getMessage(), e);<br>        }<br>        return true;<br>    }<br>    public boolean handleRequestOutFlow(MessageContext mc) {<br>        try {<br>            long now = System.currentTimeMillis();<br>            // First write wins: on a backend-failure path this callback can fire a second,<br>            // spurious time. Never let that overwrite the real dispatch timestamp.<br>            if (mc.getProperty(T_REQUEST_OUT) == null) {<br>                mc.setProperty(T_REQUEST_OUT, now);<br>            }<br>            Object endpoint = mc.getProperty(&quot;ENDPOINT_ADDRESS&quot;);<br>            String destination = endpoint != null ? (String) endpoint :<br>                (mc.getTo() != null ? mc.getTo().getAddress() : null);<br>            correlationLog.info(&quot;CorrelationId: &quot; + mc.getProperty(CORRELATION_ID_MC_PROPERTY) +<br>                &quot;, Direction: RequestOut&quot; +<br>                &quot;, HTTPMethod: &quot; + getAxis2Property(mc, &quot;HTTP_METHOD&quot;) +<br>                &quot;, Destination: &quot; + destination +<br>                &quot;, MediationInLatencyMs: &quot; + latency(mc, T_REQUEST_IN, now));<br>        } catch (Exception e) {<br>            correlationLog.error(&quot;Error logging RequestOut: &quot; + e.getMessage(), e);<br>        }<br>        return true;<br>    }<br>    public boolean handleResponseInFlow(MessageContext mc) {<br>        try {<br>            long now = System.currentTimeMillis();<br>            mc.setProperty(T_RESPONSE_IN, now);<br>            correlationLog.info(&quot;CorrelationId: &quot; + mc.getProperty(CORRELATION_ID_MC_PROPERTY) +<br>                &quot;, Direction: ResponseIn&quot; +<br>                &quot;, HTTPStatusCode: &quot; + getAxis2Property(mc, &quot;HTTP_SC&quot;) +<br>                &quot;, Origin: &quot; + mc.getProperty(&quot;ENDPOINT_PREFIX&quot;) +<br>                &quot;, BackendLatencyMs: &quot; + latency(mc, T_REQUEST_OUT, now));<br>        } catch (Exception e) {<br>            correlationLog.error(&quot;Error logging ResponseIn: &quot; + e.getMessage(), e);<br>        }<br>        return true;<br>    }<br>    public boolean handleResponseOutFlow(MessageContext mc) {<br>        try {<br>            long now = System.currentTimeMillis();<br>            correlationLog.info(&quot;CorrelationId: &quot; + mc.getProperty(CORRELATION_ID_MC_PROPERTY) +<br>                &quot;, Direction: ResponseOut&quot; +<br>                &quot;, HTTPStatusCode: &quot; + getAxis2Property(mc, &quot;HTTP_SC&quot;)<br>                // N/A when ResponseIn never fired (backend refused / timed out)<br>                +<br>                &quot;, MediationOutLatencyMs: &quot; + latency(mc, T_RESPONSE_IN, now) +<br>                &quot;, RoundTripLatencyMs: &quot; + latency(mc, T_REQUEST_IN, now));<br>        } catch (Exception e) {<br>            correlationLog.error(&quot;Error logging ResponseOut: &quot; + e.getMessage(), e);<br>        }<br>        return true;<br>    }<br>    private String latency(MessageContext mc, String fromKey, long to) {<br>        Long from = (Long) mc.getProperty(fromKey);<br>        return from != null ? String.valueOf(to - from) : NOT_AVAILABLE;<br>    }<br>    private String resolveCorrelationId(MessageContext mc) {<br>        Map headers = (Map) axis2(mc).getProperty(<br>            org.apache.axis2.context.MessageContext.TRANSPORT_HEADERS);<br>        String activityId = headers != null ? (String) headers.get(ACTIVITY_ID_HEADER) : null;<br>        return (activityId != null &amp;&amp; !activityId.isEmpty()) ?<br>            activityId : UUID.randomUUID().toString();<br>    }<br>    private String getAxis2Property(MessageContext mc, String name) {<br>        Object v = axis2(mc).getProperty(name);<br>        return v != null ? String.valueOf(v) : null;<br>    }<br>    private org.apache.axis2.context.MessageContext axis2(MessageContext mc) {<br>        return ((Axis2MessageContext) mc).getAxis2MessageContext();<br>    }<br>}</pre><h3>Building it</h3><p>The handler needs three libraries: synapse-core, axis2, and a commons-logging API. <strong>Always compile against the exact jars in your target pack&#39;s </strong><strong>repository/components/plugins directory.</strong> WSO2 ships forked versions of these libraries, and building against the gateway&#39;s own jars avoids binary-compatibility surprises.</p><pre>&lt;properties&gt;<br> &lt;apim.pack.plugins&gt;/path/to/&lt;APIM_HOME&gt;/repository/components/plugins&lt;/apim.pack.plugins&gt;<br>&lt;/properties&gt;<br>&lt;dependencies&gt;<br> &lt;dependency&gt;<br> &lt;groupId&gt;org.apache.synapse&lt;/groupId&gt;<br> &lt;artifactId&gt;synapse-core&lt;/artifactId&gt;<br> &lt;version&gt;${synapse.version}&lt;/version&gt;<br> &lt;scope&gt;system&lt;/scope&gt;<br> &lt;systemPath&gt;${apim.pack.plugins}/synapse-core_${synapse.version}.jar&lt;/systemPath&gt;<br> &lt;/dependency&gt;<br> &lt;dependency&gt;<br> &lt;groupId&gt;org.apache.axis2&lt;/groupId&gt;<br> &lt;artifactId&gt;axis2-kernel&lt;/artifactId&gt;<br> &lt;version&gt;${axis2.version}&lt;/version&gt;<br> &lt;scope&gt;system&lt;/scope&gt;<br> &lt;systemPath&gt;${apim.pack.plugins}/axis2_${axis2.version}.jar&lt;/systemPath&gt;<br> &lt;/dependency&gt;<br> &lt;dependency&gt;<br> &lt;groupId&gt;org.ops4j.pax.logging&lt;/groupId&gt;<br> &lt;artifactId&gt;pax-logging-api&lt;/artifactId&gt;<br> &lt;version&gt;${paxlogging.version}&lt;/version&gt;<br> &lt;scope&gt;system&lt;/scope&gt;<br> &lt;systemPath&gt;${apim.pack.plugins}/org.ops4j.pax.logging.pax-logging-api_${paxlogging.version}.jar&lt;/systemPath&gt;<br> &lt;/dependency&gt;<br>&lt;/dependencies&gt;</pre><p>Find the versions for your pack with:</p><pre>ls &lt;APIM_HOME&gt;/repository/components/plugins | grep -E &#39;^synapse-core_|^axis2_1|pax-logging-api&#39;</pre><p>For reference, these are the versions in a few GA packs (update levels can change them): | Pack | synapse-core | axis2 | pax-logging-api | | — -| — -| — -| — -| | 3.1.0 | 2.1.7.wso2v147 | 1.6.1.wso2v41 | 1.11.3 | | 4.1.0 | 2.1.7.wso2v271 | 1.6.1.wso2v80 | 2.1.0.wso2v4 | | 4.7.0 | 4.1.0.wso2v40_11 | 1.6.1.wso2v125_1 | 2.3.5.wso2v1 | Then run mvn clean package.</p><h3>How to use it on each APIM version</h3><p>In every version, start by copying the jar into the pack:</p><pre>cp target/LightweightObservabilityHandler-1.0.0.jar &lt;APIM_HOME&gt;/repository/components/lib/</pre><p>What follows depends on the version. <strong>The </strong><strong>deployment.toml registration differs between 3.x and 4.x</strong>, and using the wrong form fails silently: the server starts normally and the handler never runs.</p><h3>APIM 3.0.0 and 3.1.0: use the handler</h3><p>These versions have no lightweight option, so this is where the handler helps most. Add the following to deployment.toml, near the top, before the first [table] header:</p><pre>enabled_global_handlers = [&quot;external_call_logger&quot;, &quot;open_tracing&quot;, &quot;lightweight_observability&quot;]<br>[synapse_handlers.lightweight_observability]<br>name = &quot;lightweightObservabilityHandler&quot;<br>class = &quot;org.wso2.cre.handler.LightweightObservabilityHandler&quot;</pre><p>If enabled_global_handlers already exists, append &quot;lightweight_observability&quot; to it. Keep the default entries: if you remove them, those handlers are disabled. Restart the gateway. <em>(Tested end to end on 3.1.0. On 3.0.0, check after the first restart that the generated </em><em>synapse-handlers.xml contains the new handler.)</em></p><h3>APIM 3.2.0: check your update level first</h3><ul><li><strong>Update the APIM to latest:</strong> you already have the built-in feature. Add -DenableMessageTrackingLogs=true \ to bin/wso2server.sh (or .bat), just before the org.wso2.carbon.bootstrap.Bootstrap line, and restart. Use the custom handler only if you also want the latency figures computed in each line.</li></ul><h3>APIM 4.0.0: use the handler (4.x registration)</h3><p>4.0.0 has neither lightweight message tracking nor API Logs. Use the handler, and note the different registration format. The 4.x template has <strong>no </strong><strong>enabled_global_handlers key</strong>. It only creates handlers for [synapse_handlers.*] entries that have enabled = true:</p><pre>[synapse_handlers.lightweight_observability]<br>enabled = true<br>class = &quot;org.wso2.cre.handler.LightweightObservabilityHandler&quot;</pre><p>Rebuild against the 4.0.0 pack’s own jars, and restart. <em>(Check the generated </em><em>synapse-handlers.xml after restart to confirm the handler is there.)</em></p><h3>APIM 4.1.0 and later: try the built-in API Logs first</h3><p>From 4.1.0, APIM has <strong>API Logs</strong>: a lightweight, per-API feature you can switch on at runtime through the DevOps REST API or apictl. It writes to repository/logs/api.log. Its BASIC level logs the REQUEST_IN, REQUEST_OUT, RESPONSE_IN and RESPONSE_OUT flows with a correlation ID, and needs no restart. Use the custom handler on these versions when you want:</p><ul><li>the latency segments already computed in each line,</li><li>coverage of every API on the gateway without turning logging on per API, or</li><li>one consistent log format across a mix of 3.x and 4.x gateways. The registration is the same 4.x form as for 4.0.0:</li></ul><pre>[synapse_handlers.lightweight_observability]<br>enabled = true<br>class = &quot;org.wso2.cre.handler.LightweightObservabilityHandler&quot;</pre><p>Rebuild against your pack’s jars (4.7.0 ships synapse-core 4.1.0.wso2v40_11, not 2.1.7), then restart. In both 4.1.0 and 4.7.0, the AbstractSynapseHandler API and the correlation logger are unchanged, so the code itself needs no changes.</p><h3>Verify (all versions)</h3><pre>tail -f &lt;APIM_HOME&gt;/repository/logs/correlation.log</pre><p>Invoke any API. You should see four lines per request, sharing one CorrelationId.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=31991192c842" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cracking PassThrough Transport — Part 1: NIO Fundamentals]]></title>
            <link>https://medium.com/@awynimantha/cracking-passthrough-transport-part-1-nio-fundamentals-c68dc4231ef8?source=rss-7d6567f2a533------2</link>
            <guid isPermaLink="false">https://medium.com/p/c68dc4231ef8</guid>
            <dc:creator><![CDATA[Awynimantha]]></dc:creator>
            <pubDate>Tue, 30 Jun 2026 15:16:12 GMT</pubDate>
            <atom:updated>2026-06-30T15:22:51.560Z</atom:updated>
            <content:encoded><![CDATA[<p>This article is part of series exploring the inner workings of how the PassThrough transport operates in WSO2 products. This particular article covers the NIO fundamentals that form the basis of the PassThrough transport.</p><h3>Non-blocking IO</h3><p>When it comes to NIO, there are three basic things we should know:</p><p>i) Buffers, <br>ii) Channels<br>iii) Selectors</p><h4>Buffer and Channels</h4><p>Buffers work hand in glove with channels. Channels are the portals through which I/O transfers take place, while buffers serve as the sources or targets of those data transfers.</p><p>For instance, let’s consider a scenario where WSO2 MI sends a packet to the backend. The diagram below shows, at a high level, how the coordination between buffers and channels works.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/711/1*ewSVxOfUhkUvMNZXgfo28A.png" /></figure><h4>Selectors</h4><p>The Selector is the core operational unit in NIO, and it’s what makes everything efficient and fast. It implements the core concepts of the Reactor pattern [1]. Selectors make two things possible:</p><p>i) readiness selection<br>ii) I/O multiplexing.</p><p>First, it’s worth understanding why readiness selection helps handle I/O events efficiently with minimal overhead. What the Reactor pattern does so elegantly is avoid waiting for an I/O event by blocking a thread on some kind of polling mechanism, which would otherwise be costly given how frequent and fine-grained I/O events tend to be. Instead, it lets the OS kernel do that work and signal the selector once the I/O event actually occurs.</p><p>This does still cause the selector thread to block but crucially, it’s not blocking on behalf of a single connection. It’s blocking on behalf of many connections at once, which is exactly what makes I/O multiplexing possible.</p><p>By offloading the costly readiness checking work to the OS, the selector simply waits for a signal that a channel is ready, rather than doing that work itself.</p><h4>Registering</h4><p>First things first: channels need to be registered with the selector before anything else. When you call the register method, it returns a SelectionKey. This SelectionKey represents what the selector is actually interested in for that channel. Also keep in mind that not all channels can be registered with a selector for the sake of keeping things simple, let’s skip that detail for now.</p><h4>Select()</h4><p>Once registration is done, the selector is ready to perform readiness selection and multiplexing across the channels. This is where the select method comes into play. When you call this method, the selector checks in with the OS and updates the ready state of all registered channels accordingly and the method returns a count of how many keys changed as a result.</p><p><strong>Ask the OS who’s ready<br></strong>The selector checks with the operating system: “of all the sockets I’m watching, which ones actually have something to do, data to read, room to write, etc.?” This is the step where your thread might just sit and wait, doing nothing, until at least one socket has something ready (or a timeout you set runs out).</p><ol><li>When the OS says “this one’s ready,” there are two cases:<br>If this is the first time we’re seeing this channel as ready, just record what it’s ready for.</li><li>If we already noted this channel as ready from before and haven’t processed it yet, add the new info on top of the old info never erase what was already marked, only add more.</li></ol><p>So readiness builds up over time rather than resetting each round. Even if a socket stops being ready for something it was ready for last time, the old “this was ready” flag sticks around until your code actually deals with it.</p><p><strong>What the number you get back actually means<br></strong>When select() finishes, it gives you back a number. It actually means “how many sockets just became newly ready during this specific call.” If a socket was already flagged as ready from a previous round and nothing changed, it doesn’t count again. So you could legitimately get back 0, even though there are still sockets sitting around marked as ready from earlier they just didn’t change this time.</p><p>During next article we’ll discuss about how these data are read and used at application level to implement the passthrough sender.</p><p>If you want to dig deeper into NIO, <strong><em>Java NIO</em> by Ron Hitchens</strong> is a great read.</p><p>Below is a coding activity you can engage to understand how exactly does the selector can be used.</p><p><strong>Create SelectorDemo.java:</strong></p><pre>import java.io.IOException;<br>import java.net.InetSocketAddress;<br>import java.nio.channels.SelectionKey;<br>import java.nio.channels.Selector;<br>import java.nio.channels.ServerSocketChannel;<br><br>public class SelectorDemo {<br><br>    public static void main(String[] args) throws IOException {<br><br>        Selector selector = Selector.open();<br><br>        ServerSocketChannel serverChannel = ServerSocketChannel.open();<br>        serverChannel.bind(new InetSocketAddress(9090));<br>        serverChannel.configureBlocking(false);<br><br>        serverChannel.register(selector, SelectionKey.OP_ACCEPT);<br><br>        System.out.println(&quot;Listening on port 9090...&quot;);<br><br>        while (true) {<br>            int readyCount = selector.select();<br><br>            System.out.println(<br>                    &quot;select() returned: &quot; + readyCount<br>                            + &quot; | total registered keys: &quot;<br>                            + selector.keys().size()<br>            );<br>        }<br>    }<br>}</pre><p>Run it, then from another terminal: telnet localhost 9090. You should see select() returned: 1 print the moment telnet connects. Leave telnet open, don’t go further yet confirm this works before adding anything.</p><p><strong>Accept the connection and register it for reads</strong></p><p>Replace the while loop body with:</p><pre>while (true) {<br><br>    selector.select();<br><br>    Iterator&lt;SelectionKey&gt; it = selector.selectedKeys().iterator();<br><br>    while (it.hasNext()) {<br><br>        SelectionKey key = it.next();<br>        it.remove(); // Must remove manually - the selector never does this for you.<br><br>        if (key.isAcceptable()) {<br><br>            ServerSocketChannel server = (ServerSocketChannel) key.channel();<br>            SocketChannel client = server.accept();<br><br>            client.configureBlocking(false);<br>            client.register(selector, SelectionKey.OP_READ);<br><br>            System.out.println(<br>                    &quot;Accepted connection from &quot; + client.getRemoteAddress()<br>            );<br>        }<br>    }<br>}</pre><p>Run it, telnet in. You should see “Accepted connection from …” print once. Telnet should just sit there connected nothing else happens yet since you haven’t handled OP_READ.</p><p><strong>Read raw bytes and print them</strong><br>Add an else if for the readable case inside the same inner while:</p><pre>while (true) {<br><br>    selector.select();<br><br>    Iterator&lt;SelectionKey&gt; it = selector.selectedKeys().iterator();<br><br>    while (it.hasNext()) {<br><br>        SelectionKey key = it.next();<br>        it.remove();<br><br>        if (key.isAcceptable()) {<br><br>            ServerSocketChannel server = (ServerSocketChannel) key.channel();<br>            SocketChannel client = server.accept();<br><br>            client.configureBlocking(false);<br>            client.register(selector, SelectionKey.OP_READ);<br><br>            System.out.println(<br>                    &quot;Accepted connection from &quot; + client.getRemoteAddress()<br>            );<br><br>        } else if (key.isReadable()) {<br><br>            SocketChannel client = (SocketChannel) key.channel();<br>            ByteBuffer buffer = ByteBuffer.allocate(1024);<br><br>            int bytesRead;<br><br>            try {<br>                bytesRead = client.read(buffer);<br>            } catch (IOException e) {<br>                System.out.println(<br>                        &quot;Read error, closing connection: &quot; + e.getMessage()<br>                );<br>                key.cancel();<br>                client.close();<br>                continue;<br>            }<br><br>            if (bytesRead == -1) {<br><br>                System.out.println(&quot;Client closed the connection.&quot;);<br>                key.cancel();<br>                client.close();<br><br>            } else if (bytesRead &gt; 0) {<br><br>                buffer.flip();<br><br>                // Print raw byte values first.<br>                System.out.print(&quot;Raw bytes: &quot;);<br>                for (int i = 0; i &lt; bytesRead; i++) {<br>                    System.out.print((buffer.get(i) &amp; 0xFF) + &quot; &quot;);<br>                }<br>                System.out.println();<br><br>                // Decode the same bytes as text.<br>                String text = new String(buffer.array(), 0, bytesRead);<br>                System.out.println(&quot;As text: &quot; + text.trim());<br>            }<br><br>            // bytesRead == 0 means no data arrived yet; continue waiting.<br>        }<br>    }<br>}</pre><p>Run it, telnet in, type hello and press enter. You should see both lines print:<br>Raw bytes: 104 101 108 108 111 13 10 <br>As text: hello</p><p>The 13 10 at the end is \r\n — telnet sends both characters when you hit enter, which is exactly the kind of byte-level detail that’s invisible once you only look at the decoded string.</p><p>Now close the telnet session (Ctrl+] then quit, or just close the terminal). You should see “Client closed the connection.”</p><p>[1] <a href="https://en.wikipedia.org/wiki/Reactor_pattern">https://en.wikipedia.org/wiki/Reactor_pattern</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c68dc4231ef8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monitoring Ballerina Database Connections with Prometheus: A Step-by-Step Guide]]></title>
            <link>https://medium.com/@awynimantha/monitoring-ballerina-database-connections-with-prometheus-a-step-by-step-guide-7221339b2667?source=rss-7d6567f2a533------2</link>
            <guid isPermaLink="false">https://medium.com/p/7221339b2667</guid>
            <dc:creator><![CDATA[Awynimantha]]></dc:creator>
            <pubDate>Tue, 30 Jun 2026 13:08:23 GMT</pubDate>
            <atom:updated>2026-06-30T13:08:23.512Z</atom:updated>
            <content:encoded><![CDATA[<p>If you’ve ever run a Ballerina service backed by a database in production, you know that connection pool behavior is one of those things you only think about after something goes wrong. A pool that’s exhausted, a pool that resets its identity on every restart, a pool you simply can’t see — these are the kinds of problems that turn a quiet afternoon into an incident.</p><p>The good news is that recent releases of Ballerina’s data access modules have made this much easier to observe. With <strong>ballerina/sql v1.19.0</strong> and the newly released <strong>ballerinax/postgresql v1.18.0</strong>, connection pool metrics are now exposed in a way that plays nicely with Prometheus — provided you wire things up correctly.</p><p>This article walks through exactly how to do that, from updating your Ballerina version to seeing live metrics scraped by Prometheus.</p><h3>What’s New</h3><p>Two modules were released together as part of this observability push:</p><ul><li><strong>ballerina/sql v1.19.0</strong></li><li><strong>ballerinax/postgresql v1.18.0</strong></li></ul><p>One important compatibility note: if you adopt SQL v1.19.0, every module in your project needs to be updated to <strong>Ballerina 2201.13.0</strong>. This isn’t optional — mismatched versions across modules will cause build or runtime issues, so plan your upgrade as a single coordinated step rather than bumping the SQL module in isolation.</p><h3>Step 1: Enable Observability in Ballerina.toml</h3><p>The first switch you need to flip lives in your project’s Ballerina.toml file. Add the following under a [build-options] section:</p><pre>[build-options]<br>observabilityIncluded = true</pre><p>This tells the Ballerina compiler to bake observability instrumentation into your compiled artifact. Without this flag, the metrics simply won’t be collected, no matter what you configure elsewhere.</p><h3>Step 2: Configure Config.toml</h3><p>Next, you’ll need to set up three areas of configuration in Config.toml: the connection pool name, the observability/metrics toggle, and the Prometheus reporter settings.</p><h3>Connection pool naming</h3><pre>[ballerina.sql]<br>poolName = &quot;observe_demo&quot;</pre><p>This one is easy to overlook but matters a lot in practice. If you don’t explicitly set a poolName, Ballerina assigns a random UUID as the pool_name tag on your metrics. That UUID changes every time your service restarts which means your Prometheus dashboards and alerts will silently break across deployments, since the metric series identity shifts underneath them. Setting a fixed, meaningful pool name keeps your metrics stable and queryable across restarts.</p><h3>Turning on metrics</h3><pre>[ballerina.observe]<br>metricsEnabled = true<br>metricsReporter = &quot;prometheus&quot;</pre><p>This enables the metrics subsystem and tells it which reporter to use. In this case, we’re pointing it at Prometheus.</p><h3>Prometheus reporter settings</h3><pre>[ballerinax.prometheus]<br>port = 9797<br>host = &quot;0.0.0.0&quot;</pre><p>This configures the embedded Prometheus endpoint that your service will expose. Adjust the host and port values to suit your environment 0.0.0.0 is shown here so the endpoint is reachable from outside the container, which is typically what you want in containerized deployments.</p><h3>Step 3: Import the Prometheus Module</h3><p>Configuration alone isn’t enough — the Prometheus module needs to be imported somewhere in your code so it gets bundled into the compiled service. Add this import to your main.bal:</p><pre>import ballerina/http;<br>import ballerina/sql;<br>import ballerinax/postgresql;<br>import ballerinax/prometheus as _;<br><br>// Database connection configurable via Config.toml or environment.<br>configurable string dbHost = &quot;localhost&quot;;<br>configurable int dbPort = 5432;<br>configurable string dbUser = &quot;postgres&quot;;<br>configurable string dbPassword = &quot;postgres&quot;;<br>configurable string dbName = &quot;testdb&quot;;<br><br>// Connection pool — poolName picked up from [ballerina.sql] in Config.toml<br>// so Prometheus metrics get a stable pool_name tag instead of a random UUID.<br>sql:ConnectionPool connectionPool = {<br>    maxOpenConnections: 10,<br>    maxConnectionLifeTime: 1800,<br>    minIdleConnections: 2<br>};<br><br>postgresql:Client dbClient = check new (<br>    host = dbHost,<br>    port = dbPort,<br>    username = dbUser,<br>    password = dbPassword,<br>    database = dbName,<br>    connectionPool = connectionPool<br>);<br><br>service /demo on new http:Listener(8080) {<br><br>    // Simple endpoint that runs a query and returns a row count.<br>    // Hitting this repeatedly is a good way to generate pool activity<br>    // and watch the metrics move in Prometheus.<br>    resource function get count() returns int|error {<br>        sql:ParameterizedQuery query = `SELECT COUNT(*) AS total FROM information_schema.tables`;<br>        stream&lt;record {int total;}, sql:Error?&gt; resultStream = dbClient-&gt;query(query);<br><br>        int total = 0;<br>        check from record {int total;} row in resultStream<br>            do {<br>                total = row.total;<br>            };<br>        check resultStream.close();<br><br>        return total;<br>    }<br><br>    // Health endpoint to confirm the service is up.<br>    resource function get health() returns string {<br>        return &quot;OK&quot;;<br>    }<br>}</pre><p>Note the as _ — you&#39;re importing this purely for its side effect of registering the reporter, not because you&#39;ll call functions from it directly.</p><h3>Step 4: Configure Prometheus to Scrape Your Service</h3><p>With your Ballerina service now exposing metrics on port 9797, the last step is telling your Prometheus server to actually scrape it. Here’s a minimal prometheus.yml configuration to get started:</p><pre>global:<br>  scrape_interval: 10s<br>  evaluation_interval: 10s</pre><pre>scrape_configs:<br>  - job_name: &quot;ballerina&quot;<br>    static_configs:<br>      - targets: [&quot;&lt;host&gt;:9797&quot;]</pre><p>Replace &lt;host&gt; with the address of your Ballerina service — this might be localhost for local testing, a container name in a Docker Compose setup, or a Kubernetes service name in a cluster. Adjust scrape_interval and evaluation_interval based on how granular you need your metrics to be.</p><h3>Putting It All Together</h3><p>Once these four pieces are in place the build flag, the Config.toml settings, the import statement, and the Prometheus scrape config you should start seeing connection pool and service metrics flowing into Prometheus, taggable by your configured pool_name rather than a throwaway UUID. From there, you can build Grafana dashboards, set up alerting rules on pool exhaustion, or simply keep an eye on things during your next deployment.</p><p>This is a relatively small amount of configuration for a meaningful jump in visibility and it’s a good example of how Ballerina’s recent module releases are making production observability less of an afterthought and more of a built-in capability.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7221339b2667" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>