<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Viththagan Roy Nesarajah on Medium]]></title>
        <description><![CDATA[Stories by Viththagan Roy Nesarajah on Medium]]></description>
        <link>https://medium.com/@viththagan.rn?source=rss-debbfe28bd03------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*NGe14O17iyi5RnH4sxiTuA.jpeg</url>
            <title>Stories by Viththagan Roy Nesarajah on Medium</title>
            <link>https://medium.com/@viththagan.rn?source=rss-debbfe28bd03------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 09:28:45 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@viththagan.rn/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Relaying Server-Sent Events in Ballerina: What Happens When the Client Hangs Up]]></title>
            <link>https://medium.com/@viththagan.rn/relaying-server-sent-events-in-ballerina-what-happens-when-the-client-hangs-up-70c3aded2b50?source=rss-debbfe28bd03------2</link>
            <guid isPermaLink="false">https://medium.com/p/70c3aded2b50</guid>
            <category><![CDATA[ballerina]]></category>
            <category><![CDATA[server-sent-events]]></category>
            <category><![CDATA[backend-development]]></category>
            <category><![CDATA[streaming]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Viththagan Roy Nesarajah]]></dc:creator>
            <pubDate>Thu, 15 Oct 2026 09:05:53 GMT</pubDate>
            <atom:updated>2026-10-15T09:05:53.971Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Passing a stream through takes a few lines. What matters is how the stream ends.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eZo2oBdx0s95yjyqPwmeCA.png" /></figure><p>A service that sits in front of a streaming API has two jobs. It has to hand each event to its own caller the moment the event arrives, because a chat interface that shows nothing for twenty seconds and then dumps a whole answer looks broken. And it usually has to know what went through, whether that is tokens for billing, events for an audit log or bytes for a quota.</p><p>Ballerina makes the first job short. The second is where the design work is, because a stream can end in more ways than one, and each ending reaches your code by a different path. One of them skips the code you wrote for “the stream is done”, and another leaves the upstream working for a caller who has gone.</p><p><em>Written against Ballerina 2201.13.6 (Swan Lake Update 13) and the </em><em>ballerina/http package at 2.17.2. Every snippet compiles on those versions, and every behaviour described was observed from running services.</em></p><h3>A stand-in for the upstream</h3><p>To see streaming behaviour you need something that streams slowly. This service sends eight token events half a second apart, then a usage event with a count, which is the shape most LLM streaming APIs follow.</p><pre>import ballerina/http;<br>import ballerina/lang.runtime;<br> <br>class Tokens {<br>    private int sent = 0;<br>    private final string[] words = [&quot;Streams&quot;, &quot; arrive&quot;, &quot; one&quot;, &quot; event&quot;, &quot; at&quot;, &quot; a&quot;, &quot; time&quot;, &quot;.&quot;];<br> <br>    public isolated function next() returns record {|http:SseEvent value;|}? {<br>        runtime:sleep(0.5);<br>        if self.sent &lt; self.words.length() {<br>            json token = {text: self.words[self.sent]};<br>            self.sent += 1;<br>            return {value: {event: &quot;token&quot;, data: token.toJsonString()}};<br>        }<br>        if self.sent == self.words.length() {<br>            self.sent += 1;<br>            json usage = {outputTokens: self.words.length()};<br>            return {value: {event: &quot;usage&quot;, data: usage.toJsonString()}};<br>        }<br>        return ();<br>    }<br>}<br> <br>service /upstream on new http:Listener(9401) {<br>    resource function post chat(record {string prompt;} body) returns stream&lt;http:SseEvent, error?&gt;|http:BadRequest {<br>        if body.prompt == &quot;reject me&quot; {<br>            return {body: {message: &quot;prompt rejected&quot;}, headers: {&quot;retry-after&quot;: &quot;30&quot;}};<br>        }<br>        stream&lt;http:SseEvent, error?&gt; events = new (new Tokens());<br>        return events;<br>    }<br>}</pre><p>Returning stream&lt;http:SseEvent, error?&gt; from a resource is all it takes to serve Server-Sent Events. The framework sets content-type: text/event-stream, cache-control: no-cache,public and chunked transfer encoding, and writes each event as the stream produces it.</p><h3>The version that buffers</h3><p>The first proxy most people write treats the upstream response like any other response:</p><pre>final http:Client upstream = check new (&quot;http://localhost:9401/upstream&quot;);<br> <br>service /ai on new http:Listener(9090) {<br>    resource function post chat(http:Request req) returns string|error {<br>        http:Response res = check upstream-&gt;post(&quot;/chat&quot;, check req.getJsonPayload());<br>        return res.getTextPayload();<br>    }<br>}</pre><p>It compiles, it returns every event, and it is wrong. getTextPayload() waits for the whole body, so the client sees nothing for 5.2 seconds and then all nine events at once. It also answers 201 Created with content-type: text/plain, because that is what a POST resource returning a string means. The client asked for a stream and got a document.</p><figure><img alt="Timeline of nine events sent half a second apart. A proxy using getTextPayload() delivers nothing for 5.2 seconds and then all nine at once; a proxy using getSseEventStream() delivers each event when the upstream sends it." src="https://cdn-images-1.medium.com/max/1024/1*yzQ5qLwDA30AdhrKEuT3TA.png" /></figure><h3>Relaying the bytes</h3><p>If all you need is to pass the stream on, return the upstream response itself:</p><pre>resource function post chat(http:Request req) returns http:Response|error {<br>    return upstream-&gt;post(&quot;/chat&quot;, check req.getJsonPayload());<br>}</pre><p>That streams. The first event reached the client at 0.50s and the last at 4.54s, the same half-second rhythm as the upstream, and the body arrived byte for byte as the upstream wrote it.</p><p>It also brings the upstream’s headers along. The client speaks HTTP/2 to the upstream where it can, and the relayed response arrived carrying x-http2-stream-id: 3, a detail of a connection the caller never saw. And it gives you no way to look at the events, because to your code the body is opaque. Copying the body across with setByteStream(upstreamRes.getByteStream(), &quot;text/event-stream&quot;) did not help: that version delivered everything at once when the upstream finished.</p><h3>Relaying events you can read</h3><p>To see the events, ask for them as events:</p><pre>resource function post chat(http:Request req) returns stream&lt;http:SseEvent, error?&gt;|error {<br>    http:Response res = check upstream-&gt;post(&quot;/chat&quot;, check req.getJsonPayload());<br>    return res.getSseEventStream();<br>}</pre><p>Each event still reaches the client when the upstream sends it: 0.53s, then every half second up to 4.54s. The status is 200 OK, the content type is text/event-stream, and the upstream&#39;s own headers stay on their side.</p><p><strong>The response headers do not go out until the first event does.</strong> getSseEventStream() waits for the first chunk of the body, and the listener holds the status line until it has an event to write. With a two-second pause before the first event, the client sees nothing at all for two seconds, and sending a comment event of your own first does not get the headers out any sooner. For an API that thinks before it speaks, &quot;the client got a 200&quot; and &quot;the first token arrived&quot; are the same moment.</p><p>This is also a parse and a rewrite, not a copy. The upstream bytes become http:SseEvent records and the framework writes those records out again. For the usual shape, one data: line of JSON per event, nothing that matters changes. At the edges it does. An upstream event with two data lines</p><pre>data: line1<br>data: line2</pre><p>reached the client as data: line1line2. The SSE format says the two lines should be joined with a newline, and the parser joins them with nothing, so a multi-line text payload loses its line breaks. Fields came out in the framework&#39;s order rather than the upstream&#39;s, and a retry: value that was not a number was dropped. Comment lines such as : ping keep-alives did survive. If your upstream sends multi-line data, relay the bytes instead.</p><figure><img alt="Upstream bytes beside what the event relay wrote. Two data lines, line1 and line2, arrive as data: line1line2 with no newline. A ping comment survives. The id, event and data fields of one event are reordered, and retry: abc is dropped." src="https://cdn-images-1.medium.com/max/1024/1*Zs7ZQ6L6bx5VlfbCcxKwlg.png" /></figure><p>Two headers often get set by hand in proxies like this. Cache-Control: no-cache is already there, and setting it replaces the framework&#39;s value. X-Accel-Buffering: no tells nginx not to buffer one response, so it matters only if nginx, or a gateway built on it, sits in front of you. Compression is not the problem you might expect: with the client asking for gzip, the listener compressed the stream and still flushed it event by event.</p><h3>Watching events on the way past</h3><p>To count what went through, wrap the upstream stream in a class of your own whose next() looks at each event and hands it on, then build a new stream from that class. A first version, with recordUsage standing in for wherever your counts go:</p><pre>class MeteredStream {<br>    private final stream&lt;http:SseEvent, error?&gt; events;<br>    private int? reportedTokens = ();<br> <br>    isolated function init(stream&lt;http:SseEvent, error?&gt; events) {<br>        self.events = events;<br>    }<br> <br>    public isolated function next() returns record {|http:SseEvent value;|}|error? {<br>        record {|http:SseEvent value;|}? item = check self.events.next();<br>        if item is () {<br>            recordUsage(self.reportedTokens ?: 0, false);<br>            return;<br>        }<br>        if item.value.event == &quot;usage&quot; {<br>            json usage = check (item.value.data ?: &quot;{}&quot;).fromJsonString();<br>            self.reportedTokens = check usage.outputTokens;<br>        }<br>        return item;<br>    }<br>}</pre><p>Two compile errors are waiting here for anyone who writes this from memory. The stream constructor requires next() to be <strong>isolated</strong>; without it the error is invalid stream constructor. expected a subtype of &#39;object { public isolated function next() ... }&#39;. And the obvious field name source is a reserved word in Ballerina, so private final stream&lt;...&gt; source; does not parse.</p><p>The resource wraps the upstream stream before returning it:</p><pre>stream&lt;http:SseEvent, error?&gt; events = check res.getSseEventStream();<br>stream&lt;http:SseEvent, error?&gt; metered = new (new MeteredStream(events));<br>return metered;</pre><p>The typed variable is not decoration. A bare return new (new MeteredStream(events)); fails whenever the return type is a union, which for a resource that can return an error is always: cannot infer type of the object from &#39;(stream&lt;...&gt;|error)&#39;, or too many arguments in call to &#39;new()&#39; once http:Response joins the union.</p><figure><img alt="Flow from the upstream through getSseEventStream(), which turns bytes into SseEvent records, into MeteredStream, whose next() counts each event and hands it on, and out to the client one event at a time. However the stream ended, the framework then calls MeteredStream’s close(), which records usage." src="https://cdn-images-1.medium.com/max/1024/1*-PARTbat0YtxyhVZvt6HcQ.png" /></figure><p>That version works in the demo. Every event goes through and recordUsage is called with 8. It has a hole in it that only shows up when somebody closes a browser tab.</p><h3>When the client hangs up</h3><p>A client that disconnects halfway through is not an edge case for a chat interface. It is the stop button.</p><p>Here is what happens when the client drops the connection 1.3 seconds into the stream. The proxy has sent two events. When it tries to write the third, the framework logs unable to write event stream to wire with the cause Remote client closed the connection while writing outbound response entity body, and stops pulling from your stream.</p><p><strong>The end-of-stream branch never runs.</strong> next() is not called again, so it never sees () from the upstream, and recordUsage is never called. Every cancelled request is a request you did not count.</p><figure><img alt="Timeline of a client disconnect at 1.3 seconds. The client received two events; the proxy’s third write fails and close() records 3 tokens as partial, while the end-of-stream branch in next() never runs. The upstream keeps sending tokens to the end." src="https://cdn-images-1.medium.com/max/1024/1*XHnmhXRobwgztgAutspBVw.png" /></figure><p>What does run is close(). The stream calls it on the wrapped object when it is finished with it, if the class has one, and in every ending tested it was called: a normal finish, a client disconnect, an upstream error, an upstream connection that dropped, and both idle timeouts. That makes close() the one place where accounting is guaranteed to happen:</p><pre>public isolated function close() returns error? {<br>    if !self.recorded {<br>        self.recorded = true;<br>        recordUsage(self.reportedTokens ?: self.delivered, self.reportedTokens is ());<br>    }<br>    return self.events.close();<br>}</pre><p>On a normal finish the usage event has arrived and reportedTokens holds the upstream&#39;s own count. On a disconnect it has not, so the fallback is delivered, a counter that next() increments for each token event it hands on, and the second argument marks the figure as partial. The recorded flag keeps it to one call. delivered counts what was handed to the framework, so the event whose write failed is included: a client that saw two events is recorded as three.</p><p><strong>The upstream is not cancelled either.</strong> After the client left, the proxy’s connection to the upstream stayed open, and the upstream went on to send all of its tokens. close() on the stream from getSseEventStream() closes the byte stream underneath it, but the request stays open, and switching the client to HTTP/1.1 with keep-alive off made no difference. On these versions there is no call in the wrapper that aborts the upstream request. For an LLM API that means the model keeps generating, and billing, for an answer nobody is reading. Cap the generation length on the request you send upstream, so an abandoned stream is at least finite, and treat delivered as a floor rather than the real figure.</p><h3>When the upstream says no</h3><p>Streaming APIs reject requests the ordinary way: a 400 or 429 with a JSON body, before any stream starts. The pass-through version calls getSseEventStream() on that response and fails with invalid payload target type. The response is not of text/event-stream content type. The client gets a 500 carrying that message instead of the upstream&#39;s 400 and its explanation. An upstream that ignores your streaming flag and answers 200 with plain JSON fails the same way.</p><p>Check both the status and the content type before converting, and hand anything that is not a stream on as it is:</p><pre>if upstreamRes.statusCode != 200 || !upstreamRes.getContentType().startsWith(&quot;text/event-stream&quot;) {<br>    http:Response res = new;<br>    res.statusCode = upstreamRes.statusCode;<br>    res.setBinaryPayload(check upstreamRes.getBinaryPayload(), upstreamRes.getContentType());<br>    string|http:HeaderNotFoundError retryAfter = upstreamRes.getHeader(&quot;retry-after&quot;);<br>    if retryAfter is string {<br>        res.setHeader(&quot;retry-after&quot;, retryAfter);<br>    }<br>    return res;<br>}</pre><p>Returning upstreamRes itself would work too, but as with the byte relay it brings the upstream&#39;s transport headers along; the forwarded 400 arrived carrying x-http2-stream-id: 5. A fresh response carries only what you copy into it, so copy what your client needs. retry-after on a 429 is the one most likely to matter, because without it the client cannot back off properly.</p><p>This is also the last point at which the proxy can choose a status code. Once the first event has gone out with a 200, everything else, errors included, has to be said inside the stream.</p><h3>When the upstream dies halfway</h3><p>An upstream that fails cleanly sends its own event: error, and to your proxy that is just another event. It passes through, and your client handles it.</p><p>An upstream whose connection drops is different. To test it, a small server sent three events and closed the socket without finishing the chunked body. The wrapped stream’s next() returned an error, and when that propagates with check, the framework writes it to your client as an event:</p><pre>event: error<br>data: Error occurred while reading stream:Remote host closed the connection while reading inbound response body</pre><p>Then it ends the response cleanly. From the client’s side the body is complete and well formed, and the last thing in it is an error message describing your internals. The same happens to any error your own next() returns, so the check on parsing the usage event in the first version is a second way to leak one.</p><p>Catching the error in next() lets you decide what the client sees:</p><pre>if self.failed {<br>    return;<br>}<br>record {|http:SseEvent value;|}|error? item = self.events.next();<br>if item is error {<br>    log:printError(&quot;upstream stream failed&quot;, item);<br>    self.failed = true;<br>    return {value: {event: &quot;error&quot;, data: &quot;{\&quot;message\&quot;:\&quot;stream interrupted\&quot;}&quot;}};<br>}</pre><p>The client now gets one error event in a shape you chose, the real cause goes to your logs, and the next call ends the stream. close() still runs afterwards, so the partial usage is recorded.</p><p>There is a quieter version of this failure. If the upstream ends its response cleanly but its last event is missing the blank line that ends an event, there is no error at all. An upstream that sent one token event and then a usage event with no trailing blank line produced, at the client, just the token. The usage event was dropped and the stream ended as if nothing were wrong. The missing final event is the only sign, and it is exactly what partial in the code above keys off. Make your client treat a stream that ends without its final event the same way.</p><figure><img alt="Table of six ways a relayed stream ends: normal finish, client disconnect, upstream connection drop, last event cut off, client idle timeout and listener idle timeout. The end-of-stream branch of next() runs only for a normal finish and a cut-off last event; close() runs in all six." src="https://cdn-images-1.medium.com/max/1024/1*Aw24kF8L7IGecWoeb0RukA.png" /></figure><h3>Timeouts are idle timeouts</h3><p>Two timeouts sit on this path, and both measure silence, not total duration:</p><ul><li>the <strong>client</strong> timeout, on http:Client, which defaults to 30 seconds</li><li>the <strong>listener</strong> timeout, on http:Listener, which defaults to 60</li></ul><p>A stream can run for ten minutes under either as long as events keep coming. What trips them is a gap. With the client timeout lowered to 2 seconds:</p><ul><li>an upstream that sent one event and then paused for 3 seconds gave the caller that first event, then an error event reading Idle timeout triggered while reading inbound response entity body</li><li>an upstream that sent its headers at once and its first event 3 seconds later, which is how most real streaming APIs behave, gave the caller a 200 and an error event and nothing else</li><li>the Ballerina stand-in above, which holds its headers until its first event, gave the caller a 500 with Idle timeout triggered before initiating inbound response instead</li></ul><p>With the listener timeout at 2 seconds instead, the caller got its first event and then the stream simply stopped, with nothing in it to say why. The proxy logged unable to write event stream to wire, caused by Idle timeout triggered while writing outbound response entity body.</p><p>Reasoning models are the usual reason for a long first gap, since they can think for longer than 30 seconds before the first token. Set both timeouts from the longest silence your upstream can produce, not from the default. The complete listing below uses 120 seconds for each.</p><figure><img alt="Two timelines. Steady events keep a stream alive however long it runs; a single silence longer than the timeout ends it. http:Client defaults to 30 seconds between upstream events, http:Listener to 60 seconds between writes to the client." src="https://cdn-images-1.medium.com/max/1024/1*XzXwqiDsreSboFdOYdvLvw.png" /></figure><h3>Putting it together</h3><pre>import ballerina/http;<br>import ballerina/log;<br> <br>configurable string upstreamUrl = &quot;http://localhost:9401/upstream&quot;;<br>configurable string upstreamToken = &quot;local-dev&quot;;<br> <br>final http:Client upstream = check new (upstreamUrl, timeout = 120, auth = {token: upstreamToken});<br> <br>service /ai on new http:Listener(9090, timeout = 120) {<br>    resource function post chat(http:Request req) returns stream&lt;http:SseEvent, error?&gt;|http:Response|error {<br>        http:Response upstreamRes = check upstream-&gt;post(&quot;/chat&quot;, check req.getJsonPayload());<br>        if upstreamRes.statusCode != 200 || !upstreamRes.getContentType().startsWith(&quot;text/event-stream&quot;) {<br>            http:Response res = new;<br>            res.statusCode = upstreamRes.statusCode;<br>            res.setBinaryPayload(check upstreamRes.getBinaryPayload(), upstreamRes.getContentType());<br>            string|http:HeaderNotFoundError retryAfter = upstreamRes.getHeader(&quot;retry-after&quot;);<br>            if retryAfter is string {<br>                res.setHeader(&quot;retry-after&quot;, retryAfter);<br>            }<br>            return res;<br>        }<br>        stream&lt;http:SseEvent, error?&gt; events = check upstreamRes.getSseEventStream();<br>        stream&lt;http:SseEvent, error?&gt; metered = new (new MeteredStream(events));<br>        return metered;<br>    }<br>}<br> <br>class MeteredStream {<br>    private final stream&lt;http:SseEvent, error?&gt; events;<br>    private int delivered = 0;<br>    private int? reportedTokens = ();<br>    private boolean recorded = false;<br>    private boolean failed = false;<br> <br>    isolated function init(stream&lt;http:SseEvent, error?&gt; events) {<br>        self.events = events;<br>    }<br> <br>    public isolated function next() returns record {|http:SseEvent value;|}|error? {<br>        if self.failed {<br>            return;<br>        }<br>        record {|http:SseEvent value;|}|error? item = self.events.next();<br>        if item is error {<br>            log:printError(&quot;upstream stream failed&quot;, item);<br>            self.failed = true;<br>            return {value: {event: &quot;error&quot;, data: &quot;{\&quot;message\&quot;:\&quot;stream interrupted\&quot;}&quot;}};<br>        }<br>        if item is () {<br>            return;<br>        }<br>        http:SseEvent event = item.value;<br>        if event.event == &quot;token&quot; {<br>            self.delivered += 1;<br>        } else if event.event == &quot;usage&quot; {<br>            int|error tokens = readOutputTokens(event.data);<br>            if tokens is int {<br>                self.reportedTokens = tokens;<br>            } else {<br>                log:printWarn(&quot;unreadable usage event&quot;, tokens);<br>            }<br>        }<br>        return item;<br>    }<br> <br>    public isolated function close() returns error? {<br>        if !self.recorded {<br>            self.recorded = true;<br>            recordUsage(self.reportedTokens ?: self.delivered, self.reportedTokens is ());<br>        }<br>        return self.events.close();<br>    }<br>}<br> <br>isolated function readOutputTokens(string? data) returns int|error {<br>    json usage = check (data ?: &quot;&quot;).fromJsonString();<br>    return usage.outputTokens.ensureType();<br>}<br> <br>isolated function recordUsage(int tokens, boolean partial) {<br>    log:printInfo(&quot;usage&quot;, tokens = tokens, partial = partial);<br>}</pre><p>The auth field sends the token as a bearer Authorization header on every upstream call. If your upstream wants its key in a different header, pass it in the headers argument of post instead. Either way, the incoming request&#39;s own headers are not forwarded unless you copy them.</p><h3>Testing it</h3><p>An http:Client can bind a response straight to stream&lt;http:SseEvent, error?&gt;, which makes the stream as easy to test as any other response. These run against the relay with the stand-in upstream in the same package:</p><pre>import ballerina/http;<br>import ballerina/test;<br>import ballerina/time;<br> <br>final http:Client relay = check new (&quot;http://localhost:9090/ai&quot;);<br> <br>@test:Config<br>function eventsArriveAsTheyAreSent() returns error? {<br>    stream&lt;http:SseEvent, error?&gt; events = check relay-&gt;post(&quot;/chat&quot;, {prompt: &quot;hi&quot;});<br>    string[] names = [];<br>    decimal[] arrivals = [];<br>    check from http:SseEvent e in events<br>        do {<br>            names.push(e.event ?: &quot;&quot;);<br>            arrivals.push(time:monotonicNow());<br>        };<br>    test:assertEquals(names.length(), 9);<br>    test:assertEquals(names[names.length() - 1], &quot;usage&quot;);<br>    test:assertTrue(arrivals[8] - arrivals[0] &gt; 3.5d, &quot;events arrived in one burst&quot;);<br>}<br> <br>@test:Config<br>function upstreamRejectionKeepsItsStatus() returns error? {<br>    http:Response res = check relay-&gt;post(&quot;/chat&quot;, {prompt: &quot;reject me&quot;});<br>    test:assertEquals(res.statusCode, 400);<br>    test:assertEquals(check res.getJsonPayload(), {message: &quot;prompt rejected&quot;});<br>    test:assertEquals(check res.getHeader(&quot;retry-after&quot;), &quot;30&quot;);<br>}</pre><p>The timing assertion in the first test is the one that matters. A version that buffered would still deliver nine events in the right order; only the time between the first and the last event shows that the relay streamed. The disconnect and the dropped connection are harder to put in a unit test, and worth reproducing by hand once, with curl -N --max-time 1 against the relay and a look at the logs.</p><h3>What to take away</h3><p><strong>Relay bytes if you only relay, events if you need to look.</strong> Events let you read every one, at the cost of a rewrite that joins multi-line data without its newlines.</p><p><strong>Accounting goes in </strong><strong>close(), not at the end of the stream.</strong> The end-of-stream branch only runs when the stream finishes normally, and a client that leaves does not stop the upstream.</p><p><strong>Choose the status before the first event, and own every error after it.</strong> Check the upstream’s status and content type before converting, and catch errors in next() so the framework never writes its own.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=70c3aded2b50" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Prompt Caching for a Long-Running Agent: Move One Marker, Every Step]]></title>
            <link>https://medium.com/@viththagan.rn/prompt-caching-for-a-long-running-agent-move-one-marker-every-step-2276da726419?source=rss-debbfe28bd03------2</link>
            <guid isPermaLink="false">https://medium.com/p/2276da726419</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[prompt-caching]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[vercel-ai-sdk]]></category>
            <category><![CDATA[claude-ai]]></category>
            <dc:creator><![CDATA[Viththagan Roy Nesarajah]]></dc:creator>
            <pubDate>Tue, 29 Sep 2026 07:35:53 GMT</pubDate>
            <atom:updated>2026-09-29T07:35:53.912Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Where to put cache markers in an agent loop, and the small changes that quietly undo them.</em></p><figure><img alt="Two isometric desks, each with a robot agent. Left: a tall pale stack of messages, a pile of coins, a red gauge and a slow stopwatch. Right: the same stack glowing teal as cached, one new amber message on top, a single coin, a green gauge and a fast stopwatch." src="https://cdn-images-1.medium.com/max/1024/1*aGbO2_xbeebKDIbimaYwWA.png" /></figure><p>A chat completion is one request. An agent is a loop of them. Every time the model calls a tool, the loop sends the whole conversation back: the system prompt, the tool definitions, every earlier tool call and every earlier tool result, plus the one new result at the end. Step ten carries steps one to nine with it.</p><p>That makes input cost grow with the square of the number of steps. In the test agent used for this post, an eleven-step run whose final prompt was about 15,600 tokens sent 124,604 input tokens in total.</p><p>Prompt caching fixes most of that. With the right setup, the same run’s input cost 28.6% of the uncached price. With a setup that looks almost identical, it cost 44.6%. With a timestamp in the wrong place, a run that should have cached its system prompt cost 161%, more than not caching at all.The difference is where the cache markers go and how they move.</p><p><em>The examples use TypeScript with the Vercel AI SDK (</em><em>ai 7.0.114, </em><em>@ai-sdk/anthropic 4.0.63) and Claude Sonnet 5, with a comparison run on </em><em>ai 6.0.293 and </em><em>@ai-sdk/anthropic 3.0.123. Every number below comes from real API calls, and every snippet compiles against those versions.</em></p><h3>How prompt caching works, in one minute</h3><p>The provider sees each request as one long prefix, in a fixed order: tool definitions, then the system prompt, then the messages. You mark a point in that prefix with a cache breakpoint. The first request that reaches the marker writes everything up to it into the cache. A later request whose prefix is identical up to that point reads it back instead of processing it again.</p><p>Reads are cheap and writes cost a little extra. On Anthropic’s public pricing, compared with the normal input price, a cache read costs 0.1 times, a write that lives five minutes 1.25 times, and a write that lives an hour 2 times. Every read refreshes the entry’s lifetime. All the percentages in this post are input cost worked out that way (uncached tokens plus the weighted reads and writes) as a share of the same run with no caching. Output tokens are the same either way.</p><p>Cost is not the only gain. Anthropic’s documentation also notes a faster time to first token on long prompts, and cache hits are not counted against the rate limit, so the same agent can run more steps before it is throttled.</p><p>Three rules matter for everything that follows. The prefix must match exactly, byte for byte. A request can carry at most four breakpoints. And a prefix shorter than the model’s minimum is not cached at all, with no error to tell you.</p><figure><img alt="A request as one prefix: tools, system prompt, three messages and a new message. Marker 1 (1 hour) sits after the system prompt and marker 2 (moves) after the new message. Everything up to a marker can be cached, and a later request reuses it only if its prefix matches exactly." src="https://cdn-images-1.medium.com/max/1024/1*TwOXPCRSBfv41NsXGIBjog.png" /></figure><h3>The test agent, and how to see what it cached</h3><p>The agent has a system prompt and three tool definitions totalling about 6,900 tokens, and one task: read ten log files with read_file, one per step, then reply DONE. Each step adds about 860 tokens, the tool call and its result. That is eleven steps and a prompt that grows by the same amount every step, which makes the effect of each change easy to read.</p><p>The AI SDK reports cache use on every step. In onStepFinish, the usage object splits the input into three parts:</p><pre>onStepFinish: (step) =&gt; {<br>    const { noCacheTokens, cacheReadTokens, cacheWriteTokens } = step.usage.inputTokenDetails;<br>    console.log(step.stepNumber, { noCacheTokens, cacheReadTokens, cacheWriteTokens });<br>},</pre><p>step.usage.inputTokens is the sum of all three, unlike the raw Anthropic input_tokens, which counts only the uncached part. The raw response is still there as step.usage.raw, and its cache_creation field says whether a write went to the five-minute or the hour-long cache.</p><p>Each run starts its system prompt with a unique nonce, so no run reads another run’s cache, and switches thinking off, because Sonnet 5 thinks adaptively when you do not set it, which makes runs hard to compare.</p><p>Without any caching, input grew from 7,004 tokens on the first step to 15,639 on the last, all of it processed from scratch every time. That is the baseline.</p><h3>Breakpoint one: the system prompt</h3><p>The first marker goes on the system prompt, which never changes during a run. In AI SDK v7 the system prompt is passed as instructions, and it can be a message object with provider options:</p><pre>const instructions: SystemModelMessage = {<br>    role: &#39;system&#39;,<br>    content: systemPrompt,<br>    providerOptions: { anthropic: { cacheControl: { type: &#39;ephemeral&#39;, ttl: &#39;1h&#39; } } },<br>};</pre><p>The first step wrote 6,890 tokens to the cache, and every later step read those 6,890 back. The tool definitions are in that number, because they come before the system prompt in the prefix.</p><p>The run cost 55.7% of the baseline. It is a large saving for a few lines, but it stops there. The uncached part still grew by 862 tokens a step, from 978 to 8,736, because nothing after the system prompt was cached.</p><h3>Breakpoint two: the one that moves</h3><p>The growing part is the conversation, so the second marker goes on the last message. On the next step that message is no longer last; the new marker sits a little further along. The request reads everything up to the previous step’s marker from the cache and writes only the new part.</p><p>That is why one moving marker is enough. A read can only land where an earlier request wrote, and each step’s marker lands on the prefix the previous step wrote. The old markers do not need to stay.</p><figure><img alt="Four steps of one agent run. Step 1 writes the prefix to the cache. Each later step reads everything the previous step wrote and writes only its own new part, with a single marker moved to the end each time." src="https://cdn-images-1.medium.com/max/1024/1*UD53FLkTyqIHTmS89iuNjQ.png" /></figure><p>The common way to move it is to add cacheControl to the last message inside prepareStep, which runs before each step:</p><pre>function withMarker(messages: ModelMessage[]): ModelMessage[] {<br>    return messages.map((m, i) =&gt; i === messages.length - 1<br>        ? { ...m, providerOptions: { ...m.providerOptions, anthropic: { cacheControl: { type: &#39;ephemeral&#39; } } } } as ModelMessage<br>        : m);<br>}<br> <br>prepareStep: ({ messages }) =&gt; ({ messages: withMarker(messages) }),</pre><p>(This replaces the message’s whole anthropic options object; merge into it if you set other Anthropic options on messages.)</p><p>On AI SDK v6 this works. Run on ai 6.0.293, the request carried exactly two markers on every step, the system prompt&#39;s and the moving one. The uncached part stayed at 2 tokens a step, reads grew by exactly the previous step&#39;s new content, and each step wrote only its own 862 new tokens: 28.6% of the baseline.</p><h3>The same code on v7</h3><p>On v7 the same code quietly falls apart. The v7 documentation for prepareStep says why in one line: &quot;If you return a messages override, those messages carry forward to later steps.&quot; In v6 the override applied to one step only. In v7 the marker added on the first step is still on that message on the second, when another is added, and on the third, when a third is added.</p><p>On the fourth step (stepNumber 3) the count reaches five, counting the system prompt&#39;s. The SDK counts markers before sending, and from then on every step came back with the warning, one more copy each step (found 5, then found 5 and 6, and so on):</p><pre>Maximum 4 cache breakpoints exceeded (found 5). This breakpoint will be ignored.</pre><p>Every marker after the fourth is dropped, and the moving one at the end is always among them.The request goes out without error and the cache stops growing. Reads stayed flat at 8,727 tokens from the fourth step on, and the uncached part climbed again, from 864 to 6,898. The run cost 44.6% of the baseline, about one and a half times the working version, and nothing failed that anyone would notice without reading the warnings.</p><figure><img alt="The same prepareStep code on AI SDK v6 and v7 over six steps. On v6 every step sends 2 markers. On v7 each marker is carried forward, so steps send 2, 3, 4, 5, 6 and 7 markers, and from step 4 the newest ones are ignored with the warning Maximum 4 cache breakpoints exceeded." src="https://cdn-images-1.medium.com/max/1024/1*34Sq2MDddWp-qIc9xYeCPA.png" /></figure><p>The fix is to take the old markers off before putting the new one on:</p><pre>function withoutMarkers(messages: ModelMessage[]): ModelMessage[] {<br>    return messages.map((m) =&gt; {<br>        const { providerOptions, ...rest } = m as ModelMessage &amp; { providerOptions?: unknown };<br>        return rest as ModelMessage;<br>    });<br>}<br> <br>prepareStep: ({ messages }) =&gt; ({ messages: withMarker(withoutMarkers(messages)) }),</pre><p>This strips all message-level provider options, which is fine when caching is the only thing you put there; otherwise remove just the cacheControl key. With the strip in place the v7 run matched v6 again: 2 uncached tokens a step, 28.6%.</p><p>One more thing breaks when v6 agent code moves to v7: a role: &#39;system&#39; message inside messages now throws (&quot;System messages are not allowed in the prompt or messages fields. Use the instructions option instead.&quot;). Move it to instructions, as in the first snippet.</p><h3>Or let the API move it</h3><p>On the Claude API you do not have to move the marker at all. Put cacheControl in the call&#39;s own provider options and the API places it at the end of each request:</p><pre>const result = streamText({<br>    model: anthropic(&#39;claude-sonnet-5&#39;),<br>    instructions,<br>    prompt: task,<br>    tools,<br>    stopWhen: stepCountIs(12),<br>    providerOptions: {<br>        anthropic: {<br>            thinking: { type: &#39;disabled&#39; },<br>            cacheControl: { type: &#39;ephemeral&#39; },<br>        },<br>    },<br>});</pre><p>No prepareStep at all, and the numbers matched the stripped version: 2 uncached tokens a step, 28.6% of the baseline. Placing the marker yourself is still the answer where this top-level option is not available, when you want the marker somewhere other than the very end, or when you want to control the mix of cache lifetimes exactly.</p><figure><img alt="Line chart of input cost per step over 11 steps, in token equivalents. No caching climbs from 7k to 15.6k. System prompt only drops to 1.7k after the first step and climbs to 9.5k. Both markers stays between 1.8k and 2.5k after a first step of 13.9k." src="https://cdn-images-1.medium.com/max/1024/1*QiEr2N6pXwlrDRRkiiMm3Q.png" /></figure><h3>Two lifetimes, and the order they must come in</h3><p>The system prompt uses the hour-long cache and the conversation the five-minute one. Inside a single run that choice costs a little: every step reads and so refreshes the five-minute entries anyway, and the hour-long write is billed at twice the input price instead of 1.25 times. With the system prompt on the five-minute cache as well, the same run cost 24.5% of the baseline instead of 28.6%.</p><p>It pays off between turns. A user who reads the answer and replies a few minutes later finds the five-minute entries gone. A two-turn run paused for six and a half minutes between turns showed the difference. With the system prompt on the hour-long cache, the second turn’s first step read 6,654 tokens back, exactly the system prompt’s entry (this smaller test agent had one tool instead of three), and wrote only 1,802: the conversation entries that had expired, plus the new message.The same pause with everything on the five-minute cache read nothing and rewrote all 8,454 tokens. Weighted by price, that first step of the second turn cost about 3.6 times as much.</p><figure><img alt="Turn 2 after 6.5 minutes idle. With the system prompt on the hour-long cache, 6,654 tokens are read from the cache and 1,802 written again. With everything on the five-minute cache, all 8,454 are rewritten, about 3.6 times the cost for that step." src="https://cdn-images-1.medium.com/max/1024/1*fkxdRhVnTOdyhDeOj1Lnaw.png" /></figure><p>The API enforces an order: hour-long markers must come before five-minute ones in the prefix. A request with a five-minute marker on the system prompt and an hour-long one on the last message was rejected with a 400:</p><pre>a ttl=&#39;1h&#39; cache_control block must not come after a ttl=&#39;5m&#39; cache_control block.<br>Note that blocks are processed in the following order: `tools`, `system`, `messages`</pre><p>The SDK does not check this before sending, so the error arrives from the API.</p><h3>What silently breaks the cache</h3><p>Everything so far depends on the prefix staying identical. These are the ways it stops being identical without anything failing.</p><p><strong>A timestamp in the system prompt.</strong> Agents often tell the model the current time. In a run with the hour-long marker on the system prompt, a timestamp at its top, re-rendered each step, gave every step a different prefix from its first line. Not one step read anything from the cache, and every step rewrote the whole system prompt at the hour-long write price: 161.1% of the uncached baseline, against 55.7% for the same setup without it. With both markers in place and the timestamp moved to the first user message, written once when the run starts, the run cached normally, 2 uncached tokens a step.It sits in the part of the prefix that stops changing once it has been sent.</p><p><strong>Changing the tool list.</strong> Tool definitions are the first thing in the prefix, so any change to them invalidates everything after. Narrowing the tools to one with activeTools for a single step made that step read nothing and write 10,261 tokens, the system prompt included. Putting the full list back on the next step read the earlier entry again, since it was still cached, but the one change lifted the run from 28.6% to 42.1% of the baseline. A tool description that includes a path or a date does the same damage in a less obvious place.</p><p><strong>A prefix below the minimum.</strong> Anthropic documents a minimum cacheable length per model: 1,024 tokens for Sonnet 5 and 4,096 for Claude Haiku 4.5. The runs agreed. On Sonnet 5 a prefix of about 860 tokens wrote nothing and one of 1,170 was cached; on Haiku 4.5, about 3,600 wrote nothing and 4,306 was cached. Neither returns an error, only a cacheWriteTokens of 0. If a small agent shows no cache activity at all, check this first.</p><h3>Keep watching it</h3><p>A cache that stops working does not fail. It gets slowly more expensive. The numbers the loop already has are enough to notice:</p><pre>onStepFinish: (step) =&gt; {<br>    const { cacheReadTokens = 0 } = step.usage.inputTokenDetails;<br>    const ratio = step.usage.inputTokens ? cacheReadTokens / step.usage.inputTokens : 0;<br>    if (step.stepNumber &gt; 0 &amp;&amp; ratio &lt; 0.5) {<br>        console.warn(`step ${step.stepNumber}: only ${(ratio * 100).toFixed(0)}% of input came from the cache`, step.warnings);<br>    }<br>},</pre><p>After the first step, a healthy loop reads most of its input from the cache. When the ratio drops, compare the request bodies of two neighbouring steps with the markers removed; the first difference inside the shared part is what broke it. In v7, step.request.body is only filled in if you pass include: { requestBody: true } to streamText. The breakpoint warning goes to stderr by default and to step.warnings, so make sure one of them reaches a log you read.</p><figure><img alt="Bar chart of input cost for the same 11-step run as a share of no caching: both markers on five minutes 24.5%, hour-long system plus moving marker 28.6%, tool list changed for one step 42.1%, v6 prepareStep recipe on v7 44.6%, system prompt marker only 55.7%, no caching 100%, timestamp in the system prompt 161.1%." src="https://cdn-images-1.medium.com/max/1024/1*CQVNqoaqQZVwL7UCHmR4wA.png" /></figure><h3>What to take away</h3><p><strong>One marker on the system prompt, one that moves.</strong> On the Claude API, the call-level cacheControl moves the second one for you. Five minutes on both is cheapest inside one run; the hour-long cache on the system prompt pays off when users pause between turns.</p><p><strong>On AI SDK v7, strip old markers before adding a new one.</strong> prepareStep overrides now carry forward, and every marker past the fourth is silently ignored.</p><p><strong>Keep the prefix still.</strong> No timestamps in the system prompt, no per-step tool changes, and a prefix above the model’s minimum, or caching quietly costs more than it saves.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2276da726419" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ballerina HTTP Interceptors: Stop the Request Before It Reaches Your Handler]]></title>
            <link>https://medium.com/@viththagan.rn/ballerina-http-interceptors-stop-the-request-before-it-reaches-your-handler-4cda7e3cb294?source=rss-debbfe28bd03------2</link>
            <guid isPermaLink="false">https://medium.com/p/4cda7e3cb294</guid>
            <category><![CDATA[ballerina]]></category>
            <category><![CDATA[backend]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Viththagan Roy Nesarajah]]></dc:creator>
            <pubDate>Sat, 26 Sep 2026 02:56:01 GMT</pubDate>
            <atom:updated>2026-09-26T02:56:01.710Z</atom:updated>
            <content:encoded><![CDATA[<p><em>One resource function that runs for every endpoint you have, and every one you add next year.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-GMnLIz1K-kYDDQ_fG2VUw.png" /></figure><p>The bug is almost never in the auth check you wrote. It is in the endpoint somebody added a year later and forgot to put the check in.</p><p>I have been working on a Ballerina service that sits in front of an expensive upstream API, where every request has to be attributed to a user and charged against a budget before it is allowed anywhere near the upstream. That kind of check belongs to the service, not to any one endpoint. Ballerina has a specific feature for exactly that, and it is thinly documented enough to be worth writing down properly.</p><p><em>Written against Ballerina 2201.13.4 (Swan Lake Update 13) and the </em><em>ballerina/http package at 2.17.1. Every snippet below is lifted from a package that compiles on those versions; the short ones assume </em><em>import ballerina/http; and a listener called </em><em>httpListener. The ordering and error behaviour described here was observed from a running service rather than inferred.</em></p><h3>The check you keep copying</h3><p>Here is where most services start. Two endpoints, the same two lines at the top of each.</p><pre>service /ai on httpListener {<br> <br>    resource function post messages(http:Request req) returns json|error {<br>        string userId = check authenticate(req);<br>        check enforceQuota(userId);<br>        // the actual work<br>    }<br> <br>    resource function post completions(http:Request req) returns json|error {<br>        string userId = check authenticate(req);<br>        check enforceQuota(userId);<br>        // the actual work<br>    }<br>}</pre><p>Nothing here is wrong. The problem is the default: protection is opt-in, so every new endpoint arrives unprotected and stays that way until somebody remembers. The review that catches it has to happen every single time, forever.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fpFrAnB_pG4C_yHbbP4YIg.png" /></figure><p>An interceptor flips that default.</p><h3>One resource that runs before all the others</h3><p>An interceptor in Ballerina is a service class that includes one of the interceptor types and declares exactly one function: a resource function on the request side, a remote function on the response side.</p><pre>isolated service class QuotaInterceptor {<br>    *http:RequestInterceptor;<br> <br>    isolated resource function &#39;default [string... path](http:RequestContext ctx, http:Request req)<br>            returns http:NextService|http:TooManyRequests|http:Unauthorized|error? {<br>        // ...<br>        return ctx.next();<br>    }<br>}</pre><p>That resource signature is the whole trick, and it is the part that is hard to find when you go looking:</p><ul><li><strong>&#39;default</strong> is the accessor, and it matches every HTTP method. The leading quote is Ballerina&#39;s escape for using a reserved word as an identifier. As it happens default is not reserved on this version and the accessor compiles without it, but the http documentation writes &#39;default everywhere, so that is the form you will recognise.</li><li><strong>[string... path]</strong> is a rest path parameter, and it matches every path segment, at any depth.</li></ul><p>Together they mean this resource is not a route. It matches everything, which is what turns it into a pipeline stage.</p><p>*http:RequestInterceptor looks like it does nothing. In the http package it is declared as a distinct service object with no methods at all. The checking is done instead by a compiler plugin, and it is stricter than you would guess: misspell interceptResponse, or declare two resource functions in one interceptor, and you get a compile error rather than an interceptor that silently never runs.</p><p>You can also narrow the match. A request interceptor declared as resource function post messages(...) runs only for POST /ai/messages and is skipped everywhere else, which is useful when one endpoint needs a check the others do not. Interceptor paths are matched under the service&#39;s base path, just as a resource&#39;s are. The one-resource-function rule still applies, so a scoped interceptor is a separate class.</p><h3>Wiring it in</h3><p>Two changes to the service: the type, and one new function.</p><pre>listener http:Listener httpListener = new (9090);<br> <br>service http:InterceptableService /ai on httpListener {<br> <br>    public function createInterceptors() returns http:Interceptor[] {<br>        return [new QuotaInterceptor()];<br>    }<br> <br>    isolated resource function post messages(http:RequestContext ctx) returns json|error {<br>        string userId = check ctx.getWithType(USER_ID);<br>        // the actual work, and nothing about auth<br>    }<br>}</pre><p>createInterceptors returns either a single http:Interceptor or an array of them, and the array is the pipeline. For a single interceptor the expression form reads better:</p><pre>public function createInterceptors() returns http:Interceptor =&gt; new QuotaInterceptor();</pre><p>Any subtype is accepted, so naming the class directly works too. You will see all three shapes in the wild and they mean the same thing.</p><p>One thing worth knowing before you plan around it: in this version of the http package, interceptors attach to a <strong>service</strong>, not to a listener. There is no listener-wide interceptor list, so a second service sharing the same listener needs its own createInterceptors returning the same pipeline. Sharing is done by sharing the classes, not by declaring it once at the port.</p><p>The payoff is the part that matters. The next resource added to this service is covered on the day it is written, by nobody’s diligence in particular.</p><h3>Handing data down: the request context</h3><p>The interceptor has done the work of identifying the user, and the handler needs the result. That handoff is http:RequestContext.</p><pre>ctx.set(USER_ID, userId);</pre><p>and on the other side:</p><pre>string userId = check ctx.getWithType(USER_ID);</pre><p>ctx.get returns an http:ReqCtxMember that you can ensureType() yourself, and it panics outright if nothing ever set that key. getWithType returns an error you can check instead, and does the type conversion in the same step, which is the real reason to prefer it. ctx.hasKey answers whether anything set it at all.</p><p>Here is the honest part. The key is a string and the type is resolved at runtime. Nothing tells you at compile time that the handler is reading userID where the interceptor wrote userId, or that it expects a string where an int was stored. The contract between the two halves is a convention that the compiler cannot see.</p><p>Two cheap habits make it hold. Declare a constant for every key so a typo becomes a compile error:</p><pre>const string USER_ID = &quot;userId&quot;;</pre><p>And keep the interceptor and the keys it populates in one module, so the two ends of the contract are never edited far apart.</p><p>One convenience worth knowing: reading the body in the interceptor does not starve the handler. Ballerina buffers the entity, so req.getJsonPayload() can be called in the interceptor and again in the resource, and both see the payload. The cost is that the body is held in memory, which is a real consideration if yours are large.</p><h3>The return type is the control flow</h3><p>This is the piece that makes the whole pattern click.</p><pre>returns http:NextService|http:TooManyRequests|http:Unauthorized|error?</pre><p>Return ctx.next() and the request carries on to the next stage. Return one of those status records instead and that record <strong>is</strong> the response. The handler never executes.</p><p>Here is the whole interceptor, with weeklySpend and secondsUntilReset standing in for whatever you keep your counters in:</p><pre>const string USER_ID = &quot;userId&quot;;<br> <br>isolated service class QuotaInterceptor {<br>    *http:RequestInterceptor;<br> <br>    isolated resource function &#39;default [string... path](http:RequestContext ctx, http:Request req)<br>            returns http:NextService|http:TooManyRequests|http:Unauthorized|error? {<br> <br>        string|http:HeaderNotFoundError userId = req.getHeader(&quot;x-user-id&quot;);<br>        if userId is http:HeaderNotFoundError {<br>            return &lt;http:Unauthorized&gt;{body: {message: &quot;Missing identity&quot;}};<br>        }<br> <br>        decimal spent = check weeklySpend(userId);<br>        if spent &gt;= WEEKLY_BUDGET {<br>            int resetSeconds = check secondsUntilReset(userId);<br>            return &lt;http:TooManyRequests&gt;{<br>                body: {message: &quot;Weekly budget exhausted&quot;},<br>                headers: {&quot;retry-after&quot;: resetSeconds.toString()}<br>            };<br>        }<br> <br>        ctx.set(USER_ID, userId);<br>        return ctx.next();<br>    }<br>}</pre><p>The client gets a real 429, with the body and the retry-after header exactly as written, and the upstream is never called.</p><p>Returning an error is a third option and it behaves differently from both. It goes to your http:RequestErrorInterceptor if you declared one, and otherwise falls through to the response error stage, the same way a failure in the handler would.</p><p>The nice consequence is that the union in the signature is a readable list of every way this stage can end. You can tell what an interceptor is allowed to do to a request without reading its body.</p><p>One behaviour to know about: the wildcard interceptor runs even for paths under the service that have no matching resource. An over-budget request to an endpoint that does not exist comes back 429, not 404, because the gate runs after the service is chosen but before the resource is. A path outside the service’s base path never reaches it at all. For a quota gate that is the right answer. It is worth knowing if you assumed routing happened first.</p><h3>The way back out</h3><p>Responses get the same treatment, with one difference in shape: a response interceptor declares a <strong>remote</strong> function rather than a resource.</p><pre>isolated service class ResponseTagger {<br>    *http:ResponseInterceptor;<br> <br>    isolated remote function interceptResponse(http:RequestContext ctx, http:Response res)<br>            returns http:NextService|error? {<br>        res.setHeader(&quot;x-quota-checked&quot;, &quot;true&quot;);<br>        return ctx.next();<br>    }<br>}</pre><p>The pipeline is the array you return, and one array holds every kind at once:</p><pre>public function createInterceptors() returns http:Interceptor[] {<br>    return [new QuotaInterceptor(), new ResponseTagger(), new ErrorShaper()];<br>}</pre><p>Request interceptors run in the order you listed them. Response interceptors run in the reverse of that order. The pipeline is an onion: in through the layers, out through the same layers backwards.</p><figure><img alt="A request passes through QuotaInterceptor then Tracer into the post messages handler, then back out through Metrics and ResponseTagger, the reverse of the declared order. A 429 returned from QuotaInterceptor goes straight back to the client, skipping both response interceptors." src="https://cdn-images-1.medium.com/max/1024/1*GRnaWOQEssvis8w1xQp7mw.png" /></figure><p>And here is the detail that is easy to miss until it bites. <strong>When a request interceptor short-circuits, the response interceptors do not run at all.</strong> The 429 from the previous section goes straight to the client with no x-quota-checked header on it. If a response interceptor is stamping a correlation id onto every reply, every rejected request is quietly missing it, and rejected requests are exactly the ones somebody will later try to trace.</p><h3>When something throws</h3><p>There are two error stages, one for each direction, and the response-side one is where the trap is.</p><p>The version almost everybody writes first looks like this:</p><pre>isolated service class ErrorShaper {<br>    *http:ResponseErrorInterceptor;<br> <br>    isolated remote function interceptResponseError(error err) returns http:InternalServerError {<br>        return {body: {message: &quot;Something went wrong&quot;}};<br>    }<br>}</pre><p>That quietly breaks your 404s.</p><p>The http package already registers a default error interceptor of its own, the one responsible for turning an unhandled error into a status-appropriate response. Registering yours puts it in front of that default, so whatever yours returns becomes the status for every unhandled error, routing failures included. A request to a path with no matching resource fails with a routing error, that error reaches your interceptor exactly like a failure in a handler would, and the fixed http:InternalServerError above converts every 404 in the service into a 500.</p><figure><img alt="An error interceptor that returns a fixed InternalServerError turns both a 404 and a 405 into 500. One that takes the response parameter and returns it keeps each status, reshaping only the body of errors that are not routing failures." src="https://cdn-images-1.medium.com/max/1024/1*XnQb2miU4gmLCtGsc4R_fQ.png" /></figure><p>A routing failure already carries the right status and a serviceable message. The fix is to leave those alone and reshape only what is actually yours to shape:</p><pre>isolated service class ErrorShaper {<br>    *http:ResponseErrorInterceptor;<br> <br>    isolated remote function interceptResponseError(error err, http:Response res)<br>            returns http:Response {<br>        if err !is http:ResourceDispatchingError {<br>            res.setJsonPayload({message: &quot;Something went wrong&quot;}, &quot;application/json&quot;);<br>        }<br>        return res;<br>    }<br>}</pre><p>Taking http:Response as a parameter hands you the status the framework already worked out, and returning it keeps that status intact.</p><p>Match on the parent type, not on 404, because routing failures are a family. http:ResourceDispatchingError also covers a request with the wrong method, which is a 405, and an unsupported content type, which is a 415. An interceptor that special-cased 404 would flatten those two into 404 as well, which is the same bug in a smaller costume. The explicit &quot;application/json&quot; is needed because the framework&#39;s error response is text/plain and setJsonPayload leaves that header alone otherwise.</p><p>One thing you inherit along with the error stage is the logging. The package’s default interceptor is what writes the unhandled error returned from the service line, and it stops firing the moment yours sits in front of it. Log err yourself before reshaping it, or a failure becomes a tidy message and no record of what actually happened.</p><p>Two ordering facts that are worth holding on to, because they are not symmetric:</p><ul><li>A <strong>response</strong> error interceptor listed <strong>after</strong> your response interceptors runs before them, and the response it produces still travels out through them. Listed first, its output goes straight to the client and the response interceptors never run at all. Position in the array is the whole difference.</li><li>A <strong>request</strong> error interceptor (http:RequestErrorInterceptor, a resource function with the same wildcard path taking an error parameter) is the end of the line. Its response goes straight to the client without touching the response interceptors, for the same reason a short-circuit does, and it only catches errors from request interceptors listed before it.</li></ul><h3>Testing an interceptor without an upstream</h3><p>An interceptor’s entire contract is two questions: did it let the request through, and what did it put in the context. Neither needs the real service, the real upstream, or a real token.</p><p>A five-line service in the test module whose only resource reads the context back out is enough:</p><pre>service http:InterceptableService /test on new http:Listener(9099) {<br> <br>    public function createInterceptors() returns QuotaInterceptor =&gt; new QuotaInterceptor();<br> <br>    resource function get whoami(http:RequestContext ctx) returns string|error {<br>        return check ctx.getWithType(USER_ID);<br>    }<br>}<br> <br>final http:Client testClient = check new (&quot;http://localhost:9099&quot;);<br> <br>@test:Config {}<br>function testIdentityReachesTheHandler() returns error? {<br>    string userId = check testClient-&gt;get(&quot;/test/whoami&quot;, {&quot;x-user-id&quot;: &quot;u-42&quot;});<br>    test:assertEquals(userId, &quot;u-42&quot;);<br>}<br> <br>@test:Config {}<br>function testMissingIdentityIsRejected() returns error? {<br>    http:Response res = check testClient-&gt;get(&quot;/test/whoami&quot;);<br>    test:assertEquals(res.statusCode, 401);<br>}</pre><p>bal test runs that against a real listener in a second, and it covers the two failure modes that actually happen: the identity not arriving where the handler expects it, and an unauthenticated request getting through.</p><h3>What to take away</h3><p><strong>Make protection the default.</strong> The value of an interceptor is not that it saves two lines per resource. It is that the endpoint written next year is covered without anybody deciding it should be.</p><p><strong>The return type is the control flow.</strong> ctx.next() continues, a status record ends it there, and the union in the signature documents every exit this stage has.</p><p><strong>The context handoff is a runtime contract.</strong> Constants for keys, and both ends in one module, are what keep it honest.</p><p><strong>Out is the reverse of in, and short-circuits skip the way out entirely.</strong> Anything a response interceptor is responsible for will be missing from every request an interceptor ended with a status record.</p><p><strong>A blanket response error interceptor will eat your 404s.</strong> Take the http:Response parameter and return it, so the status the framework already worked out survives. Reshape the body only for errors that are not http:ResourceDispatchingError, and log the error while you are there.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4cda7e3cb294" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Rate Limiting in Ballerina: Let the Redis Key Expire]]></title>
            <link>https://medium.com/@viththagan.rn/rate-limiting-in-ballerina-let-the-redis-key-expire-145bac1de3a2?source=rss-debbfe28bd03------2</link>
            <guid isPermaLink="false">https://medium.com/p/145bac1de3a2</guid>
            <category><![CDATA[ballerina]]></category>
            <category><![CDATA[ballerinalang]]></category>
            <category><![CDATA[rate-limiting]]></category>
            <category><![CDATA[redis]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Viththagan Roy Nesarajah]]></dc:creator>
            <pubDate>Sun, 20 Sep 2026 17:27:57 GMT</pubDate>
            <atom:updated>2026-09-21T03:11:18.727Z</atom:updated>
            <content:encoded><![CDATA[<p><em>One key per user, a TTL, and no reset job anywhere.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NceEVkDPtyekgEt0Tx5jow.jpeg" /></figure><p>“Every user gets a fixed budget per week” is one sentence. Underneath it are several decisions that decide whether the limiter is correct, fast, and still standing when Redis has a bad day.</p><p>I have been working on a Ballerina service that sits in front of an expensive upstream API, the kind where every call has a real dollar cost attached, and a single enthusiastic user can run up a bill that shows up on someone’s monthly report. A cap wasn’t optional. What follows is what that turned up: the shape that works, the piece of Redis that does more work than people expect, and the two places worth being honest about rather than tidy.</p><p><em>Written against Ballerina 2201.13.4 (Swan Lake Update 13) and </em><em>ballerinax/redis 3.4.1; every snippet below compiles on those versions. </em><em>mGetOptional and </em><em>setNxEx need the connector at 3.4.1 or later, and the Redis commands used here (</em><em>SET ... NX EX, </em><em>MGET, </em><em>INCRBYFLOAT, </em><em>TTL) need a server at 2.6.12 or later.</em></p><h3>First: what are you actually limiting?</h3><p>Most rate-limiting writing assumes you’re counting <strong>requests</strong>. Hundred calls a minute, token bucket, sliding window, done.</p><p>The thing being counted here is not requests but <strong>cost</strong>. Two calls to the same endpoint can differ by two orders of magnitude depending on what they carry, so “you’ve used 60 of your 100 requests” tells a user nothing useful. What matters is how much of their budget is gone.</p><p>That one difference decides your data structure. If you’re counting requests, you need a window: a ring of buckets, or a sorted set of timestamps you trim as it slides. If you’re accumulating spend against a budget that resets on a fixed boundary, you need <strong>a number, and an expiry date</strong>. That’s it. One key.</p><pre>usage:{userId}  →  14.7302</pre><p>A string holding a running total. No sorted sets, no per-request entries to trim, no background compaction. The entire limiter is a key whose value goes up and whose life is finite.</p><p>Every snippet below reaches Redis through one client, created once at module level:</p><pre>import ballerinax/redis;<br> <br>final redis:Client redis = check new ({<br>    connection: {host: &quot;localhost&quot;, port: 6379}<br>});</pre><h3>Let TTL do the resetting</h3><p>Here’s the part worth the price of admission.</p><p>When people build a weekly cap, they usually reach for a reset job: a cron entry, a scheduled task, something that wakes up Monday morning and zeroes every counter. Now you own a scheduler. You own what happens when it doesn’t fire, and what happens when it fires twice, and what happens when it’s mid-sweep while requests are landing.</p><p>You don’t need any of it. Set the key’s TTL to the exact number of seconds between <em>now</em> and the next window boundary, and <strong>expiry is the reset</strong>. The key evaporates on Monday at 00:00 UTC on its own. The next request that comes in finds nothing there, writes a fresh value with a fresh TTL, and the cycle continues. No job, no sweep, no cleanup, nothing to monitor.</p><p>The only real work is computing that TTL:</p><pre>import ballerina/time;<br> <br>const int SECONDS_PER_DAY = 86400;<br> <br>// ISO 8601 day-of-week: 1 = Monday ... 7 = Sunday<br>isolated function isoDayOfWeek(time:Civil civil) returns int {<br>    time:Date date = {year: civil.year, month: civil.month, day: civil.day};<br>    time:DayOfWeek balDay = time:dayOfWeek(date); // 0 = Sunday ... 6 = Saturday<br>    return ((balDay + 6) % 7) + 1;<br>}<br> <br>isolated function secondsUntilNextMonday(time:Utc timestamp) returns int {<br>    time:Civil civil = time:utcToCivil(timestamp);<br>    int dow = isoDayOfWeek(civil);<br> <br>    int daysUntilMonday = dow == 1 ? 7 : (8 - dow);<br>    int secondsElapsedToday = civil.hour * 3600<br>        + civil.minute * 60<br>        + &lt;int&gt;(civil.second ?: 0.0d);<br>    int secondsRemainingToday = SECONDS_PER_DAY - secondsElapsedToday;<br> <br>    return secondsRemainingToday + ((daysUntilMonday - 1) * SECONDS_PER_DAY);<br>}</pre><p>Note time:dayOfWeek returns Sunday-based 0..6, so if you want ISO weeks you have to rotate it. That ((balDay + 6) % 7) + 1 is the whole conversion, and it&#39;s the sort of thing that&#39;s easy to get subtly wrong and not notice until a Sunday.</p><p>Then recording usage is: does the key have a TTL? If not, it’s a new window, so write the value fresh with an expiry. If it does, just add to it.</p><pre>isolated function addUsage(string userId, decimal cost) returns error? {<br>    string key = string `usage:${userId}`;<br>    int|redis:Error currentTtl = redis-&gt;ttl(key);<br> <br>    // -2 = key doesn&#39;t exist, -1 = key exists with no expiry<br>    if currentTtl is int &amp;&amp; currentTtl &lt; 0 {<br>        int ttl = secondsUntilNextMonday(time:utcNow());<br>        _ = check redis-&gt;setEx(key, cost.toString(), ttl);<br>        return;<br>    }<br> <br>    _ = check redis-&gt;incrByFloat(key, &lt;float&gt;cost);<br>}</pre><p>The &lt; 0 check covers both cases you care about: -2 means no key at all, -1 means a key with no expiry, which shouldn&#39;t happen, but if it ever does, you want to heal it rather than accumulate forever into a key that never dies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8joaxkSX-2zdxAQZc7sUqQ.png" /></figure><h3>Check the limit before your handler runs</h3><p>You don’t want a quota check pasted into the top of every resource function. Ballerina has a better answer: a <strong>request interceptor</strong>, which runs ahead of the handler and can stop the request cold.</p><pre>import ballerina/http;<br> <br>public isolated service class RateLimitInterceptor {<br>    *http:RequestInterceptor;<br> <br>    isolated resource function &#39;default [string... path](<br>            http:RequestContext ctx, http:Request req)<br>            returns http:NextService|http:TooManyRequests|http:BadRequest|error? {<br> <br>        string|http:HeaderNotFoundError userId = req.getHeader(&quot;x-user-id&quot;);<br>        if userId is http:HeaderNotFoundError {<br>            return http:BAD_REQUEST;<br>        }<br> <br>        boolean exceeded = check isOverCap(userId);<br>        if exceeded {<br>            return &lt;http:TooManyRequests&gt;{<br>                body: {message: &quot;Weekly usage limit reached&quot;}<br>            };<br>        }<br> <br>        ctx.set(&quot;userId&quot;, userId);<br>        return ctx.next();<br>    }<br>}</pre><p>Two things make this pattern worth knowing.</p><p><strong>The resource path is a wildcard.</strong> &#39;default [string... path] matches any method on any path, so the interceptor covers the whole service without you enumerating routes. Add an endpoint next month and it&#39;s protected by default, which is the right direction for a failure to lean.</p><p><strong>The return type is the control flow.</strong> Returning ctx.next() continues to the handler. Returning http:TooManyRequests <em>is</em> the response; the handler never runs, and the upstream call you were trying to protect never happens. The signature tells you every way the request can leave this function.</p><p>And because the interceptor already resolved the user, it can hand that down through ctx.set(...) instead of making the handler parse the same header again.</p><h3>Reading more than one key at a time</h3><p>Once the limiter grows past a single counter (a per-user override, a top-up balance, a marker for something in flight), the naive version does a round trip per key. On a hot path, in front of a service that’s already slow, that adds up.</p><p>MGET exists for exactly this. The catch is what happens when a key isn&#39;t there yet, which for this workload is the <em>normal</em> case: a user with no override and no top-up has neither key.</p><p>ballerinax/redis gives you two batch reads, and the distinction matters:</p><pre>// Fails if ANY key is missing: string[] can&#39;t hold a nil<br>remote function mGet(string[] keys) returns string[]|Error;<br> <br>// Missing keys come back as () in the array<br>remote function mGetOptional(string[] keys) returns string?[]|Error;</pre><p>For anything where absence is expected, you want mGetOptional. Then one round trip gets you everything, and a small helper turns the holes into defaults:</p><pre>isolated function parseOrDefault(string? val, decimal default = 0.0d) returns decimal {<br>    if val is string {<br>        decimal|error parsed = decimal:fromString(val);<br>        if parsed is decimal {<br>            return parsed;<br>        }<br>    }<br>    return default;<br>}<br> <br>isolated function getEffectiveCap(string userId) returns decimal|error {<br>    string?[] values = check redis-&gt;mGetOptional([<br>        string `cap:override:${userId}`,<br>        string `cap:bonus:${userId}`<br>    ]);<br>    decimal baseCap = parseOrDefault(values[0], defaultWeeklyCap);<br>    decimal bonus = parseOrDefault(values[1]);<br>    return baseCap + bonus;<br>}</pre><p>A note on how those two methods came to exist: mGetOptional and the setNxEx below were both gaps I ran into building this, and both are now in the connector because the fixes went upstream. If you&#39;re on ballerinax/redis 3.4.1 or later they&#39;re just there.</p><h3>One-shot markers</h3><p>A related need: “has this user already done X this window?” That might be a submitted request, a notification, whatever you only want to happen once per period.</p><p>The wrong way is check-then-set, which is two commands with a race between them. The right way is a single atomic SET key value NX EX ttl: set it only if it&#39;s absent, and give it an expiry in the same breath:</p><pre>isolated function markRequested(string userId) returns boolean|error {<br>    return check redis-&gt;setNxEx(<br>        string `quota:req:${userId}`,<br>        &quot;1&quot;,<br>        secondsUntilNextMonday(time:utcNow())<br>    );<br>}</pre><p>The boolean tells you whether <em>you</em> were the one who set it. Two concurrent calls, exactly one true. And the marker expires with the window, so it resets itself for free, the same trick as the counter.</p><h3>Now delete the calendar</h3><p>Everything above works. But look at what the fixed Monday boundary actually costs you.</p><p>A user whose first request lands on Sunday at 22:00 gets a two-hour week. That’s not a rounding error, that’s a support ticket. And on the other side, <em>everyone’s</em> quota returns at the same instant: 00:00 Monday, all users at once, all of them resuming against the expensive upstream you were trying to protect. You built a thundering herd into the design and called it a schedule.</p><p>The fix is to stop asking what day it is and start asking when <em>this user</em> began. Anchor the window to them: their week starts at their first request.</p><p>Anchoring is <strong>less</strong> code, not more. The TTL already anchors on first write; that is what a TTL is. So the whole calendar apparatus above, the ISO rotation, the Sunday-vs-Monday off-by-one, all of it, collapses into:</p><pre>_ = check redis-&gt;setEx(key, cost.toString(), WEEK_SECONDS);</pre><p>The isoDayOfWeek and secondsUntilNextMonday functions are deleted outright. Lean harder on expiry and there is simply less program.</p><p>One wrinkle: a raw anchor gives each user a reset at whatever ugly minute they first showed up: “your quota resets Thursday at 09:53”. Fine for a machine, bad for a UI. So round it to a clean boundary.</p><p>Round the <strong>reset up</strong>, not the start down. Flooring the start quietly shortens the user’s week by up to an hour; rounding the reset up means everyone gets at least their seven days and still lands on the hour:</p><pre>const int WEEK_SECONDS = 604800;<br>const int BUCKET_SECONDS = 3600;  // one hour; any divisor of the day works<br> <br>isolated function resetAt(int nowSeconds) returns int {<br>    int raw = nowSeconds + WEEK_SECONDS;<br>    int remainder = raw % BUCKET_SECONDS;<br>    return remainder == 0 ? raw : raw + (BUCKET_SECONDS - remainder);<br>}<br> <br>isolated function addUsage(string userId, decimal cost) returns error? {<br>    string key = string `usage:${userId}`;<br>    int|redis:Error currentTtl = redis-&gt;ttl(key);<br> <br>    if currentTtl is int &amp;&amp; currentTtl &lt; 0 {<br>        int nowSeconds = time:utcNow()[0];<br>        _ = check redis-&gt;setEx(key, cost.toString(), resetAt(nowSeconds) - nowSeconds);<br>        return;<br>    }<br> <br>    _ = check redis-&gt;incrByFloat(key, &lt;float&gt;cost);<br>}</pre><p>You don’t need to store the window start anywhere, either. ttl(key) already tells you how long is left, so the reset time you show the user is just now + ttl, and because you rounded, it lands on the hour.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KSBKDGeLAJRIOKrxNpRUJQ.png" /></figure><p><strong>The trap this introduces.</strong> Every other key that was expiring on the same Monday boundary (the one-shot marker, the cap override, the top-up balance) no longer shares a boundary with anything. If you leave them computing their own week, they will drift out of sync with the counter they belong to, and a marker that outlives its window silently blocks a user from ever requesting again.</p><p>Make them inherit the counter’s remaining life instead of recomputing it:</p><pre>isolated function markRequested(string userId) returns boolean|error {<br>    int ttl = check redis-&gt;ttl(string `usage:${userId}`);<br>    if ttl &lt; 0 {<br>        return false;  // no active window to mark against<br>    }<br>    return check redis-&gt;setNxEx(string `quota:req:${userId}`, &quot;1&quot;, ttl);<br>}</pre><p><strong>What you give up.</strong> Per-user windows no longer line up with calendar weeks, so “what did we spend this week” can’t be answered by reading Redis. It has to come from a durable event log, which is where that question belonged anyway; a rate-limit counter is a live gauge, not an accounting record.</p><p><strong>What it costs to migrate:</strong> nothing. Old keys carry their old TTLs and age out on their own. The first write after that creates an anchored window. There is no backfill, because there is no state to move.</p><h3>Decide your failure mode</h3><p><strong>Decide what happens when Redis is down, deliberately.</strong> Your limiter has a hard dependency on Redis, and that dependency will fail eventually. You get to choose between <em>fail-open</em> (Redis errors, let the request through) and <em>fail-closed</em> (Redis errors, reject it). Fail-open means a Redis outage becomes an unmetered free-for-all; fail-closed means a Redis outage becomes a full outage of your service. Which way to go depends entirely on what is being limited: if a few minutes of over-serving costs less than a few minutes of downtime, fail-open is defensible; if the overage is dangerous rather than merely expensive, fail-closed is. Neither is a universal answer. The one thing that’s not acceptable is arriving at it by accident because an error fell through an unhandled branch. Pick it, write it down, and log loudly when it triggers.</p><p>One corollary worth internalizing: if Redis is the source of truth for your limits rather than a cache, then flushing Redis resets everybody’s quota. Treat it accordingly.</p><h3>The ceiling is soft, not hard</h3><p><strong>This is a soft ceiling, not a transaction.</strong> Look at the order of operations: the interceptor reads usage <em>before</em> the call, and the total is incremented <em>after</em> it returns. Requests that are in flight at the same moment all read the same pre-call number, so a user sitting right under their cap can fire several concurrent requests and land meaningfully over it. It’s bounded by concurrency, not unbounded, and for a weekly budget where a small overshoot is noise, it’s fine. For anything where crossing the line actually matters, you need the check and the increment to be one atomic step: a Lua script, or a reservation model where you take the budget up front and refund the difference afterwards. Know which one you’re building.</p><h3>What to take away</h3><p>Stripped down, the whole thing is small:</p><ul><li><strong>Count what you actually care about.</strong> Cost isn’t requests, and that changes the data structure.</li><li><strong>One key per user.</strong> A number and an expiry beat a window of buckets.</li><li><strong>Let TTL be the reset.</strong> The scheduler you don’t write is the scheduler that can’t fail.</li><li><strong>Anchor the window to the user, not the calendar.</strong> It’s fairer , it spreads the herd, and it deletes code. But make every related key inherit the counter’s TTL.</li><li><strong>Enforce in an interceptor.</strong> Wildcard path, response-as-return-type, new endpoints protected by default.</li><li><strong>mGetOptional, not </strong><strong>mGet,</strong> whenever a key might legitimately not exist.</li><li><strong>setNxEx for one-shot markers.</strong> Atomic, and it expires itself.</li><li><strong>Choose your failure mode on purpose,</strong> and know your ceiling is soft.</li></ul><p>The connector docs are at <a href="https://central.ballerina.io/ballerinax/redis">ballerinax/redis</a>, and the interceptor reference is in the <a href="https://ballerina.io/learn/by-example/http-interceptors/">Ballerina HTTP docs</a>. If you build one of these and hit a gap in the connector, file it. That&#39;s how both of the functions leaned on here got added.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=145bac1de3a2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bring Ballerina into Your AI Coding Agent]]></title>
            <link>https://medium.com/@viththagan.rn/bring-ballerina-into-your-ai-coding-agent-295805af98e0?source=rss-debbfe28bd03------2</link>
            <guid isPermaLink="false">https://medium.com/p/295805af98e0</guid>
            <category><![CDATA[ballerina]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[generative-ai]]></category>
            <dc:creator><![CDATA[Viththagan Roy Nesarajah]]></dc:creator>
            <pubDate>Tue, 30 Jun 2026 16:52:29 GMT</pubDate>
            <atom:updated>2026-06-30T16:53:15.831Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Getting your AI coding agent to write real Ballerina.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TnvLt-iuJftEux2GkKai7g.png" /></figure><p>Ask a modern coding agent to write Python, and it usually just works. Ask the same agent to write <a href="https://ballerina.io/">Ballerina</a>, and you’ll often get something that <em>looks</em> right, reads fluently, and then doesn’t quite build.</p><p>That isn’t the model being bad at code. It’s the model working without context. Popular languages saturate everything an LLM has read; an integration-first language like Ballerina doesn’t. So the agent fills the gaps with confident guesses, which is exactly what you don’t want when the compiler calls your bluff.</p><p>The fix isn’t a smarter model. It’s a better-informed one. That’s what these <strong>Ballerina agent skills</strong> are: context, live data, and tooling that keep the agent anchored to real Ballerina. The coding skill at the core works in any agent; the Claude Code plugin takes it furthest, adding live library discovery, the language server, and automatic activation.</p><h3>Where the wheels come off</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FRhH0xPXxCSmpexPXa9ClQ.png" /></figure><p>If you’ve paired an AI agent with Ballerina, you’ve probably watched it do at least one of these:</p><ul><li><strong>Invent imports.</strong> A package that doesn’t exist, or the right idea under the wrong org and module name. The build won’t even start.</li><li><strong>Reinvent the standard library.</strong> Hand-rolling string, array, and map operations that <em>langlib</em> already ships.</li><li><strong>Reach for </strong><strong>json and </strong><strong>map&lt;json&gt;.</strong> Ballerina rewards typed records; an untrained agent reaches for loose maps and raw JSON out of habit from other languages.</li><li><strong>Guess at connector APIs.</strong> Need an HTTP client, a Kafka consumer, a GitHub integration? Without access to <a href="https://central.ballerina.io/">Ballerina Central</a>, the agent invents a signature that reads beautifully and doesn’t exist.</li></ul><p>Any one turns code that passes the eye test into code the compiler turns away, and you end up proofreading the same mistakes over and over. Grounding the agent is what makes that loop rarer.</p><h3>What we did about it</h3><p>Four pieces, each fixing one of the ways the agent trips up. One is a coding skill that works in any agent; the other three come with the Claude Code plugin.</p><p><strong>A coding skill: the rules of the road.</strong> The agent’s working knowledge of Ballerina: idioms, project setup, the conventions that separate compiling code from wishful code, and the build/run/test workflow. Typed records over raw json. Named arguments. configurable values for external inputs. Imports plus bal build instead of a hand-edited Dependencies.toml. When the agent writes Ballerina, this is what it writes <em>to</em>.</p><p><strong>Library discovery over MCP: a line to the real ecosystem.</strong> Instead of guessing at a connector’s API, the agent asks Ballerina Central and gets back a short, accurate summary: the right org and module, the client type, the real function signatures and records. It’s the difference between an API that sounds right and the one that’s actually published, and it means a lot fewer made-up methods.</p><p><strong>A language server: eyes on the code.</strong> The plugin wires in the real Ballerina Language Server, so the agent navigates .bal code the way an IDE does: jump to a definition, hover for a type or signature, find references. It checks an API against its real source <em>while writing</em>, instead of guessing.</p><p><strong>Hooks and a discovery agent: the glue.</strong> Activation hooks keep the agent anchored to the skill, and library lookups route to a dedicated discovery agent. Nothing to turn on; it engages when you start working in Ballerina.</p><p>So: the skill is what it knows, Central is where it looks things up, the language server lets it check its own work, and the hooks wire it all together so you don’t have to think about it.</p><h3>Getting started</h3><p><strong>In Claude Code:</strong></p><pre>/plugin marketplace add ballerina-platform/skills<br>/plugin install ballerina@ballerina-skills</pre><p>Restart the session and everything comes online together: the language server, the ballerina skill, the discovery agent, and the hooks. The MCP server ships pre-built, so there&#39;s no npm install or build step.</p><p>You’ll need <a href="https://ballerina.io/downloads">Ballerina</a> 2201.12.0 (Swan Lake Update 12) or later on your PATH, and Node.js 18+ for the bundled discovery server.</p><p><strong>On other agents:</strong></p><p>The coding skill is portable. Install it for Codex, Cursor, Gemini CLI, GitHub Copilot, and others with the Open Agent Skills CLI:</p><pre>npx skills add ballerina-platform/skills</pre><p>This brings the ballerina skill only; the language server, MCP, and hooks are Claude Code plugin features. There, library discovery falls back to bal search and reading API details from Central directly. Same grounding, different plumbing.</p><h3>What it looks like in practice</h3><p>Start with a plain request, the kind you’d actually type:</p><blockquote>“Write a Ballerina HTTP service that fetches open issues from a GitHub repository and returns them as JSON.”</blockquote><p>Here’s what happens with all this in place:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/1*_CYr-VvvjBZ2b3f5Ina_vA.png" /></figure><p><strong>1. It looks up the library instead of guessing.</strong> The discovery agent searches Central, finds the official ballerinax/github connector, and pulls back its real client API: the client type, the resource that lists issues, and the record shapes it returns. No guesswork in the signatures.</p><p><strong>2. It writes idiomatic Ballerina.</strong> Typed records, a configurable for the token, named arguments, error handling with check: the conventions the skill enforces, not habits from some other language.</p><pre>import ballerina/http;<br>import ballerinax/github;</pre><pre>configurable string githubToken = ?;</pre><pre>final github:Client githubClient = check new ({<br>    auth: {token: githubToken}<br>});</pre><pre>type IssueSummary record {|<br>    int number;<br>    string title;<br>    string state;<br>    string url;<br>|};</pre><pre>service /repos on new http:Listener(8080) {<br>    resource function get [string owner]/[string repo]/issues()<br>            returns IssueSummary[]|error {<br>        github:Issue[] issues = check githubClient-&gt;/repos/[owner]/[repo]/issues(<br>            queries = {state: &quot;open&quot;, perPage: 50}<br>        );<br>        IssueSummary[] summaries = from github:Issue issue in issues<br>            select {<br>                number: issue.number,<br>                title: issue.title,<br>                state: issue.state,<br>                url: issue.htmlUrl<br>            };<br>        return summaries;<br>    }<br>}</pre><p>Every bit of that (the bearer-token ConnectionConfig, the -&gt;/repos/[owner]/[repo]/issues resource, the IssuesListForRepoQueries filters, the Issue fields) is the real, published API, not a best guess. That&#39;s the part an agent without this usually gets wrong.</p><p><strong>3. It adds the import and lets </strong><strong>bal build do the rest.</strong> The import lines plus bal build are enough; Ballerina resolves and downloads the dependency from Central on its own. No hand-edited lockfile.</p><p><strong>4. It builds, runs, and checks its work.</strong> It compiles, runs the service, and (if you asked for tests) runs bal test, then fixes any failures before handing it back.</p><p>The payoff: a service far more likely to compile and run on the first pass, built on a real connector with its real API, instead of a draft you debug into existence.</p><h3>How it actually works</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hOcsCgXn_8fsCrSyRU0B0g.png" /></figure><p>Here’s how each piece actually works.</p><p><strong>The skill: progressive disclosure, not a wall of rules.</strong> SKILL.md is a short router (write code, validate, troubleshoot) that points to deeper files only when needed: the full conventions, a langlib reference, and a troubleshooting index that maps a symptom (an HTTP error, a data-binding mismatch, a dependency conflict) to the one page for it. The agent pulls in detail on demand instead of carrying it all in context, which keeps it focused and the token budget sane.</p><p><strong>The MCP server: tens of KB of docs into a usable summary.</strong> Its one real job is get_library: it grabs a package&#39;s full API docs from Central and rewrites them as a compact <strong>Ballerina-syntax</strong> string (types, clients, functions, services) instead of sprawling JSON. Raw docs can run tens of kilobytes; the compact version is a fraction of that, which is what makes it practical to hand to the model.</p><p>The discovery agent does the thinking on top: one bal search to find candidates, pick the best-maintained one, call get_library, then trim it down to just the signatures the task needs. When two packages overlap (say, an older trigger package versus a connector with its own listener), it goes with the current one instead of blending two APIs into something that won&#39;t compile. And if a build breaks on a library&#39;s own API, it reads the resolved package source on disk for the real answer instead of guessing again.</p><p><strong>The language server: navigation straight from the compiler.</strong> The plugin registers bal start-language-server over LSP for .bal files, giving the agent compiler-backed navigation: hover for a type and signature, goToDefinition / goToImplementation to inspect a connector&#39;s real shape, documentSymbol / workspaceSymbol to find declarations. That&#39;s how it confirms an API against the source as it writes. Catching errors is a separate step: bal build from the shell is the source of truth there, not the language server.</p><p><strong>The hooks: activation without ceremony.</strong> A PreToolUse hook nudges the agent back to the skill when you start editing or running things, and a PostToolUse hook records when the skill activates. It all kicks in on its own: you ask for Ballerina, and the grounding is already there.</p><h3>A few design choices</h3><ul><li><strong>One repository, one plugin.</strong> Everything lives at the root of <a href="https://github.com/ballerina-platform/skills">ballerina-platform/skills</a>; Claude Code discovers the skill, agents, hooks, MCP, and LSP from the plugin root, with nothing to wire up.</li><li><strong>The MCP server ships bundled.</strong> Built ahead of time with esbuild and committed, so installing never compiles anything on your machine.</li><li><strong>Open source, Apache 2.0.</strong> The skill, the code rules, the discovery server: all in the open, all open to pull requests.</li></ul><h3>Where it goes next</h3><p>This is an actively evolving project, and it keeps improving with use: we sharpen the skill and the plugin continuously, folding in what we learn from real Ballerina builds. The roadmap widens the same idea: first-class support across more agents, deeper Ballerina Central coverage, and richer API summaries.</p><p>If your AI agent has been fumbling Ballerina, this is what closes the gap. Try it, open issues, and send PRs at <a href="https://github.com/ballerina-platform/skills">github.com/ballerina-platform/skills</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=295805af98e0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stop Writing Docs. Record Them.]]></title>
            <link>https://medium.com/@viththagan.rn/stop-writing-docs-record-them-2ccc4f8bbc40?source=rss-debbfe28bd03------2</link>
            <guid isPermaLink="false">https://medium.com/p/2ccc4f8bbc40</guid>
            <category><![CDATA[documentation]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[claude-code]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[developer-productivity]]></category>
            <dc:creator><![CDATA[Viththagan Roy Nesarajah]]></dc:creator>
            <pubDate>Wed, 22 Apr 2026 16:39:49 GMT</pubDate>
            <atom:updated>2026-04-22T17:05:16.365Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Because writing docs shouldn’t be harder than building the feature.</blockquote><p>As software engineers, there’s one task we all quietly avoid — writing documentation. You’ve built the feature, tested it, shipped it — and now someone wants a step-by-step guide with screenshots. The energy just isn’t there. 😅</p><p>So we reach for AI. And it helps — sort of.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DusP6GfWSC83isNz9yWszg.png" /></figure><h3>The Problem Nobody Talks About</h3><p>Here’s how documentation usually goes, even with AI in the picture:</p><p>You finish building something. You want to document it. You open your screen, do the thing, and start <strong>manually taking screenshots</strong>. You crop them, rename them (screenshot_01_final_ACTUAL_final.png), and dump them into a folder.</p><p>Then you switch to your AI assistant and start describing what you did — “first I clicked this, then I filled in that” — and it generates some nice-looking markdown. Great.</p><p>But now you’re the glue. The screenshots are in one folder. The AI-written steps are somewhere else. You’re manually matching them — which screenshot goes with which step? Did you capture that one? Did the cursor block the button label?</p><p>The writing problem is mostly solved. The <strong>coordination problem</strong> is still all yours.</p><h3>What If You Just… Recorded It?</h3><p>Here’s the idea: instead of doing the thing, taking screenshots, writing steps, and stitching it together yourself — what if you just <strong>recorded a short screen walkthrough</strong>, handed it to Claude Code, and got back a finished markdown doc with matched, cropped screenshots?</p><p>That’s exactly what I built.</p><h3>Meet the Skill</h3><p>I’ve packaged the entire workflow as a Claude Code skill. Add the marketplace, install one plugin, and you’re done.</p><p>To install it, you can follow the guide below — which happens to be a good example of what the skill actually produces.</p><blockquote><em>📝 </em><strong><em>A little meta:</em></strong><em> The installation guide below was auto-generated by the skill from a 63-second screen recording. Every screenshot, every step, every crop — written by the skill itself.</em></blockquote><h3>Install and Update the sw-dev Plugin</h3><p><strong>Prerequisites</strong></p><ul><li>Claude Code installed</li></ul><p><strong>1. Add the marketplace</strong></p><pre>/plugin marketplace add RNViththagan/claude-plugins</pre><p>The Add Marketplace dialog will appear. Claude Code clones the repository from GitHub. Press <strong>Enter</strong> to confirm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fG-Myih_EpgOv_lr7l7pJw.png" /><figcaption>Add Marketplace</figcaption></figure><p><strong>2. Install the plugin</strong></p><pre>/plugin install sw-dev@RNViththagan</pre><p>The Plugin Details screen opens. Select your preferred install scope:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1VZ_Wx4pe39hFvbA-Uk8sw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_Y6WM0La_sBmfhI9ugJ19Q.png" /><figcaption>Plugin Details with install scope options</figcaption></figure><p>Press <strong>Enter</strong> to install.</p><p><strong>3. Reload plugins</strong></p><pre>/reload-plugins</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0Ecmx0RE2UGpTE7TxJ5m3A.png" /><figcaption>Reload plugins</figcaption></figure><p>The plugin is now active.</p><p><strong>4. Update the plugin</strong></p><p>Run /plugin, go to the <strong>Installed</strong> tab, select sw-dev, and choose <strong>Update now</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eYx9mtQJDjyGpfasoE2OsQ.png" /><figcaption>Installed tab showing Update now option</figcaption></figure><p>After updating, run /reload-plugins again.</p><h3>Using It</h3><p>The easiest way is to just mention your video in the conversation:</p><blockquote>“Here’s a recording of the onboarding flow: /path/to/recording.mov — can you generate docs for it?&quot;</blockquote><p>Claude Code recognises the intent and triggers the skill automatically. No slash command needed.</p><p>If you prefer to invoke it explicitly:</p><pre>/sw-dev:doc-from-video</pre><p>Either way, Claude walks through the analysis, shows you an outline for review, extracts screenshots, and writes the doc. All intermediate files are cleaned up when you confirm you’re done.</p><p>The output is a folder with a markdown file and an images/ directory — ready to publish, commit to your repo, or paste into Confluence.</p><h3>Under the Hood</h3><p>If you’re curious how it actually works, the skill is less about “AI writes docs” and more about a reliable pipeline that mirrors how a careful human would do it — just faster.</p><h3>1. Extract frames from the video</h3><p>ffmpeg pulls one frame per second from the recording as low-resolution JPEGs. That&#39;s enough to understand what&#39;s happening on screen without flooding the context window with thousands of images. For fast-paced recordings, it bumps to 2fps. For deeper analysis, it re-extracts specific sections at 4fps.</p><h3>2. Analyse in passes, take timestamped notes</h3><p>This is the key design decision. Rather than generating the doc directly from frames, Claude <strong>writes detailed notes first</strong>.</p><p>It does a rough pass — sampling every 10th frame to get the big picture — then goes section by section, reading every frame in each range. For every new screen or dialog, it writes a full inventory: every visible field, label, button, and layout element. For screens it’s already seen, it only notes what changed.</p><p>Everything gets a timestamp. Every potential screenshot gets flagged.</p><p>The notes become the <strong>source of truth</strong>. On a long video, this prevents details from early sections getting lost by the time the doc is written. It also means the final generation step works from a complete, structured foundation — not a blur of frames it vaguely remembers.</p><h3>3. Pick the right screenshots — cursor-aware</h3><p>Screenshots from any random frame tend to look messy — the cursor is hovering over a label you need to read, a dropdown is half-open, or a spinner is mid-rotation.</p><p>For each flagged timestamp, Claude extracts a short burst of frames (5fps over a ±2 second window) at the original resolution. Then the <strong>cursor-aware</strong> selection kicks in: it picks the frame where the cursor isn’t blocking a label or button, no dropdown is mid-open, and no spinner is visible.</p><p>If the relevant content is just one panel or dialog, it crops to that region. Otherwise it keeps the full screen.</p><h3>4. Review, then generate</h3><p>Before writing anything, Claude shows you an outline and waits for your confirmation — so you can rename steps, skip sections, or adjust what gets documented. Then it writes the final markdown with screenshots placed <em>after</em> the steps they illustrate.</p><p>The skill is open source at <a href="https://github.com/RNViththagan/claude-plugins"><strong>RNViththagan/claude-plugins</strong></a> — take a look if you want to see how it’s built, or contribute back.</p><p>If it saves you from one documentation session, a ⭐ feels like a fair trade 😉</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2ccc4f8bbc40" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Claude Code’s Plan Mode: Streamlining AI-Driven Development]]></title>
            <link>https://medium.com/@viththagan.rn/claude-codes-plan-mode-streamlining-ai-driven-development-d396a5ea1fc4?source=rss-debbfe28bd03------2</link>
            <guid isPermaLink="false">https://medium.com/p/d396a5ea1fc4</guid>
            <category><![CDATA[co-pilot]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[planning]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[claude-code]]></category>
            <dc:creator><![CDATA[Viththagan Roy Nesarajah]]></dc:creator>
            <pubDate>Fri, 17 Oct 2025 12:21:59 GMT</pubDate>
            <atom:updated>2025-10-17T12:21:59.963Z</atom:updated>
            <content:encoded><![CDATA[<p>As a developer passionate about building intelligent coding agents, I’ve been diving into how AI can enhance development workflows. Claude Code’s Plan Mode stands out as a structured approach that ensures thoughtful planning, user validation, and precise execution. In this analysis, I’ll walk through Plan Mode’s workflow, focusing on how tools like ExitPlanMode and TodoWrite orchestrate a seamless process, supported by other tools like WebSearch, Read, Write, Edit, and Bash. Using a simplified Python TODO app as a case study, I’ll present the workflow step-by-step to show how Claude Code balances AI autonomy with human oversight for reliable results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hR-OB0E4wvg2JIV2vnPeaw.png" /></figure><h3>Why Structured Workflows Matter</h3><p>Unplanned coding often leads to errors: unclear requirements, overlooked edge cases, or code needing heavy refactoring. Plan Mode addresses this with a disciplined sequence — research, plan, validate, execute — mimicking a seasoned developer’s approach. By integrating user feedback and task tracking, it ensures alignment, much like a sprint planning session, but with AI-driven efficiency. A key feature is that Plan Mode restricts the agent to read-only tools (e.g., WebSearch, Read) until the user approves the plan via ExitPlanMode, preventing premature changes.</p><h3>Presenting the Plan</h3><p>ExitPlanMode is pivotal for ensuring user alignment before execution. It’s used when a task requires implementation planning, such as building a feature like the TODO app, but not for research tasks like exploring a codebase. By presenting a completed plan to the user for approval, ExitPlanMode acts as a gatekeeper, unlocking write-enabled tools only after validation. This mimics a design review, ensuring the agent’s plan aligns with user intent before any code is written.</p><h3>Tracking Progress</h3><p>TodoWrite drives execution by managing a task list for complex, multi-step tasks (3+ steps or user-requested lists). It tracks progress with statuses—pending, in_progress, completed—updating in real-time as tasks are executed. For example, it marks a task as in_progress before execution and completed after success, ensuring clarity. Skipped for trivial tasks like printing “Hello World,” it excels in scenarios like the TODO app, where multiple steps (file creation, testing) require organization, acting like a developer’s project tracker.</p><h3>The Plan Mode Workflow: Step by Step</h3><p>Plan Mode moves from planning to execution with clear checkpoints, leveraging a suite of tools. Below, I trace the flow, integrating ExitPlanMode and TodoWrite alongside task execution.</p><h3>1. User Activates Plan Mode</h3><p>The process begins with a user prompt including a system reminder that activates Plan Mode, restricting the agent to read-only tools:</p><pre>{<br>  &quot;content&quot;: [<br>    {<br>      &quot;text&quot;: &quot;&lt;system-reminder&gt;Plan mode is active. Use only read-type tools until user approval...&lt;/system-reminder&gt;&quot;,<br>      &quot;type&quot;: &quot;text&quot;<br>    },<br>    {<br>      &quot;text&quot;: &quot;simple todo app in python&quot;,<br>      &quot;type&quot;: &quot;text&quot;<br>    }<br>  ],<br>  &quot;role&quot;: &quot;user&quot;<br>}</pre><p>This ensures the agent plans without modifying files or running commands, akin to a developer outlining a solution.</p><h3>2. Agent Researches and Drafts Plan</h3><p>The agent uses read-only tools like WebSearch to gather context, adhering to the system reminder’s restriction:</p><pre>{<br>  &quot;input&quot;: {<br>    &quot;query&quot;: &quot;simple todo app python command line 2024&quot;<br>  },<br>  &quot;name&quot;: &quot;WebSearch&quot;,<br>  &quot;type&quot;: &quot;tool_use&quot;<br>}</pre><p>Results shape a plan, outlining features (e.g., CRUD operations) and steps (e.g., create todo.py). This step ensures informed planning without premature actions.</p><h3>3. Agent Presents Plan for Approval</h3><p>The agent submits the plan using ExitPlanMode:</p><pre>{<br>  &quot;input&quot;: {<br>    &quot;plan&quot;: &quot;# Simple Todo App Plan\n## Overview\nCommand-line app with CRUD...\n## Steps\n1. Create todo.py...&quot;<br>  },<br>  &quot;name&quot;: &quot;ExitPlanMode&quot;,<br>  &quot;type&quot;: &quot;tool_use&quot;<br>}</pre><p>The user approves, and the tool returns:</p><pre>{<br>  &quot;content&quot;: &quot;User has approved your plan. You can now start coding...&quot;,<br>  &quot;type&quot;: &quot;tool_result&quot;<br>}</pre><p>This checkpoint ensures alignment before write-enabled tools are unlocked.</p><h3>4. Agent Initializes Task List</h3><p>With approval, the agent initializes a task list using TodoWrite:</p><pre>{<br>  &quot;input&quot;: {<br>    &quot;todos&quot;: [<br>      {&quot;content&quot;: &quot;Create main todo.py file&quot;, &quot;status&quot;: &quot;pending&quot;, &quot;activeForm&quot;: &quot;Creating main todo.py file&quot;},<br>      {&quot;content&quot;: &quot;Test all functionality&quot;, &quot;status&quot;: &quot;pending&quot;, &quot;activeForm&quot;: &quot;Testing all functionality&quot;}<br>    ]<br>  },<br>  &quot;name&quot;: &quot;TodoWrite&quot;,<br>  &quot;type&quot;: &quot;tool_use&quot;<br>}</pre><p>This sets up a tracker for multi-step tasks, like a developer’s issue board.</p><h3>5. Agent Executes Task and Tracks Progress</h3><p>The agent marks a task as in_progress with TodoWrite:</p><pre>{<br>  &quot;input&quot;: {<br>    &quot;todos&quot;: [<br>      {&quot;content&quot;: &quot;Create main todo.py file&quot;, &quot;status&quot;: &quot;in_progress&quot;, &quot;activeForm&quot;: &quot;Creating main todo.py file&quot;},<br>      ...<br>    ]<br>  },<br>  &quot;name&quot;: &quot;TodoWrite&quot;,<br>  &quot;type&quot;: &quot;tool_use&quot;<br>}</pre><p>It creates a file using Write:</p><pre>{<br>  &quot;input&quot;: {<br>    &quot;content&quot;: &quot;#!/usr/bin/env python3\nimport argparse...\nif __name__ == &#39;__main__&#39;:\n    main()&quot;,<br>    &quot;file_path&quot;: &quot;/tmp/temp_project/todo.py&quot;<br>  },<br>  &quot;name&quot;: &quot;Write&quot;,<br>  &quot;type&quot;: &quot;tool_use&quot;<br>}</pre><p>Upon success, TodoWrite marks it completed:</p><pre>{<br>  &quot;input&quot;: {<br>    &quot;todos&quot;: [<br>      {&quot;content&quot;: &quot;Create main todo.py file&quot;, &quot;status&quot;: &quot;completed&quot;, &quot;activeForm&quot;: &quot;Creating main todo.py file&quot;},<br>      ...<br>    ]<br>  },<br>  &quot;name&quot;: &quot;TodoWrite&quot;,<br>  &quot;type&quot;: &quot;tool_use&quot;<br>}</pre><p>This cycle ensures transparent progress tracking.</p><h3>6. Agent Validates Task and Tracks Progress</h3><p>The agent tests using Bash:</p><pre>{<br>  &quot;input&quot;: {<br>    &quot;command&quot;: &quot;cd /tmp/temp_project &amp;&amp; python3 todo.py --help&quot;,<br>    &quot;description&quot;: &quot;Test help command&quot;<br>  },<br>  &quot;name&quot;: &quot;Bash&quot;,<br>  &quot;type&quot;: &quot;tool_use&quot;<br>}</pre><p>If an error occurs (e.g., python vs. python3), it retries using tools like Edit. Successful output confirms functionality:</p><pre>{<br>  &quot;content&quot;: &quot;usage: todo.py [-h] [--add ADD] [--list]...&quot;,<br>  &quot;type&quot;: &quot;tool_result&quot;<br>}</pre><p>TodoWrite updates the testing task’s status, maintaining clarity.</p><h3>7. Agent Completes Remaining Tasks</h3><p>The agent loops through tasks (e.g., creating a README with Write or editing files with Edit), updating TodoWrite:</p><pre>{<br>  &quot;input&quot;: {<br>    &quot;todos&quot;: [<br>      {&quot;content&quot;: &quot;Create README with usage examples&quot;, &quot;status&quot;: &quot;completed&quot;, &quot;activeForm&quot;: &quot;Creating README with usage examples&quot;},<br>      {&quot;content&quot;: &quot;Test all functionality&quot;, &quot;status&quot;: &quot;in_progress&quot;, &quot;activeForm&quot;: &quot;Testing all functionality&quot;}<br>    ]<br>  },<br>  &quot;name&quot;: &quot;TodoWrite&quot;,<br>  &quot;type&quot;: &quot;tool_use&quot;<br>}</pre><p>This cycle continues until all tasks are completed.</p><h3>Visualizing the Flow</h3><p>The diagram below illustrates Plan Mode’s workflow:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/15f0a45d454ca2df42162640f8a7a2b0/href">https://medium.com/media/15f0a45d454ca2df42162640f8a7a2b0/href</a></iframe><ul><li><strong>User</strong>: Submits prompt, approves plan.</li><li><strong>Agent</strong>: Researches, executes, tracks progress.</li><li><strong>Tools</strong>: WebSearch, Read, ExitPlanMode, TodoWrite, Write, Edit, Bash, and others drive the process.</li></ul><h3>Why This Workflow Excels</h3><p>Plan Mode’s strengths include:</p><ul><li><strong>Structured Planning</strong>: The system reminder restricts the agent to read-only tools (e.g., WebSearch, Read) until ExitPlanMode approval, ensuring thorough preparation.</li><li><strong>Human Oversight</strong>: ExitPlanMode aligns plans with user intent, reducing errors.</li><li><strong>Transparent Tracking</strong>: TodoWrite provides real-time status updates for clarity.</li><li><strong>Tool Cohesion</strong>: Tools like WebSearch, Read, Write, Edit, Bash, and others form a robust pipeline for planning, execution, and validation.</li></ul><p>This feels like pair programming with an AI that prioritizes alignment. Complex tasks may need multiple planning rounds, but Plan Mode’s structure is robust.</p><h3>Key Findings on Claude Code’s Plan Mode</h3><ul><li><strong>Read-Only Enforcement</strong>: The system reminder’s restriction to read-only tools (e.g., WebSearch, Read) during Plan Mode ensures deliberate planning until ExitPlanMode approval, preventing premature changes.</li><li><strong>Plan Validation</strong>: ExitPlanMode ensures user approval for implementation tasks, acting as a critical alignment checkpoint.</li><li><strong>Task Tracking Efficiency</strong>: TodoWrite enables systematic execution by updating task statuses, ensuring transparency and error recovery for complex tasks.</li><li><strong>Comprehensive Tool Pipeline</strong>: The integration of ExitPlanMode, TodoWrite, Write, Edit, Bash, WebSearch, Read, and other tools creates a cohesive workflow, balancing planning and execution.</li></ul><p>These findings highlight how Plan Mode delivers a disciplined, user-aligned coding process.</p><h3>Final Thoughts</h3><p>Claude Code’s Plan Mode, powered by ExitPlanMode and TodoWrite, transforms AI-assisted coding into a structured, collaborative process. By enforcing read-only planning, user validation, and transparent task tracking, it ensures reliable code. Based on Claude Code, this workflow sets a high standard for AI-driven development. Have you explored similar workflows? Share your thoughts in the comments.</p><p><em>Note: Analysis based on Claude Code v1</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d396a5ea1fc4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Smart Sub-Agents for File and Context Searching in Coding AI Agents]]></title>
            <link>https://medium.com/@viththagan.rn/smart-sub-agents-for-file-and-context-searching-in-coding-ai-agents-b78d9b0ee662?source=rss-debbfe28bd03------2</link>
            <guid isPermaLink="false">https://medium.com/p/b78d9b0ee662</guid>
            <category><![CDATA[context-management]]></category>
            <category><![CDATA[copilot-programming]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[ai-coding]]></category>
            <dc:creator><![CDATA[Viththagan Roy Nesarajah]]></dc:creator>
            <pubDate>Wed, 01 Oct 2025 16:15:21 GMT</pubDate>
            <atom:updated>2025-10-01T16:54:05.683Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Explore how dedicated sub-agents optimize file and context searching in coding AI agents, tackling memory management and security challenges effectively.</em></p><figure><img alt="Generated from Nano Banana AI" src="https://cdn-images-1.medium.com/max/1024/1*QWju3nyn-gqbyea0_6gUyw.png" /></figure><p>Managing context efficiently is crucial in the world of coding AI agents — AI-powered tools that assist developers by generating, refactoring, and testing code. Drawing from general principles in agent architecture, such as minimizing context overload and enhancing task isolation, I’ve been investigating how to handle file and context searching without overwhelming the main agent’s memory. The key? Dedicated sub-agents for search tasks. Let’s break this down, inspired by real-world examples like Claude’s code tools, and address the challenges of memory management and security.</p><h3>Why Sub-Agents for Searching?</h3><p>Coding AI agents often need to search files or retrieve context — think grepping for keywords, globbing paths, or reading file contents to inform code generation. A naive approach dumps all this into the main agent’s context window, leading to bloat and potential errors as conversations grow.</p><p>Enter sub-agents: lightweight, specialized components that offload these tasks. For instance, Claude Code uses tools like grep and glob for quick searches, plus a general-purpose sub-agent for complex queries, research, or multi-step operations. This sub-agent can hunt for keywords or files iteratively, returning only the essentials to the main agent.</p><p>The benefit? It keeps the main agent’s context lean by avoiding exhaustive search logs or intermediate results. This aligns with core principles of context engineering: share only what’s necessary to avoid compounding errors, as actions (like searches) carry implicit decisions that could mislead if not isolated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/534/1*lXMQO4e1HTsgec90x8oV8A.png" /></figure><h3>The Memory Trade-Off</h3><p>A challenge arises if the sub-agent is stateless, forgetting data after each task, which limits optimization potential. In complex workflows, one search might uncover details to refine the next — e.g., an initial scan revealing multiple matches could guide a follow-up to prioritize results.</p><p>Solution: Store the sub-agent’s actions and search history in a dedicated context memory managed to support optimization. This allows sub-agents to leverage prior search context, enhancing efficiency without overloading the main agent.</p><h3>Handling File Modifications</h3><p>Coding AI agents dynamically modify files — editing code, running tests, then iterating. If the sub-agent caches file contents, outdated data can lead to stale search results and bugs.</p><p>To address this, implement update mechanisms:</p><ul><li><strong>Timestamps or Hashes</strong>: Track the last read time or use a hash (e.g., SHA-256) to detect changes, prompting the sub-agent to refresh the context on modification.</li><li><strong>Event-Driven Sync</strong>: Use orchestration hooks to broadcast updates, keeping the sub-agent’s memory current without constant polling.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/1*t9HfpY2fmYGpg5yJYlICUA.png" /></figure><p>This ensures accuracy and efficiency, avoiding errors from outdated context.</p><h3>Security Considerations in File Handling</h3><p>Searching often involves sensitive files, like configs with API keys or env vars, where exposing data to the main agent’s LLM context risks privacy breaches, especially in tasks like copying secrets.</p><p>One idea is to use a stateless sub-agent to process sensitive files and return metadata (e.g., “Key copied: true”). While this could help isolate sensitive operations, it has potential issues if the sub-agent relies on an LLM, as sensitive data might still be exposed. A logical alternative is a non-LLM tool within the sub-agent, using regex or pattern matching to extract metadata (e.g., key names) without revealing values, returning confirmations like “Key exists: true.” Add user confirmation prompts before modifications to enhance security. The key takeaway: Keep sensitive data out of LLM contexts entirely. However, handling complex tasks may not be feasible with a non-LLM approach.</p><h3>Wrapping Up</h3><p>Sub-agents for file and context searching balance memory optimization, iterative efficiency, and security updates. As AI models advance, these methods may evolve, but starting with isolated sub-agents avoids common pitfalls. These are my insights from my learnings; better approaches may emerge as the field progresses.</p><p>For deeper dives, check out Cognition’s blog on context engineering for multi-agent and sub-agent systems and this insightful YouTube discussion on how Zed built their coding agent panel.</p><p><em>References:</em></p><ul><li><a href="https://cognition.ai/blog/dont-build-multi-agents">Don’t Build Multi-Agents</a></li><li><a href="https://www.youtube.com/watch?v=24rAEgYd6jw">YouTube: Building Reliable Coding Agents</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b78d9b0ee662" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build Subscription Apps Fast with Ballerina’s PayPal Subscriptions Connector]]></title>
            <link>https://medium.com/ballerina-techblog/build-subscription-apps-fast-with-ballerinas-paypal-subscriptions-connector-002a4d09683b?source=rss-debbfe28bd03------2</link>
            <guid isPermaLink="false">https://medium.com/p/002a4d09683b</guid>
            <category><![CDATA[saas-development]]></category>
            <category><![CDATA[paypal-api]]></category>
            <category><![CDATA[api-integration]]></category>
            <category><![CDATA[subscription]]></category>
            <category><![CDATA[ballerina-connector]]></category>
            <dc:creator><![CDATA[Viththagan Roy Nesarajah]]></dc:creator>
            <pubDate>Mon, 07 Jul 2025 07:52:48 GMT</pubDate>
            <atom:updated>2025-07-18T07:35:21.421Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>This article was written using <a href="https://ballerina.io/">Ballerina SwanLake Update 12 (2201.12.0)</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*akFxu_ObiMS5K0u_1vDnOg.png" /></figure><p>Want to add subscriptions to your app? Whether it’s a SaaS platform, a membership site, or a content service, managing payments can be tricky. The ballerinax/paypal.subscriptions connector for Ballerina makes it super easy to integrate PayPal subscriptions. If you’re new to Ballerina, this post will show you how simple it is to get started with a quick example.</p><h3>What Are Ballerina Connectors?</h3><p>Ballerina is a programming language built for integration. Ballerina connectors are part of the Ballerina Library that helps seamlessly integrate with different systems, such as PayPal’s. They handle the tough stuff, authentication, data formatting, so you can write clean, safe code. The ballerinax/paypal.subscriptions connector lets you create and manage subscriptions with just a few lines, perfect for building subscription-based apps.</p><h3>Why Use This Connector?</h3><p>Subscriptions power apps like Netflix or your favorite SaaS tool. With the ballerinax/paypal.subscriptions connector, you can:</p><ul><li>Create subscription plans with custom prices.</li><li>Sign up customers effortlessly.</li><li>Manage subscriptions without hassle.</li></ul><p>It’s built for Ballerina’s clear, beginner-friendly syntax.</p><h3>What Can You Do with This Connector?</h3><p>The ballerinax/paypal.subscriptions connector simplifies managing PayPal subscriptions with a range of powerful features:</p><ul><li><strong>Create subscription plans</strong>: Set up plans with custom pricing and billing cycles, tailored to your app’s needs.</li><li><strong>Manage plans</strong>: View, update, activate, deactivate, or adjust pricing for existing plans to keep them aligned with your business.</li><li><strong>Enroll users in subscriptions</strong>: Let users subscribe to plans with minimal code, enabling seamless sign-ups.</li><li><strong>Handle subscriptions</strong>: Update, revise, suspend, cancel, or reactivate subscriptions, and capture payments or track transaction histories for full control.</li></ul><p>These capabilities make it easy to build and manage subscription-based apps.</p><h3>Setting Up PayPal Credentials</h3><p>To use the connector, you need a PayPal Developer account and credentials:</p><p>1. Visit the <a href="https://developer.paypal.com/dashboard">PayPal Developer Dashboard</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XoP6ecfGeaMIW8JoM-TycQ.png" /></figure><p>2. Create a Business account (note: some features may vary by region).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XnWKgYcQZbukmUszu6VYQw.png" /></figure><p>3. Go to “Apps and Credentials,” create a new REST API app, and select your Business account.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*llB-0yy09XVJOMLt_Uq5tQ.png" /></figure><p>4. Copy the clientId and clientSecret displayed for your app. Store them securely.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WaOJd8abDbvszpYi6UlpKA.png" /></figure><p>Follow PayPal’s developer docs for detailed setup instructions.</p><h3>A Simple Example: Pro Plan Subscription</h3><p>Let’s build a SaaS app with a “Pro Plan” for $10/month and let users subscribe. The code below first creates the subscription plan and then enrolls a user in it.</p><p>Set up PayPal credentials and API URLs in a configuration file:</p><pre>clientId = &quot;&lt;your_client_id&gt;&quot;<br>clientSecret = &quot;&lt;your_client_secret&gt;&quot;<br>productId = &quot;&lt;your_product_id&gt;&quot;<br><br># Use sandbox URLs for testing or production URLs (https://api-m.paypal.com) for live apps<br>tokenUrl = &quot;https://api-m.sandbox.paypal.com/v1/oauth2/token&quot;<br>serviceUrl = &quot;https://api-m.sandbox.paypal.com&quot;</pre><p>Import modules, define configurations including API URLs, and initialize the PayPal client:</p><pre>import ballerina/io;<br>import ballerinax/paypal.subscriptions as paypal;</pre><pre>configurable string clientId = ?;<br>configurable string clientSecret = ?;<br>configurable string productId = ?;<br>configurable string tokenUrl = ?;<br>configurable string serviceUrl = ?;<br><br>final paypal:Client paypal = check new ({<br>    auth: {<br>        clientId,<br>        clientSecret,<br>        tokenUrl<br>    }<br>}, serviceUrl);</pre><p>Create a function to define a $10/month Pro Plan:</p><pre>function createProPlan() returns paypal:Plan|error {<br>    paypal:PlanRequestPOST plan = {<br>        product_id: productId,<br>        name: &quot;Pro Plan&quot;,<br>        description: &quot;Monthly subscription for pro features&quot;,<br>        status: &quot;ACTIVE&quot;,<br>        billing_cycles: [<br>            {<br>                frequency: {<br>                    interval_unit: &quot;MONTH&quot;,<br>                    interval_count: 1<br>                },<br>                tenure_type: &quot;REGULAR&quot;,<br>                sequence: 1,<br>                total_cycles: 0,<br>                pricing_scheme: {<br>                    fixed_price: {<br>                        value: &quot;10.00&quot;,<br>                        currency_code: &quot;USD&quot;<br>                    }<br>                }<br>            }<br>        ],<br>        payment_preferences: {auto_bill_outstanding: true}<br>    };<br>    return paypal-&gt;/plans.post(plan);<br>}</pre><p>Create a function to enroll a user in the subscription:</p><pre>function subscribeUser(string planId) returns paypal:Subscription|error {<br>    paypal:SubscriptionRequestPost subscription = {<br>        plan_id: planId,<br>        subscriber: {<br>            name: {<br>                given_name: &quot;Viththagan&quot;,<br>                surname: &quot;R&quot;<br>            }<br>        }<br>    };<br>    return paypal-&gt;/subscriptions.post(subscription);<br>}</pre><p>Call the functions to create the plan and subscribe a user:</p><pre>public function main() returns error? {<br>    paypal:Plan createdPlan = check createProPlan();<br>    io:println(&quot;Created Plan: &quot;, createdPlan.id);<br><br>    paypal:Subscription createdSub = check subscribeUser(createdPlan.id);<br>    io:println(&quot;Subscription created: &quot;, createdSub.id);<br>}</pre><p>Run it with bal run, and you’ll see:</p><pre>Created Plan: P-987654321<br>Subscription created: I-123456789</pre><p>The connector makes this process quick and easy!</p><h3>Why Ballerina?</h3><p>Ballerina makes integrations easy:</p><ul><li><strong>No Mistakes</strong>: Type safety catches errors early at compile-time.</li><li><strong>Simple Setup</strong>: Store API keys securely.</li><li><strong>Clear Code</strong>: Write code that’s easy to read.</li></ul><p>This connector simplifies building subscription apps with clean, efficient code.</p><h3>Try It Out</h3><p>Ready to create your subscription-based app? Dive into the ballerinax/paypal.subscriptions connector on <a href="https://central.ballerina.io/ballerinax/paypal.subscriptions/1.0.0">Ballerina Central</a>. Explore practical examples and contribute ideas at the <a href="https://github.com/ballerina-platform/module-ballerinax-paypal.subscriptions">GitHub repository</a>. Follow PayPal’s developer docs to set up your credentials, and visit the <a href="https://ballerina.io/learn/">Ballerina documentation</a> to discover more integration possibilities. With Ballerina’s simple syntax, building powerful apps is a breeze start today!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=002a4d09683b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ballerina-techblog/build-subscription-apps-fast-with-ballerinas-paypal-subscriptions-connector-002a4d09683b">Build Subscription Apps Fast with Ballerina’s PayPal Subscriptions Connector</a> was originally published in <a href="https://medium.com/ballerina-techblog">Ballerina Swan Lake Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>