<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Amila De Silva on Medium]]></title>
        <description><![CDATA[Stories by Amila De Silva on Medium]]></description>
        <link>https://medium.com/@jaadds?source=rss-68142244b10b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*0EVhZ-oEnhVgcjrR</url>
            <title>Stories by Amila De Silva on Medium</title>
            <link>https://medium.com/@jaadds?source=rss-68142244b10b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 03:11:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jaadds/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Visibility and Ordering: Navigating the Java Memory Model]]></title>
            <link>https://jaadds.medium.com/visibility-and-ordering-navigating-the-java-memory-model-a90c249872b9?source=rss-68142244b10b------2</link>
            <guid isPermaLink="false">https://medium.com/p/a90c249872b9</guid>
            <category><![CDATA[concurrency]]></category>
            <category><![CDATA[java]]></category>
            <dc:creator><![CDATA[Amila De Silva]]></dc:creator>
            <pubDate>Mon, 27 Jul 2026 17:36:46 GMT</pubDate>
            <atom:updated>2026-07-29T02:00:29.230Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hWjgtFQTs8ap-f5V-roCvw.png" /></figure><h3><strong>Can you spot the bug?</strong></h3><p>So, you know a bit about concurrency, right?</p><p>Take a look at the following program and see if you can predict what it prints.</p><p>The code is intentionally simple. Ignore why it’s doing work in a tight loop, for the moment. The interesting part is what happens when the main thread asks the worker thread to stop.</p><pre>public class Demo {<br>   static  boolean running = true;<br><br><br>   public static void main(String[] args) throws InterruptedException {<br>       Thread worker = new Thread(() -&gt; {<br>           long count = 0;<br>           while (running) {          // read the flag on every pass<br>               count++;<br>           }<br>           System.out.println(&quot;Worker stopped after &quot; + count + &quot; iterations&quot;);<br>       });<br><br><br>       worker.start();<br>       Thread.sleep(1000);            // let the worker warm up<br><br><br>       System.out.println(&quot;Main: setting running = false&quot;);<br>       running = false;               // ask the worker to stop<br><br><br>       worker.join();                 // wait for it to finish<br>       System.out.println(&quot;Main: done&quot;);<br>   }<br>}</pre><p>The worker thread continuously increments a counter. The main thread waits for a second, sets running to false, and then waits for the worker to exit.</p><p>At first glance, there doesn’t seem to be anything particularly complicated here. There are no locks, no synchronized blocks, and no obvious race conditions. One thread writes a boolean, another thread reads it.</p><p>So the output should look something like this, right?</p><pre>Main: setting running = false<br>Worker stopped after 4,582,991,234 iterations<br>Main: done</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mz7hWHGiri5swWFwZ-myqA.png" /><figcaption>Every simple concurrency bug begins exactly like this.</figcaption></figure><p>Unfortunately… that’s probably <strong>not</strong> what you’ll see.</p><p>On most JVMs running in server mode, after a few executions, the program simply hangs.</p><p>It prints:</p><pre>Main: setting running = false</pre><p>…and then nothing.</p><p>join() never returns. The worker thread never stops. The program sits there until you terminate it. If you haven’t tried it already, run the program a few times in your IDE before reading further.</p><p>The more interesting question isn’t <em>that</em> it hangs.</p><p>It’s <strong>why</strong>.</p><p>After all, the worker reads running on every single iteration. The main thread changes the value. Why doesn’t the worker ever notice?</p><p>If that question made you scratch your head, you’ll definitely learn something out of this article.</p><h3>Why does it happen?</h3><p>Our intuition about shared variables is usually something like this: One thread writes a value into memory. Another thread reads that value from memory.</p><p>If that’s how the JVM behaved, this program would always terminate.</p><p>The problem is that the JVM doesn’t execute Java code the way we imagine it does.</p><p>Modern JVMs are designed to squeeze every bit of performance out of modern hardware. To do that, both the JIT compiler and the CPU aggressively optimize your code. They’ll cache frequently used values in CPU registers, reorder instructions, eliminate redundant memory reads, and perform other optimizations, as long as a <strong>single thread</strong> can’t observe any difference.</p><p>While those optimizations are great for performance, they make surprising outcomes once multiple threads start sharing state.</p><p>Let’s go back to our loop.</p><pre>long count = 0;<br>while (running) {<br>    count++;<br>}</pre><p>To us, it looks like running is read from memory every time the loop executes.</p><p>The JVM doesn’t have to do that.</p><p>After the JIT compiler observes that this thread never writes to running, it’s free to assume the value won’t change from the thread’s perspective. Instead of repeatedly loading it from memory, it may keep the value in a CPU register and reuse it on every iteration.</p><p>From the worker thread’s point of view, nothing has changed.</p><p>From the main thread’s point of view, running became false a second ago.</p><p>The two threads now have different views of the same variable.That’s exactly why the worker keeps spinning forever. At first, this behaviour feels broken, but it isn’t.</p><p>It’s allowed by the Java Memory Model (JMM).</p><p>According to <a href="https://docs.oracle.com/javase/specs/jls/se21/html/jls-17.html#jls-17.4.5">JLS 17.4.5</a>, the JMM defines <strong>what one thread is guaranteed to observe about another thread’s actions</strong>. Unless the program establishes a <em>happens-before</em> relationship between two threads, Java provides <strong>no guarantee</strong> that one thread will immediately observe another thread’s writes.</p><p>In other words, the JVM is allowed to optimize aggressively because, from the language’s perspective, those optimizations are perfectly legal.</p><p>The guarantee that one thread’s changes become visible to another is <strong>not automatic</strong>.</p><p>You have to ask for it explicitly.</p><p>One of the simplest ways to do that is with volatile.</p><h3>The fix: volatile</h3><p>The only change we need is this:</p><pre>static volatile boolean running = true;</pre><p>That single keyword completely changes the contract between your program and the JVM.</p><p>By declaring a field as volatile, you’re telling the JVM:</p><blockquote>This variable is shared between threads. Don’t optimize away the visibility guarantees.</blockquote><p>A write to a volatile field becomes visible to other threads that subsequently read the same field. The JVM also inserts the necessary memory barriers so the compiler and the CPU can’t apply certain re-orderings around those reads and writes.</p><p>With that one change, the worker thread eventually observes running == false, exits the loop, and the program terminates as expected.</p><p>So we’ve learned the first important property of volatile:</p><blockquote><strong>Visibility.</strong> A write performed by one thread becomes visible to other threads reading the same volatile field.</blockquote><p>In addition to visibility, it also provides ordering guarantees, and that’s what makes it a fundamental building block in many high-performance Java libraries.</p><p>However before proceeding to other use cases, let’s see how visibility guarantees are actually helping production code</p><h3>Visibility: publishing a single value</h3><p>Imagine a background thread continuously sending requests to a remote server.</p><p>At some point, another thread needs to tell it to stop.</p><p>That’s almost exactly what Apache Kafka’s producer does. Check <a href="https://github.com/apache/kafka/blob/525b278288acbf74de366284257789c300e9252f/clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java#L108">clients/…/producer/internals/Sender.java</a></p><p>The relevant part of the implementation looks like this:</p><pre>public void run() {<br>   while (running) {<br>       try {<br>           runOnce();<br>       } catch (Exception e) {<br>           log.error(&quot;Uncaught error in kafka producer I/O thread: &quot;, e);<br>       }<br>   }<br>}        // I/O thread reads<br><br><br>public void initiateClose() { this.running = false; }       // another thread writes</pre><p>One thread repeatedly checks the running flag while another thread changes it.</p><p>If running were an ordinary field, the same problem we saw earlier could happen again. The sending thread might never observe the update and continue running indefinitely.</p><p>Instead they have the running attribute marked volatile</p><pre>private volatile boolean running;<br><br><br>public void run() { while (running) { /* ... */ } }        // I/O thread reads<br><br><br>public void initiateClose() { this.running = false; }       // another thread writes</pre><p>Once the flag becomes volatile, a write performed by initiateClose() is guaranteed to become visible to the thread executing run(). As soon as that thread observes running == false, it exits the loop and shuts down cleanly.</p><p><strong><em>Another benefit: atomic 64-bit writes</em></strong></p><p>While we’re on the topic of visibility, there’s another guarantee volatile provides that is much less well known.</p><p>According to <a href="https://docs.oracle.com/javase/specs/jls/se8/html/jls-17.html#jls-17.7">JLS 17.7</a>, the JVM is allowed to treat ordinary 64-bit long and double writes as two separate 32-bit operations. If another thread reads the value in the middle of that update, it can observe a torn value, a combination that was never actually written.</p><p>Declaring the field volatile changes that guarantee.</p><p>A volatile long or volatile double is always read and written atomically. Readers either observe the old value or the new one, never a partially updated value.</p><p>This isn’t the reason most developers reach for volatile, but it’s an important guarantee to keep in mind when working with shared 64-bit fields.</p><p>So far, we’ve only been talking about <strong>visibility</strong> — making sure one thread eventually sees another thread’s changes.</p><p>That’s useful, but it isn’t what makes volatile so powerful.</p><p>The second guarantee is <strong>ordering</strong>.</p><p>A volatile write doesn’t just publish the value of that field; it also publishes everything that happened before it.</p><p>This turns out to solve a whole class of concurrency problems that have nothing to do with a single boolean flag. Let’s look at a few of them.</p><h3>Ordering: volatile publishes more than just itself</h3><p>Ordering isn’t a use case by itself. It’s the mechanism that makes several useful concurrency patterns possible.</p><p>Let’s look at three common ones.</p><h4>Pattern 1: Publishing surrounding fields around a flag</h4><p><strong><em>The Problem</em></strong></p><p>Imagine one thread is preparing a report while another thread waits for it to become available.</p><pre>class Report {<br>   private int rows;<br>   private long checksum;<br>   private String status;<br>   private boolean ready;<br><br><br>   // called by the producer thread<br>   void publish(int rows, long checksum, String status) {<br>       this.rows = rows;            // (1) plain writes<br>       this.checksum = checksum;    // (2)<br>       this.status = status;        // (3)<br>       this.ready = true;           // (4)<br>   }<br><br><br>   // called by consumer threads<br>   boolean tryRead() {<br>       if (!ready) {                // plain read<br>           return false;            // not published yet<br>       }<br>      <br>       System.out.println(rows + &quot; rows, checksum=&quot; + checksum + &quot;, status=&quot; + status);<br>       return true;<br>   }<br>}</pre><p>If both methods execute on the same thread, everything works exactly as you’d expect.</p><p>rows, checksum, and status are assigned first. Only then is ready set to true.</p><p>The consumer never sees ready == true before the rest of the data has been initialized.</p><p>Now introduce a second thread. Our intuition says the same ordering should still hold.</p><p>Unfortunately, the Java Memory Model doesn’t make that guarantee.</p><p>Just because the producer executes those assignments in one order doesn’t mean another thread has to observe them in that order. The compiler and the CPU are free to reorder or delay ordinary field accesses whenever there’s no synchronization preventing them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B4SGvOD24GPtpYaCasHRSw.png" /><figcaption>Variables can be seen in a different order by different threads</figcaption></figure><p>That means the consumer can observe this impossible-looking state:</p><ul><li>ready == true</li><li>rows == 0</li><li>checksum == 0</li><li>status == null</li></ul><p>In other words, it sees the flag saying “the report is ready” while still observing stale values for the report itself.</p><p>It sounds impossible. But it’s completely legal.</p><blockquote>Without a <em>happens-before</em> relationship, Java doesn’t promise that another thread will observe your writes in program order. The Java Memory Model only guarantees ordering when you explicitly ask for it.</blockquote><p>Right now, we’ve only used ordinary field writes.</p><p>So the JVM is free to optimize.</p><p>(If you’re skeptical, check the jcstress results <a href="https://jaadds.github.io/volatile-repro/problem2-ordering/jcstress/results/arm/index.html">here</a>. It demonstrate that this outcome really does occur on modern hardware. Test code for reproducing the sample is <a href="https://github.com/jaadds/volatile-repro/tree/main/problem2-ordering">here</a>.)</p><p><strong><em>The fix</em></strong></p><p>The solution is to declare the ready attribute volatile.</p><pre>private volatile boolean ready;</pre><p>The write to volatile field acts as a <strong>publication barrier</strong>. The JVM is no longer allowed to move the earlier writes — rows, checksum, status — to after the volatile write.</p><p>Likewise, when another thread reads ready and observes true, Java guarantees that it also observes everything that happened <strong>before</strong> the producer set ready.</p><p>That includes rows, checksum, and status.</p><p>Notice what happened here. We didn’t make those three fields volatile. We didn’t lock anything. We didn’t synchronize either method.</p><p>One volatile field safely published all of them.</p><p>That’s why this pattern appears so frequently in concurrent Java code.</p><h4>Pattern 2: Safe publication of a lazily-built object</h4><p><strong><em>The Problem</em></strong></p><p>Not every visibility problem revolves around primitive values or simple flags. Sometimes the thing being shared is an <strong>entire object</strong>.</p><p>Consider the following example.</p><pre>class Service {<br>   private Config config;                 // not volatile<br>   private final Object lock = new Object();<br><br><br>   void start() {<br>       if(config == null) {<br>           synchronized (lock) {              // write IS locked -- mutation is thread-safe<br>               if (config == null) {<br>                   config = new Config();<br>               }<br>           }<br>       }<br>   }<br>   Config config() {<br>       return config;                     // lock-free read -- the actual hole<br>   }<br>}</pre><p>At first glance, this looks perfectly safe.</p><p>The object is created inside a synchronized block, so only one thread can initialize it. We don’t have to worry about multiple Config instances being created.</p><p>So where’s the bug?</p><p>The synchronization only protects <strong>construction</strong>. It doesn’t protect <strong>publication</strong>.</p><p>The config() method returns the reference without acquiring the same lock. That means another thread can observe the reference as soon as it’s published.</p><p>The surprising part is that a <strong>non-null</strong> reference doesn’t necessarily mean the object behind it has been fully constructed.</p><p>How is that possible?</p><p>The seemingly simple statement</p><pre>config = new Config();</pre><p>isn’t a single operation.</p><p>Conceptually, it involves three steps:</p><ol><li>Allocate memory for the new Config object.</li><li>Run the constructor to initialize its fields.</li><li>Publish the reference by assigning it to config.</li></ol><p>We naturally expect those steps to happen in that order.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pMnj1cqOnXE8PS1yKh0TSQ.png" /><figcaption>One thread may see an object before it’s fully initialized</figcaption></figure><p>Without a happens-before relationship, Java doesn’t guarantee that another thread will observe them in that order.</p><p>The reference can become visible before the constructor’s effects become visible.</p><p>As a result, another thread can call config(), receive a perfectly valid-looking reference, and still observe default values inside the object because its initialization hasn’t been safely published.</p><p>It sounds unlikely, but it isn’t just a theoretical possibility. The jcstress results <a href="https://jaadds.github.io/volatile-repro/problem3-unsafe-publication/jcstress/results/arm/com.example.repro.ConfigCachePublication.html">here</a> demonstrate exactly this behaviour. Relevant code is available <a href="https://github.com/jaadds/volatile-repro/tree/main/problem3-unsafe-publication">here</a>.</p><p><strong><em>The fix</em></strong></p><p>The fix is simply making the config reference volatile.</p><pre>private volatile Config config;</pre><p>Making the reference volatile establishes the necessary happens-before relationship between the thread constructing the object and the thread reading it.</p><p>Once another thread observes a non-null value in config, Java guarantees that it also observes everything that happened before the reference was published — including the constructor completing.</p><p>In other words, readers either see <strong>no object</strong> or <strong>a fully initialized object</strong>. They never see one that’s only partially constructed.</p><p><strong><em>A real-world example</em></strong></p><p>One example is Kafka’s <a href="https://github.com/apache/kafka/blob/525b278288acbf74de366284257789c300e9252f/clients/src/main/java/org/apache/kafka/common/requests/MetadataResponse.java#L215-L223">MetadataResponse</a>, which lazily initializes an internal holder object only when metadata is first requested.</p><p>The holder contains derived metadata such as topic information, broker nodes, controller information, and other internal structures. Because this object is built lazily and then shared across threads, Kafka stores the reference in a volatile field. That guarantees every thread either sees no holder at all or a fully initialized one, never an object that’s only half constructed.</p><h4>Pattern 3: Publishing an immutable snapshot</h4><p><strong><em>The Problem</em></strong></p><p>In the previous pattern, we used a volatile reference to safely publish a single object.</p><p>This pattern takes the same idea one step further.</p><p>Instead of publishing one object, we’ll publish an <strong>entire snapshot</strong> of an application’s state.</p><p>Imagine a pricing service that maintains an acceptable price range.</p><pre>class PriceGuard {<br>   private volatile int min;   // even volatile on EACH field...<br>   private volatile int max;   // ...does not make the PAIR consistent<br>   // invariant: min &lt;= max<br><br><br>   void update(int newMin, int newMax) {   // one writer (e.g. a market-data thread)<br>       this.min = newMin;                  // (1)<br>       this.max = newMax;                  // (2)<br>   }<br><br><br>   boolean isAcceptable(int price) {        // many reader threads<br>       return price &gt;= min &amp;&amp; price &lt;= max; // reads min, then max -- two separate reads<br>   }<br>}</pre><p>At first glance, this looks thread-safe. Both fields are declared volatile, so every read sees the latest value.</p><p>So what’s the problem?</p><p>The problem isn’t visibility anymore. It’s <strong>consistency</strong>.</p><p>min and max aren’t independent values. Together they represent a single logical state: a valid price range.</p><p>The writer updates them one after the other.</p><p>The reader also reads them one after the other.</p><p>Those two operations can interleave.</p><p>For example, imagine the range changes from:</p><p>100–110 to 200–210</p><p>A reader might observe:</p><p>min = 200, max = 110</p><p>That’s a combination that never actually existed. Each individual read is perfectly up to date. The pair isn’t.</p><p>This is an important lesson about volatile.</p><p>A volatile field guarantees the visibility of <strong>that field</strong>. It does <strong>not</strong> magically make several related fields behave like one atomic value.</p><p><strong><em>The fix</em></strong></p><p>Instead of publishing two independent fields, publish <strong>one immutable object</strong> that contains both.</p><pre>class PriceGuard {<br>   private volatile Band band = new Band(100, 110);   // ONE volatile reference<br><br><br>   void update(int newMin, int newMax) {<br>       band = new Band(newMin, newMax);               // one atomic reference swap<br>   }<br><br><br>   boolean isAcceptable(int price) {<br>       Band b = band;                                  // ONE volatile read -&gt; consistent pair<br>       return price &gt;= b.getMin() &amp;&amp; price &lt;= b.getMax();<br>   }<br>}<br><br><br>class Band {         <br>   private int min;<br>   private int max;<br><br><br>   Band(int min, int max) {<br>       this.min = min;<br>       this.max = max;<br>   }<br><br><br>   public int getMin() {<br>       return min;<br>   }<br><br><br>   public int getMax() {<br>       return max;<br>   }<br>}</pre><p>Now the reader performs exactly one volatile read.</p><p>That single read gives it a complete snapshot of the price range.</p><p>It either sees:</p><p>100–110 or200–210</p><p>It never sees a mixture of the two.</p><p>The Band object itself is effectively immutable. Once created, its contents never change.</p><p>Updating the range simply means constructing a new Band instance and publishing it through the volatile reference.</p><p>This is a common pattern in concurrent code because it allows readers to remain completely lock-free while still observing a consistent view of the data.</p><p><strong><em>Why does this work?</em></strong></p><p>Notice what’s really being published here. We’re not relying on volatile to keep min and max in sync. Instead, we build the entire state first, then publish it with a <strong>single </strong><strong>volatile write</strong>.</p><p>That write establishes the happens-before relationship we discussed earlier.</p><p>By the time another thread observes the new Band reference, Java guarantees that the object’s construction has completed and all of its fields are visible. The entire snapshot appears atomically from the reader’s perspective.</p><p><strong><em>A real-world example</em></strong></p><p>This pattern shows up in Apache Kafka’s metadata handling.</p><p>The <a href="https://github.com/apache/kafka/blob/525b278288acbf74de366284257789c300e9252f/clients/src/main/java/org/apache/kafka/clients/Metadata.java">Metadata</a> class maintains the client’s view of the Kafka cluster — brokers, topics, partitions, leaders, and other metadata shared across many threads.</p><p><a href="https://github.com/apache/kafka/blob/525b278288acbf74de366284257789c300e9252f/clients/src/main/java/org/apache/kafka/clients/MetadataSnapshot.java#L47-L56">MetadataSnapshot</a> is the immutable read-optimized data holder that Metadata publishes. Since it’s the snapshot, the view should consistently update.</p><p>Metadata class keeps an volatile reference to <a href="https://github.com/apache/kafka/blob/525b278288acbf74de366284257789c300e9252f/clients/src/main/java/org/apache/kafka/clients/Metadata.java#L79">metadataSnapshot</a>.</p><pre>private volatile MetadataSnapshot metadataSnapshot = MetadataSnapshot.empty();</pre><p>The snapshot is largely immutable, with most of its fields declared final. Readers simply fetch the current snapshot and use it without any additional synchronization.</p><p>That gives Kafka two important properties:</p><ul><li>readers never block,</li><li>and every reader observes a consistent view of the cluster.</li></ul><p>It’s the same pattern we just implemented in PriceGuard, applied to a much larger object graph.</p><h3>volatile and Atomicity: Where volatile Stops Helping</h3><p>So far, volatile has been looking pretty impressive. It solved our visibility problem.It safely published objects. It lets us publish multiple fields with a single synchronization point.</p><p>At this point it’s tempting to think:</p><blockquote><strong>Can I just use volatile whenever multiple threads share a variable?</strong></blockquote><p>Not quite. There’s one important problem that volatile does <strong>not</strong> solve: <strong>Atomicity</strong>.</p><p>Let’s look at another example.</p><pre>class RequestCounter {<br>   private volatile long count = 0;<br><br><br>   void record() {<br>       count++;                      <br>   }<br>   long total() {<br>       return count;<br>   }<br>}</pre><p>Looks harmless enough. The counter is volatile, so every thread should see the latest value. If several threads call record() concurrently, what do you expect the final count to be?</p><p>Most people would say:</p><p>Every request increments the counter, so after 1000 requests the count should be 1000.</p><p>Unfortunately, that’s not guaranteed.</p><p><strong><em>The problem isn’t visibility</em></strong></p><p>The mistake is assuming that count++ is a single operation when It isn’t. Behind the scenes, the JVM performs three separate steps:</p><ul><li>Read count</li><li>Add one</li><li>Write the new value back</li></ul><p>Each of those individual operations is perfectly visible because count is volatile. The problem is that <strong>the three operations together are not atomic.</strong></p><p>Imagine two threads executing record() at almost the same time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mCoyXePV48OEFTS2JgPmHQ.gif" /><figcaption>Two threads incrementing a single counting the same time</figcaption></figure><p>Both threads successfully completed their work. Yet the counter only increased by one. One increment simply disappeared.</p><p>This is called a <strong>read-modify-write race</strong>, and it’s one of the most common concurrency bugs in Java. volatile can’t prevent it because every individual read and write is perfectly legal. What it cannot do is make multiple operations behave as one indivisible action.</p><h4>When is volatile enough?</h4><p>Notice how all of the examples we’ve looked at earlier had one thing in common. One thread wrote a value.</p><p>Other threads observed it. None of the writes depended on the previous value.</p><p>That’s exactly the kind of problem volatile was designed to solve.</p><p>The moment your update depends on the current value, or if you’re performing concurrent writes, you’ve moved beyond visibility.</p><p>Then you need <strong>atomicity</strong>.</p><p>That usually means using synchronization, locks, or classes such as AtomicInteger, AtomicLong, or LongAdder, depending on the problem you’re solving.</p><h3>Conclusion</h3><p>At this point we’ve seen what volatile is really for.</p><p>It isn’t a lightweight replacement for synchronized, and it isn’t a magic keyword that makes code thread-safe.</p><p>Instead, it solves two very specific problems.</p><ul><li><strong>Visibility</strong> — ensuring that one thread’s writes become visible to another.</li><li><strong>Ordering</strong> — ensuring that writes before a volatile write are observed before the volatile value itself.</li></ul><p>Those two guarantees are enough to solve an entire class of concurrency bugs. That’s why you’ll find volatile used throughout high-performance Java libraries, from stop flags and lazy initialization to immutable snapshots and state publication.</p><p>But volatile has a clear boundary.</p><p>The moment correctness depends on <strong>multiple operations happening as one indivisible action</strong>, visibility is no longer the problem. <strong>Atomicity</strong> is.</p><p>Knowing where that boundary lies is one of the most valuable concurrency lessons in Java.</p><p>PS: If this article sparked your interest in Java concurrency, I highly recommend <strong><em>Java Concurrency in Practice</em></strong>. Despite its age, it remains one of the best books for building a solid understanding of concurrent programming in Java.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a90c249872b9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes Under Load: Common Scaling Issues and How to Fix Them]]></title>
            <link>https://jaadds.medium.com/kubernetes-under-load-common-scaling-issues-and-how-to-fix-them-f9b027aed268?source=rss-68142244b10b------2</link>
            <guid isPermaLink="false">https://medium.com/p/f9b027aed268</guid>
            <category><![CDATA[scalability]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Amila De Silva]]></dc:creator>
            <pubDate>Mon, 03 Nov 2025 03:46:40 GMT</pubDate>
            <atom:updated>2025-11-03T03:46:40.747Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GU7zBMovAvegz9yVtof5vg.png" /></figure><h3>Introduction</h3><p>Black Friday is around the corner and it’s the season systems are tested to their limits.</p><p>Kubernetes, with its built-in autoscaling and orchestration capabilities, is often expected to handle such load seamlessly. However some subtle configuration issues can cause service disruptions and intermittent failures, which do not appear during routine traffic volumes.</p><p>In this article we’re exploring several issues that can surface when running Kubernetes clusters under high load, the reasons behind them, and practical approaches to resolving them.</p><h3>Scaling issues in Multi-Container Pods</h3><h4>Problem</h4><p>When running multi-container pods, you might notice that they don’t always scale when expected.</p><p>For example, one container inside the pod may be hitting its CPU limit, but the pod itself doesn’t scale.</p><h4>Cause</h4><p>The Horizontal Pod Autoscaler (HPA) , by default operates based on the metrics type Resource and, evaluates them at the <strong>pod level</strong>.<br> In multi-container pods, this can hide contention within a single container.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PbuzbFJ9ahFEKT8Z8bV_Lw.png" /><figcaption>Even though small container hits the CPU limit pod doesn’t scale.</figcaption></figure><p>If one container reaches 100 % CPU utilization while the others remain idle, the pod’s <strong>average</strong> CPU usage can still fall below the HPA threshold. As a result, the Pod won’t scale out, even though one container is fully saturated.</p><h4>Fix</h4><p>You can fix this issue by configuring the HPA to use the ContainerResource metric type. This makes scaling decisions per container, making sure that any overloaded container triggers additional replicas when needed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ynRDQNfyFNf7yVik4RXYFw.png" /><figcaption>ContainerResource metric computes utilization at each container level.</figcaption></figure><p>You can read more about this configuration in the <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#container-resource-metrics">HPA documentation</a>.</p><h3>Container Crashes Due to Memory Limits</h3><h4>Problem</h4><p>When a container exceeds its defined memory limit, Kubernetes terminates it with an OOMKilled (Out Of Memory Killed) error. This issue is particularly common in JVM-based applications under high load.</p><h4>Cause</h4><p>Applications using fixed heap size parameters, such as:</p><p>java -Xms1g -Xmx2g -jar app.jar</p><p>can allocate more memory than the container is permitted to use.</p><p>While the application may start successfully, heavy load and delayed garbage collection can eventually push memory consumption beyond the container limit, triggering termination.</p><h4>Fix</h4><p>To avoid this, use container-aware JVM parameters that automatically adjust heap allocation relative to the container’s available memory:</p><p>java -XX:InitialRAMPercentage=35.0 -XX:MaxRAMPercentage=70.0 -jar app.jar</p><p>These options automatically scale heap allocation relative to the container’s memory limit, preventing sudden OOM kills even as container configurations evolve.</p><p>You can read more about these parameters <a href="https://www.baeldung.com/java-jvm-parameters-rampercentage">here</a>.</p><h3>Pod Evictions Due to Memory Pressure</h3><h4>Problem</h4><p>Pods may get evicted when a node runs low on critical resources such as memory, disk space or process identifiers (PID). Evicted pods disappear immediately without the chance to shut down gracefully, which can disrupt workloads and cause transient failures.</p><h4>Cause</h4><p>The kubelet may evict pods due to one of the following pressures:</p><ul><li><strong>Memory pressure:</strong> Node memory drops below a critical threshold (usually around 100 MiB).</li><li><strong>Disk pressure:</strong> The node runs out of disk space for provisioning new pods, downloading images, or writing container logs.</li><li><strong>PID pressure:</strong> The node exhausts the available process identifiers.</li></ul><p>In my experience <strong>memory pressure</strong> is the most common cause during high-load conditions. This usually happens when:</p><ul><li>Containers don’t have defined resource limits, allowing them to consume memory without bounds.</li><li>The gap between the resource request and limit is excessively large, leading the scheduler to over-commit node resources.</li></ul><p>Both these scenarios can starve the other containers running in the node.</p><p>Because Kubernetes schedules pods based on <strong>requests</strong>, not <strong>limits</strong>, pods might fit during scheduling but exceed node capacity later when they reach their limits — forcing the kubelet to evict them.</p><p>When a pod is evicted, the event appears in its status. Running kubectl describe pod &lt;pod-name&gt; will show the reason in the description.</p><pre>kubectl describe po gitlab-task-runner-backup-1568592000-shkw6 -n gitlab<br>...<br>Status:         Failed<br>Reason:         Evicted<br>Message:        The node was low on resource: memory. Container task-runner-backup was using 3715668Ki, which exceeds its request of 350M. <br>...<br>Events:<br>  Type     Reason   Age   From                               Message<br>  ----     ------   ----  ----                               -------<br>  Warning  Evicted  68m   kubelet, aks-opsminion-38596230-5  The node was low on resource: memory. Container task-runner-backup was using 3715668Ki, which exceeds its request of 350M.</pre><h4>Fix</h4><p>Kubernetes decides which pods to evict based on how much resource usage exceeds their requests. Typically:</p><ul><li><strong>BestEffort</strong> and <strong>Burstable</strong> pods are evicted first. Because their resource usage is likely to exceed the request.</li><li><strong>Guaranteed</strong> pods are evicted last.</li></ul><p>You can find the full eviction order in the official documentation: — <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/#pod-selection-for-kubelet-eviction">Pod Selection for Kubelet Eviction</a></p><p><strong>To reduce evictions:</strong></p><ul><li>For critical workloads, set <strong>CPU and memory requests equal to their limits</strong> to qualify them for the <strong>Guaranteed QoS</strong> class. Guaranteed pods are far less likely to be evicted. Read more:<a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/"> Pod QoS Classes</a></li><li>Assign a <strong>higher PriorityClass</strong> to critical workloads to further reduce the likelihood of eviction.</li><li>If you want to give containers some headroom (limits &gt; requests), keep the gap small. A large difference increases the risk of eviction. In practice, running multiple smaller replicas is often more effective than assigning excessive limits to a few pods.</li></ul><h3>Downtime Caused due to CPU Contention</h3><h4>Problem</h4><p>Unlike memory exhaustion, when a container reaches its CPU limit, it is throttled rather than terminated. However, CPU throttling can delay responses to health probes and cause temporary downtime or degraded service performance.</p><h4>Cause</h4><p>Under heavy CPU load, containers may experience throttling, delaying responses to <strong>readiness</strong> or <strong>liveness</strong> probes.</p><p>If readiness probes fail repeatedly, Kubernetes removes the pod from the Service endpoint list. If all pods in a deployment experience this, incoming requests result in connection errors.</p><p>Similarly, repeated liveness probe failures can cause containers to restart. If the application does not shutdown gracefully, in-flight requests can fail during restarts.</p><p>You can verify whether probe failures caused the disruption by inspecting Kubernetes events or running kubectl describe pod &lt;pod-name&gt; to check for Liveness probe failed or Readiness probe failed messages.</p><h4>Fix:</h4><p><strong>Review CPU usage and increase allocation</strong></p><p>The most obvious fix is to increase CPU allocation — but before you do that, check the long-term CPU utilization. If usage consistently stays near 90% of the requested value, then increasing the CPU request and limit makes sense.</p><p><strong>Scale horizontally using HPA</strong></p><p>Use a Horizontal Pod Autoscaler (HPA) to spin up new replicas as utilization rises. Distributing load across multiple replicas helps reduce CPU pressure on individual pods and lowers the risk of throttling.</p><p><strong>Increase maxReplicas if HPA has reached its limit</strong></p><p>If CPU throttling continues even with HPA enabled, it may mean the pods have already reached the maximum replica count. In that case, increase the maxReplicas value to allow further horizontal scaling.</p><p><strong>Check the maximum node pool size</strong></p><p>If new pods stay in the <strong>Pending</strong> state, check whether the cluster’s node pool can actually provision additional nodes. If Node pool limit is reached, pods will stay in the Pending state forever.</p><p><strong>Optimize CPU-bound code paths</strong></p><p>Sometimes, tweaking cluster-level configs can only get you so far — the real bottleneck might be in the application itself. Review application code and look for opportunities to optimize expensive computations. These optimizations can help maintain responsiveness even under high load.</p><h3>Node Provisioning Delays</h3><h4>Problem</h4><p>Sometimes workloads don’t scale out as quickly as expected, even though the HPA is functioning correctly.</p><p>You may notice new pods stuck in the Pending state for several minutes, waiting for nodes to become available.</p><p>This typically indicates delays in node provisioning.</p><h4>Cause</h4><p>When the Cluster Autoscaler decides to add nodes, it must go through several steps — provisioning a VM, starting the kubelet, and configuring components like pod networking.</p><p>Depending on the environment, this can take anywhere from 30 seconds to a few minutes. These delays become particularly noticeable during sudden traffic spikes, where by the time new nodes are ready, the surge may have already caused degraded performance or brief downtime.</p><p>Since these delays are largely unavoidable, it’s better to reserve some compute capacity in advance, making sure resources are readily available when scaling is required. This can be achieved by over-provisioning the cluster using low-priority pods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qTOSBV48QdN91TFmHz9k5Q.gif" /><figcaption>High-priority pod preempting lower-priority pod</figcaption></figure><h4>Fix</h4><p>You can reserve resources by creating low-priority pods that occupy node capacity until higher-priority workloads need it. When demand increases, these low-priority pods are preempted, triggering a cluster scale-up in parallel.</p><ol><li>Create a PriorityClass</li></ol><pre>apiVersion: scheduling.k8s.io/v1<br>kind: PriorityClass<br>metadata:<br>  name: low-priority<br>value: -10<br>globalDefault: false<br>preemptionPolicy: Never<br>description: &quot;This priority class should be used for dummy service pods only.&quot;</pre><p>2. Deploy a Dummy Pod with the PriorityClass:</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: capacity-reservation<br>  namespace: default<br>  labels:<br>    app: overprovisioning<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: overprovisioning<br>  template:<br>    metadata:<br>      labels:<br>        app: overprovisioning<br>    spec:<br>      containers:<br>      - name: pause<br>        image: registry.k8s.io/pause<br>        resources:<br>          requests:<br>            cpu: &quot;4&quot;<br>            memory: &quot;4Gi&quot;<br>          limits:<br>            cpu: &quot;4&quot;<br>            memory: &quot;4Gi&quot;<br>      priorityClassName: low-priority</pre><p>When the cluster is already at full capacity and a high-priority pod is scheduled, the low-priority pods will be preempted to make room. Their eviction also triggers the <strong>Cluster Autoscaler</strong> to scale up and add new nodes to the cluster.</p><blockquote>Note: The priority value is intentionally set to -10. According to the <a href="https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#how-does-cluster-autoscaler-work-with-pod-priority-and-preemption">Cluster Autoscaler FAQ</a>, the cutoff for scale-up triggering is -10. Any pod with a lower priority value won’t trigger scale-up.</blockquote><p>This over-provisioning pattern helps workloads start quickly, even during traffic surges. It’s especially useful if your cluster uses <a href="https://learn.microsoft.com/en-us/azure/virtual-machines/spot-vms">Spot Instances</a>, making sure that nodes are available at your bid price when new capacity is needed.</p><p>For more information, refer to the Kubernetes documentation: <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/">Pod Priority and Preemption</a></p><h3><strong>Conclusion</strong></h3><p>Kubernetes is built to scale, but scaling smoothly under pressure depends mainly on how the cluster is configured. When traffic surges, even minor configuration gaps — a misconfigured HPA metric, an unbounded memory limit, or a slow node spin-up — can turn into real production issues.</p><p>Most of the problems described here are not Kubernetes bugs — they’re tuning opportunities. With the right configuration and a bit of proactive testing, your clusters can scale predictably and handle high load without surprises.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f9b027aed268" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tuning an Azure Stream Analytics Pipeline: A Step-by-Step Guide]]></title>
            <link>https://jaadds.medium.com/tuning-an-azure-stream-analytics-pipeline-a-step-by-step-guide-5dc48b9ca2cb?source=rss-68142244b10b------2</link>
            <guid isPermaLink="false">https://medium.com/p/5dc48b9ca2cb</guid>
            <category><![CDATA[azure-stream-analytics]]></category>
            <category><![CDATA[azure-event-hub]]></category>
            <category><![CDATA[stream-processing]]></category>
            <dc:creator><![CDATA[Amila De Silva]]></dc:creator>
            <pubDate>Thu, 17 Jul 2025 01:55:45 GMT</pubDate>
            <atom:updated>2025-08-21T14:59:15.402Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RgZYtIID2JUTvzzBjPnOLA.png" /></figure><h3>Introduction</h3><p>In our <a href="https://medium.com/p/bc33cdc6c2a6">previous post</a>, we explored how to identify bottlenecks in streaming analytics jobs and strategies to scale them effectively.</p><p>In this post, we’re going to put that theory into practice by scaling a pipeline from handling<strong> 500 events per second to 2,000 events per second.</strong></p><p>We’ll walk step by step through diagnosing bottlenecks, tuning resources, and verifying results using Azure metrics and charts.</p><h3>The setup</h3><p>This experiment uses the exact configuration from<a href="https://github.com/Azure-Samples/streaming-at-scale/tree/main/eventhubs-streamanalytics-eventhubs"> this Azure sample</a>.</p><p>The pipeline has three main components:</p><ul><li><strong>Input Event Hub</strong> — receives raw events</li><li><strong>Stream Analytics Job</strong> — processes the data</li><li><strong>Output Event Hub</strong> — gathers processed results</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jfHDQRejNDGT6kDdCV-QFw.png" /><figcaption>Figure: Event Processing pipeline from the client to the Output Event Hub</figcaption></figure><h4>The Client</h4><p>We use a client that mimics IoT devices by sending temperature and CO2 readings. It runs as an Azure Container Instance.</p><p>By adjusting parameters like the number of unique devices and their publishing intervals, we can precisely control the event rate.</p><p>Here’s a sample event JSON:</p><pre>{<br>    &quot;eventId&quot;: &quot;b81d241f-5187-40b0-ab2a-940faf9757c0&quot;,<br>    &quot;complexData&quot;: {<br>        &quot;moreData0&quot;: 57.739726013343247,<br>        &quot;moreData1&quot;: 52.230732688620829,<br>        &quot;moreData2&quot;: 57.497518587807189,<br>        &quot;moreData3&quot;: 81.32211656749469,<br>        &quot;moreData4&quot;: 54.412361539409427,<br>        &quot;moreData5&quot;: 75.36416309399911,<br>        &quot;moreData6&quot;: 71.53407865773488,<br>        &quot;moreData7&quot;: 45.34076957651598,<br>        &quot;moreData8&quot;: 51.3068118685458,<br>        &quot;moreData9&quot;: 44.44672606436184,<br>        [...]<br>    },<br>    &quot;value&quot;: 49.02278128887753,<br>    &quot;deviceId&quot;: &quot;contoso-device-id-000154&quot;,<br>    &quot;deviceSequenceNumber&quot;: 0,<br>    &quot;type&quot;: &quot;CO2&quot;,<br>    &quot;createdAt&quot;: &quot;2019-05-16T17:16:40.000003Z&quot;<br>}</pre><h4><strong>The Stream Analytics Job</strong></h4><p>This job isn’t just a pass-through — it does non-trivial processing:</p><ul><li>For each event, it finds the minimum value over a 1-minute window.</li><li>Computes the difference between the current value and that minimum.</li><li>If the difference exceeds a threshold, it raises an alert.</li></ul><p>All output events are sent to the output Event Hub. The complete query is available <a href="https://github.com/jaadds/azure-stream-analytics-tuning/blob/main/query.sql">here</a>.</p><h4>Initial Configuration</h4><ul><li>Both Event Hubs have <strong>2 partitions</strong> each.</li><li>We start with <strong>1 Throughput Unit (TU)</strong> for both input and output Event Hubs.</li><li>We also start with <strong>1 Streaming Unit (SU)</strong> for the Stream Analytics Job.</li></ul><h3>The Baseline — 500 events/sec</h3><p>We’ll start by testing our pipeline at <strong>500 events per second</strong> to see how it performs under a moderate load.</p><p>Take a look at the input Event Hub metrics:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L9KlnqKMOACXgkvSD9Z_WQ.png" /><figcaption>Figure: Input Event Hub — Incoming &amp; Outgoing Messages at 500 events/sec</figcaption></figure><p>In this chart, the incoming messages (published by our client) and the outgoing messages (read by the Stream Analytics Job) are both straight, parallel lines. That tells us publishing and consumption are smooth, without any throttling or interruptions.</p><p>Next, let’s look at how the Stream Analytics Job itself handles this rate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dYuTmMv-cRDa5tpImxkHzQ.png" /><figcaption>Figure: Stream Analytics Job — Backlogged Input Events at 500 events/sec</figcaption></figure><p>Apart from a small spike right at the start — likely from the job warming up — the backlog stays flat. That means the job is processing events as fast as it receives them, without any accumulation or delay.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-Rlj_RC8CAZBTBZmVE7Luw.png" /><figcaption>Figure: Stream Analytics Job — Watermark Delay at 500 events/sec</figcaption></figure><p>Checking the watermark delay confirms this: there’s no sign of the job falling behind over time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3ca2n4XR42Ge1LpT8WJyUg.png" /><figcaption>Figure: Streaming Analytics Job — CPU Utilization at 500 events/sec</figcaption></figure><p>Finally, CPU usage hovers comfortably around 40–50%, indicating the job has plenty of spare capacity.</p><p>Overall, this configuration handles <strong>500 events per second</strong> easily, with no backlog growth, no spikes in delay, and CPU usage well under control.</p><h3>Increase inputs to 2k events</h3><p>Now let’s see what happens when we increase the input rate to <strong>2,000 events per second</strong>, but keeping the same initial configuration.</p><p>Right away, the effects are clear.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0ibsLQOUva7HTVbxbG55zw.png" /><figcaption>Figure: Backlog Growth with 2K Events/sec</figcaption></figure><p>Over just 30 minutes, the backlog grows steadily from 100k to 500k events. The job simply can’t keep up with the incoming load, and events are piling up.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*unyB1cXZLlkMr9Ay0xEEbQ.png" /><figcaption>Figure: Watermark Delay with 2K Events/sec</figcaption></figure><p>Watermark delay shows the same problem, reaching a peak of 23 minutes. The job is falling further and further behind over time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nFJ3DTkTCtKq9jc4aUbH9g.png" /><figcaption>Figure: CPU Usage at 2K Events/sec</figcaption></figure><p>The CPU doesn’t stay at 100% constantly, but it’s highly unstable, with sharp peaks up to 90%. This kind of spiky usage suggests the job is overloaded and struggling to keep pace.</p><p>But it’s not just the Stream Analytics Job that’s in trouble. Let’s take a look at the Input Event Hub.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1ipzzaXJzkT4MZvZY7mscg.png" /><figcaption>Figure: Incoming &amp; Outgoing Messages — Event Hub Under Stress</figcaption></figure><p>Here, the once-straight message lines have turned into erratic waves. Incoming and outgoing rates fluctuate wildly, sometimes dropping to zero before shooting back up. Plus, we see a high number of throttled-out requests. This is classic evidence of an Event Hub under pressure — it can’t ingest or deliver messages reliably at this rate.</p><p>Given that a single TU can only handle around 1,000 messages per second, trying to force 2,000 through it was never going to work. This is our first clear sign that the pipeline is under stress and needs more resources.</p><p>In the next sections, we’ll walk through systematically removing these bottlenecks one at a time — so our pipeline can reliably handle 2,000 events per second just as smoothly as it did 500.</p><h3>Clearing the incoming path</h3><p>When the Event Hub throttles, fewer events actually make it through to the Stream Analytics Job. Therefore our first priority is to to fix that bottleneck, and let events flow freely from the client to the Stream Analytics Job.</p><p>So, let’s see what happens when we increase the TUs from 1 to 2.</p><h4>Assigning 2 TUs to the Input Event Hub</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WTXJiHSsmq016HahRwNQ7Q.png" /><figcaption>Figure: Incoming and Outgoint Messages with 2 TUs</figcaption></figure><p>After increasing to 2 TUs, there’s clear improvement. The Incoming Messages line is now linear, without the spikes and dips we saw before. That tells us ingestion throttling is gone — the client can publish events freely, and they flow smoothly into the Event Hub.</p><p>However, the Outgoing Messages line still shows fluctuations. That suggests throttling is now happening as the Stream Analytics Job reads from the Event Hub.</p><p>It’s worth remembering that with 2 TUs allocated, the entire Event Hub can support up to 8,192 egress events per second. That’s way higher than the actual output events.</p><p>So why do we still see those fluctuations?</p><p>To answer that, let’s check the bandwidth.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VfqdvZIjK5FlBp-bcy-ngQ.png" /><figcaption>Figure: Incoming and Outgoing Bytes with 2 TUs</figcaption></figure><p>Looking at the chart, we can see Outgoing Bytes consistently hover near 200 MB per minute and occasionally spike past 240 MB, which is the maximum bandwidth 2 TUs can support.</p><p>This suggests that even with 2 TUs, we’re hitting the bandwidth ceiling. That’s why throttling still happens intermittently.</p><p>Clearly, we need even more capacity. So let’s increase to <strong>3 TUs</strong> and see what happens.</p><h4>From 2TUs to 3 — More power to the Input Event Hub</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wcn2Mi_wrHTo-3lnuHZFSQ.png" /><figcaption>Figure: Incoming and Outgoing Messages with 3 TUs</figcaption></figure><p>After increasing to 3 TUs, the difference is obvious. We no longer see Throttled Requests. The Incoming Message line remains smooth, and the Outgoing Message line is much more stable, though still with some fluctuation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gN4-1LhV0NdK1PqgiZOySA.png" /><figcaption>Figure: Incoming/Outgoing Bytes with 3 TUs</figcaption></figure><p>Both incoming and outgoing bytes are now within acceptable limits. We’ve essentially removed the bottleneck at Input Event Hub .</p><p>However, there’s still one thing to notice: even though the Event Hub is no longer throttling, the output lines still fluctuate a bit.</p><p>That suggests the Stream Analytics Job itself may now be struggling to consume messages uniformly.</p><p>Since we’ve fixed the Event Hub side, let’s turn our attention to the Stream Analytics Job and see what’s going on there.</p><h3>Increasing Processing Power</h3><p>Let’s take a closer look at how the Stream Analytics Job performs when it’s still running with only 1 Streaming Unit (SU).</p><h4>Struggling with Just One Streaming Unit</h4><p>To recap, let me re-state the configuration of the pipeline:</p><ul><li>3 TUs for the Input Event Hub</li><li>1 SU for the Stream Analytics Job</li><li>1 TU for the Output Event Hub</li></ul><p>Here’s what happens to the backlog and watermark delay with this configuration;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L36QkpaQI8VFZEt90pCCHA.png" /><figcaption>Figure: Backlog Growth with 1 SU</figcaption></figure><p>You can see that the backlog grows steadily over time. This indicates the job can’t process incoming events as quickly as they’re arriving.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J4phBr2DNiObG3VvHv7HZA.png" /><figcaption>Figure: Watermark Delay Increasing with 1 SU</figcaption></figure><p>The watermark delay rises significantly, showing the system is falling further behind as it struggles to process events in real-time.</p><p>Why is this happening? The CPU chart gives us a clue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oSi6Qjy_bYMpR1V94Hs-ng.png" /><figcaption>Figure: CPU Pegged at 100% with 1 SU</figcaption></figure><p>CPU utilization is pinned near 100% the whole time. Unlike when the job was handling 500 events/sec with CPU at a comfortable 45%, now it’s completely saturated.</p><p>That CPU saturation explains the growing backlog and the increasing watermark delay. The job simply doesn’t have enough resources to process 2,000 events per second efficiently.</p><p>Clearly, the next step is to give the job more muscle by increasing its Streaming Units.</p><h4>More power to Process</h4><p>So, let’s double the job’s capacity by increasing the SUs from 1 to 2. How does that change things?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aDQAt9EA2cDbflJo1nY8bg.png" /><figcaption>Figure: Backlog Behavior with 2 SUs</figcaption></figure><p>After this change, the backlog no longer grows out of control. Instead, it remains largely stable with only occasional, manageable spikes peaking around 5,000 events.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sBzBqUbubNLYxOWmzynWiA.png" /><figcaption>Figure: CPU Utilization with 2 SUs</figcaption></figure><p>The average CPU usage also drops, indicating the job now has enough headroom to handle the load without being constantly maxed out.</p><p>But it’s not just the job’s internal stats that improve. Let’s also check the Event Hub message flow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H3APlRUm_yEGHVW7VJHvTQ.png" /><figcaption>Figure: Incoming and Outgoing Messages from Input Event Hub</figcaption></figure><p>Here we see that both Incoming and Outgoing Message lines are smooth and stable. The fluctuations we observed earlier with Outgoing Message line has completely disappeared.</p><p>All of this suggests that with 2 Streaming Units, the Stream Analytics Job can finally keep up with 2,000 events per second without falling behind or overloading.</p><p>With this adjustment, the pipeline is now performing significantly better, and we’re much closer to our target of seamless high-throughput processing.</p><h3>Improving Outgoing Message flow</h3><p>Now that our Stream Analytics Job is consuming events smoothly, it’s time to check the final stage of the pipeline: writing to the Output Event Hub.</p><p>We need to make sure processed events are sent out at a steady, reliable rate without hitting any new bottlenecks.</p><h4>Analyzing the Output Event Hub</h4><p>First, let’s take a look at the message flow in the outgoing Event Hub with <strong>1 TU</strong> allocated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bEUAKlrO70u2Xe4oXqociQ.png" /><figcaption>Figure: Outgoing Messages in Output Event Hub with 1 TU</figcaption></figure><p>Here, you can see the Incoming Messages occasionally exceed the capacity limit for 1 TU — about 60,000 messages per minute. These spikes are a clear sign that we’re pushing the Event Hub right up to (and sometimes beyond) what it can comfortably handle.</p><p>But message count is only part of the picture. What about the actual bandwidth being consumed?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*StOZ8eeJCAVOrnwyb5K9dA.png" /><figcaption>Figure: Outgoing Event Hub Ingress Bandwidth</figcaption></figure><p>Each TU offers 1 MB/s of ingress bandwidth, or roughly 60MB per minute.</p><p>In this chart, the incoming bandwidth hovers right at that limit, and at times even exceeds it. This is strong evidence that the Stream Analytics Job is trying to push data faster than the Event Hub can accept, which inevitably leads to throttling.</p><h4><strong>Tuning the Outgoing Event Hub</strong></h4><p>When we were tuning the Input Event Hub, we had an advantage:</p><ul><li>We controlled the client’s publishing rate.</li><li>We knew exactly how many events per second would arrive.</li><li>We could plan TU allocation based on those numbers.</li></ul><p>But here, it’s different. The Stream Analytics Job itself is the publisher.</p><p>It decides:</p><ul><li>How many events to send.</li><li>How large each message batch is.</li></ul><p>It can dynamically change these to maximize channel utilization. That makes its output rate less predictable.</p><p>As a result, we can’t perfectly calculate the required TUs ahead of time. Instead, we have to watch for throttling errors and increase TUs until those errors disappear.</p><h4>The Problem with Partitions</h4><p>So increasing 2 TUs should increase the capacity right?</p><p>But there’s a catch.</p><p>This <strong>Stream Analytics Job is non-parallel</strong>. Therefore all the partitions in the output Event Hub aren’t utilized uniformly. Check for output stages from different experiments to understand what I’m saying;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jdgWJj0ASWZp6K0rR365iA.png" /><figcaption>Figure: Output Events from Streaming Job with 1SU — 500 events/sec</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZLAbXbo5mAX0dDdu5_VWUA.png" /><figcaption>Figure: Output Events from Streaming Job with 1SU — 2000 events/sec</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*03R7qaRq_scgyZQcRwn0qA.png" /><figcaption>Figure: Output Events from Streaming Job with 2SUs — 2000 events/sec</figcaption></figure><p>Regardless of how many partitions the Event Hub has, the job writes everything to just one partition — partition 0.</p><p>Since the Event Hub has two partitions, when we add more TUs overall, they’ll get split across partitions. To truly double the capacity for that single active partition, we need enough TUs allocated so that partition 0 alone gets 2 TUs.</p><p>Since there are 2 partitions, the simplest way is to increase total TUs to at least 4. This ensures partition 0 can get 2 TUs all to itself.</p><h4>4 TUs for the Output Event Hub</h4><p>So, let’s see what happens after increasing the Output Event Hub to 4 TUs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mq7gwDzr_YKEBiQYg-cdcw.png" /><figcaption>Figure: Incoming &amp; Outgoing Messages from Outgoing Event Hub with 4 TUs</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RWgOt71_L_pEXLpUmw9FyQ.png" /><figcaption>Figure: Incoming &amp; Outgoing Bytes from Outgoing Event Hub with 4 TUs</figcaption></figure><p>The difference is clear. Throttling errors disappear, and the Incoming Messages forms an almost perfect straight line.</p><p>If we check the output events from the Streaming Analytics job, we can see a perfect line there too.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NID5qjskY0mhuOvMXKWp0w.png" /><figcaption>Output Events from Streaming Job — with 4TUs assigned to Outgoing Event Hub</figcaption></figure><p>No more oscillations. No more drops to zero. Just smooth, consistent throughput.</p><h3>Final Observations</h3><p>After all our tuning, here’s the final configuration that reliably supports 2,000 events per second end-to-end:</p><ul><li><strong>Input Event Hub:</strong> 3 TUs</li><li><strong>Stream Analytics Job:</strong> 2 SUs</li><li><strong>Output Event Hub:</strong> 4 TUs</li></ul><p>With these settings in place, the pipeline handles 2,000 events/sec just as comfortably as it did 500.</p><p>No backlogs building up over time.</p><p>No spikes in watermark delay.</p><p>No sustained CPU overload.</p><p>But don’t just take my word for it. Let’s look at the <strong>metrics</strong> and see how similar everything looks compared to our original 500 events/sec baseline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gh2_D-9QtTIQTCp5jL5qkw.png" /><figcaption>Figure: Incoming and Outgoing Messages from Input Event Hub</figcaption></figure><p>Here you can see both incoming and outgoing message lines are smooth and parallel. There’s no sign of throttling. This indicates that ingestion is flowing perfectly, even at the higher rate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xXjRlxXSOzj8l59e3N3pFQ.png" /><figcaption>Figure: Backlog Over Time with Final Settings</figcaption></figure><p>Apart from a small spike at the very start (which is typical when the job first warms up), there’s <strong>no sustained growth in backlog</strong>. The job is processing events just as quickly as it receives them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CFfjGcHK2LRH4KC2Pstxqg.png" /><figcaption>Figure: Watermark Delay with Final Settings</figcaption></figure><p>The watermark delay remains low and stable throughout, showing the job isn’t falling behind.</p><h3>Wrap-up</h3><p>In this example, the goal wasn’t just to show raw performance numbers, but to walk you through the mindset and process of systematically tuning an event processing pipeline.</p><p>By carefully identifying and removing bottlenecks at every stage — from ingestion, to processing, to output — we turned a pipeline that was falling behind at 2,000 events/sec into one that handled that load just as reliably as it did 500.</p><p>And while our demo job was intentionally non-parallel for teaching purposes, these same lessons on TU &amp; SU allocation, bottleneck detection, and methodical tuning apply directly to real-world streaming workloads.</p><p>If you found this guide helpful, I’d love to hear from you. Feel free to leave a comment, share it with your team, or let me know what other streaming topics you’d like to see explored next.</p><p>Thanks for reading, and happy tuning!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5dc48b9ca2cb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scaling Stream Analytics Jobs: Identifying and Fixing Bottlenecks]]></title>
            <link>https://jaadds.medium.com/scaling-stream-analytics-jobs-identifying-and-fixing-bottlenecks-bc33cdc6c2a6?source=rss-68142244b10b------2</link>
            <guid isPermaLink="false">https://medium.com/p/bc33cdc6c2a6</guid>
            <category><![CDATA[stream-processing]]></category>
            <category><![CDATA[azure-event-hub]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[azure-stream-analytics]]></category>
            <dc:creator><![CDATA[Amila De Silva]]></dc:creator>
            <pubDate>Sun, 11 May 2025 04:58:06 GMT</pubDate>
            <atom:updated>2025-05-11T04:58:06.059Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BKWQ6NfWNy5RmhLFJplbwg.png" /></figure><h3>Introduction</h3><p>If you’ve used Azure Stream Analytics to build event processing pipelines, chances are you’ve found yourself randomly increasing Streaming Units in an attempt to scale up your job.</p><p>You may have also encountered situations where your Stream Analytics job had plenty of idle resources, yet still accumulated a significant backlog. In this article, we’ll explore how to systematically identify bottlenecks in your Stream Analytics jobs and eliminate them effectively.</p><h3><strong>Thinking in Terms of Pipelines</strong></h3><p>While our main focus is on scaling the <strong>Stream Analytics job</strong>, it’s important to understand that what we’re really aiming for is to scale the <strong>larger pipeline</strong> it belongs to. A Stream Analytics job typically reads from an input, performs some processing, and writes to an output. So, when you’re trying to scale the pipeline end-to-end, you also need to clear any bottlenecks that occur at <strong>both the input and the output </strong>to ensure events can flow through smoothly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FWxQsm0UJwyfxt3_SSskww.png" /><figcaption>A multi-staged pipeline with multiple Stream Analytics Jobs doing distinct processing</figcaption></figure><p>That’s why, throughout this article, we won’t just look at the job in isolation — we’ll also examine how it interacts with its inputs and outputs. In the examples provided here, we’re specifically using <strong>Event Hubs</strong> as both the input and output sources. However, the techniques discussed should apply just as well to other input and output types too.</p><p>In the rest of the article, we’ll focus on how to scale this Event Hub, Streaming Analytics Job combination. We’ll explore which metrics from Stream Analytics and Event Hubs can help you pinpoint bottlenecks — and how to use those insights to increase processing capacity smoothly.</p><h3>Understanding the Basics — TUs and SUs</h3><p>Before we dive into the details, let’s agree on a few terms to keep things simple:</p><ul><li>The <strong>Event Hub</strong> that sits closer to the event producer will be referred to as the <strong>Upstream Event Hub</strong>.</li><li>The <strong>Stream Analytics Job</strong> will read from this Upstream Event Hub.</li><li>The <strong>Event Hub</strong> closer to the event sink — often the source for another analytics job — will be referred to as the <strong>Downstream Event Hub</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*klUCZNMWoqXWZ3YEccHiGw.png" /><figcaption>Events flowing from Source to Sink through EventHubs and a Stream Analytics Job</figcaption></figure><p>Now, let’s look at the key capacity units in this setup — <strong>Throughput Units (TUs)</strong> for Event Hubs and <strong>Streaming Units (SUs)</strong> for Stream Analytics — and how they impact performance.</p><h4>EventHubs &amp; Throughput Units</h4><p>Each Event Hub lives inside an <strong>Event Hub namespace</strong>. Think of the Event Hub as a topic and the namespace as the physical host — similar to how a Kafka topic lives on a Kafka broker.</p><p>The capacity of an Event Hub is defined by the number of <strong>Throughput Units (TUs)</strong> assigned to its namespace. Each TU provides:</p><ul><li>Up to 1 MB per second or 1,000 events per second for ingress</li><li>Up to 2 MB per second or 4,096 events per second for egress</li></ul><p>You can find more details in the official documentation<a href="https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-scalability#throughput-units"> here</a>.</p><p>If your workload is high throughput, it’s common practice to create a <strong>dedicated namespace per Event Hub</strong> to isolate and scale them independently. That’s what we assume in the examples throughout this article.</p><p>TUs control how much data you can push into and pull out of Event Hubs. If your producers or consumers exceed the TU limit, Event Hub will <strong>throttle</strong> the connections — leading to delays and potential data loss if not handled properly.</p><h4>Streaming Analytics Job &amp; Streaming Units</h4><p><strong>Streaming Units (SUs)</strong> define how much processing power your Stream Analytics job has. Each SU provides a fixed amount of CPU, memory, and network bandwidth — but Microsoft doesn’t publish the exact specs.</p><p>Jobs vary. Some are <strong>CPU-intensive</strong>, while others are more <strong>I/O-bound</strong>. Depending on the nature of your workload, you’ll need to allocate a suitable number of <strong>Streaming Units (SUs)</strong> to ensure the job can keep up with the incoming events.</p><p>There’s also the concept of a <strong>Streaming Node</strong>, which you can think of as a virtual machine where the job actually runs. The relationship between SUs and Streaming Nodes depends on the <strong>version</strong> of SUs you’re using. This<a href="https://learn.microsoft.com/en-us/azure/stream-analytics/stream-analytics-streaming-unit-consumption"> documentation</a> explains the details.</p><p>But for simplicity, in this article, we’ll assume adding a single Streaming Unit is equivalent to adding Streaming Node<strong> </strong>— and use those terms interchangeably.</p><h3>How much Processing Power do I need?</h3><p>Whenever you hit a bottleneck in your pipeline, there are a few changes you can do: you can increase the <strong>Throughput Units (TUs)</strong> for the upstream and downstream Event Hubs, and the <strong>Streaming Units (SUs)</strong> for the Stream Analytics job. These are the most straightforward adjustments you can make, without modifying your job’s query to improve parallelism.</p><p>However, keep in mind that<strong> increasing SUs doesn’t always result in a noticeable performance boost</strong> — especially if you’re only adding one or two units. Depending on how your job is structured, you might need a larger jump in SUs to see a meaningful increase in throughput. That’s because SUs determine how your job’s processing elements are distributed across Streaming Nodes — and more than just a couple of SUs may be needed to achieve an efficient allocation.</p><p>This is where the <strong>Job Diagram view</strong> and <strong>Job Simulation view</strong> come in handy. These tools can help you understand how your job is utilizing resources and how well it’s scaling.</p><h4>Job Diagram view</h4><p>You’ll find this under the <strong>Monitoring</strong> section once your job is running. It offers two perspectives:</p><ul><li><strong>Logical View</strong> — This shows how the various steps in your query are logically arranged. Unless you change the query itself, this layout stays the same. You can click on each step to understand what part of the query it represents. It’s great for visualizing how data flows through your job and estimating event consumption and output rates. <a href="https://learn.microsoft.com/en-us/azure/stream-analytics/stream-analytics-job-logical-diagram-with-metrics">This article</a> provides a good overview of the Logical View.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/899/1*b4scTjU0VBWCNqRrLjHL-Q.png" /><figcaption>Sample Job Diagram — Logical View</figcaption></figure><ul><li><strong>Physical View</strong> — This shows how your job is distributed across Streaming Nodes and how resources are consumed. A time slider at the top lets you select specific moments to view resource usage at that point in time. This can give you insight into whether certain nodes are overworked while others are sitting idle. You can find more details about the Physical View <a href="https://learn.microsoft.com/en-us/azure/stream-analytics/stream-analytics-job-physical-diagram-with-metrics#identify-the-cause-of-overloaded-cpu-or-memory">here</a>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/899/1*PjXZwJJTZirQXh0ACoi8VA.png" /><figcaption>Physical Job Diagram view for the same Job</figcaption></figure><h4>Job Simulation view</h4><p>Located in the <strong>Query Editor</strong>, the Job Simulation view is more of a development-time tool. It combines elements of both the logical and physical views — but without showing real-time resource usage.</p><p>In this view, you can experiment with different SU allocations and see how processing elements are distributed across Streaming Nodes. It even shows lower-level details like marshaling and unmarshaling logic — things not visible in the Logical View.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5uIPq3kuTDo2xmBb6HvPPQ.png" /><figcaption>Job Simulation View — with 2 SUs assigned</figcaption></figure><p>This view is especially helpful when you’re trying to break through a bottleneck. You can simulate how the job would be distributed under different SU counts and pick a configuration where processing elements are more <strong>evenly spread out across nodes</strong>. Then, apply that SU count to your job to improve throughput. For a detailed breakdown of how to use this view effectively, refer to the <a href="https://learn.microsoft.com/en-us/azure/stream-analytics/optimize-query-using-job-diagram-simulator">official documentation</a>.</p><p>Next, we’ll look at how to use metrics to identify bottlenecks — and how these views can guide you in scaling the pipeline more effectively.</p><h3>Tips to identify bottlenecks</h3><p>While checking CPU and memory usage is a good starting point for spotting bottlenecks in a Stream Analytics job, it often doesn’t tell the full story. In many cases, these metrics alone won’t be conclusive enough to confirm whether there’s an actual performance issue. That’s why it’s often more reliable to look at <strong>event-specific metrics</strong> like <strong>backlogged input events</strong> and <strong>watermark delay</strong> — both designed to reflect processing slowdowns more directly.</p><h4>Check Backlogged input events from the Streaming Analytics Job</h4><p>It’s normal to see a bit of backlog while the Job runs. But if the number of backlogged input events keeps growing, it’s a strong signal that the job isn’t processing incoming events fast enough.</p><p>A growing backlog often points to slow processing. So let’s break down a few specific causes:</p><p><strong><em>Common Causes of a Growing Backlog</em></strong></p><p><em>Not Enough Streaming Units (SUs)</em></p><p>If the Stream Analytics job doesn’t have enough SUs allocated, it simply can’t keep up with the incoming event rate.</p><ul><li>This is especially true if you’re using <strong>fractional SU allocations</strong> (like 1/3 or 2/3) under StandardV2 pricing. These hit their limits faster as traffic scales.</li><li>If you see the job’s <strong>CPU usage consistently nearing 90%</strong>, it’s a strong indication that more SUs are needed.</li></ul><p><em>Upstream Event Hub Is Throttling</em></p><p>Sometimes the problem isn’t in the job itself, but in the <strong>upstream Event Hub</strong>. If the job tries to pull more events than the Event Hub can deliver, connections may be throttled.</p><ul><li>This happens when the Event Hub <strong>doesn’t have enough Throughput Units (TUs)</strong> to serve the required data rate.</li><li>You can confirm this by checking Event Hub metrics for <strong>Throttled Requests</strong>, which will spike in this scenario.</li></ul><p>We’ll look more into this in the next section.</p><p><em>Skewed Partition Distribution</em></p><p>If events aren’t evenly distributed across partitions, some partitions may backlog even if others are underutilized.</p><ul><li>One solution is to <strong>increase the number of partitions</strong>, allowing better parallelism and more balanced workloads.</li><li>Alternatively, <strong>changing the partition key</strong> can sometimes yield a more even distribution.</li></ul><p><strong><em>When the Cause Isn’t Obvious: Digging Deeper</em></strong></p><p>Sometimes, even when the backlog is growing, the root cause isn’t immediately obvious — especially if you’re relying solely on default CPU and memory metrics. These views typically show average utilization, which can be misleading when multiple Streaming Units (SUs) are involved. Idle or lightly loaded nodes can skew the average, masking overloaded ones.</p><p>In such cases, switching to <strong>Max aggregation</strong> can surface hidden hotspots.</p><p>In the example shown below, the job was running with two Streaming Units and was experiencing a growing backlog. At first glance, average CPU utilization hovered around 56%, which didn’t raise any red flags. But after switching to Max CPU, it became clear that one of the Streaming Nodes was spiking to 95% — a clear sign of a resource bottleneck.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hkcT6wIgM5rIf__MdL0IYg.png" /><figcaption>Changing Average Utilization to Max reveals hidden Hotspots</figcaption></figure><p>To further investigate, we turned to the <strong>Job Diagram</strong>, which provides visibility into how Streaming Nodes are allocated and how each one is performing.</p><p>Since the <strong>Max CPU</strong> view revealed uneven load, we examined the diagram to see which node was under strain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yAib6MHGEEtCgPIYclPuew.png" /><figcaption>Job Diagram shows CPU Hotspot on StreamingNode1 — though Average CPU Utilisation is 56%</figcaption></figure><p>This view confirmed that <strong>StreamingNode1</strong>, which also responsible for input ingestion, was consistently hitting <strong>91% CPU</strong> — which was causing the bottleneck.</p><h4>Observe Watermark delay</h4><p>Watermark delay is another key metric that can help you detect bottlenecks in your streaming pipeline.</p><p>Put simply, Watermark delay measures the <strong>time difference between when an event is processed</strong> (i.e., the current wall clock time at the processing node) and <strong>when that event originally occurred</strong>.</p><p>The event’s “occurred time” can be either:</p><ul><li><strong>Application-defined</strong> — explicitly set within the event payload, or</li><li><strong>Inferred</strong> — based on the event’s arrival time at the upstream Event Hub</li></ul><p>Since streams are unbounded and don’t have an end (unlike arrays or files), you can’t define a metric like completion percentage. Instead, metrics like Watermark delay give us a sense of how “fresh” the events being processed are.</p><ul><li><strong>Low Watermark delay</strong> means the job is nearly in sync with the incoming stream — it’s processing events shortly after they arrive.</li><li><strong>High or increasing Watermark delay</strong> suggests the job is falling behind — events are arriving faster than they’re being consumed, which typically means input is <strong>piling up</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y-I0NPaoDOKCN0UTDwAsBA.png" /><figcaption>Watermark Delay &amp; Backlog — The further left the read pointer is, the lower the delay tends to be</figcaption></figure><p>As Watermark delay increases, you’ll often see <strong>Backlogged Input Events</strong> increasing too. In other words, both metrics typically rise together and point to the same underlying issue: <strong>processing slowness</strong>.</p><p>This<a href="https://learn.microsoft.com/en-us/azure/stream-analytics/stream-analytics-time-handling"> documentation</a> provides a solid explanation of Watermark delay and how it fits with the other set of time concepts used in Azure Stream Analytics.</p><p>Also, Microsoft has a<a href="https://learn.microsoft.com/en-us/azure/stream-analytics/stream-analytics-job-analysis-with-metric-dimensions"> great guide here</a> explaining possible causes for increased Watermark delay and what you can do to address them.</p><p>In short, if you notice Watermark delay rising, consider revisiting the same areas you would investigate for growing input backlogs: insufficient SUs, Event Hub throttling, or partition skew.</p><h4>Keep an eye on Throttled Requests</h4><p>As you send more events through your processing pipeline, it’s important to monitor the <strong>Throttled Requests</strong> metric on both the <strong>Upstream</strong> and <strong>Downstream Event Hubs</strong>. When an Event Hub reaches its capacity limits — either by hitting its maximum number of requests or its bandwidth threshold — it starts rejecting additional connections, which is captured in this metric.</p><p>To provide a reference point, here’s a snapshot from an <strong>upstream Event Hub</strong> operating in a <strong>steady state</strong>, where throughput is stable and no throttling is observed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s-vfzy0QgZAwoiTu8INc2A.png" /><figcaption>Metrics view when the Event Hub was operating at steady state</figcaption></figure><p>Now, let’s look at two common scenarios where throttling can occur:</p><p><strong><em>Upstream Clients Pushing Events Too Fast</em></strong></p><p>This happens when an <strong>event producer</strong> (such as a sensor) tries to send events at a rate higher than the Event Hub can handle.</p><ul><li>In this case, the <strong>Incoming Messages</strong> metric often becomes unstable and shows a <strong>fluctuating or sawtooth-like pattern</strong>, in contrast to the flat, steady line seen during normal operation.</li><li>The <strong>Outgoing Messages</strong> metric will usually mirror this fluctuation.</li></ul><p>However, it’s important to note: this type of throttling <strong>only impacts the producer</strong> — it doesn’t cause any direct delays in the downstream <strong>Stream Analytics job</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qSIzb1-tI2GZgANqMXPj5Q.png" /><figcaption>Incoming Messages showing a fluctuating line</figcaption></figure><p><strong><em>Stream Analytics Job (or Other Consumers) Consuming Too Aggressively</em></strong></p><p>Throttling can also occur on the <strong>consumption side</strong>, when the Stream Analytics job — or any other downstream consumer — tries to read more events than the Event Hub can serve.</p><p>If your <strong>incoming stream is steady</strong> (as shown by a linear <strong>Incoming Messages</strong> line), throttling will appear in the form of <strong>irregular line in the Outgoing Messages</strong> metric.</p><p>When this happens:</p><ul><li>The Stream Analytics job can’t read fast enough.</li><li>Unconsumed events build up.</li><li>This leads to an increase in <strong>Backlogged Input Events</strong> on the Stream Analytics job.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FFrEpyzdatVFoOoAr-vOAw.png" /><figcaption>Throttling due to excessive consumption. Outgoing messages fluctuates, Incoming messages remain stable</figcaption></figure><p>In either case — whether it’s due to publishing or consumption — <strong>adding more Throughput Units (TUs)</strong> to the Event Hub is the way to resolve throttling and restore smooth flow through your pipeline.</p><p>Alternatively, if throttling occurs due to aggressive publishing, you can reduce the number of requests by <strong>batching multiple events</strong> before sending. Most Event Hub SDKs support this batching mechanism. While batching reduces the number of messages (and request overhead), it will increase the bandwidth consumption, so keep an eye on the <strong>Incoming Bytes</strong> metric when applying this strategy.</p><h3>Conclusion</h3><p>Scaling a Stream Analytics job isn’t just about assigning more Streaming Units — it’s about understanding how your job fits into the larger pipeline and addressing bottlenecks across inputs, processing, and outputs. By using metrics like <strong>Backlogged Input Events</strong>, <strong>Watermark Delay</strong>, and <strong>Throttled Requests</strong>, and by making use of tools like the <strong>Job Diagram</strong> and <strong>Job Simulation view</strong>, you can take a more systematic approach to scaling.</p><p>Since each job behaves differently, it’s important to monitor how your job responds to changing loads and allocate capacity units accordingly. With a clearer understanding of where bottlenecks can arise — and how to spot them early — you’ll be better equipped to build streaming pipelines that scale smoothly.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bc33cdc6c2a6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Gracefully Terminating Pods in Kubernetes — Part 2: PreStop Hooks]]></title>
            <link>https://jaadds.medium.com/gracefully-terminating-pods-in-kubernetes-part-2-prestop-hooks-41fbd3a5318e?source=rss-68142244b10b------2</link>
            <guid isPermaLink="false">https://medium.com/p/41fbd3a5318e</guid>
            <category><![CDATA[pods]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[resilience]]></category>
            <category><![CDATA[lifecycle-hook]]></category>
            <dc:creator><![CDATA[Amila De Silva]]></dc:creator>
            <pubDate>Fri, 28 Feb 2025 05:23:28 GMT</pubDate>
            <atom:updated>2025-02-28T05:23:28.985Z</atom:updated>
            <content:encoded><![CDATA[<h3>Gracefully Terminating Pods in Kubernetes — Part 2: PreStop Hooks</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*V_rq5mCjLXYD1_bFLS7W3g.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@alexpresa?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Alex Presa</a> on <a href="https://unsplash.com/photos/a-cruise-ship-in-a-body-of-water-NwS_Pv_LztY?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a></figcaption></figure><p>In our <a href="https://medium.com/p/fb0d60c7e983">previous article</a>, we explored how to gracefully terminate pods by handling SIGTERM. In this post, we’ll look at another approach: handling terminations gracefully using a <strong>preStop handler</strong>.</p><h3>What is a PreStop Handler?</h3><p>Kubernetes provides a hook to execute custom logic as a container within the pod shuts down.</p><p>When a preStop hook handler is defined, it is executed first as soon as the pod is marked for termination. Any tasks required for gracefully shutting down the particular container can be executed as part of this handler. For a detailed breakdown of the steps that run during pod termination, refer to <a href="https://jaadds.medium.com/gracefully-terminating-pods-in-kubernetes-handling-sigterm-fb0d60c7e983#c1d8">How Kubernetes Terminates Pods</a>.</p><p>A preStop<strong> </strong>handler can be defined for each container in a pod. The typical format of the<strong> </strong>handler is as follows:</p><pre>lifecycle:<br>  preStop:<br>    exec:<br>      command: [&quot;/bin/sh&quot;,&quot;-c&quot;,&quot;nginx -s quit; while killall -0 nginx; do sleep 1; done&quot;]</pre><p>A preStop<strong> </strong>hook supports multiple implementations:</p><ul><li><strong>exec</strong> — Executes a specific command inside the container.</li><li><strong>http</strong> — Calls an HTTP endpoint within the container.</li><li><strong>sleep</strong> — Pauses the container for a specified duration.</li></ul><p>While you can simulate the behavior of sleep through exec (by running the sleep command), using sleep as a native handler saves you the trouble of including the sleep binary in your container image.</p><p>For more details on implementing a preStop handler, check the official documentation on <a href="https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#hook-handler-implementations">Kubernetes Container Lifecycle Hooks</a>.</p><h3><strong>Beyond Inducing Delays</strong></h3><p>While preStop<strong> </strong>handlers are widely used to introduce delays for draining active connections, they serve broader purposes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-Szzs1aJpvN3kzI-W4edew.gif" /><figcaption>PreStop handler inducing a delay to drain active connections</figcaption></figure><p>Unlike SIGTERM, which is delivered to the main process in the container, the preStop<strong> </strong>handler operates at the container level. This allows it to coordinate processes inside a container or manage interactions across containers within a pod. Here are some common use cases:</p><h4>1. Delaying Termination to Drain Active Connections</h4><p>By introducing a delay, the container can remain running until all active connections are completed. This prevents abrupt request failures as the container shuts down. Additionally, this gives enough time for new connection requests to be routed to the new pod.</p><p>Example command:</p><pre>lifecycle:<br>  preStop:<br>    exec:<br>      command: [&quot;/bin/sh&quot;, &quot;-c&quot;, &quot;sleep 10&quot;]</pre><p>Check out how <strong>Elasticsearch</strong> implements this method for shutting down containers gracefully: <a href="https://www.elastic.co/guide/en/cloud-on-k8s/2.6/k8s-prestop.html">Elastic’s Guide on PreStop Hooks</a>.</p><h4>2. Coordinating with Other Pods in StatefulSets or ReplicaSets</h4><p>When the pod is part of a distributed application, it can use the preStop handler to trigger actions that ensure health of the entire application.</p><p>If the terminating pod is a primary replica or leader, or if its presence is crucial for maintaining quorum, the preStop handler can initiate a leader election or a vote to re-establish quorum. These actions help reduce<strong> </strong>potential stability issues during shutdown.</p><p>For example, in the <a href="https://github.com/bitnami/charts/blob/main/bitnami/postgresql-ha/templates/postgresql/hooks-scripts-configmap.yaml">Bitnami PostgreSQL Helm Chart</a>, a preStop handler initiates a leader election call, making sure a new leader seamlessly takes over. Even though the application is capable of electing a leader upon detecting a missing one, preemptive triggers like these minimize instability.</p><h4>3. Initiating a Graceful Shutdown for Processes with Slow Termination</h4><p>For bulky applications like Java servers (e.g., <strong>JBoss</strong> or <strong>Tomcat</strong>), a preStop hook can be used to initiate an early shutdown. When combined with an extended graceful termination period, the preStop handler ensures that the application gets enough time to clean up resources properly, preventing abrupt terminations.</p><p>Check this discussion on <strong>Stack Overflow</strong> for an example:<br><a href="https://stackoverflow.com/questions/52011947/graceful-shutdown-of-tomcat-on-a-kubernetes-cluster">Graceful Shutdown of Tomcat on Kubernetes</a>.</p><h4>4. Handling SIGTERM When Source Code Cannot Be Modified</h4><p>If the container runs a binary for which you don’t have the source code, a preStop<strong> </strong>handler can execute steps to ensure a smooth shutdown.</p><ul><li>If the application provides an alternative way to trigger a proper shutdown, the preStop<strong> </strong>handler can call that.</li><li>Otherwise, it can induce<strong> </strong>a delay to drain requests and minimize the impact of an abrupt shutdown.</li></ul><h3><strong>Hands-On Example: Fixing Abrupt Termination with a PreStop Hook</strong></h3><p>Let’s revisit the scenario from our previous post, where our <a href="https://jaadds.medium.com/gracefully-terminating-pods-in-kubernetes-handling-sigterm-fb0d60c7e983#0abd"><strong>Go application</strong> exited abruptly</a>, causing request failures. This time, instead of modifying the application code, we’ll use a preStop<strong> </strong>hook to introduce a delay.</p><p>Here’s how to define a preStop hook in your pod specification:</p><pre>lifecycle:<br>  preStop:<br>    exec:<br>      command: [&quot;/bin/sh&quot;, &quot;-c&quot;, &quot;sleep 10&quot;]</pre><p>This makes sure that the pod waits for 10 seconds before receiving SIGTERM. The delay allows Kubernetes to redirect traffic to other pods, reducing the likelihood of failed requests.</p><p>Complete Deployment manifest:</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  labels:<br>    app: greeting-app<br>  name: greeting-app<br>  namespace: default<br>spec:<br>  replicas: 2<br>  selector:<br>    matchLabels:<br>      app: greeting-app<br>  strategy:<br>    rollingUpdate:<br>      maxSurge: 25%<br>      maxUnavailable: 25%<br>    type: RollingUpdate<br>  template:<br>    metadata:<br>      labels:<br>        app: greeting-app<br>    spec:<br>      containers:<br>      - image: jaadds/go-sample:1.0.0<br>        imagePullPolicy: IfNotPresent<br>        name: go-sample<br>        lifecycle:<br>          preStop:<br>            exec:<br>              command: [&quot;/bin/sh&quot;, &quot;-c&quot;, &quot;sleep 10&quot;]</pre><p>If you repeat the experiment from the previous post, you’ll see that the requests don’t fail while re-starting the deployment.</p><h3><strong>Do preStop hooks replace </strong><strong>SIGTERM handling?</strong></h3><p>In some cases, gracefully stopping in-flight requests is all that’s needed for graceful application termination. However, if the application holds resources like open database connections, file handles, or storage locks, it’s crucial to close these resources properly before termination. This requires handling SIGTERM.</p><p>Handling SIGTERM addresses certain edge cases in request draining that delays alone cannot fully address. For example, consider a client using keep-alive connections. Even with traffic rerouting, the persistent connection remains pinned to the terminating pod. Unless the application explicitly closes the connection in a SIGTERM handler, it will remain open until the app is forcefully killed by SIGKILL.</p><h3><strong>Final Thoughts</strong></h3><p>In this two-part article, we looked at two ways of handling pod termination gracefully. Even though we talked about them as two approaches, SIGTERM handling and preStop handlers are not mutually exclusive; they are complementary. You can use them to handle cleanup tasks at different levels.</p><p>For example, you can clean up resources held by the application through the SIGTERM handler and use the preStop handler for tasks that don’t involve changing the code — transient tasks like inducing delays. This allows you to manage code-level changes through the SIGTERM handler, while adjusting transient settings like delay times without modifying the application code.</p><p>If you’re interested in other methods to improve the resilience of applications in Kubernetes, you can find more tips in my previous article</p><p><a href="https://jaadds.medium.com/building-resilient-applications-on-kubernetes-9e9e4edb4d33">Building Resilient Applications on Kubernetes</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=41fbd3a5318e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monetizing APIs with WSO2 API Manager]]></title>
            <link>https://jaadds.medium.com/monetizing-apis-with-wso2-api-manager-7af0438230d?source=rss-68142244b10b------2</link>
            <guid isPermaLink="false">https://medium.com/p/7af0438230d</guid>
            <category><![CDATA[stripe]]></category>
            <category><![CDATA[wso2]]></category>
            <category><![CDATA[api-management]]></category>
            <category><![CDATA[monetization]]></category>
            <dc:creator><![CDATA[Amila De Silva]]></dc:creator>
            <pubDate>Mon, 27 Jan 2025 01:08:41 GMT</pubDate>
            <atom:updated>2025-01-27T01:08:41.764Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hjCRjWbWFMG5hLgE5kMGcA.jpeg" /></figure><p>WSO2 API Manager 3.0.0 provides first-class support to monetize APIs. It gives the flexibility to plug in any billing engine, and out-of -the-box the product integrates with Stripe — which is a service that allows individuals and businesses to make and accept payments online. Keeping up with market demands, API Manager 3.0.0 supports two widely popular models to charge consumers (i.e., application developers).</p><ul><li>Charging per subscription</li><li>Charging per request</li></ul><h3>Charging per subscription</h3><p>A fixed amount is charged from the application developer at the end of the billing cycle, where a certain number of calls are allowed per month. Irrespective of whether they use these or not, they will be charged.</p><p>Subscription-based charging can be further broken down into two models. Freemium and tiered pricing.</p><p>In tiered pricing, different request quotas can be sold for different prices. For example, when implementing this model, you can create three tiers: Bronze, Silver, and Gold. They can be priced at USD50, USD100 and USD150, which would offer bundles of 1000, 2000, and 3000 requests per hour.</p><p>When it comes to the freemium model, you offer a free tier with a stringent limit on the request quota and a paid tier that offers a higher quota. The freemium model allows you to introduce a free tier, with 100 requests per hour. Depending on your requirement, you can choose to either block the additional requests or continue the flow when the quota depletes.</p><h3>Charging per request</h3><p>Price is calculated based on the number of requests made. This model is also known as a “pay-as-you-go” model, where there’s no fixed payment. If the pricing is set at USD 0.01 for a single request, and then if 1000 requests are made during the month, a user would be charged USD 10. There is no fee if no requests are made.</p><p>Let’s see how API Manager can be used to implement these models.</p><h3>Using WSO2 API Manager for charging</h3><p>With the 3.0.0 release, WSO2 API Manager supports monetization as a first-class feature.</p><p>This is achieved by leveraging the strengths of external billing engines, such as Stripe, for billing and collecting, while relying on WSO2 API Manager’s capabilities to gather and publish usage data.</p><p>If you are familiar with these features and are excited about trying them out, you can download the latest product from <a href="https://wso2.com/api-management/">here</a> and start following the <a href="https://apim.docs.wso2.com/en/latest/learn/api-monetization/monetizing-an-api/#monetizing-an-api">doc</a>. However, if you find the document too technical, this article aims to provide a more high-level overview, explaining the need for different configurations and how certain features in Stripe are used while implementing billing.</p><p><strong>Integration with Stripe</strong></p><p>WSO2 API Manager uses Stripe to manage interactions between API providers and Consumers. From Stripe’s perspective, WSO2 API Manager operates as a platform. Therefore, as a part of the configuration, an admin for WSO2 API Manager needs to create an account and configure it as the platform account. API providers would also have to create accounts in Stripe and link it with the platform account using Stripe Connect, which gives the ability to the admin to charge consumers on behalf of an API provider.</p><p><strong>How the integration works</strong></p><p>When charging for API usage, two parties are involved: the API provider, who hosts the API and who is responsible for the upkeep and management of the API, and the API consumer, the party that uses the API. API consumers would use an API through an application. In a typical paying scenario, API consumers would pay API providers for using the API. We use features like accounts, customers, and subscriptions available in Stripe to depict this relationship.</p><p>As the first step of configuring, the admin needs to create an account and configure it as the platform account. API providers would also have to create accounts in Stripe and link it with the Platform Account using Stripe Connect, which gives the ability to the admin to charge consumers on behalf of the API provider. The section on <a href="https://apim.docs.wso2.com/en/latest/Learn/APIMonetization/monetizing-an-api/#a-configure-the-billing-engine">Configuring the Billing Engine</a> provides details on this.</p><p>Once you have done the configuration, you can start charging for APIs simply by creating billing plans. You log into the admin portal and create a billing plan, filling in details such as how much you need to charge and what charging model you need to follow. You can also specify the request count and whether to block the invocations once the allotted quota is depleted.</p><p>You can refer to <a href="https://apim.docs.wso2.com/en/latest/Learn/APIMonetization/monetizing-an-api/#step-1-create-a-subscription-policy">Creating a Subscription Polic</a>y for more details on this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/0*cQ9eyYH2vQj3DAQS" /><figcaption>Creating a Tier with a Billing Plan</figcaption></figure><p>While creating billing plans, you get to choose whether you are going to charge for the subscription or charge per request. But, based on how you engaged the billing plans, you can provide a freemium model or the tiered pricing model.</p><p>Then, API providers would log into the publisher portal and select the billing plans they want to enable. While saving the API, they also have to provide an ID that maps the account at Stripe’s end. By this point, you have created an API that can be charged for.</p><p>This is covered in detail under <a href="https://apim.docs.wso2.com/en/latest/Learn/APIMonetization/monetizing-an-api/#step-2-enable-monetization">Enabled Monetization</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SQos0knz4sZSOGsg" /><figcaption>Applying a Paid Tier to the API</figcaption></figure><p>To use these APIs, application developers (API consumers) would have to log into the developer portal and subscribe to those. There are a few additional steps, such as entering payment methods and billing information, but, finally, application developers will be charged based on the plan they subscribe to.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LOIPav58uMweWO7w" /><figcaption>Subscribing to Application with a Paid Tier</figcaption></figure><p>For more details look at <a href="https://apim.docs.wso2.com/en/latest/Learn/APIMonetization/monetizing-an-api/#step-3-subscribe-to-a-monetized-api">Subscribe to a Monetized API</a></p><p>Now that the basic configuration has been done, let’s consider how charging happens.</p><p>As requests flow through the API Manager, stats on those are collected and stored by the Analytics framework in real-time. Long before billing was introduced, WSO2 API manager had the framework to gather usage data and aggregate them according to different requirements. And the new billing feature simply uses the same underlying framework to gather and process usage data. Gathered data is published to Stripe in a pre-defined frequency.</p><p><a href="https://apim.docs.wso2.com/en/latest/Learn/APIMonetization/monetizing-an-api/#step-4-send-usage-data-to-the-billing-engine">This section </a>gives you more details on configuring the Data Publishing part.</p><p>That’s all about enabling Billing for APIs. If you are ready try out the new features you can get the distribution from <a href="https://wso2.com/api-management/">here</a> and the docs from <a href="https://apim.docs.wso2.com/en/latest/learn/api-monetization/monetizing-an-api/">here</a>. But if you are curious to know as to what happens underneath, the next section gives more details about how API Manager and Stripe works hand in hand.</p><p><strong>What happens underneath</strong></p><p>You don’t need to understand the internals to make billing work. However, if you are looking to customize the flows, this section tries to shed more light on how different elements in Stripe are used to implement billing.</p><p>We use the following features/elements available in Stripe.</p><p><strong>Product</strong> — Represents a real-world product or a service. A single product can be defined with different pricing plans.</p><p><strong>Customer</strong> — A party who uses a product. In order to proceed with charging, billing details need to be provided.</p><p><strong>Subscription</strong> — Before using a product or service, a customer needs to subscribe with a pricing plan.</p><p>When creating a billing plan through the admin portal, a corresponding pricing plan will be created on Stripe. This will be created via the Platform Account (the account created by the Admin), so that the plans are available to different API providers.</p><p>And then, the API provider can enable monetization and select a plan for the API. When doing this, a matching product with the API’s name will be created in Stripe; the selected pricing plans will be added into the product. These products are created in the API provider’s account. Recall that when configuring, we had to specify the connected account ID, which is used in this case to create the products and add pricing plans on behalf of the API provider.</p><p>Next is the part performed by the API consumer (or the application developer). When an app developer logs into the Dev portal and creates an application, a matching customer object is created in Stripe. So, for each application created, there would be a separate customer object, bearing the payment details specified by the API consumer.</p><p>At the time of subscribing to the API, a subscription will be created on Stripe, under the product corresponding to the API, with the billing plan selected. This ensures that the API accesses are charged according to the billing plan as usage data is published.</p><p>There’s something to be said about publishing usage data. While the requests flow through the gateway, relevant billing plans and associated data get pushed to the analytics nodes. Now, analytics would summarize this data by different time intervals and update the DBs. But unless a command is issued to push this Data, Stripe will not receive any updates about the usage. Depending on how frequently you run the billing cycles, you can schedule a job to push usage data.</p><p>We hope that you got a good idea about the new Monetization feature and how WSO2 API Manager integrates with Stripe. If you have any questions on Monetization or in general about API Management, feel free to contact us through <a href="https://wso2.com/contact">this</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7af0438230d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Gracefully Terminating Pods in Kubernetes: Handling SIGTERM]]></title>
            <link>https://jaadds.medium.com/gracefully-terminating-pods-in-kubernetes-handling-sigterm-fb0d60c7e983?source=rss-68142244b10b------2</link>
            <guid isPermaLink="false">https://medium.com/p/fb0d60c7e983</guid>
            <category><![CDATA[termination]]></category>
            <category><![CDATA[sigterm]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[resilience]]></category>
            <dc:creator><![CDATA[Amila De Silva]]></dc:creator>
            <pubDate>Sun, 26 Jan 2025 11:28:15 GMT</pubDate>
            <atom:updated>2025-06-02T01:08:09.444Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*osXLwI_uPyWE99YW" /><figcaption>Photo by <a href="https://unsplash.com/@hdbernd?utm_source=medium&amp;utm_medium=referral">Bernd 📷 Dittrich</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>Kubernetes (K8s) provides a powerful platform for hosting and scaling applications, removing the burden of infrastructure management from developers. However, this flexibility comes with its challenges: K8s often deletes pods to handle various deployment tasks. Whether you’re rolling out a new version of the application, scaling down a deployment after a seasonal traffic spike, or taking a node offline for maintenance, the pods running your application are often shut down in the process.</p><p>If you’re transitioning from a VM-based setup to K8s, you might find this frequent shutdown surprising. And because applications on K8s can shut down more often, it’s crucial to handle terminations gracefully to properly save application state and avoid potential issues.</p><h3>How Kubernetes Terminates Pods</h3><p>When Kubernetes deletes a pod, it triggers a series of steps to stop the applications running inside it. Understanding these steps help to identify how best to shutdown applications gracefully. Let’s walk through the process:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EHAOuPlrC8vfefkqKPt6yQ.png" /><figcaption>Pod Termination Steps</figcaption></figure><ul><li>When a pod is flagged for deletion, the kubelet on the node triggers the pre-stop handlers for all containers in the pod. This step only happens if a pre-stop handler is defined at the container level. At the same time, the countdown for the graceful shutdown period (defined by terminationGracePeriodSeconds) begins.</li><li>After the pre-stop hook finishes, the kubelet sends a SIGTERM signal to the containers. This is a polite request for the application to terminate. However, applications that don’t handle SIGTERM might continue to run.</li><li>If the application doesn’t terminate before the grace period ends, the kubelet sends a SIGKILL signal, forcefully stopping the containers and any running processes inside them.</li><li>Once the containers are terminated, the Kubernetes control plane removes the pod, and it’s no longer visible to any client.</li></ul><p>You can find more details on this process, from the<a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination"> Kubernetes docs</a>.</p><h4>How to Handle Terminations Gracefully</h4><p>To make sure your application exits cleanly when a pod is deleted, you can use one (or both) of the following approaches:</p><ul><li><strong>Intercepting </strong><strong>SIGTERM</strong>: Applications can listen for the SIGTERM signal and perform cleanup tasks, such as closing connections, draining in-flight requests, or flushing data.</li><li><strong>Implementing a Pre-Stop Hook</strong>: This allows you to run custom actions before the SIGTERM signal is sent, such as notifying other services or coordinating other containers in the pod.</li></ul><p>In this post, we’ll discuss in detail how to handle SIGTERM. For an in-depth discussion on Pre-Stop hook handling, check out the following article.</p><p><a href="https://jaadds.medium.com/gracefully-terminating-pods-in-kubernetes-part-2-prestop-hooks-41fbd3a5318e">Gracefully Terminating Pods in Kubernetes — Part 2: PreStop Hooks</a></p><p>But you might be wondering why you even need to bother handling the SIGTERM, when your application is running just fine.</p><h3>But My Requests Aren’t Failing</h3><p>A telltale sign of abrupt shutdowns is service downtime during updates to the K8s deployment. However, if you don’t see any failed requests, does that mean your application is shutting down properly? Not necessarily.</p><h4>Different Apps, Different Reactions to SIGTERM</h4><p>In the previous section, we learned that kubelet issues a SIGTERM when shutting down a pod. When SIGTERM isn’t handled, apps behave differently depending on the language they’re written in:</p><ul><li><strong>Some apps exit immediately:</strong> For example, Go, C, Rust, and Java apps typically shut down as soon as they receive SIGTERM. This can cause bursts of errors during deployment restarts or scale down events, as in-flight requests get abruptly cut off.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sRPo63zaT_wNGqRQ0LRpQw.gif" /><figcaption>Pod crashing upon receiving SIGTERM — Inflight requests fail causing service downtime</figcaption></figure><ul><li><strong>Some apps ignore </strong><strong>SIGTERM:</strong> Node.js, Python, Ruby, and C# apps might just keep running, completely ignoring the signal. These apps get forcefully stopped by SIGKILL when the grace period runs out. While you might not see request failures because your application runs long enough for in-flight requests to complete, the shutdown is still abrupt — and that can harm your app’s stability.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*26ZmlIV2T5j7CbIvurGxKQ.gif" /><figcaption>Pod continuing to run upon SIGTERM, eventually getting killed by SIGKILL</figcaption></figure><p>Even if your requests don’t fail, abrupt terminations can still affect the stability of your application. They can lead to:</p><ul><li><strong>Garbage Data</strong>: Data might not be written properly, causing incomplete records or inconsistencies.</li><li><strong>Connection Leaks</strong>: Open connections to databases or services may linger, consuming resources.</li><li><strong>File Corruption</strong>: Files might not be flushed or closed, leading to integrity issues.</li></ul><h4>Signs Your App Isn’t Shutting Down Gracefully</h4><p>Here are some other common signs that your application might be shutting down abruptly:</p><ul><li><strong>Pods linger in the Terminating state:</strong> If pods take about 30 seconds (which is the default terminationGracePeriodSeconds) to delete upon issuing a kubectl delete pod, it’s often a sign that the app isn’t responding to SIGTERM and is running until SIGKILL forces it to stop. If you keep checking the pod status with kubectl get pods, you may also notice STATUS column transitioning from Terminating to Error (In a proper shutdown transition would be from Terminating to Complete)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QrCzQCwjQYUbGC3RjbBucA.gif" /><figcaption>Change in Pod Status as the deployment re-starts</figcaption></figure><ul><li><strong>docker stop doesn’t stop the app immediately:</strong> If you run your container locally with Docker and it doesn’t exit upon issuing a docker stop,it’s a red flag.</li><li><strong>Non-zero exit code in kubelet logs:</strong> After a termination, the exit code of the container is set to a non-zero value, typically to 137 if the container was killed. You’ll see a log similar to this — kubelet.go:2068] “Post-termination container state” pod=”default/ubuntu-5f5f97b667-cnsp9&quot; podUID=”c62a67a2–25ec-49db-8e77–352a549e60ae” containers=[{“Name”:”ubuntu”,”State”:”exited”,”ExitCode”:137,”FinishedAt”:”2025–01–13T05:30:43.105854299Z”}]. To check this, you’ll need to change the verbosity of kubelet logs as explained<a href="https://stackoverflow.com/questions/55739315/how-to-enable-kubelet-logging-verbosity"> here</a>.</li></ul><p>Since we now know the importance of handling shutdowns gracefully, let’s look at how to do that by intercepting the SIGTERM.</p><h3>Handling SIGTERM</h3><p>Kubernetes sends SIGTERM to the main process inside a container, giving the application a chance to shut down gracefully. Let’s look into an example of how to do this in practice.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/746/1*T6KmW-E67hYx0OK2xtABFg.png" /><figcaption>Kubelet issues SIGTERM to the main process in each container</figcaption></figure><h4>Example: Handling SIGTERM in Go</h4><p>Let’s look at a Go application example. First, we’ll observe its behavior without SIGTERM handling, then modify it to handle SIGTERM.</p><p><strong>Step 1: Observe default behavior</strong></p><p>We’ll begin with a simple Go program that listens on port 8080 and responds with “Hello, World” to any GET request.</p><pre>package main<br><br>import (<br> &quot;fmt&quot;<br> &quot;net/http&quot;<br>)<br><br>func main() {<br> // Define the handler function for the root path<br> http.HandleFunc(&quot;/hello&quot;, func(w http.ResponseWriter, r *http.Request) {<br>     // Respond with a greeting<br>     fmt.Fprintln(w, &quot;Hello World!&quot;)<br> })<br><br> // Specify the port to listen on<br> port := &quot;8080&quot;<br> fmt.Printf(&quot;Starting server on port %s...\n&quot;, port)<br><br> // Start the HTTP server<br> if err := http.ListenAndServe(&quot;:&quot;+port, nil); err != nil {<br>     fmt.Printf(&quot;Error starting server: %s\n&quot;, err)<br> }<br>}</pre><p>Let’s run this code and see how it handles traffic upon a restart.</p><p>You can use the following Dockerfile to build an image from this. Or you can use the pre-built image jaadds/go-sample:1.0.0 for the experiment. To build the docker image, paste the code into a file named main.go and run docker build -t go-sample . .</p><pre># Stage 1: Build the application<br>FROM golang:1.20 as builder<br>WORKDIR /app<br># Enable static compilation<br>ENV CGO_ENABLED=0 GOOS=linux<br># Copy all application files<br>COPY . .<br># Build the Go application<br>RUN go build -o main main.go<br><br># Stage 2: Create a lightweight image for the application<br>FROM alpine:latest<br>WORKDIR /root/<br># Copy the compiled binary from the builder stage<br>COPY --from=builder /app/main .<br># Expose the port on which the application will run<br>EXPOSE 8080<br># Set the entrypoint<br>ENTRYPOINT [&quot;/root/main&quot;]</pre><p>Next, we’re going to create a deployment out of this image, and open it up for receiving traffic. If you don’t have a K8s environment setup locally, you can use a Playground environment provided by <a href="https://killercoda.com/playgrounds">KillerCoda</a>.</p><p>Let’s create a deployment with two replicas.</p><pre>kubectl create deploy greeting-app --image=jaadds/go-sample:1.0.0 --replicas 2</pre><p>After the pods start running, we can create a service to expose the deployment.</p><pre>kubectl expose deploy greeting-app --type NodePort --port 8080</pre><p>We need to find the port, the service is exposed from</p><pre>controlplane $ kubectl get svc<br>NAME              TYPE      CLUSTER-IP    EXTERNAL-IP    PORT(S)        AGE<br>greeting-app    NodePort  10.96.122.213    &lt;none&gt;     8080:32740/TCP    14s<br>kubernetes      ClusterIP   10.96.0.1      &lt;none&gt;       443/TCP         11d</pre><p>You can hit the service with curl to check if it works fine</p><pre>controlplane $ curl http://localhost:32740/hello<br>Hello World!</pre><p>Now let’s see what happens if we keep invoking the service while restarting the deployment. To continuously hit the service we can use <a href="https://httpd.apache.org/docs/2.4/programs/ab.html">Apache Bench</a>.</p><p>Let’s run this command in one terminal</p><pre>ab -r -t 20 http://localhost:32740/hello</pre><p>-r flag keeps the client running amidst failures, -t flag specifies how long to run the test.</p><p>In another terminal we can re-start the deployment with the following command.</p><pre>kubectl rollout restart deploy/greeting-app</pre><p>You’ll get something similar to the below</p><pre>controlplane $ ab -r -t 20 http://localhost:32740/hello<br>This is ApacheBench, Version 2.3 &lt;$Revision: 1843412 $&gt;<br>Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/<br>Licensed to The Apache Software Foundation, http://www.apache.org/<br><br>Benchmarking localhost (be patient)<br>Completed 5000 requests<br>Finished 9410 requests<br><br><br>Server Software:     <br>Server Hostname:     localhost<br>Server Port:         32740<br><br>Document Path:       /hello<br>Document Length:     13 bytes<br><br>Concurrency Level:   1<br>Time taken for tests:   20.001 seconds<br>Complete requests:   9410<br>Failed requests:     456<br>   (Connect: 0, Receive: 166, Length: 166, Exceptions: 124)<br>Total transferred:   1201720 bytes<br>HTML transferred:    120172 bytes<br>Requests per second: 470.48 [#/sec] (mean)<br>Time per request:    2.125 [ms] (mean)<br>Time per request:    2.125 [ms] (mean, across all concurrent requests)<br>Transfer rate:       58.68 [Kbytes/sec] received<br><br>Connection Times (ms)<br>           min  mean[+/-sd] median   max<br>Connect:     0 1  10.7   1 1027<br>Processing:  0 1   3.5   1  187<br>Waiting:     0 1   3.1   1  187<br>Total:       0 2  11.3   1 1030<br><br>Percentage of the requests served within a certain time (ms)<br>  50%   1<br>  66%   2<br>  75%   2<br>  80%   3<br>  90%   4<br>  95%   6<br>  98%   9<br>  99%  12</pre><p>As per the output there are failed requests</p><pre>Failed requests:     456<br>   (Connect: 0, Receive: 166, Length: 166, Exceptions: 124)</pre><p>This happens because the application exits abruptly as the deployment re-starts, giving no time for the inflight requests to complete. This is the default behavior for a Go application, when SIGTERM is not handled.</p><p><strong>Step 2: Adding </strong><strong>SIGTERM Handling</strong></p><p>Now, let’s modify the program to handle SIGTERM. By waiting 10 seconds before shutting down, we can make sure the application has enough time to complete in-flight requests.</p><p>We’re going to modify the code like below;</p><pre>package main<br><br>import (<br> &quot;context&quot;<br> &quot;fmt&quot;<br> &quot;net/http&quot;<br> &quot;os&quot;<br> &quot;os/signal&quot;<br> &quot;syscall&quot;<br> &quot;time&quot;<br>)<br><br>func main() {<br> // Define the handler function<br> http.HandleFunc(&quot;/hello&quot;, func(w http.ResponseWriter, r *http.Request) {<br>  fmt.Fprintln(w, &quot;Hello World!&quot;)<br> })<br><br> // Create an HTTP server<br> server := &amp;http.Server{Addr: &quot;:8080&quot;}<br><br> // Channel to listen for OS signals<br> signalChan := make(chan os.Signal, 1)<br> signal.Notify(signalChan, syscall.SIGINT, syscall.SIGTERM)<br><br> // Start the server<br> go func() {<br>  fmt.Println(&quot;Starting server on port 8080...&quot;)<br>  if err := server.ListenAndServe(); err != nil &amp;&amp; err != http.ErrServerClosed {<br>   fmt.Printf(&quot;Error starting server: %s\n&quot;, err)<br>  }<br> }()<br><br> // Wait for SIGTERM or SIGINT<br> &lt;-signalChan<br> fmt.Println(&quot;\nSIGTERM received. Waiting 10 seconds to drain requests...&quot;)<br><br> // Wait 10 seconds to allow ongoing requests to complete<br> time.Sleep(10 * time.Second)<br><br> // Gracefully shut down the server<br> ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)<br> defer cancel()<br> if err := server.Shutdown(ctx); err != nil {<br>  fmt.Printf(&quot;Error during shutdown: %s\n&quot;, err)<br> }<br><br> fmt.Println(&quot;Server stopped gracefully.&quot;)<br>}</pre><p>You can build this with the same Dockerfile used earlier, or you can use this image jaadds/go-sample-graceful:1.0.0</p><p>Let’s update the deployment to have the new image and re-run the test.</p><pre>kubectl set image deploy/greeting-app go-sample=jaadds/go-sample-graceful:1.0.0</pre><p>After the new image applies, we can run the same ab command while re-starting the deployment.</p><p>This time you won’t see any failures.</p><pre>controlplane $ ab -r -t 20 http://localhost:32740/hello<br>This is ApacheBench, Version 2.3 &lt;$Revision: 1843412 $&gt;<br>Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/<br>Licensed to The Apache Software Foundation, http://www.apache.org/<br><br>Benchmarking localhost (be patient)<br>Completed 5000 requests<br>Completed 10000 requests<br>Finished 10876 requests<br><br><br>Server Software:     <br>Server Hostname:     localhost<br>Server Port:         32740<br><br>Document Path:       /hello<br>Document Length:     32 bytes<br><br>Concurrency Level:   1<br>Time taken for tests:   20.172 seconds<br>Complete requests:   10876<br>Failed requests:     0<br>Total transferred:   1620524 bytes<br>HTML transferred:    348032 bytes<br>Requests per second: 539.16 [#/sec] (mean)<br>Time per request:    1.855 [ms] (mean)<br>Time per request:    1.855 [ms] (mean, across all concurrent requests)<br>Transfer rate:       78.45 [Kbytes/sec] received<br><br>Connection Times (ms)<br>           min  mean[+/-sd] median   max<br>Connect:     0 1   1.1   1   19<br>Processing:  0 1   3.2   1  200<br>Waiting:     0 1   2.9   1  200<br>Total:       0 2   3.5   1  201<br><br>Percentage of the requests served within a certain time (ms)<br>  50%   1<br>  66%   2<br>  75%   2<br>  80%   2<br>  90%   4<br>  95%   6<br>  98%   9<br>  99%  11<br> 100% 201 (longest request)</pre><p>In this sample, we simply introduced a delay. However, in a real application, the process might involve calls to other backends and opening connections to databases, brokers, or files. When handling SIGTERM, you can:</p><ul><li>Drain active connections.</li><li>Flush buffers for open files and remote connections, ensuring in-flight data is properly saved.</li><li>Cleanly close files and connections to databases or brokers.</li></ul><p>While we’ve seen now that handling SIGTERM allows the application to shut down gracefully, is it possible for the application to still shut down abruptly even when SIGTERM is handled?</p><h4>Does Your App Receive SIGTERM?</h4><p>Kubernetes sends SIGTERM to the process with PID 1 inside the container. If your app isn’t PID 1, it won’t receive the signal and the code to handle graceful termination won’t execute. This can happen if;</p><ul><li>Your application was started by a script that launches it as a child process, causing it to lose PID 1.</li><li>You were using CMD or ENTRYPOINT instructions in the shell form to create the image.</li></ul><p>For example, if you used script like below below to start the application</p><pre>#!/bin/sh<br><br># Print a message indicating the script has started<br>echo &quot;Starting the application...&quot;<br><br># Execute the compiled Go binarys<br>./main</pre><p>it’ll run inside the container like this;</p><pre>PID   USER   TIME  COMMAND<br> 1    root   0:00   {entrypoint.sh} /bin/sh /root/entrypoint.sh<br> 6    root   0:00   ./main<br>13    root   0:00   ps</pre><p>Here, the shell runs as the PID 1, and the actual application runs with PID 6.</p><p><strong>Avoiding PID Issues</strong></p><ol><li><strong>Use </strong><strong>exec in Startup Scripts:</strong> Change the line calling the program from ./main to exec ./main</li><li><strong>Use the </strong><strong>exec Form in Dockerfile:</strong> For ENTRYPOINT or CMD, prefer the exec form over the shell form. For example:</li></ol><ul><li>Shell Form: ENTRYPOINT /root/main</li><li>Exec Form: ENTRYPOINT [&quot;/root/main&quot;]</li></ul><p>For more details on these forms, you can refer <a href="https://docs.docker.com/reference/dockerfile/#shell-and-exec-form">Dockerfile documentation</a> and the <a href="https://www.docker.com/blog/docker-best-practices-choosing-between-run-cmd-and-entrypoint/">Docker blog</a>. After building an image, as a best practice, check whether your container shuts down upon receiving a SIGTERM. You can do this by issuing a docker stop and checking whether container shuts down.</p><h3>Conclusion</h3><p>In this post, we discussed the importance of handling SIGTERM to gracefully terminate applications in Kubernetes. By intercepting SIGTERM and ensuring proper cleanup, you can avoid issues like data corruption, connection leaks, and resource wastage. Graceful shutdowns not only enhance application stability but also improve user experience, making them an essential practice in Kubernetes environments.</p><p>Terminating pods gracefully is one step in improving stability of your applications. If you’re interested in reading about other steps, you can check my post on building resilient Applications on K8s —</p><p><a href="https://jaadds.medium.com/building-resilient-applications-on-kubernetes-9e9e4edb4d33">Building Resilient Applications on Kubernetes</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fb0d60c7e983" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Value of NGINX Upstreams]]></title>
            <link>https://jaadds.medium.com/the-value-of-nginx-upstreams-682831474864?source=rss-68142244b10b------2</link>
            <guid isPermaLink="false">https://medium.com/p/682831474864</guid>
            <category><![CDATA[nginx]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[performance-testing]]></category>
            <category><![CDATA[nginx-ingress-controller]]></category>
            <category><![CDATA[black-friday]]></category>
            <dc:creator><![CDATA[Amila De Silva]]></dc:creator>
            <pubDate>Tue, 26 Nov 2024 19:28:54 GMT</pubDate>
            <atom:updated>2024-11-27T15:54:24.896Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>How we learned it the hard way</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nFqlQH-o44AHLAA-uIuW1w.png" /></figure><h3>Introduction</h3><p>Sometimes the best lessons are the ones learned the hard way, and that’s exactly what happened while tuning the NGINX Ingress Controller. What started as a simple goal — optimizing performance — quickly turned into a journey of uncovering hidden pitfalls and discovering the true value of NGINX upstreams.</p><p>Along the way, we ran into unexpected bottlenecks, tackled challenges related to connection handling, and stumbled upon some useful tools to troubleshoot them. If you’ve ever struggled with TIME-WAIT connections, resource-intensive processes, or optimizing throughput in high-traffic scenarios, you’ll find this useful. Let’s dive into what we learned the hard way so you don’t have to.</p><h3>The Task</h3><p>One of our customers was preparing for Black Friday, and was expecting a traffic spike. On a regular day, they handle around 800–1200 requests per second, but for Black Friday, they were expecting up to 15,000 requests per second.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NRRyNw1FpT_D9Vug-ANfMw.png" /><figcaption>The setup — Kubernetes Cluster with NGINX Ingress Controller at the edge</figcaption></figure><p>In this setup, all API traffic flows through an edge Ingress Controller. Normally, two replicas are enough to handle peak traffic, and they scale up to three under spikes. Based on these numbers, the customer estimated they’d need about 25 replicas to handle the surge. That’s when my colleague and I were called in to see if we could squeeze more performance out of each instance, ideally cutting down the number of replicas needed.</p><h3>Testing Approach</h3><p>We had following resources allocated to the Ingress Controller:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/310/1*5vPKPyVj3uDV-AZNhuYZ3w.png" /><figcaption>Resources allocated to the Ingress Controller</figcaption></figure><p>We had Horizontal Pod Autoscaling (HPA) enabled, but since our goal was to maximize the throughput of a single replica, we limited it to just one Ingress Controller instance. To simulate the load, we used a JMeter script.</p><h3>First Shot — Tuning the Ingress Controller</h3><p>At first glance, we noticed that the Ingress Controller was still using its default settings. We thought that simply adjusting a few parameters might be enough to hit our target throughput — easy enough, right? Here’s what we changed:</p><pre>  keep-alive: &quot;120&quot;<br>  keep-alive-requests: &quot;10000&quot;<br>  max-worker-connections: &quot;65536&quot;<br>  upstream-keepalive-connections: &quot;500&quot;<br>  upstream-keepalive-time: 1h<br>  worker-processes: &quot;8&quot;</pre><p>But little did we know, this would only mark the beginning of a long day.<br>We ran a 15-minute test with these settings and got the following results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*YyMlhI-MHDRjiOoVN28giA.png" /><figcaption>Results for the first test</figcaption></figure><p>Despite our hopes, these changes only got us a throughput of 781 requests per second — and to make matters worse, the test showed a 40% error rate.</p><p>A quick check on CPU usage revealed it was maxing out.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MP0XQJs-MAj9_otH" /><figcaption>CPU Usage for the First Test</figcaption></figure><h4>Investigation</h4><p>On the client side, most errors were 502s, the kind NGINX throws when it can’t reach the upstream. We knew the upstream was available, so we dug into the NGINX error logs and found:</p><pre>2024/11/01 07:21:52 [crit] 53#53: *2335756 connect() to 172.19.68.121:8080 failed (99: Address not available) while connecting to upstream, client: 10.100.9.55, server: *.toysrus.perfserver.dev, request: &quot;POST /playground/perf-proxy/v1.0/ HTTP/1.1&quot;, upstream: &quot;http://172.19.68.121:8080/c8d10e2d-ad7f-4575-9084-4023b7212067/playground/perf-proxy/v1.0/&quot;, host: &quot;c8d10e2d-ad7f-4575-9084-4023b7212067-perf.toysrus.perfserver.dev&quot;</pre><p>After some searching, we found that this issue was caused by a lack of available ports. When NGINX forwards requests, it needs to bind a socket to a local port to connect to the target server. By default, Linux only allows ~28,000 local ports, and during our load test, we ran out. NGINX couldn’t establish connections because no more ports were available.</p><p>As highlighted in<a href="https://stackoverflow.com/questions/14144396/nginx-proxy-connect-to-ip80-failed-99-cannot-assign-requested-address"> this Stack Overflow question</a>, the solution was to increase the available port range.</p><p>We increased the limit to allow 64,000 ports and gave it another shot. Here’s how we set it up with an initContainer:</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>  name: api-ingress-controller<br>spec:<br>  template:<br> spec:<br>   containers:<br>   - image: nginx/nginx-ingress:1.3.0<br>     name: controller<br>     ...<br>   initContainers:<br>   - command:<br>     - sh<br>     - -c<br>     - |<br>       sysctl -w net.core.somaxconn=65535<br>       sysctl -w net.ipv4.ip_local_port_range=&quot;1024 65535&quot;<br>     image: busybox<br>     imagePullPolicy: Always<br>     name: setsysctl<br>     resources: {}<br>     securityContext:<br>       privileged: true<br>   ...</pre><p>Since both containers share the same network namespace, changes done through the initContainer are available on the main container.</p><p><strong>Pro Tip</strong>: If you haven’t checked the open file limit, that’s another area worth investigating. It can be a bottleneck for high-load scenarios too.</p><p>With high hopes of getting a better throughput, we ran the test for a second time.</p><h3>The Second attempt — Increasing ports</h3><p>After another 15 minutes of testing, we got the following results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*hP3X2kjdFPS1o202ZFN1lg.png" /><figcaption>Results for the second run</figcaption></figure><p>Good news: no errors this time! Increasing the port range worked.</p><p>Bad news: the throughput still wasn’t where we wanted it to be. It was better, but far from great.</p><p>The real problem was the CPU — it was still maxed out. There were clear signs of CPU throttling during peak load. Sure, we could have just bumped up the CPU allocation, but we decided to step back a bit and investigate the issue, instead of throwing more resources at the problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8suqJugaSPUFDIjD" /><figcaption>CPU Usage for the second run — CPU was still maxing out</figcaption></figure><h4>Where’s All the CPU Going?</h4><p>The Ingress Controller accepts traffic through a TLS listener and forwards it to the backend. Only the incoming connections are encrypted; the traffic between the Ingress Controller and the backend is plain text. So, apart from reading from a TLS socket and writing to a plain text socket, the Ingress Controller shouldn’t be doing much processing.</p><p>We were also using keep-alive connections on both ends:</p><ul><li>JMeter was configured to use keep-alive, so the same TCP connection should handle multiple HTTPS requests.</li><li>On the backend side, we enabled keep-alive via the config-map, making sure the same connection was reused.</li></ul><p>Since it wasn’t clear where the extra CPU was being spent, we decided to monitor the Ingress Controller using strace.</p><h4>Using strace to monitor ingress controller</h4><p>strace is a Linux diagnostic tool that intercepts and records system calls. It’s great for identifying which syscalls are taking up the most time. Unfortunately, the container running the Ingress Controller didn’t have strace installed, and since it ran as the www-data user (with uid — 101), we couldn’t just install it.</p><p>Our solution? Add a debug container to the same pod. By sharing the process namespace between containers, we could monitor the processes running in the Ingress Controller from the debug container. (Learn more about this technique<a href="https://kubernetes.io/docs/tasks/configure-pod-container/share-process-namespace/"> here</a>.)</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: api-ingress-controller<br>spec:<br>  template:<br> spec:<br>   shareProcessNamespace: true<br>   containers:<br>   - command:<br>     - sleep<br>     - 1d<br>     image: ubuntu:latest<br>     name: debug<br>     stdin: true<br>     tty: true<br>     securityContext:<br>       privileged: true<br>       runAsUser: 0<br>   - env:<br>     image: nginx/nginx-ingress:1.3.0<br>     name: controller<br>     ...<br>   initContainers:<br>   - command:<br>     - sh<br>     - -c<br>     - |<br>       sysctl -w net.core.somaxconn=65535<br>       sysctl -w net.ipv4.ip_local_port_range=&quot;1024 65535&quot;<br>     image: busybox<br>     imagePullPolicy: Always<br>     name: setsysctl<br>     resources: {}<br>     securityContext:<br>       privileged: true<br>   ...</pre><p>After getting inside the debug container, we ran the following strace command on the NGINX worker process: ( note the process id is for a nginx-worker process — the master process didn’t return anything)</p><pre>strace -f -p 184 -cw</pre><p>Here’s what we found:</p><pre>% time  seconds     usecs/call  calls      errors   syscall<br>------ ----------- ----------- --------- --------- ----------------<br> 72.53   31.662792     7142        4433      4433    connect<br>  8.05    3.514438     1154        3044              epoll_pwait<br>  3.07    1.339601      144        9282              getpid<br>...</pre><p>The overwhelming majority of time was spent on <a href="https://man7.org/linux/man-pages/man2/connect.2.html">connect </a>system call. This syscall is used to establish a connection between the client and server or between NGINX and the backend. Seeing NGINX repeatedly establishing new connections was concerning, especially since we’d configured keep-alive connections.</p><p>Since we know the issue is with creating connections, we thought of probing a bit into the network side.</p><h4>Investigating the Network with ss</h4><p>To dig deeper, we used the ss <a href="https://www.linode.com/docs/guides/ss/">(socket statistics) utility </a>to check network behavior:</p><pre>kubectl exec deploy/api-ingress-controller -c debug -- ss -s</pre><p>Here’s the output:</p><pre>Total: 592<br>TCP:   50517 (estab 505, closed 49960, orphaned 0, timewait 47725)<br><br>Transport Total       IP        IPv6<br>RAW          0         0         0<br>UDP          8         8         0<br>TCP         557       532       25<br>INET        565       540       25<br>FRAG         0         0         0</pre><p>What stood out was the <strong>large number of </strong><strong>TIME-WAIT connections</strong>. Not sure if this was normal, we did some research and came across this comprehensive<a href="https://vincent.bernat.ch/en/blog/2014-tcp-time-wait-state-linux"> article by Vincent Bernat</a>. That’s when we started to understand how these connections were dragging down the performance</p><h4>The Dark Side of TIME-WAIT</h4><p>TIME-WAIT is a built-in TCP mechanism designed to prevent delayed packets from interfering with new connections. Once the party initiating the termination sends the final acknowledgment (ACK) — confirming the connection is fully terminated — it puts the connection into the TIME-WAIT state. The connection remains in this state for <strong>2 × Maximum Segment Lifetime</strong> (typically 2 minutes). During this time, the local port cannot be reused.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*R_tlfJhEvoP-kGI9" /><figcaption>NGINX initiates Termination. Connection enters TIME-WAIT state following the last ACK.</figcaption></figure><p>Here’s the catch: with so many connections stuck in TIME-WAIT, the OS has to hunt for a new port for every new connection. The larger the pool of ports stuck in TIME-WAIT, the longer it takes for the OS to find and allocate a new port — which ultimately slows down the process of establishing new connections.</p><p>One of Vincent’s suggestions was to enable the net.ipv4.tcp_tw_reuse property. This lets the OS reuse connections in the TIME-WAIT state for new connections. Instead of increasing the CPU, we decided to give this a shot.</p><h3>The Third Attempt — Enabling TIME-WAIT Reuse</h3><p>This time, we set net.ipv4.tcp_tw_reuse=1 via the initContainer to enable TIME-WAIT reuse. We also rolled back the max port setting we applied earlier. Here’s what our updated initContainer looked like:</p><pre>   initContainers:<br>   - command:<br>     - sh<br>     - -c<br>     - |<br>        sysctl -w net.ipv4.tcp_tw_reuse=1</pre><p>With the change in place, we kicked off another run, and — finally — a breakthrough! <strong>3300 requests per second</strong>. That’s right, the bottleneck was gone. With this improvement, we could hit the target throughput with just <strong>5 replicas</strong>, so we didn’t bother pushing further.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*mGYZrU-zFiylVTZvYYIXyw.png" /><figcaption>Results for the third run</figcaption></figure><h4>CPU Utilization</h4><p>To check how efficiently the deployment was performing, we monitored CPU usage. Here’s what we observed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OmcQhnJZkNrQCW8l" /><figcaption>CPU Utilization for the third run — utilisation is around 700m</figcaption></figure><p>The CPU utilization settled at <strong>700–800 milicores</strong>, a much better scenario than before.</p><p>We also re-ran strace to analyze system calls, and here’s the new output:</p><pre>strace -p 411 -f -cw<br>% time  seconds    usecs/call  calls        errors      syscall<br>------ ----------- ----------- --------- --------- ----------------<br> 62.56   61.502207      981      62672                  epoll_pwait<br>  7.93    7.799705       30     254447                  madvise<br>  4.66    4.576808       41     110984                  write<br>  3.59    3.527924       31     110627       55228      read<br>  ...<br>  2.25    2.212580       40      54668       54668      connect<br>  ...</pre><p>The connect syscall, which previously consumed significant CPU time, is no longer the highest. Instead, <a href="https://man7.org/linux/man-pages/man2/epoll_wait.2.html"><strong>epoll_pwait</strong></a> — used for waiting for connections — moved to the top. NGINX was now spending more time waiting for new connections than creating them — which is a good sign.</p><h4>Network Behavior</h4><p>Next, we checked in on the network behavior using ss. Here’s what we found:</p><pre>kubectl exec -n choreo-apim deploy/api-ingress-controller -c debug -- ss -s<br>Total: 695<br>TCP:   16741 (estab 601, closed 16073, orphaned 15, timewait 15755)<br><br>Transport Total     IP        IPv6<br>RAW          0         0         0<br>UDP          8         8         0<br>TCP         668       643       25<br>INET        676       651       25<br>FRAG         0         0         0</pre><p>The TIME-WAIT connections? They’d dropped significantly. Even with the port range reset to the default (~28,000), there were plenty of ports to go around. No more expensive searches for a free port, and no more bottlenecks.</p><h3>The Realization</h3><p>We were happy that the network-level changes got us to the throughput we needed. But as we reflected on the whole process, one question kept bugging us: <strong>why didn’t the NGINX-level settings work?</strong></p><p>If keep-alive connections were being used properly, we shouldn’t have run out of ports, and TIME-WAIT connections wouldn’t have become an issue. Something wasn’t adding up, so we went back to our configs to investigate.</p><p>That’s when it hit us: <strong>how we were using NGINX was part of the problem.</strong></p><p>We’d been using a custom virtual host (vhost) definition. There’s a section in this configuration that dynamically calculates the target URL based on the request URL. Instead of using the typical Ingress resources, we were routing traffic using this custom vhost setup. In short, we were using the Ingress Controller more like a regular NGINX server.</p><p>Here’s the relevant snippet from our vhost config:</p><pre>server {<br>     listen    443 ssl;<br>     server_name  &quot;*.toysrus.perfserver.dev&quot;;<br><br>     ...<br><br>     location / {<br>         resolver kube-dns.kube-system.svc.cluster.local ipv6=off;<br>         set $target &#39;&#39;;<br><br>         access_by_lua_block {<br><br>             ...<br>             ngx.var.target = &quot;api-router.&quot; .. ngx.var.NAMESPACE .. &quot;.svc.cluster.local:8080/&quot; .. requestUri<br>             ...<br>         }<br>         proxy_set_header X-Request-Id $req_id;<br>         proxy_pass  http://$target;<br><br>     }<br><br>}</pre><p>In the location block, you’ll notice we’re using a Lua function to dynamically compute the target, and proxy_pass is then used to forward the request. But here’s the catch: <strong>when you do this, NGINX doesn’t use keep-alive connections to the backend.</strong> Every single request is sent through a new TCP connection.</p><h4>Understanding the Configs</h4><p>To figure this out, it helps to understand how config-map properties are applied. Here’s a quick breakdown:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eCXXq5SwqVdWrWAbL-ZXag.png" /><figcaption>How properties in config map are applied to the NGINX</figcaption></figure><ul><li><strong>Properties like </strong><strong>keep-alive and </strong><strong>keep-alive-requests</strong> apply to connections between JMeter and the Ingress Controller.</li><li><strong>Properties starting with </strong><strong>upstream</strong> apply to connections between the Ingress Controller and the backend.</li></ul><p>Now, here’s the thing: properties like upstream-keepalive-connections only work if you define an <strong>upstream block</strong> in your configuration. Without an upstream block, keep-alive connections aren’t used, and every HTTP request spawns a new TCP connection.</p><p>So, in our case, each HTTP request was going through a separate TCP connection because we weren’t using an upstream block. That explained everything — the port exhaustion, the TIME-WAIT bottleneck, and why our initial settings didn’t have any effect.</p><h4>Testing with an Upstream Block</h4><p>To see the impact, we tweaked our vhost config to hardcode the target inside an upstream block. Here’s what we changed:</p><pre>upstream backend {<br> server api-router.default.svc.cluster.local:8080;<br> keepalive 32;<br>}<br><br>server {<br> listen 443 ssl;<br> server_name &quot;*.toysrus.perfserver.dev&quot;;<br><br> location / {<br>     proxy_set_header X-Request-Id $req_id;<br>     proxy_pass http://backend;<br> }<br>}</pre><p>With this config, the same TCP connection was reused for multiple HTTP requests. The results? A throughput of <strong>4059 req/sec</strong>. This was without any of the network level changes (no TIME-WAIT reuse, and no change in maximum number of ports allowed).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*n3CLetVZVbOf-lBEvMacGg.png" /><figcaption>Results for the Fourth Run</figcaption></figure><p>Here’s what the network looked like during this test:</p><pre>Total: 714<br>TCP:   6755 (estab 643, closed 6060, orphaned 0, timewait 2466)<br><br>Transport Total   IP    IPv6<br>RAW         0     0      0<br>UDP         0     0      0<br>TCP       695   670     25<br>INET      695   670     25<br>FRAG        0     0      0</pre><p>The number of TIME-WAIT connections dropped significantly, along with the number of closed connections. This means fewer active connections were being used to send HTTP requests to the backend, and the system was operating efficiently.</p><h3>Wrap Up</h3><p>Using an upstream block made all the difference. It allowed us to reuse TCP connections, significantly reducing the number of TIME-WAIT and closed connections. In hindsight, if we’d configured the Ingress Controller this way from the start, we could have avoided a lot of headaches.</p><p>To make the comparison easier, here’s a breakdown of the different settings we tried and the throughput improvements they brought:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/986/1*jILuEA8ijopPFVh77KlOMg.png" /><figcaption>Throughput variation with different settings</figcaption></figure><p>Although these changes came in different forms, they all revolved around connection establishment. While we’ve always known that creating new connections can be expensive, we didn’t expect it to catch us off guard quite like this.</p><p>On the bright side, this exercise taught us a lot:</p><ul><li>We now know how to use strace effectively to identify processing bottlenecks.</li><li>ss is a great tool for understanding network behavior.</li></ul><p>One key takeaway is this: when a process is consuming excessive CPU but only modestly increases throughput, connection establishment should be one of the first things to check. From there, tuning connection pooling parameters can often lead to big gains.</p><p><strong>One Last Thing</strong></p><p>This post ended up being longer than I anticipated, but I hope you found it helpful!</p><p>And one more request: In our case, the server attribute in the upstream block needs to be defined dynamically, and we haven’t cracked that yet. If you’ve got ideas or have solved this yourself, <strong>please leave a comment — I’d love to hear from you!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=682831474864" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Resilient Applications on Kubernetes]]></title>
            <link>https://jaadds.medium.com/building-resilient-applications-on-kubernetes-9e9e4edb4d33?source=rss-68142244b10b------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e9e4edb4d33</guid>
            <category><![CDATA[readinessprobe]]></category>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[resilience]]></category>
            <category><![CDATA[fault-tolerance]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Amila De Silva]]></dc:creator>
            <pubDate>Sun, 23 Jun 2024 16:46:15 GMT</pubDate>
            <atom:updated>2024-06-23T16:46:15.040Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SANHcT7tfaIbyT1-" /></figure><h3>Introduction</h3><p>Features that Kubernetes offers, such as the ability to change infrastructure without bringing down the entire cluster, automatic healing of applications, and dynamic scaling based on traffic, have helped organizations realize the benefits of cloud-native applications. However, the dynamic nature that enables these features raises the question of whether Kubernetes is stable enough to run mission-critical, high-throughput applications. For example, the possibility of pods going down at any time suggests that traffic and user experience can be affected. While this can happen without proper configuration, with the right probes, pod distribution strategies, and scaling options, any application can run robustly on Kubernetes. In this post, we discuss approaches to reduce downtime and improve the resiliency of applications running on Kubernetes.</p><h3>Configure proper health probes</h3><p>Health probes are the way to inform the Kubernetes control plane whether your application is healthy, ready to receive traffic, or still booting up. While properly configured health probes can improve your application’s responsiveness, improper configurations can cause unnecessary downtime. Health probes are a part that gets often misconfigured, so it’s important to understand the role of different health probes.</p><h4>Purpose of different probes</h4><p><strong>Readiness Probe</strong>: Checked before routing traffic to the pod. If the readiness probe fails, the pod no longer receives traffic.</p><p><strong>Liveness Probe</strong>: Used to determine whether the container within the pod is running. If the liveness probe fails, the container is restarted.</p><h4>Readiness Probe</h4><p>Set the readiness probe to pass when the pod has completely started up and is ready to receive traffic. If you need to warm up the application by priming caches or initializing database connection pools, do so before passing the readiness probe.</p><p>You only need a readiness probe if your pod is serving traffic through a Kubernetes service. If you’re executing a task like running a batch job or consuming from a queue, you don’t need a readiness probe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/947/1*lkgB_6kKpQxnQdQ5ogpcrw.gif" /><figcaption>Readiness probe failure</figcaption></figure><p><strong>Fail readiness when</strong></p><ul><li>The container is overwhelmed by traffic and needs to complete ongoing requests before accepting new traffic.</li><li>The application receives SIGTERM. We’ll discuss this more in the <a href="#8b89">next section</a>.</li></ul><p><strong>On Choosing a probe</strong></p><p>A good candidate would be a dummy endpoint that utilizes the same thread pools as the normal requests. When the request handling thread pools are fully utilized, requests to readiness would also fail, cutting down traffic to the pods.</p><p>To further assess the readiness of the application, you can use various metrics, such as the utilization of database connection pools, threads in use, and concurrent requests. When any of these metrics indicate the application’s exhaustion, you can fail readiness to prevent further degradation.</p><p>Depending on the type of service, you can use a Command, HTTP Probe, TCP Probe, or <a href="https://medium.com/google-cloud/grpc-health-probes-with-kubernetes-1-24-b5bd26253a4c">gRPC Probe</a>. Avoid TCP probes if possible, as they only check if your service is reachable through the port. Since it won’t reach the application level, it won’t give a correct indication of the application’s ability to accept new requests. Generally, use HTTP probes for web applications and microservices/applications exposing REST or GraphQL interfaces. For services with a gRPC interface, use gRPC probes.</p><p><strong>A few tips</strong></p><ul><li>Log the time taken for the readiness probe in the application code. This will help to fine tune the thresholds if probe failures become too frequent.</li><li>Avoid calling backend-services inside the readiness probe as this can cause the probe to time out and fail. The purpose of the probe is to assert whether the application is ready to take traffic. If backend-service is down it should be handled through proper error handling by the application.</li><li>In case the application is solely dependent on backend-services and their statuses are needed before passing the readiness probe, fetch the statuses asynchronously.</li></ul><h4>Liveness probe</h4><p>The liveness probe determines whether the application is running or alive. An application not being ready doesn’t mean it isn’t alive (it may be processing in-flight requests without accepting new ones), but if the application isn’t alive, it can never be ready.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/699/1*fOsFk824Q9QmxzJKRgI63Q.gif" /><figcaption>Liveness probe failure</figcaption></figure><p><strong>Fail liveness when</strong></p><ul><li>The container encounters an issue that cannot be resolved without restarting.</li><li>The application transitions to a frozen state from which it cannot recover by waiting or cooling down. For example a DB connection pool held up by due to a slow and unresponsive DB server</li><li>Connectivity to a critical backend service (like a database or queue) fails and cannot be re-established without restarting.</li></ul><p><strong>On Choosing a probe</strong></p><p>In my opinion, a liveness probe should be used, if the application can be recovered from a re-start. A model liveness probe would return a very simple ( or a hard coded) response that won’t fail unless something is completely wrong with the application.</p><p>If your application needs a restart to establish connectivity with backend services ( like databases or queues) then checking connectivity to these before passing liveness can be a good practice. However, note that cloud native applications developed with modern frameworks such as Express, Spring Boot don’t typically require re-starts to re-establish connectivity. Libraries that handle backend connection pools can do this on their own. If you don’t see the need for a liveness probe, it’s better to avoid one.</p><h4>Things to note</h4><ul><li>Don’t use the same configurations for liveness and readiness probes. The requirements for the two probes are different, and the actions you take in the readiness probe might cause the liveness probe to fail prematurely.</li><li>If you must use the same endpoint for both probes, ensure you have different configurations (initialDelay, timeout, period, etc.) for each.</li><li>Kubernetes doesn’t maintain an order while executing the probes, so make sure the readiness probe fails before the liveness probe. If you’re tolerating three failures for the readiness probe, tolerate five for the liveness probe.</li></ul><h3>Handle Pod Termination gracefully</h3><p>Another common point where requests can fail is when the pod is terminating. Unless the pod termination is gracefully handled, requests sent to a terminating pod can fail.</p><p>When the containers running within the pod don’t gracefully shut down (i.e., they abruptly stop or crash), inflight requests are affected. Additionally, the Kubernetes Service Object associated with the pod will keep sending traffic until the application exits, leading some requests to receive connection refused errors.</p><p>The way to prevent abrupt failures is to handle the SIGTERM signal properly. To understand how that works, let’s examine the pod termination lifecycle in more detail.</p><figure><img alt="Pod Termination steps" src="https://cdn-images-1.medium.com/max/1024/1*EHAOuPlrC8vfefkqKPt6yQ.png" /><figcaption>Pod Termination Steps</figcaption></figure><ol><li>When the pod is marked for deletion, the kubelet calls the prestop handlers of all containers defined in the pod. Along with this, the countdown of the graceful shutdown period ( defined by terminationGracePeriodSeconds) begins.</li><li>After the prestop hook finishes, the kubelet issues SIGTERM to all containers in the pod. SIGTERM is a request to terminate the application running in the container. However the application can keep running even after receiving SIGTERM.</li><li>After the graceful shutdown period is over, SIGKILL is issued to all containers. If there are already running applications, this command will bring them to a halt.</li><li>The Kubernetes control plane removes the pod. After this point, the pod won’t be visible through any client.</li></ol><p>Normally, SIGTERM is the first signal containers receive when the pod starts terminating. If a prestop handler is provided, it gets called first. In some situations, specifying a preStop handler helps achieve a graceful pod termination. Let’s first see how to handle SIGTERM and then how a preStop handler achieves the same.</p><h4>Handling SIGTERM</h4><p>The surest way to stop receiving traffic is to fail the readiness probe when the pod starts terminating by catching the SIGTERMsignal.</p><p><strong>Side note</strong>: Handling SIGTERM properly is arguably the single most important thing to improve resiliency. Why? Because it’s through this signal that the application gets notified it’s about to be taken down. The dynamic nature of Kubernetes requires it to terminate pods frequently for various reasons, like rolling out a new application version, scaling down replicas, cluster maintenance tasks, and more. Not handling SIGTERM means you’re paving the way for the dynamic nature of Kubernetes to disrupt the stability of your application.</p><p>Upon receiving SIGTERM, applications need to catch it and perform a graceful shutdown by doing the following;</p><ol><li>Failing the readiness probe either by marking a condition explicitly or by closing the request accepting thread pool.</li><li>Letting inflight requests complete.</li><li>Closing any two way connections like WebSockets or gRPC.</li><li>Closing the DB connection, connections to queues</li></ol><p>When the readiness probe fails, the Service Object attached to the pod, no longer considers the pod’s IP to send requests. This effectively cut down traffic to the pod immediately.</p><p>To receive SIGTERM, the application inside the container should be running with process ID 1. If you’ve been using a script to start the application, the script will be the process with PID 1. In that case, you need to save the application’s PID and relay the SIGTERM back to the application using the PID.</p><h3>Tune RollingUpdates</h3><p>Downtimes can also occur when rolling out a new version of the application. When a deployment is scaled up to its maximum replica count and a rollout happens, the pods eligible to handle traffic can go down, causing some requests to fail.</p><p>By setting maxUnavailable to 0, you can ensure that new pods are scheduled before taking any of the existing pods down.</p><pre>strategy:<br>    rollingUpdate:<br>      maxSurge: 1<br>      maxUnavailable: 0<br>    type: RollingUpdate</pre><p>Note that as soon as the new pods are running, the old pods are taken down. The control plane doesn’t check whether the new pods are ready to receive traffic. If you need to defer terminating old pods until the new pods are ready to take traffic, you can use <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-readiness-gate">readiness gates</a>. Currently this feature is only supported out of the box by the AWS Load Balancer. Check <a href="https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.1/deploy/pod_readiness_gate/">this link</a> to find more on using readiness gates with AWS Load Balancer.</p><h3>Define Pod Disruption Budgets</h3><p>While performing cluster maintenance tasks like updating the Kubernetes version or upgrading the nodes of the cluster, pods can get terminated or rescheduled on different nodes. A Pod Disruption Budget (PDB) guards the deployment from these disruptions by allowing it to retain a minimum number of pods. The following configuration mandates 50% of the pods to be available during a disruption:</p><pre>apiVersion: policy/v1<br>kind: PodDisruptionBudget<br>metadata:<br>  name: pdb-front–end<br>spec:<br>  minAvailable: 50%<br>  selector:<br>	matchLabels:<br>  	app: front–end</pre><p>If it’s a stateless application you’re guarding, better to use percentages for minAvailable, maxUnavailable attributes. This is because, over time, the number of replicas needed to run the application can increase, and the PDB definition can soon out date.</p><p>If the application is stateful and needs to maintain a minimum number of pods to achieve quorum, then it’s better to specify attributes as numbers.</p><p>Also note that PDBs aren’t considered for all disruptions. For example, deleting a deployment with kubectl doesn’t trigger PDB. Typically the node-related changes like kubectl delete &lt;node&gt;, kubectl drain , kubectl cordon will evaluate PDB. This <a href="https://medium.com/@xpiotrkleban/poddisruptionbudgets-situations-where-they-are-respected-and-ignored-in-kubernetes-29ebaabac175">blog post</a> talks in detail about voluntary disruptions that go through PDB. And you can refer to the official <a href="https://kubernetes.io/docs/tasks/run-application/configure-pdb/">K8s document</a> for more recommendations on defining PDBs.</p><h3>Distributing Pods across nodes</h3><p>Should a node-level failure occur, having pods spreaded across different nodes will help reduce impact of the failure. You can achieve this by using pod affinity or topology spread constraints.</p><h4>Pod Anti-affinity</h4><p>Using anti affinity rules, you can instruct Kubernetes scheduler not to schedule two pods in the same node.</p><pre> affinity:<br>  podAntiAffinity:<br>     requiredDuringSchedulingIgnoredDuringExecution:<br>       - labelSelector:<br>           matchExpressions:<br>             - key: app<br>               operator: In<br>               values:<br>               - transaction-service<br>         topologyKey: kubernetes.io/hostname  </pre><p>According to this no two pods with the label app set to transaction-service will run on nodes which have the same value for kubernetes.io/hostname.</p><p>Depending how strictly you need to enforce this behavior, you can choose between the policies requiredDuringSchedulingIgnoredDuringExecution or preferredDuringSchedulingIgnoredDuringExecution. When the former is used, kube-scheduler cannot schedule the pod unless the condition is met. If it can’t find a new node that doesn’t have a transaction-service app, it’ll wait for a new node. With the latter, it’ll try to prevent co-locating transaction-service pods, but if a suitable node isn’t available, it will violate the condition and schedule.</p><h4>Topology Spread Constraints</h4><p>Suppose you’re running on a managed Kubernetes cluster like EKS or AKS, and you need to distribute your pods evenly across different availability zones. You can use topologySpreadConstraints for that. Availability zones is a method cloud infrastructure providers use to isolate failures. By spreading pods across availability zones, you can make the application highly available.</p><p>Even with a proper anti-affinity rule to place pods on different nodes, you applications can get scheduled like this;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/0*t2sjLU4avyC1UtFm" /><figcaption>Two pods scheduled to the same Zone</figcaption></figure><p>With a toplogySpreadConstraint defined like this, you can have an even distribution.</p><pre>topologySpreadConstraints:<br> - labelSelector:<br>     matchLabels:<br>       app: transaction-service<br>   maxSkew: 1<br>   topologyKey: topology.kubernetes.io/zone<br>   matchLabelKeys:<br>  	- pod-template-hash<br>   whenUnsatisfiable: DoNotSchedule</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/0*eK9-3Phm3ZCmShKR" /><figcaption>Pods scheduled into different zones</figcaption></figure><p>You might be wondering why you cannot use anti-affinity in this case, simply by changing the topology key to topology.kubernetes.io/zone. Unless you carefully configure the policy and parameters, it can pose restrictions on scaling the application. For example, if you simply change the anti-affinity policy we saw, just by changing the topology key, then your application won’t scale beyond three replicas. Therefore, it’s better to use affinity when your purpose is to co-locate a pod with another, or to avoid placing pods together. Use topology spread constraints to distribute pods across nodes, zones, regions.</p><h3>Conclusion</h3><p>Even though it might seem that achieving resiliency and performing zero-downtime deployments with Kubernetes is impossible, with careful planning and proper configuration, the dynamic nature of the platform can be harnessed to run mission-critical applications. By following the strategies explained here — such as configuring proper health probes, handling pod termination gracefully, and defining pod disruption budgets — you can significantly reduce downtimes and improve the overall stability of your applications. Additionally, configuring Cluster Autoscalers and Pod Autoscaling can help scale the cluster and pods to handle traffic surges more effectively.</p><p>As Kubernetes continues to evolve, staying updated with the latest features and best practices will further strengthen your ability to build and maintain resilient applications.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e9e4edb4d33" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Running Spring Boot Applications on Choreo]]></title>
            <link>https://jaadds.medium.com/running-spring-boot-applications-on-choreo-8c4742df06f7?source=rss-68142244b10b------2</link>
            <guid isPermaLink="false">https://medium.com/p/8c4742df06f7</guid>
            <category><![CDATA[choreo]]></category>
            <category><![CDATA[spring-boot]]></category>
            <category><![CDATA[microservices]]></category>
            <dc:creator><![CDATA[Amila De Silva]]></dc:creator>
            <pubDate>Thu, 02 Nov 2023 03:30:59 GMT</pubDate>
            <atom:updated>2023-11-07T00:35:11.807Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/995/1*BDah-O56Vwl2o0d8SZp60g.png" /></figure><p>With the introduction of <a href="https://wso2.com/choreo/changelog/component-creation-with-buildpacks/">Buildpacks</a>, Choreo now offers the flexibility to host services written in various languages, enabling development teams to leverage the benefits of Choreo while working on polyglot applications. With these recent updates, deploying applications written in Node, Java, Go, or Ruby on Choreo has become effortless.</p><p>In this post, I will guide you through the steps to deploy a Spring Boot application on Choreo. Before we begin, please make sure that you have the following prerequisites in place.</p><h3>Prerequisites</h3><ul><li>An active Choreo account. Ensure you have not exceeded your free component limit. <a href="https://console.choreo.dev/signup">Create a new account here</a> if you don’t have one already</li><li>Java 17 or higher.</li></ul><h3>The Spring Boot Application</h3><p>For this guide, I’ve chosen a popular Spring Boot example: employee service. This service facilitates the management of employee details. You can access the code for this application <a href="https://github.com/jaadds/employee/tree/base">here</a>. Feel free to fork it and change it.</p><p>Our Spring Boot application provides a simple REST API with the following four operations;</p><pre>@RestController<br>@RequestMapping(&quot;/api&quot;)<br>public class EmployeeController {<br>    @Autowired<br>    private EmployeeService employeeService;<br><br>    @ResponseStatus(HttpStatus.OK)<br>    @GetMapping(&quot;/employees&quot;)<br>    public List&lt;Employee&gt; getEmployees() {<br>        return employeeService.getEmployees();<br>    }<br><br>    @ResponseStatus(HttpStatus.CREATED)<br>    @PostMapping(&quot;/employees&quot;)<br>    public Employee createEmployee(@RequestBody EmployeeRequest employee) {<br>        return employeeService.createEmployee(employee);<br>    }<br><br>    @ResponseStatus(HttpStatus.OK)<br>    @PutMapping(&quot;/employees/{employeeId}&quot;)<br>    public Employee updatedEmployee(@RequestBody EmployeeRequest employee, @PathVariable int employeeId) {<br>        return employeeService.updateEmployee(employeeId, employee);<br>    }<br><br>    @ResponseStatus(HttpStatus.NO_CONTENT)<br>    @DeleteMapping(&quot;/employees/{employeeId}&quot;)<br>    public void deleteEmployee(@PathVariable int employeeId) {<br>        employeeService.deleteEmployee(employeeId);<br>    }<br>}</pre><p>I have enabled an in-memory H2 database with the hope of replacing it with a proper DB later. When you read through the code, you’ll notice that I have used JPA for data persistence.</p><p>To test the application locally, clone the repository and run the application using the following commands:</p><pre>git clone git@github.com:jaadds/employee.git<br>cd employee<br>./mvnw spring-boot:run</pre><p>You should see the application starting up.</p><pre>  .   ____          _            __ _ _<br> /\\ / ___&#39;_ __ _ _(_)_ __  __ _ \ \ \ \<br>( ( )\___ | &#39;_ | &#39;_| | &#39;_ \/ _` | \ \ \ \<br> \\/  ___)| |_)| | | | | || (_| |  ) ) ) )<br>  &#39;  |____| .__|_| |_|_| |_\__, | / / / /<br> =========|_|==============|___/=/_/_/_/<br> :: Spring Boot ::                (v3.1.4)<br><br>2023-10-23T15:46:41.186+05:30  INFO 20491 --- [           main] c.example.employee.EmployeeApplication   : Starting EmployeeApplication using Java 19.0.2 with PID 20491 <br>2023-10-23T15:46:41.188+05:30  INFO 20491 --- [           main] c.example.employee.EmployeeApplication   : No active profile set, falling back to 1 default profile: &quot;default&quot;<br>2023-10-23T15:46:41.654+05:30  INFO 20491 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode.<br>2023-10-23T15:46:41.699+05:30  INFO 20491 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 37 ms. Found 1 JPA repository interfaces.<br>2023-10-23T15:46:42.166+05:30  INFO 20491 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)<br>2023-10-23T15:46:42.178+05:30  INFO 20491 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]<br>....<br></pre><p>Let’s give it a try:</p><pre>curl http://localhost:8080/api/employees | jq</pre><p>You should get the following output:</p><pre>[<br>  {<br>    &quot;id&quot;: 1,<br>    &quot;firstName&quot;: &quot;Bilbo&quot;,<br>    &quot;lastName&quot;: &quot;Baggins&quot;,<br>    &quot;email&quot;: &quot;bilbo@bagend.me&quot;<br>  }<br>]</pre><p>Let’s add a second employee and check the results:</p><pre>curl -H &quot;Content-Type: application/json&quot; -X POST http://localhost:8080/api/employees -d &#39;{&quot;firstName&quot;:&quot;Frodo&quot;,&quot;lastName&quot;:&quot;Baggins&quot;,&quot;email&quot;:&quot;frodo@bagend.me&quot;}&#39;</pre><p>Output:</p><pre>{&quot;id&quot;:2,&quot;firstName&quot;:&quot;Frodo&quot;,&quot;lastName&quot;:&quot;Baggins&quot;,&quot;email&quot;:&quot;frodo@bagend.me&quot;}</pre><p>Now that we know our service is working, let’s continue with deploying it on Choreo.</p><h3>Preparing the Repository for Deployment</h3><p>Choreo needs certain details to create an endpoint for your service, such as the service port, context path, and service definition. Additionally, you need to specify the endpoint’s visibility level, choosing between public (accessible over the internet) or internal (available only to services running within Choreo). All these details should be defined in the <strong>endpoints.yaml</strong> file.</p><p>For this tutorial, we will expose our employee service as a public endpoint.</p><p>Create a <strong>.choreo/endpoints.yaml</strong> file in your project and add the following content:</p><pre>version: 0.1<br>endpoints:<br>- name: Employee API<br>  port: 8080<br>  type: REST<br>  networkVisibility: Public<br>  context: /<br>  schemaFilePath: openapi.yaml</pre><p>This is a short description of the attributes. For a detailed explanation, you can refer to the <a href="https://wso2.com/choreo/docs/develop-components/configure-endpoints/#learn-the-endpointsyaml-file">official documentation</a>.</p><pre><strong>name</strong>             : The name Choreo will use to create the endpoint. This name will also be used in the Developer Portal if you decide to publish a Managed API later.<br><strong>port</strong>             : The port on which your service is running. Update this value if you have changed the port using the `server.port` property.<br><strong>type</strong>             : The type of API/interface you are exposing. Valid values are REST, GraphQL, and gRPC. Since this is a REST service, we use REST.<br><strong>networkVisibility</strong>: The level of visibility for the service over the network. Valid values are Project, Organization, and Public. We&#39;ll set this to Public for this tutorial.<br><strong>context          </strong>: The context path of your application. If you have changed the context path using the `server.servlet.context-path` property, use that value here. Since we haven&#39;t changed it for this tutorial, we&#39;ll leave it as /.<br><strong>schemaFilePath   </strong>: The schema of the Service. For a REST API, this will be the OpenAPI Specification. For GraphQL and gRPC, you can specify SDL and proto files, respectively. The location is read from the repository root, not from the `./.choreo` folder.</pre><p>Since we specified a schema file path, we need to generate and place the OpenAPI Specification in the specified location.</p><h4>Generating the OAS</h4><p>While it’s possible to manually create the OpenAPI Specification (OAS) for a small API like this, Spring’s rich ecosystem provides tools to do this easily. Add the <strong>springdoc-openapi</strong> dependency to your <strong>pom.xml</strong>, and start your application.</p><pre> &lt;dependency&gt;<br>    &lt;groupId&gt;org.springdoc&lt;/groupId&gt;<br>    &lt;artifactId&gt;springdoc-openapi-starter-webmvc-ui&lt;/artifactId&gt;<br>    &lt;version&gt;2.1.0&lt;/version&gt;<br> &lt;/dependency&gt;</pre><p>Get the OAS in YAML format using the following command:</p><pre>curl http://localhost:8080/v3/api-docs.yaml &gt; openapi.yaml</pre><p>Commit these changes to your repository.</p><h3>Deploying Changes to Choreo</h3><p>Now that everything is set up, it’s time to deploy our Spring Boot application on Choreo. In case you missed any step, you can cross-check your work with the <a href="https://github.com/jaadds/employee/tree/main">main branch</a>. In addition to the steps mentioned here, I’ve updated the <strong><em>snakeyaml</em></strong> dependency to avoid security scan failures.</p><p>Before deploying our Spring Boot Application on Choreo, we’ll have to create a component on Choreo. As our Application behaves as a service, we’ll be creating a <strong><em>Service</em></strong> type component.</p><h4>Creating a Service Component</h4><ol><li>Select the “Service” Tile.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m2R8SKn_AZSiLPQovcPO5w.png" /><figcaption>Project Listing — Select Service component from here</figcaption></figure><p>2. Provide the following details:</p><pre><strong>Component Name</strong>          : Employee Service<br><strong>Description</strong>             : Service for managing employee details  <br><strong>Organization</strong>            : Your GitHub account ID  <br><strong>Repository</strong>              : The repository containing your code  <br><strong>Branch</strong>                  : The branch with the deployable changes (in this case, &quot;main&quot;)  <br><strong>Build Preset</strong>            : Java  <br><strong>Java Project Directory</strong>  : Leave the default as is  <br><strong>Language Version  </strong>      : Select the Java version from your POM (only LTS versions are available)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VKT4-iy6bzYOIkyBbsAz6Q.png" /><figcaption>Create service wizard — Select the language and other details</figcaption></figure><p>3. Click “Create” to finalize the component.</p><h4>Deploying the Service</h4><ol><li>Navigate to the <strong>Deploy</strong> page.</li><li>In the <strong>Build Area</strong> section, verify component details and the latest commit.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EgwZcnEHVPP9llCayLK_GQ.png" /><figcaption>Deploy Page — Your commits and builds will appear here</figcaption></figure><p>3. Click “Configure &amp; Deploy”, which will initiate a three-step wizard.</p><ul><li>First screen: Screen for adding environment variables. We’ll skip this for now.</li><li>Second screen: For specifying file mounts. We can skip this too.</li><li>Final step: Verify details from `endpoints.yaml` and click “Deploy.”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m882KvFBjwDsMUETwaJjfQ.jpeg" /></figure><p>4. Clicking Deploy will initiate the build process. Choreo will compile the source code, build a container out of it, do a vulnerability scan and deploy it to the development environment. This will take a minute or two to complete.</p><p>5. Wait for the status to change to “Active.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t-r1ie-7N4VUgk0WRvwuNw.png" /><figcaption>Service getting deployed to development environment</figcaption></figure><h4>Viewing Logs</h4><p>Once you see the Active label on the Development environment, your service should be up and running. If you want to further confirm whether the service got started properly, you can check the logs.</p><p>Go to the <strong>Observability</strong> section to view logs. You’ll see logs similar to those you saw when running locally.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7GiWaAYMp6u6AN-iM1EsOw.png" /><figcaption>Logs in Observability page</figcaption></figure><h4>Testing the Service</h4><p>Let’s test the service. The <strong>Test</strong> section provides a variety of tools for this purpose. For our case, we will choose the Swagger Console. You may notice that the endpoint you specified in the YAML file appears in the drop-down menu. If you have defined multiple endpoints, they will all be displayed in this drop-down. Additionally, observe that an API Key has been generated. This is because when exposing endpoints for consumption, Choreo automatically secures them, requiring consumers to use a proper token.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5ZlMDFid1F0_-H8qQpq-dQ.png" /><figcaption>Swagger Console on Test page</figcaption></figure><p>You can expand on a resource and click Execute.</p><p>This will give the response like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cBR8jZusT6QFvu1ZyU_sbQ.png" /><figcaption>A successful invocation</figcaption></figure><p><strong>Congratulations</strong>!! You’ve successfully deployed a Spring Boot application on Choreo.</p><h3>What’s Next?</h3><p>Learning and experimenting with the platform will help you to use Choreo better and develop production-grade applications in no time. Here are a few things you can do to familiarize yourself with the platform:</p><ol><li>Add logs to your resources and use the Logs view in Choreo to trace requests end-to-end. This can be invaluable for debugging and understanding the behavior of your application in a real-world scenario.</li><li>Promote your Service to Production and check whether everything runs smoothly. Experience how easily you can propagate changes across environments.</li><li>Publish your service as an API in the Developer Portal. Manage, document, and share your API with other developers or teams.</li><li>Follow the <a href="https://wso2.com/choreo/docs/">official documentation of Choreo</a> to learn more about concepts like <a href="https://wso2.com/choreo/docs/choreo-concepts/project/">Projects</a>, <a href="https://wso2.com/choreo/docs/choreo-concepts/component/">Components</a>, and <a href="https://wso2.com/choreo/docs/choreo-concepts/environments/">Environments</a>.</li></ol><p>Stay tuned for the next post, where I will talk about connecting a DB and securely storing configs for you application.</p><p>Happy Coding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8c4742df06f7" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>