<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Pramuditya Ananta Nur on Medium]]></title>
        <description><![CDATA[Stories by Pramuditya Ananta Nur on Medium]]></description>
        <link>https://medium.com/@pramuditya?source=rss-d7e5708a3155------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*aGpQsN1vsFQj7qNF.jpg</url>
            <title>Stories by Pramuditya Ananta Nur on Medium</title>
            <link>https://medium.com/@pramuditya?source=rss-d7e5708a3155------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 01:52:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@pramuditya/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[He Got the Diploma. I Got the Lesson]]></title>
            <link>https://pramuditya.medium.com/he-got-the-diploma-i-got-the-lesson-2b294052168d?source=rss-d7e5708a3155------2</link>
            <guid isPermaLink="false">https://medium.com/p/2b294052168d</guid>
            <category><![CDATA[engineering-manager]]></category>
            <category><![CDATA[lessons-learned]]></category>
            <category><![CDATA[family]]></category>
            <category><![CDATA[fatherhood]]></category>
            <dc:creator><![CDATA[Pramuditya Ananta Nur]]></dc:creator>
            <pubDate>Sat, 27 Jun 2026 03:01:02 GMT</pubDate>
            <atom:updated>2026-06-27T03:01:02.451Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2B_ACbpAgtkrixVA" /><figcaption>Photo by <a href="https://unsplash.com/@timmossholder?utm_source=medium&amp;utm_medium=referral">Tim Mossholder</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The chairs were small. Everything was small — the diplomas made from construction paper, the mortarboards that kept slipping sideways, the certificates that listed achievements no HR department would ever recognize: <em>Most Curious</em>. <em>Best Helper</em>. <em>Always Smiling</em>.</p><p>He stood in line with the other children, fidgeting with his robe, looking for our faces in the audience the way children do when they’re performing — half proud, half checking that you’re watching. I was watching. His mother was recording. Both of us doing what parents do at these moments: pretending to be composed while something moves through us that doesn’t have a name.</p><p>Then it landed.</p><p>This wasn’t just his graduation.</p><p>I had been sitting in that school too. For four years, without a curriculum, without grades, without anyone telling me whether I was passing. No syllabus. No office hours. No retry mechanism when I got something wrong. My student was a child who didn’t know he was teaching me, and my subject was something I’m still not sure I’ve mastered.</p><p>He graduated playgroup. I’m on a journey, learning something I’m still trying to name.</p><p>—</p><h4>The Minister of Everything</h4><p>Let me be precise about how this school actually ran.</p><p>I was not the primary educator. I was the one who showed up — consistently, with love, and not much else to offer in those early years when showing up was genuinely the most I could manage.</p><p>His mother ran the school. She ran it as five simultaneous ministries, and she ran all of them without delegating any of them to anyone, least of all to me.</p><p><em>Menteri agama</em> — faith formation. She was the one awake before the cartoons started, making sure <em>iqra</em> happened before screens. She taught him how to pray before he understood what prayer was, knowing that the body learns before the mind does. She decided which values came first and held that line every day without making it feel like a law.</p><p><em>Menteri pendidikan</em> — the actual curriculum. She tracked what he was learning, what he still needed, which books were at the right level and which ones would stretch him just enough. She noticed when he was bored before he noticed himself. She adjusted before he even knew there was a problem.</p><p><em>Menteri militer</em> — structure. The schedules that held. The consistency that meant he knew, without being told, what came next. Structure isn’t control; it’s the scaffolding that lets children feel safe enough to explore. She built it and maintained it every single day.</p><p><em>Menteri hukum</em> — the discipline system. What happens when rules break. How consequences are delivered so they teach rather than injure. She calibrated this continuously, which is a harder job than it sounds, because the right consequence at four is the wrong consequence at five, and you have to keep adjusting without losing the child’s trust.</p><p><em>Menteri keuangan</em> — the invisible logistics. The math behind a family that nobody sees. Who manages the enrollment, the fees, the schedule conflicts, the coordination between what a child needs and what’s actually possible. She ran this too.</p><p>My contribution: I showed up. I was present. I tried to be someone he could count on, and on most days I managed it. But I was not the architect of his early life. She was.</p><p>Gratitude has to be specific to be real. “She did so much” costs nothing. Naming what she actually did — that’s the honest accounting.</p><blockquote>Terima kasih, istriku. Untuk segalanya yang tidak pernah kelihatan tapi selalu terasa.</blockquote><h4>The Why Machine</h4><p>He asks <em>why</em> the way other children ask for snacks — constantly, relentlessly, and with the expectation of being satisfied.</p><p>The first answer is never enough. He follows it. <em>But why does that happen? But why does that matter? But why did they decide that?</em> He pulls the thread until he reaches something that feels final — some bedrock principle where the chain of causation runs out — and then he stops, nods, and moves on. He has found what he was looking for. The mechanism, not just the answer.</p><p>I am an engineer. My instinct when he started doing this was to treat it like a debugging session — answer the question, close the ticket, move on. What I didn’t understand at first is that he wasn’t debugging. He was building a model of the world, and he needed the full architecture, not just the surface behavior.</p><p>I tried to close the loop once. He was asking about something — I don’t remember what, but the chain was long, and it was 7pm, and I had been in Jira for eight hours — and I gave him a final-sounding answer that wasn’t actually final. He looked at me. “But <em>why</em>?” Not challenging. Genuinely asking. He could tell I had stopped before the end.</p><p>I gave him the real answer. He nodded. Done.</p><p>I put away the debugger after that. This wasn’t a system to be fixed. This was a mind working exactly the way a good mind should: refusing to accept the surface when the structure underneath is what actually matters.</p><p>The child who won’t accept “because I said so” will eventually understand <em>why</em> rules exist — which is a fundamentally deeper form of wisdom than obedience through authority. I want him to understand things all the way down. Even when it’s 7pm and I’m tired.</p><p>Especially then.<br> —</p><h4>The Negotiator</h4><p>He found the loopholes. Every rule we made, he studied it with the attention of someone preparing a legal brief. He learned the letter of the policy. Then he probed the edges of the spirit.</p><p><em>“You said I couldn’t have screen time before dinner. Dinner is over.”</em></p><p><em>“You said one cookie. This is a mini cookie. That’s different.”</em></p><p><em>“You said I had to finish my vegetables. I finished the ones on my plate. You didn’t say anything about the ones still in the bowl.”</em></p><p>My first instinct — which I am not proud of — was to close every loophole. Tighten every rule. Anticipate every technicality and foreclose it before he could find it. Make the system airtight.</p><p>What I eventually understood is that closing loopholes is the wrong response to a child who interrogates rules. The right response is to explain the <em>why</em> before they ask. Give them the principle, not just the policy. When he understands that the rule about screen time exists because uninterrupted family time matters — not because screens are evil, but because connection is the point — then the negotiation becomes a values conversation. And those are worth having.</p><p>He and the why machine are the same brain working on different problems. One asks why the world works the way it does. The other asks why rules apply to him specifically. Same curiosity. Different domain. I stopped treating it as defiance the day I realized it was the same trait I want him to keep for the rest of his life: the refusal to accept things on authority alone, the insistence on understanding the reason underneath the instruction.</p><p>I’m still working on giving him the principle instead of just the rule. I don’t always get there. But I’m trying.<br> —</p><h4>Team Player and Boxing Challenger</h4><p>His younger brother doesn’t know yet that he’s lucky to have him.</p><p>He will know someday. Right now he’s working with incomplete information.</p><p>There are moments — and I have watched them with something that looks like awe — where my older son is the most protective person in the room. His brother trips, and he’s there before either of us. He can’t reach something, and help appears without being asked. He explains things with a patience he does not extend to adults. In those moments, he is the older brother in the best possible sense: someone who knows more and chooses to share it instead of leveraging it.</p><p>And then, ten minutes later, he is the reason his brother is crying.</p><p>I want to tell him this is something he’ll grow out of. I don’t know if that’s true. What I know is that he is experiencing — at four years old, without any vocabulary for it — the full complexity of loving someone. He loves his brother and competes with him simultaneously. He protects him and also, occasionally, checks whether he can defend himself. He hasn’t learned to hide that contradiction yet.</p><p>Honestly? I’m not sure he should.</p><p>Relationships between people who genuinely care about each other are not smooth. They are a continuous negotiation between closeness and friction, between protection and competition. What he’s doing with his brother at four is what adults do at thirty-five — he just hasn’t learned to perform consistency yet. The adults who look like they have it figured out are mostly just better at performing it.</p><p>He’ll learn the performance. I hope he keeps the honesty underneath it.</p><h4>What We Want for You</h4><p><em>This part is for him. You can listen.</em></p><p>We looked at the standard definitions — grades, rank, prestigious university, high salary, impressive title — and found them insufficient. Not wrong, exactly. Just not enough. They measure outputs without describing the person. They tell you where someone ended up without saying anything about who they are.</p><p>So we built our own. This is what we actually want for you:</p><ul><li><strong>Curiosity about how fundamental things work, and the ability to connect the dots.</strong> Not the ability to recall answers — the need to understand mechanisms. The person who knows <em>why</em> the rule exists is more valuable, more trustworthy, and more interesting than the person who just knows the rule.</li><li><strong>Courage to try, and to rise when you fall.</strong> Not resilience as a slogan. The specific willingness to attempt something you might fail at — and then, when you fail, to get back up with the same openness intact. The falling isn’t the failure. Staying down is.</li><li><strong>Preparation plans for both routes.</strong> Most people only plan the success path. Knowing what you’ll do when it works <em>and</em>when it doesn’t is a different and rarer kind of readiness. You don’t have to be afraid of failure if you’ve already thought about what happens next.</li><li><strong>Empathy.</strong> Not the performance of sympathy at the right moments. The actual ability to ask yourself <em>what is it like to be them</em> — and mean it.</li><li><strong>Happiness.</strong> Stated simply, without apology. Not achievement. Not recognition. The interior experience of being genuinely okay with your life on an ordinary Tuesday. That’s harder than it sounds, and it matters more than almost anything else on this list.</li><li><strong>Decisions based on data and research.</strong> Before forming an opinion, before taking action — ask: <em>How do we actually know this?</em> Not as cynicism. As discipline.</li><li><strong>God above everything.</strong> The anchor that keeps all the other values from drifting when the rest of life is turbulent. This one isn’t negotiable, and it isn’t just ritual — it’s the source that makes everything else on this list mean something beyond personal preference.</li><li><strong>Discipline.</strong> Not the military version. The version that means you do the thing you decided mattered, even when you don’t want to. Especially when you don’t want to.</li><li><strong>Teamwork.</strong> Knowing your role. Showing up for others. Understanding that some things can only be built together, and that being the one who shows up reliably is worth more than being the one who shows up brilliantly once.</li><li><strong>Accountability when you’re wrong.</strong> The courage to say <em>that was my mistake</em> — without deflecting, minimizing, or explaining it away. This is one of the rarest things in the world. Grow up to be someone who does it.</li><li><strong>Creativity.</strong> Seeing what isn’t there yet and having the nerve to try to make it.</li><li><strong>Courage to be different according to the values you actually hold.</strong> The hardest one. Being willing to stand outside the consensus when it conflicts with what you genuinely believe — not as rebellion, but as conviction. This only works if the values are truly yours, not borrowed from whoever you’re trying to impress.</li></ul><p>That’s the list. It’s not on any report card. But it’s what we’re trying to build.</p><h4>Kepada Anakku</h4><p>You graduated playgroup. I learned something that doesn’t have a diploma.</p><blockquote>Terima kasih telah membersamai aku belajar menjadi seorang ayah.</blockquote><p>Four years of watching you become someone. Four years of trying to become the father you deserved, with incomplete information and no preview of what came next. Every stage you entered was a stage I entered too — without a guidebook, without a rehearsal, usually a half-step behind you just trying to keep up.</p><p>And I’m sorry.</p><p>I’m sorry for every time I raised my voice when what I actually felt was fear dressed up as anger. For the moments where the correction came out wrong and landed somewhere it shouldn’t have. For the nights I was too tired, too distracted, too inside my own head to show up the way you needed. For the times I was harsh when I should have been patient, and impatient when I should have been quiet.</p><p>I can’t take those moments back.</p><p>What I can tell you is what was underneath them: love. Always love. Even when it came out badly shaped, even when it didn’t look like love from where you were standing — the reason was always love.</p><blockquote>Mendidik terkadang terlihat setega itu, tapi itu adalah karena rasa cinta.</blockquote><p>I hope you carry the intention more than the incident. I hope what stays with you is not the friction but the warmth underneath it.</p><p>I don’t know what the next stage is called. I don’t know what kind of father you’ll need me to be when you’re ten, or fifteen, or thirty. But I know that you’re the one teaching it to me — one stage at a time, one <em>why</em> at a time, one small chair and sideways mortarboard at a time.</p><p>You crossed something today.</p><p>So did I.</p><p>—</p><p><em>Pramuditya Ananta Nur— writing at the intersection of engineering, leadership, and the things that actually matter. ❤️</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2b294052168d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Logs Were Lying to You — And Here’s What I Did About It]]></title>
            <link>https://pramuditya.medium.com/the-logs-were-lying-to-you-and-heres-what-i-did-about-it-dda2f3fb8c07?source=rss-d7e5708a3155------2</link>
            <guid isPermaLink="false">https://medium.com/p/dda2f3fb8c07</guid>
            <category><![CDATA[elk]]></category>
            <category><![CDATA[triage]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[logging-and-monitoring]]></category>
            <category><![CDATA[prometheus-monitoring]]></category>
            <dc:creator><![CDATA[Pramuditya Ananta Nur]]></dc:creator>
            <pubDate>Mon, 08 Jun 2026 04:31:02 GMT</pubDate>
            <atom:updated>2026-06-11T05:33:21.033Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Logs Were Lying to You — And Here’s What I Did About It</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4ibQPny-gzl-pRZj" /><figcaption>Photo by <a href="https://unsplash.com/@chrisyangchrisfilm?utm_source=medium&amp;utm_medium=referral">Chris Yang</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>2 AM. The ping arrives.</h4><p>“Service degraded. L1 is escalating.”</p><p>You open the laptop. Eyes half-closed. Coffee not yet brewed.<br>You open Kibana.</p><p>And there it is — 80,000 log lines, scrolling. A wall of JSON. Each line is a story, but nobody told you which page to open. You start filtering. status_code:503. Okay, 1,200 hits. timestamp:[now-15m TO now]. Still 800. You pick one at random, squint at the raw message, and try to reconstruct what happened from a sea of text that was never written for you to read under pressure.</p><p>45 minutes later, the incident is resolved. Not because you found the root cause — because it fixed itself. And you log the post-mortem: <em>“Investigated via ELK. Root cause unclear.”</em></p><p>That was me. For longer than I’d like to admit.</p><h4>The Old World: ELK Was Our Single Source of Truth</h4><p>In the beginning, the architecture was clean and reasonable.</p><p>Every service produces logs. Logs go to a central log shipper (Logstash or Filebeat). Logs land in Elasticsearch. Kibana reads Elasticsearch. Engineers read Kibana. Alerts fire when something in Kibana crosses a threshold.</p><p>For success rate monitoring, the approach was: parse every HTTP request and response, extract the status code, response body, and build a KQL query that counts 5xx responses over a time window. If the ratio crosses 5%, you get paged.</p><p>It worked. Kind of.</p><p>But over time, three things became impossible to ignore.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MMRRK-H9_zaUiGYtAqKDEQ.png" /><figcaption>The incident resolved itself. We have the logs. We just can’t read them yet</figcaption></figure><p><strong>The latency problem.</strong> Logs are produced by the application, buffered, shipped, parsed, indexed, and <em>then</em> queryable. From event to alert, you’re looking at 3–5 minutes of pipeline lag — sometimes more when the indexing backlog grows. Five minutes is an eternity when a payment service is throwing 503s.</p><figure><img alt="The Elasticsearch bill does not care about your intentions" src="https://cdn-images-1.medium.com/max/1024/1*Z_EmF1U85dxITUrG3n4aIw.png" /><figcaption>The Elasticsearch bill does not care about your intentions</figcaption></figure><p><strong>The volume problem.</strong> We weren’t logging errors. We were logging <em>everything</em> — every request, every response, every retry, every internal state transition. “More logs = more visibility” sounded sensible until we got the Elasticsearch bill. We were paying to store noise, then paying again in query time to filter it out.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IGfWTnSawEPnUG0b7Bk8iA.png" /><figcaption>The logs tell you HTTP codes. The business wants to know how many transactions failed. These are not the same question</figcaption></figure><p><strong>The language problem.</strong> This one is subtle, and it’s the one that caused the most friction. Kibana queries look like this:</p><pre>http.response.status_code:5* AND service.name:&quot;payment-service&quot;</pre><p>That’s engineer-speak. L1 support reads it and sees hieroglyphics. When they escalate to L2, L2 has to translate: “So, how many transactions failed?” Nobody knows. The logs tell you <em>HTTP codes</em>. They don’t tell you <em>business outcomes</em>.</p><h4>The Turning Point: The Incident That Changed Everything</h4><p>There was a specific incident — the kind that doesn’t kill you but does make you rethink your life choices.</p><p>A service started returning timeouts intermittently. Not all requests — maybe 3%. Not all endpoints — mostly on one flow. L1 noticed it because users were complaining. They opened Kibana. They searched for errors. They found… nothing obvious. The logs showed the request coming in. The logs showed the response going out. The status code was 200.</p><p>Wait. 200?</p><p>The service was catching the timeout internally and returning a polite “sorry, something went wrong” message with a 200 status code, because the developer who wrote it wanted to avoid triggering the upstream retry logic. A reasonable micro-decision. An absolute disaster for observability.</p><p>Our entire monitoring relied on HTTP status codes parsed from logs. And the service was lying to us through perfectly valid 200 responses.</p><p>We couldn’t see the problem. Not because the logs weren’t there. Because the logs weren’t measuring what we actually cared about.</p><p>That was the day we decided to build metrics that spoke the truth.</p><h4>The New World: Custom Metrics, Prometheus, and Grafana</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3pdcu9vKoIBkE4RZCvfZGA.png" /><figcaption>The HTTP status code is what the protocol says happened. The metric is what actually happened</figcaption></figure><p>The shift wasn’t complicated technically. It was a mindset shift first, and then a plumbing job.</p><p>The key insight: <strong>stop measuring what the HTTP protocol says, and start measuring what the business says.</strong></p><p><strong>The Setup<br></strong>Every service runs on Spring Boot. Spring Boot has Actuator. Actuator has /actuator/prometheus. This endpoint exposes all registered metrics in Prometheus exposition format — and with Micrometer, you can register <em>custom</em> metrics with about five lines of code.</p><p>On the infrastructure side, every service pod in Kubernetes gets three annotations:</p><pre>annotations:<br>  prometheus.io/scrape: &quot;true&quot;<br>  prometheus.io/path: /actuator/prometheus<br>  prometheus.io/port: &quot;8080&quot;</pre><p>That’s it. Prometheus auto-discovers pods with these annotations and starts scraping every n seconds. No log pipeline. No indexing lag. Metrics are available in near real-time.</p><p>Grafana connects to Prometheus. Dashboards query Prometheus. Alerts query Prometheus.</p><p>The pipeline went from this:</p><pre>App → Logs → Logstash → Elasticsearch → Kibana → (you, squinting at KQL)</pre><p>to this</p><pre>App → /actuator/prometheus → Prometheus (15s scrape) → Grafana</pre><p><strong>The Metrics We Built</strong><br>The critical design decision was naming. We didn’t name metrics after code concepts. We named them after business concepts.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/22dc567a3526d3b051f8731fd9808c9d/href">https://medium.com/media/22dc567a3526d3b051f8731fd9808c9d/href</a></iframe><p>L1 support opens Grafana, sees “Failed Purchase: 1,247 in 15 mins,” and doesn’t need a translator. The metric name <em>is</em> the business language.</p><p>The timeout case that broke us before? external_call_timeout_total increments <em>inside the service logic</em> — before the HTTP response is constructed. It doesn&#39;t matter what status code goes out. The metric is honest.</p><h4>The Hybrid: Metrics See It, Logs Explain It</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FYQNh1Dk0J3UYH8Kgg6AeA.png" /><figcaption>one of these gets you back to sleep by 2:30 AM, the other doesn’t.</figcaption></figure><p>Here’s the thing people get wrong about this shift: we didn’t delete the logs.</p><p>Logs are not the enemy. Overreliance on logs as the <em>detection mechanism</em> is the problem.</p><p>The new mental model:</p><blockquote><strong>Metrics are the radar. Logs are the scalpel.</strong></blockquote><p>Radar tells you there’s something in the sky. Scalpel is how you figure out what, exactly, is wrong.</p><p>In practice, this means:</p><ol><li><strong>Alert on metrics.</strong> Prometheus fires when external_call_success_rate drops below 99% for 5 minutes. You get paged. You open Grafana.</li><li><strong>Triage on metrics.</strong> Grafana shows you which service, which endpoint, which metric spiked. You have context before you’ve read a single log line.</li><li><strong>Diagnose with logs.</strong> <em>Now</em> you open Kibana — with a specific service name, a specific time window, a specific transaction ID. You’re not fishing. You’re dissecting.</li></ol><p>The order used to be: <strong>open Kibana → search blindly → hope something jumps out</strong>. <br>The new order is: <strong>metric tells you what broke → log tells you why. Two different tools</strong>. <br>Two different jobs. Neither one does both well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yAADODg2yYsB-iRkIfxCPg.png" /><figcaption>We’re not breaking up with Kibana. We’re just changing the relationship. Kibana is for diagnosis now. It deserved better than being the first thing we opened at 2 AM.</figcaption></figure><h4>The SOP: What Happens When Something Actually Breaks</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6En1mSE_V_xOQ_EMUbNqag.png" /><figcaption>The SOP is not overhead. The SOP is the observability system.</figcaption></figure><p>Metrics and dashboards are meaningless if the team doesn’t know what to do when a metric turns red. So we formalized the runbook.</p><p>Every incident follows five steps:</p><p><strong>Step 1 — Detect.</strong><br>Prometheus alert fires. Grafana annotation appears. The metric that crossed the threshold tells you which service and which failure type. Examples:</p><ul><li>external_call_timeout_total spiking → timeout issues, probably downstream dependency</li><li>external_call_duplicate_total spiking → upstream retry storm or client-side retry bug</li><li>external_call_validation_error_total spiking → bad input, possibly a client deploy with missing required fields</li></ul><p><strong>Step 2 — Identify.</strong><br>Open the Grafana dashboard for the affected service. Look at the time correlation: did the spike start at a deploy? A downstream restart? A traffic surge? The dashboard should answer <em>when</em> and <em>at what scale</em> without you reading a single log.</p><p><strong>Step 3 — Dig down.</strong><br>Take the service name, the time window, and the failing transaction type into Kibana. Search with context, not hope. If you have a trace ID on the metric (via exemplars), you can jump directly to the relevant log cluster.</p><p><strong>Step 4 — Fix.</strong><br>Resolve the immediate issue. Rollback if it’s a deploy-correlated spike. Rate-limit if it’s a duplicate request storm. Patch if it’s a validation bug. Confirm fix via Grafana: the metric should recover within one scrape interval.</p><p><strong>Step 5 — Prevent.</strong><br>Post-incident: did this alert fire fast enough? If the lag was more than 2 minutes, tighten the Prometheus scrape interval or add a pre-alert threshold. Did the runbook have a gap? Update the SOP. Was this a known failure mode? Add a dedicated metric for it so next time it’s named and visible from the first second.</p><p>The goal is that <em>no incident should surprise the same team twice.</em></p><h4>The Optimization: What Actually Changed</h4><p>After running the new setup for a few months, the numbers told a cleaner story than any log query ever could.</p><p><strong>Performance.</strong> Mean time to detect (MTTD) dropped. Not because engineers got smarter — because the detection mechanism no longer had a 1–5 minute indexing lag. Prometheus scrapes every n seconds. The gap between an anomaly happening and an alert firing went from “sometime this morning” to “within a minute.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1XPO3bFA5kW6_FoilKoAMg.png" /><figcaption>7 days of logs for diagnosis. 30 days of metrics for trend analysis. Same visibility, dramatically different invoice.</figcaption></figure><p><strong>Cost.</strong> The Elasticsearch bill went down. Not because we deleted logs — because we stopped treating every log line as an observability signal that needed to be retained, indexed, and queryable forever. Logs are now kept for 7 days (for diagnosis). Metrics, being lightweight time-series data, are kept for 30 days. The volume economics flip: you’re keeping the lightweight representation longer and the heavyweight representation shorter. That’s the right way around.</p><p><strong>Traceability.</strong> A Grafana dashboard with labeled metrics is a shared artifact. It can be shown to a product manager, a support lead, or an auditor. “Here’s our transaction success rate over the last 30 days” is a sentence a human can parse. “Here’s our ELK query for 5xx in the last 30 days” is not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DwqX9AWcNl-lBARe2tef6A.png" /><figcaption>The goal was never fewer incidents. The goal was fewer incidents that needed an engineer. We got both.</figcaption></figure><p><strong>Team autonomy.</strong> L1 can now check service health without escalating to an engineer. L2 can triage before they even open Kibana. Engineers get woken up for things that actually need them. The on-call rotation got quieter. Not because incidents disappeared — because the people closest to the problem could handle more of them on their own.</p><h4>What’s Next</h4><p>We standardized this in one service first. Then two. Then the whole product.</p><p>The next phase is federation: instead of each team managing their own Prometheus instance and their own Grafana folder, we’re moving toward a shared observability platform. One place to find metrics for every service. One alert rule format. One SOP that all L1 teams can follow regardless of which service is having a bad day.</p><p>The phase after that is correlation: connecting metrics to traces (via Datadog) so that when Grafana shows a timeout spike, you can click through to a distributed trace that shows you exactly which microservice in the chain started the delay. Metrics tell you <em>what</em>. Traces tell you <em>where in the chain</em>.</p><p>We’re not done. But we know what done looks like.</p><h4>Lessons Learned</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*6i5fVXNZMaYnE6kqRn33MA.png" /><figcaption>One of these is readable at 2 AM. The other is a puzzle. Observability is not the place for puzzles</figcaption></figure><p><strong>Name things for the people who will read them under pressure.</strong> If your metric name requires a senior engineer to interpret, it’s not observable — it’s encrypted.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*APqNu8G8tE-_Jd4qT2LQAQ.png" /><figcaption><em>She sees the difference. Especially at 2 AM. She definitely sees the difference.</em></figcaption></figure><p><strong>Detect with metrics, diagnose with logs.</strong> Trying to do both with the same tool means you’ll be bad at both.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qtifrDwo-aAL1wOC9XdYsw.png" /><figcaption>The metric tells you something is on fire. The SOP tells you where the extinguisher is.</figcaption></figure><p><strong>The SOP is part of the observability system.</strong> A metric without a runbook is a fire alarm with no evacuation plan.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*o44U5TCjSCrtWGnVeeVhuw.png" /><figcaption>Ship the 80%. The remaining 20% will tell you exactly what to build next</figcaption></figure><p><strong>Small instruments are better than no instruments.</strong> We didn’t build a perfect observability system. We built one that was 80% better than what we had. That 80% is where most of the incidents live.</p><h3>Key Takeaways</h3><ul><li>Custom metrics expose business truth; HTTP status codes expose protocol truth. They’re not the same thing.</li><li>/actuator/prometheus + Kubernetes pod annotations = zero-config scraping.</li><li>Metrics are cheap to store, fast to query, and legible to non-engineers. Logs are expensive to store, slow to query, and legible only to engineers with context.</li><li>The hybrid wins: metrics for detection, logs for diagnosis. Neither replaces the other.</li><li>Standardize the SOP before the next incident. Post-incident learning is valuable, but pre-incident preparation is cheaper.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zjSPBWMMPdN91vGZ.jpg" /></figure><p>If your on-call runbook starts with “open Kibana and start scrolling,” you might be in the same place I was. It’s fixable. And it doesn’t require rewriting your services from scratch.</p><p>It just requires asking: what does the business actually need to know — and are we measuring that, or are we measuring what’s convenient?</p><blockquote><strong><em>The logs know the answer. They’re just not telling you.</em></strong></blockquote><p>— <br><strong>🤝 Let’s Fix Something Together</strong></p><p>I wrote this because I lived it — and I’m still building in this space.</p><p>If your team is tired of “<em>root cause unclear</em>” post-mortems, if your L1 needs an engineer just to read an alert, or if your Elasticsearch bill is quietly eating your infrastructure budget — I’d love to talk.</p><p>I’m currently <strong><em>open to collaboration, new opportunities</em></strong><em> or </em><strong><em>just a conversation</em></strong> about why your monitoring stack is lying to you at 2 AM.</p><p>If this resonated, reach out. I work best with teams that are ready to build systems that actually tell the truth.</p><p><em>Find me on </em><a href="https://id.linkedin.com/in/pramuditya-ananta-nur-013859136"><em>LinkedIn</em></a><em> — or just leave a comment. The metrics will tell me if you showed up </em>😉<em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dda2f3fb8c07" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[2025 Retrospectives: Rascal Engineer point of view]]></title>
            <link>https://pramuditya.medium.com/2025-retrospectives-rascal-engineer-point-of-view-a1c7f605096d?source=rss-d7e5708a3155------2</link>
            <guid isPermaLink="false">https://medium.com/p/a1c7f605096d</guid>
            <category><![CDATA[teamwork]]></category>
            <category><![CDATA[retrospectives]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Pramuditya Ananta Nur]]></dc:creator>
            <pubDate>Wed, 31 Dec 2025 16:56:55 GMT</pubDate>
            <atom:updated>2025-12-31T16:56:55.866Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZZlmTgPVdHc-OzzJ" /><figcaption>Photo by <a href="https://unsplash.com/@clemono?utm_source=medium&amp;utm_medium=referral">Clem Onojeghuo</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>As I look back at 2025, the most significant refactoring didn’t happen in my IDE — it happened in my mindset. I spent this year transitioning from a “<em>Builder</em>” who obsesses over technical depth to an “<em>Owner</em>” who understands leverage.</p><p>After sitting down with my “<em>intellectual sparring partner</em>” to look toward my future self, these are the hard-won truths that defined my year.</p><h4>Leverage Over Depth</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wVUcSfxzauW9H6Lq" /><figcaption>Photo by <a href="https://unsplash.com/@mitchel3uo?utm_source=medium&amp;utm_medium=referral">Mitchell Luo</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><blockquote>Your growth ceiling is no longer technical depth — it’s leverage.</blockquote><p>My growth ceiling is no longer about how deep I can go into a specific framework. It’s about leverage. Leverage means:</p><ul><li>Explaining complex things so simply that others can act on them.</li><li>Designing systems that others can safely use without me being in the room.</li><li>Teaching without the “hahhh” frustration.</li></ul><p>I’ve learned that <em>you don’t win by being right; you win by making others right with you</em>. If the team succeeds because of a framework I built, that is a 10x return on my time compared to just finishing a ticket.</p><h4>The Trap of Cleverness</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WolUAV9eN4k2H9h0" /><figcaption>Photo by <a href="https://unsplash.com/@abengs84?utm_source=medium&amp;utm_medium=referral">Anders Bengs</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><blockquote>Clever systems impress engineers. Boring systems survive years.</blockquote><p>In the past, I sought to prove my intelligence through complex systems. In 2025, I realized that if you can’t explain your architecture simply, you don’t own it yet. I’ve shifted my focus from proving to look as the smartest person in the room to proving the most reliable. Reliability is quiet; cleverness is loud. At this stage of my journey, I prefer the silence of a system that just works.</p><h4>The People Protocol</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CDJD0yWNJ_YW4vDl" /><figcaption>Photo by <a href="https://unsplash.com/@ryoji__iwata?utm_source=medium&amp;utm_medium=referral">Ryoji Iwata</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><blockquote>Not everyone is slow — they’re optimizing for a different incentive</blockquote><p>I used to think “<em>people problems</em>” were distractions from “<em>real work</em>.” Now I know that my future stress won’t come from bugs — it will come from people<strong>.</strong></p><p>I’ve had to accept that not everyone is “slow”; they are often just optimizing for different incentives. When I feel irritated that an environment rewards speed over correctness, I’ve learned to name the <em>compromise</em>. What hurts is not the compromise itself — it’s pretending it doesn’t exist<strong>.</strong> By calling out trade-offs instead of just saying “<em>this is wrong</em>,” I’ve maintained my shield while staying intellectually honest.</p><h4>Guarding Against My Own Bias</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Txv2GyOjp5Wfvm5z" /><figcaption>Photo by <a href="https://unsplash.com/@christianlue?utm_source=medium&amp;utm_medium=referral">Christian Lue</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><blockquote>You think you’re the only one who sees the “right” way</blockquote><p>As I’ve grown more confident, the risk of confirmation bias has increased. I often think I’m the only one who sees the “<em>right</em>” way. My retrospective for 2025 has been a constant exercise in calling out my own bias.</p><p><em>Code rots, but reasoning scales</em>. My notes on why I made a decision have become more valuable than the lines of code I wrote. I refuse to become “<em>efficient but shallow</em>.” I will keep asking why, but <em>I have learned to stop weaponizing that question against my colleagues</em>. — My apologies if my feedback hit a sarcastic chord; I’m just deeply invested in getting the logic right. To be honest, I’m still wrestling with whether this is the optimal decision, and I’d value a quick stress test on the current plan 😊</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rGCmxXVvj6e1DSlNOYEAEA.png" /></figure><p>If I had to summarize 2025 in one sentence: I am moving from being a builder of things to an owner of outcomes. — <em>I’m re-evaluating my current decision because I’m looking at the long-term outcome, and frankly, the math isn’t adding up yet</em> 😉</p><p>This transition is uncomfortable. It is quieter. It is heavier. It requires being calm under stupidity and influential when the stakes are high. I have the architecture now. I have the skills. I have the <em>shield</em>.</p><p>The 2025 journey has taught me to stop worrying about the world’s response and start focusing on my own initiate action<strong>.</strong> The architecture is ready, and the load is look like “manageable”.</p><p>The question for 2026 is no longer whether the system can handle the load. The question is: <strong>What am I going to build that actually matters?</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a1c7f605096d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Art of the Impossible: How I Rescued a Microservice from the Performance Abyss]]></title>
            <link>https://pramuditya.medium.com/the-art-of-the-impossible-how-i-rescued-a-microservice-from-the-performance-abyss-12fea9be707d?source=rss-d7e5708a3155------2</link>
            <guid isPermaLink="false">https://medium.com/p/12fea9be707d</guid>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[performance-tuning]]></category>
            <category><![CDATA[jmeter]]></category>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[microservices]]></category>
            <dc:creator><![CDATA[Pramuditya Ananta Nur]]></dc:creator>
            <pubDate>Fri, 15 Nov 2024 06:32:39 GMT</pubDate>
            <atom:updated>2024-11-28T00:00:36.546Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*31TppNLrfMJBX7po" /><figcaption>Photo by <a href="https://unsplash.com/@carlheyerdahl?utm_source=medium&amp;utm_medium=referral">Carl Heyerdahl</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Let’s face it, we’ve all been there: a sluggish microservice. But fear not! With my specialized magic, I turned that sluggish beast into a performance powerhouse.</p><h4>Old Schema</h4><p>this microservice is related to <a href="https://medium.com/@pramuditya/a-tale-of-how-i-tame-memory-glutton-system-92f212312df2"><em>my previous post</em></a>, which is to generate a random unique number for each category. For each random unique number that has been used, it’ll be marked and inserted into a <em>used_sequence</em> table. Here is the previous schema the microservice used</p><pre>CREATE TABLE IF NOT EXISTS public.used_sequences<br>(<br>    category varchar(20),<br>    used_sequence bigint,<br>    CONSTRAINT used_sequences_pkey PRIMARY KEY (category, used_sequence)<br>);</pre><p>in this old schema, I’m doing 3 performance tests using JMeter using 3 different scenarios: 100 threads for 5 mins, 150 threads for 5 mins, and the last using 200 threads for 5 mins, here is the result for 3 scenarios using old schema:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b4ivD0R_c1HEYAOzN27Dhg.jpeg" /><figcaption>100 threads for 5 mins using old schema</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SFDx0T6xBlr1Rx6aTk4-0Q.jpeg" /><figcaption>150 threads for 5 mins using old schema</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z4e4OAvvdUujM6ktN0tgcQ.jpeg" /><figcaption>200 threads for 5 mins using old schema</figcaption></figure><p>The experiment revealed a significant degradation in performance metrics. As concurrency increased, the system exhibited signs of resource contention and scheduling overhead, leading to a substantial decline in throughput by 2.5x and a marked increase in response latency by 4x. A clear indication that our microservice was not optimized for high-throughput, low-latency workloads. Which mean the current microservices did not work well under an oppressive workload and stressful environment. 🤪</p><h4>New Schema</h4><p>Now, I’m tuning the table schema, changing from using composite primary key to sequence primary key, and also adding unique index constraint for <em>category</em> and <em>used_sequence</em> column.</p><pre>CREATE TABLE IF NOT EXISTS public.new_used_sequences<br>(<br>    id bigserial,<br>    category varchar(20),<br>    used_sequence bigint,<br>    created_date timestamp without time zone,<br>    modified_date timestamp without time zone,<br>    version integer,<br>    CONSTRAINT new_used_sequences_pkey PRIMARY KEY (id)<br>);<br><br>CREATE UNIQUE INDEX IF NOT EXISTS new_used_sequences_uidx<br>    ON public.new_used_sequences USING btree<br>    (category COLLATE pg_catalog.&quot;default&quot; ASC NULLS LAST, used_sequence ASC NULLS LAST);</pre><p>And the showtime, I’m using the same scenarios as before to do the performance test.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J1HqhmzffmwDaQLQ7Tdq6g.jpeg" /><figcaption>100 threads for 5 mins using new schema</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zc1kyHZtD-iw137W3fS4cg.jpeg" /><figcaption>150 threads for 5 mins using new schema</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FIRvvjMSQpiKJ8YRMDVuuw.jpeg" /><figcaption>200 threads for 5 mins using new schema</figcaption></figure><p>The microservice exhibited steady throughput under moderate load, with a manageable <strong>2x</strong> increase in latency. The <em>99th</em> percentile response time, however, experienced a significant reduction, plummeting from a sluggish <strong>6 seconds</strong> to a brisk <strong>1.1 seconds</strong>. When faced with a 2x initial workload, the system’s performance soared, achieving a <strong>6x </strong>throughput increase, showcasing its resilience and efficiency.</p><h4><strong>Conclusion</strong></h4><p>Here are a few points why just changing the schema can have <strong>6x</strong> better results:</p><ul><li>Composite indexes are generally larger and more complex than single-column indexes. This increased size can impact disk I/O and memory usage, especially during insert operations.</li><li>PostgreSQL needs to maintain and update indexes during insert, update, and delete operations. Composite indexes require more extensive maintenance, potentially slowing down these operations.</li><li>When inserting a new row with a composite primary key, PostgreSQL must check for uniqueness across all columns in the composite key. This involves a more complex query and potentially more disk I/O.</li><li>PostgreSQL can optimize storage layout by clustering rows based on their primary key. A sequence-based primary key often allows for efficient clustering, as new rows are inserted sequentially.</li><li>Composite primary keys can lead to more frequent page splits, as rows may not be inserted in a sequential order based on the primary key. Page splits can degrade performance, especially during high-write workloads.</li><li>While composite primary keys offer unique identification based on multiple columns, they can introduce performance overhead, especially during high-write workloads. In many cases, a sequence-based primary key can provide a good balance between performance and data integrity.</li></ul><h4><strong>What’s next?</strong></h4><p>The next part I want to try is to optimize the code and scalability part.<br>See ya, and happy working ~</p><h4>Outro</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7uc6w-9XA01TFUXU.jpg" /></figure><blockquote><strong>I’m not just a software engineer, I’m a performance artist. And when the servers are under load? That’s when the real show begins. 😎</strong></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=12fea9be707d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A tale of how I tame memory glutton system]]></title>
            <link>https://pramuditya.medium.com/a-tale-of-how-i-tame-memory-glutton-system-92f212312df2?source=rss-d7e5708a3155------2</link>
            <guid isPermaLink="false">https://medium.com/p/92f212312df2</guid>
            <category><![CDATA[memory-management]]></category>
            <category><![CDATA[jvm]]></category>
            <category><![CDATA[system-optimization]]></category>
            <category><![CDATA[resource-optimization]]></category>
            <category><![CDATA[tech-debt]]></category>
            <dc:creator><![CDATA[Pramuditya Ananta Nur]]></dc:creator>
            <pubDate>Wed, 28 Aug 2024 03:03:40 GMT</pubDate>
            <atom:updated>2024-08-28T03:03:40.527Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*BhnbePZdPTkn7vSk.jpg" /><figcaption>out of memory system</figcaption></figure><p>I recently embarked on a mission to optimize a memory-hungry microservice responsible for generating unique random numbers within specific ranges. As the data volume surged, the system’s memory footprint ballooned to a staggering ~8GB on startup. It was time to tame this memory glutton before it devoured resources intended for other critical services.</p><h4>Overview of System</h4><p>The system is designed to generate unique random numbers within specified ranges for various categories. Each category has a predefined maximum range and may have reserved certain numbers and will be used if some kind of criteria match. Here is the constraint of the system:</p><ul><li>this system must generate unique numbers in ranges for certain categories, let’s say I have 8 categories [A, B, C, D, E, F, G, H].</li><li>each category has a maximum range number, ex: “A” from 0 to 9.999.999, “B” from 1.000.000 to 999.999.999 and so on</li><li>each category has preserved a unique number, if any. ex: “A” has preserved from a range of 100.000 to 500.000 unique numbers and will be used if some kind of criteria match.</li></ul><p>Upon the startup, the system:</p><ul><li>Retrieves existing used numbers for each category from a database.</li><li>then, it caches “n” of a unique number that doesn&#39;t exist in previous steps in memory.</li><li>then, it also queries the database and finds if this category has preserved a unique number that has not been used yet, if any it will also stored in memory.</li></ul><p>When an external system requests a unique number for a category, the system:</p><ul><li>Prioritizes preserved numbers if match some criteria.</li><li>If no preserved numbers are available, it retrieves a unique number from the cache and marks it as used.</li><li>A scheduler is in place to periodically replenish the cached unique numbers when their count falls below a predetermined threshold.</li></ul><h4>Testing</h4><p>To test the system’s capabilities, I simulated a demanding scenario with 8 categories, each containing 1 million used unique numbers and 1000 reserved sequences.</p><p>The original system struggled:</p><ul><li>taking a sluggish <strong>56</strong> seconds to start up and reaching a CPU peak of <strong>92%</strong></li><li>Memory consumption was equally concerning, with a hefty <strong>1.46GB </strong>allocated for objects and a staggering <strong>7.39GB</strong> for database queries.</li></ul><figure><img alt="startup time and cpu peak" src="https://cdn-images-1.medium.com/max/1024/1*DsO2Gm4DQ-nuThh6wr6MLQ.png" /><figcaption>startup time and cpu peak</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IpmXLNNIFyfkv3YxduS8wg.png" /><figcaption>memory allocation of object</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FuE9q9N6ffO0n4dFoaP4YA.png" /><figcaption>memory allocation to query used unique number</figcaption></figure><p>After implementing optimizations, the system’s performance underwent a transformation:</p><ul><li>Startup time plummeted to a mere <strong>5</strong> seconds, and CPU usage dropped to a manageable <strong>58%.</strong></li><li>Memory allocation for objects shrank to a more reasonable <strong>348.89MB</strong>, and database query costs decreased to <strong>2.31GB</strong>. These improvements demonstrate the system’s newfound efficiency and ability to handle demanding workloads.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZTCQ6oCd-VPkkYg2WkxHJg.png" /><figcaption>startup time and cpu peak</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ii3u4GtkBemOeZImOSkTdg.png" /><figcaption>memory allocation of object</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w2j6wUPEvQA6ShQ2UlzfJw.png" /><figcaption>memory allocation to query used unique number</figcaption></figure><p>I also tested the scenario with 8 categories, each having 1 million remaining unique numbers and 9 million records of each category in the database, the system generated unique numbers for all categories within 3 minutes. This performance was deemed acceptable considering the constraints.</p><h4>Optimization</h4><p>To optimize the system, I implemented the following strategies:</p><ul><li><strong>Selective Database Queries:</strong> By querying only the necessary columns (in this case, the used sequences), I significantly reduced the amount of data transferred between the system and the database, resulting in faster query times and lower memory usage.</li><li><strong>Cache Management:</strong> After generating unique numbers and storing them in the cache, I cleared the <em>usedSequenceList</em> variable to free up memory and prevent unnecessary data retention.</li></ul><h4>Conclusion</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/444/0*Hl85K3-mQ0zHD82h.png" /><figcaption>select * is the root of all evil. lol.</figcaption></figure><p>By implementing targeted optimizations, I was able to improve the performance of the system responsible for generating unique random numbers<strong>.</strong></p><p>Key strategies included optimizing database queries to retrieve only necessary data, effectively managing the cache of unique numbers, and minimizing unnecessary data retention. These optimizations resulted in significant reductions in startup time, CPU usage, and memory consumption, enabling the system to handle demanding workloads more efficiently and effectively.</p><p>See ya.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=92f212312df2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Effortless Error Resilience: Simplifying Kafka DLT Handling]]></title>
            <link>https://pramuditya.medium.com/effortless-error-resilience-simplifying-kafka-dlt-handling-f1c0f0902b3a?source=rss-d7e5708a3155------2</link>
            <guid isPermaLink="false">https://medium.com/p/f1c0f0902b3a</guid>
            <category><![CDATA[java]]></category>
            <category><![CDATA[spring-boot]]></category>
            <category><![CDATA[dlt]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[reactive]]></category>
            <dc:creator><![CDATA[Pramuditya Ananta Nur]]></dc:creator>
            <pubDate>Sun, 28 Jan 2024 04:16:59 GMT</pubDate>
            <atom:updated>2024-01-28T04:16:59.240Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nRsU_ADjs0hqWmY3" /><figcaption>Photo by <a href="https://unsplash.com/@i_am_simoesse?utm_source=medium&amp;utm_medium=referral">Simona Sergi</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>Dead Letter Topic, What is it?</strong></p><p>Imagine you’re the mail carrier of a bustling city, and you’ve got a job to deliver letters to various addresses. Now, in our Kafka world, messages are like these letters, and topics are like different neighborhoods.</p><p>The Dead Letter Topic in Kafka is like the place where undeliverable letters end up — it’s the lost and found for messages that couldn’t reach their destination. We call it the “Limbo Lounge for Lost Letters.”</p><p>So, let’s say you’re trying to send a message (letter) to a topic (neighborhood), but there’s something funky about it — maybe the address is wrong, or the recipient isn’t available. Instead of letting it get lost forever, Kafka is a responsible mail service. It reroutes these undeliverable messages to the Dead Letter Topic, where they can hang out until someone figures out what went wrong.</p><p>In essence, the Dead Letter Topic is Kafka’s safety net, catching those messages that didn’t quite make it to their intended destination, giving you a chance to inspect, debug, and figure out why your messages are taking unexpected detours.</p><p><strong>How I handle it?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*97uJdmojewdV68pixCQJhg.png" /><figcaption>kafka dlt flow illustration</figcaption></figure><p>Pay attention on illustration above:</p><ul><li>in the event of the receiver encountering a processing failure with an event topic, it gracefully appends a “<em>-retry</em>” suffix to the topic, accompanied by “<em>retryTime”</em> headers reflecting the retry counter.</li><li>this sequence concludes when certain conditions are met — either the receiver successfully processes the retry topic, or it reaches the maximum retry counter without successfully processing the event.</li><li>upon reaching the maximum retry counter with the receiver unable to process the event, a denouement unfolds: the publication of a “<em>-dlt</em>” topic. Your system can leverage this dlt topic to trigger alerts or other pertinent actions.</li></ul><p>Through this intricately designed mechanism, we establish a robust event consumption process, complete with a safety net for potential errors. However, it is imperative to ensure that our receiver adeptly manages both the original topic and the retry topic as idempotent processes. This guarantees a resilient and fault-tolerant event handling system.</p><p><strong>Talk is cheap, show me the code</strong></p><ul><li>To make things a breeze, I’ve whipped up a Java library using Reactor Kafka. Using it is a walk in the park — just toss this dependency into your Maven project. But before that make sure your project meet this requirements:<br>- Java 17<br>- Spring 2.7.16<br>- PostgreSQL</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/df59d6471712ff7e5a3f1453983a787d/href">https://medium.com/media/df59d6471712ff7e5a3f1453983a787d/href</a></iframe><ul><li>Then, sprinkle a little magic by configuring your Kafka host for both publishers and subscribers on <em>application.yml</em></li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/29aed77038b9408c9c697aaaaf09d470/href">https://medium.com/media/29aed77038b9408c9c697aaaaf09d470/href</a></iframe><ul><li>Write some nifty code to juggle events and retry them. Oh, and let’s not forget the safety net for those “oops” moments with DLT events.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0315a49584f8d7fcde63fe29b0d8268a/href">https://medium.com/media/0315a49584f8d7fcde63fe29b0d8268a/href</a></iframe><ul><li>And voila! You’re done, it’s as easy as pie. If you feel like stirring the pot with some retry or DLT events, just toss an exception into the <strong>handleEvent</strong> reactive chain, and it automatically publish retry / dlt event based on <em>retryTime</em> header.</li></ul><p><strong>Advance Configuration</strong></p><p>this is the detail of available configuration for this library</p><ul><li>publisher configuration</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/910d8e381c0534cd6cbab35797a81b24/href">https://medium.com/media/910d8e381c0534cd6cbab35797a81b24/href</a></iframe><ul><li>subscriber configuration</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f0e71c3c0fc7c8bbf1ab7f9719c19223/href">https://medium.com/media/f0e71c3c0fc7c8bbf1ab7f9719c19223/href</a></iframe><p><strong>Conclusion</strong></p><p>In a nutshell, that’s my secret sauce for gracefully handling those hiccups in Kafka event consumption. For a peek at the action, swing by <a href="https://github.com/nantaaditya/example-app/tree/feature/kafka">my GitHub project</a>. Lastly, if you feel that this library has been helpful, Maybe you can support a cup of coffee for me <a href="https://paypal.me/nantaaditya"><strong>here</strong></a><strong>. </strong>Catch you later!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f1c0f0902b3a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unlocking the Native Image Enigma]]></title>
            <link>https://pramuditya.medium.com/unlocking-the-native-image-enigma-082e3a9e46d2?source=rss-d7e5708a3155------2</link>
            <guid isPermaLink="false">https://medium.com/p/082e3a9e46d2</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[native-image]]></category>
            <category><![CDATA[spring-boot]]></category>
            <category><![CDATA[developer-experience]]></category>
            <category><![CDATA[graalvm]]></category>
            <dc:creator><![CDATA[Pramuditya Ananta Nur]]></dc:creator>
            <pubDate>Mon, 06 Nov 2023 04:41:48 GMT</pubDate>
            <atom:updated>2023-11-06T04:41:48.473Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/705/0*xt9EKhzjQLtH5ZkR.jpg" /></figure><p>So, you’re developing some cool stuff, and you’ve heard about this magical thing called “<strong>native-image</strong>.” Well, buckle up, because we’re about to take you on a wild ride through the native-image wonderland.</p><p>Native-image is like that one quirky friend who always brings something interesting to the party. It takes your Java applications and compiles them into a nifty native binary. No more JVM startup lag — we’re in the fast lane now!</p><p>Picture this: You’re coding away, and suddenly, your app starts up so fast that you’ll wonder if it had one too many espressos. The only problem is that your app is now a binary file, and you’re pretty sure it’s trying to challenge your C++ colleagues to a speed race.</p><p>But here’s the catch — native-image can be a bit of a diva. It’s picky about what libraries it wants to play with. It’s like that one kid at the playground who insists on having the fanciest toys. If your code uses something exotic, native-image might throw a tantrum and refuse to cooperate.</p><p>For this thrilling test drive, I’ve crafted a humble Spring project that executes a database DDL migrator, which you can find on <a href="https://github.com/nantaaditya/db-migration/tree/feature/native-image">my GitHub</a>.</p><ul><li>Our quest begins with the quest to compare the build times of native images and their non-native counterparts within Docker. Now, let’s roll up our sleeves and dive into the comparison:</li></ul><pre>mvn install -DskipTests<br>docker build -f .docker/Dockerfile -t nantaaditya/db-migration:0.0.1 .</pre><p>it tooks around 5.354s, from maven install ‘till build the docker image</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*llPi6x3BUOk53D-58KilEA.png" /><figcaption>mvn install build time</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j3N99BvFPrtzaJ_KZRoWLg.png" /><figcaption>non-native image build time</figcaption></figure><p>and let’s try for native image build:</p><pre>docker buildx build -f .docker/NativeDockerfile -t nantaaditya/db-migration:0.0.1-native .</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V0fDKNCY-uDKR2Stdam8zA.jpeg" /><figcaption>native image build time</figcaption></figure><p>When it comes to native images, we embarked on an epic build journey that lasted a whopping 485.2 seconds. That’s right, you read it correctly — it’s bordering on the realm of pure craziness!</p><p>Now, what can we deduce from this eye-popping build time? Well, let’s not beat around the bush — native image build times are no sprint; they’re more like a marathon.</p><p>In this light, it’s safe to say that, for our development needs, native image may not be the knight in shining armor we hoped for. Why, you ask? Because spending such an eternity on the build process is like watching paint dry.</p><p>Imagine if you have a treasure trove of features waiting to be brought to life. You’d be burning daylight, building and testing away, without much progress to show for it. It’s like trying to cross a vast desert with a teaspoon — not the most efficient way to quench your thirst for development.</p><p>So, as we gaze upon the clock ticking away, it’s evident that for development, the native image journey might require some serious time investment. Is it worth it, or are there alternative routes to explore? That’s the million-dollar question we’ll tackle next.</p><ul><li>Now, let’s venture into the thrilling arena of startup time and resource consumption, pitting our contenders, the non-native image and the native image, against each other. Buckle up!<br>To fire up the non-native image and see how fast it revs its engine, use this command:</li></ul><pre>docker run --env-file .env/local.env --name db-migration -p 1000:1000 nantaaditya/db-migration:0.0.1</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5eYUJgOzH1viVq3ib0npRw.jpeg" /><figcaption>non-native image startup time</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_FI1uJWhhtXh_xecHYsl6w.jpeg" /><figcaption>non-native image resource consumption</figcaption></figure><p>For the non-native image, we clocked a startup time of approximately 2.9 seconds. But wait, there’s more! The resource consumption for a single API request on the CPU soared to around 15%, and the idle memory usage settled at a hefty 432MB. It’s as if our typical Java program is on an all-you-can-eat resource buffet!</p><p>But hold onto your hats, because the native image is about to give us a run for our money. Running it with this command:</p><pre>docker run --env-file .env/local.env --name db-migration-native -p 1000:1000 nantaaditya/db-migration:0.0.1-native</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3CKvpWbgkjTxcaV5gMRr1Q.jpeg" /><figcaption>native image startup time</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tLgBN_zYMbS__QJbEcZrwg.jpeg" /><figcaption>native image resource consumption</figcaption></figure><p>Prepare to be amazed! The startup time for the native image is a lightning-fast 0.27 seconds. It’s so quick it might make your coffee seem slow! And the resource consumption for the CPU? A mere 0.4%, making it almost look like it’s on a coffee break. The idle memory usage is a svelte 82MB, giving us the feeling that our application is practicing a little Marie Kondo tidying up.</p><p>So, in the startup time and resource consumption showdown, the native image emerges as a true speedster and a resource-saver. It’s like watching a race between a hare and a tortoise, with the hare sprinting to victory in record time.</p><p>When we put these two points of comparison under the microscope, the question that looms large is, “Is it worth considering for production use?”</p><p>In my humble opinion, it all comes down to your priorities and trade-offs. If you’re willing to make a deal with the time-devouring build process, the native image could be a production contender.</p><p>However, when it comes to development, it’s a different story. The extended build times may make it an unwelcome guest at the development party. After all, time is of the essence when you’re experimenting, iterating, and crafting new features.</p><p>So, in the grand scheme of things, the decision hinges on your specific needs and the role you envision for native images in your software journey. Production, with its focus on performance and efficiency, might find a good companion in native images, while the swift pace of development could prefer lighter, more agile alternatives.</p><p>Choose your path wisely, my coding comrades, and may your decisions lead to tech adventures that serve you well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/0*_cNVrGYEJP9UrKI6.jpg" /><figcaption>native-image cost</figcaption></figure><p>In conclusion, native-image is like that rollercoaster you can’t resist riding, even though you know it’s going to be a wild and unpredictable journey. So, if you’re ready to take on the challenge and add a touch of chaos to your dev life, give native-image a whirl!</p><p>Just remember, when your app becomes a binary superhero, don’t be surprised if it starts wearing a cape.</p><p>Happy coding, my engineering comrades, and may the native-image odds be ever in your favor!</p><p>Next, I will cover about the performance for this two image and do strest test, So don’t miss it out. Thank you.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=082e3a9e46d2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Big Data Conundrum: How to Tame Your Big Data]]></title>
            <link>https://blog.devgenius.io/the-big-data-conundrum-how-to-tame-your-big-data-243fcf951be?source=rss-d7e5708a3155------2</link>
            <guid isPermaLink="false">https://medium.com/p/243fcf951be</guid>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Pramuditya Ananta Nur]]></dc:creator>
            <pubDate>Thu, 24 Aug 2023 10:16:01 GMT</pubDate>
            <atom:updated>2023-08-24T10:16:01.211Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xPviM1Gs5uhrkfhf" /><figcaption>Photo by <a href="https://unsplash.com/@tofi?utm_source=medium&amp;utm_medium=referral">Tobias Fischer</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Database is a crucial component in any application, as it serves to store important data which can be retrieved and processed as needed. However, as an application evolves over time with the addition of features or schema, the amount of data stored in a database can grow significantly. As the size of the database grows, several issues can arise, including the slow query processing time, despite the use of indexing. Additionally, inserting data into a database with a large volume of data can also become a slow and cumbersome process. In light of these challenges, several solutions can be implemented to address these issues.</p><ol><li><strong>Index</strong></li></ol><p>Indexing is a first option that you must consider on large tables because it can significantly improve the performance of queries and data retrieval operations. In a large table, searching for data can be time-consuming and resource-intensive, especially if the table contains millions or billions of rows. An index is a data structure that is created on one or more columns of a table, which allows the database engine to quickly locate the rows that match a query or data retrieval operation. This can lead to faster query performance, reduced I/O operations, improved concurrency, and faster data retrieval.</p><p>However, indexing also comes with some trade-offs, such as increased disk space and maintenance overhead, as well as potential performance issues if the index is not designed or used properly. Therefore, when working with large tables, it is important to carefully consider the use of indexing and its potential benefits and drawbacks. If you want to know best practice on how to choose an index, read <a href="https://medium.com/dev-genius/sql-101-index-c293e973dd8">my previous post</a>.</p><p>2. <strong>Archiving</strong></p><p>Archiving data is an important consideration for organizations that need to manage large volumes of data efficiently. By archiving older data that is no longer actively used, or separating active transactional data from completed transactions, organizations can free up valuable resources and improve the overall performance of their database.</p><ul><li>Improved query performance: Archiving older data from a big table can improve query performance by reducing the amount of data that needs to be scanned by the database engine. This can speed up query processing times and improve overall application performance.</li><li>Reduced storage costs: Archiving older data from a big table can also reduce storage costs by freeing up disk space that is no longer needed. This can be especially important for large tables that are growing rapidly, as it can help to keep storage costs under control.</li><li>Regulatory compliance: In some industries, such as finance or healthcare, regulations may require organizations to retain data for a certain period of time. Archiving older data from a big table can help organizations to meet these requirements without incurring excessive storage costs.</li><li>Simplified data management: Archiving older data from a big table can simplify data management by reducing the amount of data that needs to be backed up, migrated, or otherwise managed. This can make it easier for IT teams to manage the data lifecycle and ensure that critical data is properly protected and available when needed.</li></ul><p>Overall, archiving data on a big table can offer several benefits, including improved query performance, reduced storage costs, regulatory compliance, and simplified data management. It is an essential practice for organizations that need to manage large volumes of data efficiently while maintaining data quality and compliance.</p><p>3. <strong>Partitioning</strong></p><p>Partitioning tables is a powerful tool for organizations that need to manage large tables efficiently and improve database performance. By dividing a table into smaller, more manageable pieces, partitioning can help to optimize query performance and reduce latency, while also allowing for more efficient use of resources. <br>However, choosing the right partitioning strategy can be a complex process that requires careful consideration of a range of factors, including data access patterns, data size, and hardware resources. Best practices for choosing a partitioning strategy include understanding the unique needs of your organization and its data, carefully evaluating the available partitioning options, and working with experienced professionals to design and implement a partitioning strategy that is tailored to your specific needs. With the right approach, partitioning can be an effective tool for managing large tables and optimizing database performance, helping to ensure that your organization can continue to meet its data management needs as it grows and evolves over time.</p><ul><li>Improved query performance: Partitioning a large table can improve query performance by dividing the table into smaller, more manageable partitions. This allows the database engine to scan only the relevant partitions when processing a query, which can significantly reduce the amount of data that needs to be scanned and improve query performance.</li><li>Simplified data management: Partitioning a large table can simplify data management by allowing IT teams to manage and maintain smaller partitions instead of a single large table. This can make it easier to optimize and tune the database for performance, backup and restore data more efficiently, and manage the data lifecycle more effectively.</li><li>Increased scalability: Partitioning a large table can also increase scalability by allowing IT teams to add or remove partitions as needed to accommodate growth or changing business needs. This can help to avoid performance issues and ensure that the database can continue to meet the needs of the organization as it grows.</li><li>Better resource utilization: Partitioning a large table can improve resource utilization by enabling IT teams to allocate resources, such as memory or CPU, to specific partitions. This can help to ensure that resources are used more efficiently and that performance is optimized for each partition.</li></ul><p>Overall, partitioning a table is an important strategy for managing large tables efficiently and effectively. It can improve query performance, simplify data management, increase scalability, and improve resource utilization. As such, it is an essential practice for organizations that need to manage large volumes of data efficiently and ensure that their databases can continue to meet their needs as they grow and evolve.</p><p>4. <strong>Scaling</strong></p><p>Managing large amounts of data can be a challenge for any organization, particularly when it comes to optimizing database performance. While strategies such as adding indexes, archiving, and partitioning data can help to improve performance in some cases, these approaches may not always provide the desired results. In such situations, scaling the database may be the ultimate solution for managing large data and ensuring optimal performance.</p><p>Scaling a database involves increasing its capacity to accommodate growing amounts of data and support more users and transactions. This can be achieved through a range of techniques, including vertical scaling, horizontal scaling, and distributed databases.</p><p>Vertical scaling involves increasing the resources of a single database server, such as adding more memory or processing power, to improve performance. This approach is often used for databases that have reached the limit of their current hardware capabilities.</p><p>Horizontal scaling, on the other hand, involves adding more servers to distribute the workload and increase capacity. This approach is often used for databases that need to handle a large number of concurrent transactions or support a growing number of users.</p><p>Distributed databases involve spreading data across multiple servers or nodes, allowing for greater scalability and improved performance. This approach is often used for organizations that require high levels of availability, as well as those that need to process large amounts of data quickly.</p><p>While scaling a database can be a complex process, it offers numerous benefits for organizations that need to manage large amounts of data. These benefits include:</p><ul><li>Improved performance: Scaling a database can help to improve query response times, reduce latency, and ensure that the database can handle a growing volume of transactions and users.</li><li>Greater flexibility: Scaling a database can allow for greater flexibility in terms of how data is stored and accessed, as well as how resources are allocated and used.</li><li>Higher availability: Scaling a database can help to ensure that data is available when it is needed, even in the face of unexpected events or high levels of demand.</li><li>Lower costs: Scaling a database can help to reduce costs associated with managing and maintaining hardware and software, as well as improving overall resource utilization.</li></ul><p>When it comes to choosing a scaling strategy, organizations should consider a range of factors, including their unique data management needs, hardware capabilities, and budget constraints. In conclusion, managing large amounts of data can be a complex and challenging task, particularly when it comes to optimizing database performance. While strategies such as adding indexes, archiving, and partitioning data can help in some cases, scaling the database may be the ultimate solution for organizations that need to manage large data and ensure optimal performance. By choosing the right scaling strategy and working with experienced professionals, organizations can ensure that their databases can continue to meet their needs as they grow and evolve over time.</p><p>5. <strong>Sharding</strong></p><p>Sharding must be the last option you must consider, an approach to database partitioning that involves dividing a large table into smaller, more manageable pieces that can be stored on different servers. Sharding can be a useful strategy for managing large tables because it allows for more efficient use of resources, improved scalability, and better performance.</p><p>The main advantages of sharding are:</p><ul><li>Improved performance: By distributing data across multiple servers, sharding can help to improve query performance and reduce latency. This is particularly important for large tables that would otherwise require a significant amount of time to search through.</li><li>Improved scalability: Sharding allows for more efficient use of resources, which can help to improve scalability and handle increasing amounts of data over time. This can be particularly useful for growing businesses that need to accommodate increased traffic and demand.</li><li>Better resource utilization: By dividing data across multiple servers, sharding can help to improve resource utilization and optimize the use of available resources such as CPU, memory, and storage.</li></ul><p>However, there are also some potential drawbacks to sharding that should be considered:</p><ul><li>Increased complexity: Sharding can add complexity to a database architecture, which can make it more difficult to manage and maintain over time. This can be particularly challenging for organizations that lack the necessary expertise or resources to manage a sharded database effectively.</li><li>Increased risk: Sharding can also increase the risk of data loss or corruption, particularly if data is not properly managed or replicated across multiple servers.</li><li>Higher costs: Sharding can also be expensive, particularly if additional hardware or infrastructure is needed to support the sharded database. This can be a significant cost for smaller organizations or those with limited resources.</li></ul><p>Overall, sharding can be a useful strategy for managing large tables, but it is important to weigh the potential benefits and drawbacks before implementing it. Organizations should carefully consider their specific needs and resources before deciding whether to pursue sharding as a solution for managing their databases.</p><p>In conclusion, managing big data can be a daunting task, but with the right strategies in place, businesses can effectively harness the power of data without sacrificing performance. Additionally, leveraging distributed processing and parallel computing frameworks can help distribute workloads and accelerate data processing. If you have any strategy to handle big data, please share your strategy on the comment section. Thank you.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=243fcf951be" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/the-big-data-conundrum-how-to-tame-your-big-data-243fcf951be">The Big Data Conundrum: How to Tame Your Big Data</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Level-Up your Spring Integration Test]]></title>
            <link>https://blog.devgenius.io/level-up-your-spring-integration-test-942ab6f9ccab?source=rss-d7e5708a3155------2</link>
            <guid isPermaLink="false">https://medium.com/p/942ab6f9ccab</guid>
            <category><![CDATA[spring-framework]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[integration-testing]]></category>
            <category><![CDATA[sping-boot]]></category>
            <category><![CDATA[junit-5]]></category>
            <dc:creator><![CDATA[Pramuditya Ananta Nur]]></dc:creator>
            <pubDate>Mon, 06 Mar 2023 04:56:46 GMT</pubDate>
            <atom:updated>2023-03-06T04:56:46.403Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UBhPJSBgHjN5LDlA" /><figcaption>Photo by <a href="https://unsplash.com/@austindistel?utm_source=medium&amp;utm_medium=referral">Austin Distel</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>As a software engineer, I’ve learned that one of the most arduous tasks in developing a program is testing it. Crafting code for testing requires preparing data, which can be tedious and time-consuming. And let’s be honest, sometimes the data we create doesn’t fully reflect the program’s intended functionality. To add to the challenge, understanding test cases can be difficult, especially when some engineers prioritize code coverage over testing for functional requirements. I’ll admit, I’ve been guilty of this myself!</p><p>But perhaps the most frustrating aspect of creating unit tests is when the component being tested relies on other components. This means we’ll have to do a lot of mocking to ensure that the required components are included in the test environment. These complexities are why I believe that integration testing is essential when creating a program that’s as close to bug-free as possible.</p><p>In summary, as a software engineer, I understand the importance of testing. However, I also know that it can be a grueling and time-consuming process. By prioritizing integration testing alongside other forms of testing, we can ensure that our programs meet their functional requirements, and any bugs are caught early on in the development process.</p><p>For those who have been following my Medium articles from the beginning, you’ll know that one of the biggest pain points of creating integration tests is having to write repetitive lines of code for each test case. This can consume up to 20–40% of the total time spent on a task. That’s why I created a simple library that streamlines and speeds up the process of creating integration tests in Spring Framework-based programs.</p><h4>Requirement &amp; Set Up</h4><p>Because the library I created uses Java 17, JUnit5, and Spring 2.7.8, please make sure that your project also meets these minimum requirements. As I was too lazy to upload this library to Maven, I decided to use GitHub Packages for easier access.</p><ul><li>First, add my GitHub Packages library to your project.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0c8cf00421d9a18b944daf63df9dbfda/href">https://medium.com/media/0c8cf00421d9a18b944daf63df9dbfda/href</a></iframe><ul><li>After adding the necessary dependencies and repositories, the next step is to add the required test properties to the test resource.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a9e317204ddc46cf5974a2f069f14d4c/href">https://medium.com/media/a9e317204ddc46cf5974a2f069f14d4c/href</a></iframe><p>the use of test property above:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8e0a7e60b8757e5288eb80446dafc934/href">https://medium.com/media/8e0a7e60b8757e5288eb80446dafc934/href</a></iframe><p>What needs to be considered is that when your Spring project doesn’t use the default port, the baseUrl needs to be replaced according to the port used.</p><ul><li>The next step is to create a test case. The step is very easy, all we have to do is create a .yaml file at the path /resources/test with the suffix -integration.yaml. Where the contents are as follows:</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a284eb78b0d37c2ea18a60291c90ee3a/href">https://medium.com/media/a284eb78b0d37c2ea18a60291c90ee3a/href</a></iframe><p>Some things to note are, we can create multiple integration.yaml files as long as the groupName is unique between files. In addition there are two options for asserting the response, namely by asserting each JSON path or by asserting the JSON response.</p><p>example ofintegration.yaml file:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1c0bca8c17b479702dc6d7d035f5059f/href">https://medium.com/media/1c0bca8c17b479702dc6d7d035f5059f/href</a></iframe><ul><li>And finally, for the code to run, you need to create a class to run the test case.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b5ab70193a38fa8868555875e1613773/href">https://medium.com/media/b5ab70193a38fa8868555875e1613773/href</a></iframe><p>After everything is set up, we can run the created integration tests. In the example above, we can directly run the test cases in one group by calling the function runGroupAPITest() with the groupName parameter from the integration.yaml file that has been created. Alternatively, we can also run a specific test case by selecting it and calling the function runSingleAPITest() with the groupName and test case id parameters from the integration.yaml file.</p><h4><strong>Conclusion</strong></h4><p>With this library, software engineers like myself can focus on creating test cases and only need to add a few lines of testing code. Moreover, test cases using this model are, in my opinion, easier to read and understand the test case’s purpose. As this library is still in beta, I would appreciate feedback on what can be improved or added to enhance the developer experience in creating integration tests faster and better. Lastly, if you feel that this library has been helpful, Maybe you can support a cup of coffee for me <a href="https://paypal.me/nantaaditya"><strong>here</strong></a>. Please stay tuned for other libraries that I will create. Thank you.</p><p><strong>Example Project<br></strong><a href="https://github.com/nantaaditya/example-app">https://github.com/nantaaditya/example-app</a>s</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=942ab6f9ccab" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/level-up-your-spring-integration-test-942ab6f9ccab">Level-Up your Spring Integration Test</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SQL 101: Index]]></title>
            <link>https://blog.devgenius.io/sql-101-index-c293e973dd8?source=rss-d7e5708a3155------2</link>
            <guid isPermaLink="false">https://medium.com/p/c293e973dd8</guid>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[index]]></category>
            <category><![CDATA[software-engineer]]></category>
            <dc:creator><![CDATA[Pramuditya Ananta Nur]]></dc:creator>
            <pubDate>Sat, 28 Jan 2023 15:36:45 GMT</pubDate>
            <atom:updated>2023-01-31T09:10:23.553Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fHrKT11qhGwC0E9Z" /><figcaption>Photo by <a href="https://unsplash.com/es/@markuswinkler?utm_source=medium&amp;utm_medium=referral">Markus Winkler</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>As a backend engineer, sometimes I found out that my apps respond quite slowly, and usually the bottleneck is on database. One of solutions that we know to speed up queries is to do indexing, but sometimes indexes that have been created don’t really provide much improvement, or even make it slow when doing insert / update query. Therefore, before creating an index there are several things that must be considered, and I’ll share some best practices in creating an index.</p><p><strong>Create the right index for your query</strong></p><p>Before creating an index there are a number of things that must be considered, some of them are related to which columns we will query, usually I will use these criteria for indexing columns:</p><ul><li>column that will be used for filter query.</li><li>column that will be used for sorting query.</li><li>column that will be used for join query.</li><li>column that will be used for group query.</li><li>column that will be used as combination for filter, sorting, join, and group query.</li></ul><p>But keep in mind that not all columns that meet the criteria above must be indexed. Usually I also consider how often the query is performed. In PostgreSQL we can also use help of query analyzer, we can compare the response time between before the index is added and after the index is added using <em>explain analyze</em>:</p><pre>EXPLAIN ANALYZE SELECT column1, column2, columnN FROM table WHERE column1 = &#39;criteria&#39;;</pre><p>And also keep in mind:</p><ul><li>Determine whether we need a single-column index or a multi-column index, usually it depends on query to be performed. Usually I use to create an index that will cover almost all scenario.</li><li>In a multi-column index, order of columns that are indexed and those that are queried has an effect, so check wisely.</li></ul><pre>CREATE INDEX idx_tableZ_columnA_columnB ON table_z(column_a DESC, column_b);<br><br>// is not equal to<br><br>CREATE INDEX idx_tableZ_columnB_columnA ON table_z(column_b, column_a DES);</pre><ul><li>We must know where the index will not be used in query, for example when we use <em>like</em>, and <em>function</em>. Suppose we have an index like below, and use the <em>function</em> in query. Then the index that has been created will have no effect.</li></ul><pre>CREATE INDEX idx_tableZ_columnA ON table_z(column_a);</pre><pre>SELECT * FROM table_z WHERE LOWER(column_a) = &#39;john doe&#39;;<br>SELECT * FROM table_z WHERE column_a LIKE &#39;%john%&#39;;</pre><ul><li>Avoid using indexes on columns that have low cardinality (uniqueness), because it usually doesn’t provide much improvement and in some cases will slow down queries. This is because the index will be scanned every time a query is performed.</li><li>Use <em>concurrently</em> to create a new index on live database, this is to avoid write locks on tables.</li></ul><pre>CREATE INDEX CONCURRENTLY idx_tableZ_columnA ON table_z(column_a);</pre><ul><li>Consider using a partial index, a partial index is usually used to index only a certain subset of rows in a table. Usually I use a partial index to query a column with a certain value without having to create an index for all rows, or when it turns out that the column created by the index has many <em>null</em> values. So usually I will use a partial index because it will reduce the index size.</li></ul><pre>CREATE INDEX idx_tableZ_columnA ON table_z(column_a) WHERE column_a IS NOT NULL;</pre><ul><li>And maybe the last thing before creating an index, things to consider is whether we really need to create the index or not. For example, in a table with 10 indexes, when we do an insert query, the database will insert 11 times, 1 for the table, and the other for the indexes that we have created. So consider again when creating an index, because this will affect the insert or update query.</li></ul><p><strong>Choose the right type of Index</strong></p><p>In almost all common cases we can use the default index type, which is <em>B-Tree</em>, but in certain cases we also have to determine the type of index that match our needs. In here I will not explain more detail about the type of index, you can read for yourself in the documentation for each SQL database, as in <a href="https://www.postgresql.org/docs/current/indexes-types.html">PostgreSQL</a>.</p><p><strong>Maintain your Index.</strong></p><p>After we create the index, the thing that is also important is doing maintenance on the index that we have created. Some of the things that can be done after creating an index are:</p><ul><li>Do performance monitoring after the index is created, this is to ensure that the index that has been created can provide performance improvement on queries.</li><li>Update statistics after creating / deleting index. This is for the query optimizer to have an updated information about the indexed column. To do this update just use the command:</li></ul><pre>ANALYZE table;<br>ANALYZE table COLUMNS column;</pre><ul><li>Also check whether there are unused indexes, this is because these unused indexes use up storage and slow down insert / update queries. We can check the use of unused indexes in PostgreSQL with a query made by <a href="https://hakibenita.com/postgresql-unused-index-size">Haki Benita</a>, thanks.</li></ul><pre>SELECT<br>    relname,<br>    indexrelname,<br>    idx_scan,<br>    idx_tup_read,<br>    idx_tup_fetch,<br>    pg_size_pretty(pg_relation_size(indexrelname::regclass)) as size<br>FROM<br>    pg_stat_all_indexes<br>WHERE<br>    schemaname = &#39;public&#39;<br>    AND indexrelname NOT LIKE &#39;pg_toast_%&#39;<br>    AND idx_scan = 0<br>    AND idx_tup_read = 0<br>    AND idx_tup_fetch = 0<br>ORDER BY<br>    pg_relation_size(indexrelname::regclass) DESC;</pre><ul><li>Do <em>vacuum</em> regularly to avoid <em>bloat</em>. <em>Bloat</em> is when the index / table has more pages than it should need. This is usually because we update / delete so it becomes <em>dead tuples</em>. With <em>vacuum</em> we reclaim storage used by dead tuples. On index we can also do a <em>reindex</em> to recalculate the index that has been created and clean up unused space.</li></ul><p><strong>Conclusion</strong></p><p>So for the conclusion, above are usually my considerations before creating an index, and what’s also important is always do monitoring after we create the index, this is to check whether the index that has been created actually provides performance improvement or vice versa. Because the index itself is like two sides of a coin, on the other hand it provides an improvement when performing read queries, but on the other hand it will also affect insert / update queries. So be wise when creating index. ❤</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c293e973dd8" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/sql-101-index-c293e973dd8">SQL 101: Index</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>