<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mohinighanghas on Medium]]></title>
        <description><![CDATA[Stories by Mohinighanghas on Medium]]></description>
        <link>https://medium.com/@mohinighanghas1099?source=rss-96ae8e0a6f28------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*QINa-CjtZHg-sONy</url>
            <title>Stories by Mohinighanghas on Medium</title>
            <link>https://medium.com/@mohinighanghas1099?source=rss-96ae8e0a6f28------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 11:22:12 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mohinighanghas1099/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How We Reduced Java Cache Loading Time by 80%: From 50 Minutes to 10 Minutes]]></title>
            <link>https://medium.com/@mohinighanghas1099/how-we-reduced-java-cache-loading-time-by-80-from-50-minutes-to-10-minutes-918a7741a0bd?source=rss-96ae8e0a6f28------2</link>
            <guid isPermaLink="false">https://medium.com/p/918a7741a0bd</guid>
            <category><![CDATA[garbage-collection]]></category>
            <category><![CDATA[query-optimization]]></category>
            <category><![CDATA[cache]]></category>
            <category><![CDATA[performance-optimization]]></category>
            <category><![CDATA[java]]></category>
            <dc:creator><![CDATA[Mohinighanghas]]></dc:creator>
            <pubDate>Thu, 19 Nov 2026 09:45:43 GMT</pubDate>
            <atom:updated>2026-11-19T09:45:43.519Z</atom:updated>
            <content:encoded><![CDATA[<p><em>How eliminating redundant database fetches and unnecessary garbage collection requests helped us process nearly 3 million records more efficiently.</em></p><p>Performance bottlenecks are not always caused by slow database queries, inefficient algorithms, or insufficient infrastructure.</p><p>Sometimes, the real problem is much simpler: <strong>doing the same work repeatedly without realizing how expensive it becomes at scale.</strong></p><p>Recently, while investigating a performance issue in a Java application powered by Drools, we discovered two seemingly harmless implementation decisions that were significantly slowing down our cache-loading process.</p><p>The application was taking approximately <strong>40–50 minutes to process nearly 3 million records</strong>.</p><p>Surprisingly, the database query was not the primary bottleneck.</p><p>After profiling the complete processing pipeline, we identified two major issues:</p><ul><li>The same retailer dataset was being fetched multiple times across hierarchy-processing stages.</li><li>Explicit garbage collection was being requested after every batch of records.</li></ul><p>By addressing these two issues, we reduced the total cache-loading time to approximately <strong>10 minutes — an improvement of nearly 80%.</strong></p><p>And we achieved this without changing business logic, upgrading hardware, or scaling infrastructure.</p><p>Let’s understand what happened.</p><h3>1. Understanding the Business Context</h3><p>Our application uses <strong>Drools</strong>, a Java-based rule engine, to evaluate business rules involving retailer visibility, hierarchy mapping, assignments, and other business-specific operations.</p><p>Since these rules are executed frequently, querying the database during every rule evaluation would introduce unnecessary latency.</p><p>To avoid that, relevant data is loaded into an in-memory cache before rule execution begins.</p><p>The cache initialization process handles a significant volume of data:</p><p>DataApproximate VolumeRetailers1.5 millionSite records1–1.5 million<strong>Total2.5–3 million</strong></p><p>During cache initialization, the application performs several operations:</p><ul><li>Fetches retailer and site information.</li><li>Maps retailers across multiple sales hierarchy levels.</li><li>Executes Drools-based processing.</li><li>Builds in-memory cache structures.</li><li>Maintains retailer-to-user associations.</li><li>Prepares data for downstream rule evaluation.</li></ul><p>Since this cache is a critical dependency, any delay directly affects application startup and cache refresh operations.</p><h3>2. The Problem: Why Was Cache Loading Taking 50 Minutes?</h3><p>The cache refresh process was consistently taking between <strong>40 and 50 minutes</strong>.</p><p>This resulted in:</p><ul><li>Longer application startup times.</li><li>Delayed data availability.</li><li>Increased resource utilization.</li><li>Prolonged cache refresh windows.</li><li>Additional load on dependent systems.</li></ul><p>Our initial assumption was straightforward:</p><p><em>The database query must be slow.</em></p><p>But instead of immediately optimizing SQL, we decided to investigate the complete execution pipeline.</p><h3>3. Investigating the Actual Bottleneck</h3><p>The first step was analyzing the primary query responsible for fetching retailer and site information.</p><p>The result was unexpected.</p><p><strong>Database query execution time: Approximately 1 minute.</strong></p><p>A query taking one minute was certainly worth monitoring, but it could not independently explain an end-to-end processing time of 40–50 minutes.</p><p>This was our first indication that the actual bottleneck existed somewhere beyond database retrieval.</p><p>We started examining the entire cache-loading workflow.</p><h3>Understanding the existing execution flow</h3><p>The process looked something like this:</p><pre>Database Query<br>      |<br>      v<br>Fetch Retailers &amp; Sites<br>      |<br>      v<br>Batch Processing (1,000 Records)<br>      |<br>      v<br>Drools Rule Evaluation<br>      |<br>      v<br>Sales Hierarchy Mapping<br>      |<br>      v<br>Cache Population<br>      |<br>      v<br>System.gc()<br>      |<br>      v<br>Next Batch</pre><p>The application processed approximately 3 million records in batches of 1,000.</p><p>That meant:</p><pre>Total Records = 3,000,000</pre><pre>Batch Size = 1,000</pre><pre>Number of Batches = 3,000,000 / 1,000</pre><pre>               = 3,000 batches</pre><p>This calculation became particularly important.</p><p>Anything executed inside the batch-processing loop was potentially being repeated 3,000 times.</p><p>During the investigation, we identified two major inefficiencies.</p><h3>4. Bottleneck #1: Fetching the Same Data Multiple Times</h3><p>While reviewing the hierarchy-processing logic, we discovered an architectural inefficiency.</p><p>The retailer dataset remained unchanged across multiple hierarchy-processing stages.</p><p>However, the application was repeatedly fetching the same retailer information.</p><h3>Existing implementation</h3><p>The flow was effectively:</p><pre>Hierarchy Level 1 → Fetch Retailers</pre><pre>Hierarchy Level 2 → Fetch Retailers Again</pre><pre>Hierarchy Level 3 → Fetch Retailers Again</pre><pre>Hierarchy Level 4 → Fetch Retailers Again</pre><p>Although each hierarchy level had different processing requirements, the underlying retailer dataset was identical.</p><p>This meant repeatedly paying the cost of:</p><ul><li>Database query execution.</li><li>Database connection utilization.</li><li>ResultSet traversal.</li><li>Object creation.</li><li>Data mapping.</li><li>Repeated memory allocation.</li></ul><p>The business logic was different.</p><p><strong>The data was not.</strong></p><p>We were retrieving and processing information that was already available.</p><h3>Optimization #1: Fetch Once, Reuse Everywhere</h3><p>Instead of fetching the same retailer dataset separately for each hierarchy level, we modified the implementation to retrieve it once and reuse it across all processing stages.</p><p>The optimized approach became:</p><pre>Database<br>                |<br>                v<br>     Fetch Retailers &amp; Sites<br>                |<br>                v<br>       Store Dataset in Memory<br>                |<br>       +--------+--------+<br>       |        |        |<br>       v        v        v<br>    Level 1  Level 2  Level 3<br>       |        |        |<br>       +--------+--------+<br>                |<br>                v<br>        Cache Population</pre><p>This eliminated redundant data retrieval and unnecessary repeated mapping.</p><p>The key takeaway here is simple:</p><p><strong>The fastest database query is often the one you don’t need to execute.</strong></p><h3>5. Bottleneck #2: The Hidden Cost of System.gc()</h3><p>The second issue was even more interesting.</p><p>While reviewing the batch-processing implementation, we noticed an explicit garbage collection request inside the processing loop.</p><p>The code was similar to:</p><pre>for (Batch batch : batches) {</pre><pre>    processBatch(batch);</pre><pre>    System.gc();<br>}</pre><p>At first glance, this might seem harmless.</p><p>In fact, it was likely introduced with the intention of controlling memory consumption while processing millions of records.</p><p>However, its placement was problematic.</p><h3>Understanding the scale</h3><p>We already knew that the application processed approximately 3 million records in batches of 1,000.</p><p>That meant:</p><pre>Total Records      = 3,000,000<br>Batch Size         = 1,000<br>Number of Batches  = 3,000</pre><pre>GC Requests        = 1 per batch</pre><pre>Total GC Requests  ≈ 3,000</pre><p>In other words, the application was requesting garbage collection approximately 3,000 times during a single cache refresh cycle.</p><h3>But wasn’t garbage collection taking only 300–400 milliseconds?</h3><p>This was one of the interesting observations during profiling.</p><p>An individual garbage collection event appeared relatively inexpensive.</p><p>The average duration we observed was approximately 300–400 milliseconds.</p><p>That might not seem significant in isolation.</p><p>But performance bottlenecks are not always about the cost of one operation.</p><p>They are also about how frequently that operation executes.</p><h3>Calculating the cumulative overhead</h3><p>Using an average observed duration of 350 milliseconds:</p><pre>Average GC Duration = 350 ms</pre><pre>Number of GC Requests = 3,000</pre><pre>Total Time = 3,000 × 350 ms</pre><pre>           = 1,050,000 ms</pre><pre>           = 1,050 seconds</pre><pre>           = 17.5 minutes</pre><p>That’s approximately <strong>17.5 minutes of cumulative garbage collection time</strong>, based on the observed event durations.</p><p>A seemingly small operation was potentially accounting for a substantial portion of the total processing time.</p><p>And this was just one of the inefficiencies we had identified.</p><h3>6. Why Explicit Garbage Collection Can Hurt Java Performance</h3><p>Java applications run on the JVM, which manages memory allocation and garbage collection automatically.</p><p>Modern garbage collectors, including:</p><ul><li>G1 Garbage Collector (G1GC)</li><li>Z Garbage Collector (ZGC)</li><li>Shenandoah</li><li>Parallel GC</li></ul><p>use runtime information to determine when and how garbage collection should happen.</p><p>They consider factors such as:</p><ul><li>Heap occupancy.</li><li>Object allocation rate.</li><li>Object lifetime.</li><li>Memory pressure.</li><li>Available heap space.</li><li>Collector-specific performance goals.</li></ul><p>The JVM uses this information to manage memory according to the selected garbage collector and its configuration.</p><h3>What happens when we call System.gc()?</h3><p>Consider:</p><pre>System.gc();</pre><p>This requests that the JVM perform garbage collection.</p><p>However, an important distinction is that <strong>System.gc() is a request, not a guarantee of a full garbage collection.</strong></p><p>The actual behavior depends on the JVM, garbage collector, and configuration.</p><p>For example, the JVM option -XX:+DisableExplicitGC can disable the processing of explicit GC requests.</p><p>Nevertheless, repeatedly issuing these requests inside a performance-critical processing loop can interfere with the garbage collector’s normal scheduling and introduce unnecessary collection activity.</p><h3>The problem with frequent GC requests</h3><p>Our processing flow effectively looked like this:</p><pre>Process Batch<br>     |<br>     v<br>Request Garbage Collection<br>     |<br>     v<br>Process Next Batch<br>     |<br>     v<br>Request Garbage Collection<br>     |<br>     v<br>Process Next Batch<br>     |<br>     v<br>Request Garbage Collection</pre><p>This pattern repeated thousands of times.</p><p>Instead of allowing the JVM to manage garbage collection based on actual memory pressure, the application was repeatedly requesting it after every batch.</p><p>The important lesson is that garbage collection should not be treated as a routine cleanup operation that must run after every batch.</p><h3>Optimization #2: Let the JVM Manage Garbage Collection</h3><p>We removed the explicit System.gc() invocation from the batch-processing loop.</p><p>The application was then allowed to rely on the JVM’s configured garbage collector to manage memory based on runtime conditions.</p><p>This removed unnecessary explicit GC requests and allowed the application to spend more time performing useful business processing.</p><h3>7. The Final Results: 50 Minutes Reduced to 10 Minutes</h3><p>After implementing both optimizations, we measured the cache-loading performance again.</p><p>The improvement was significant.</p><p>MetricBeforeAfterRecords processed~3 million~3 millionBatch size1,0001,000Cache loading time40–50 minutes~10 minutesBusiness logicUnchangedUnchangedInfrastructureUnchangedUnchanged</p><h3>Performance improvement</h3><p><strong>Before:</strong> 50 minutes</p><p><strong>After:</strong> 10 minutes</p><p>&lt; 80% reduction in cache-loading time.</p><p>The improvements came from two relatively simple changes:</p><ol><li>Fetch retailer and site information once and reuse it across hierarchy-processing stages.</li><li>Remove unnecessary explicit garbage collection requests from the batch-processing loop.</li></ol><p>We achieved this without:</p><ul><li>Changing business rules.</li><li>Modifying rule definitions.</li><li>Changing the database schema.</li><li>Increasing infrastructure capacity.</li><li>Upgrading hardware.</li></ul><p>The biggest improvement came from eliminating unnecessary work rather than adding more resources.</p><h3>8. Key Engineering Lessons</h3><p>This investigation reinforced several important lessons about Java performance optimization.</p><h3>1. Don’t Blame the Database Too Early</h3><p>A slow database query is an obvious suspect when an application takes too long to process data.</p><p>But optimizing SQL will not solve a bottleneck that exists somewhere else in the application.</p><p>Always profile the complete execution pipeline:</p><pre>Database<br>   ↓<br>Data Retrieval<br>   ↓<br>Object Mapping<br>   ↓<br>Business Logic<br>   ↓<br>Rule Evaluation<br>   ↓<br>Cache Construction<br>   ↓<br>JVM Activity</pre><p>Measure before deciding where to optimize.</p><h3>2. The Fastest Query Is the One You Don’t Execute</h3><p>Developers often focus on making database queries faster.</p><p>But sometimes, the biggest improvement comes from eliminating unnecessary queries altogether.</p><p>If the underlying dataset has not changed, repeatedly retrieving it across different processing stages is often unnecessary.</p><p>Reuse data wherever it makes architectural sense.</p><h3>3. System.gc() Is Not a Performance Optimization</h3><p>Explicit garbage collection is not a substitute for proper memory management.</p><p>Modern JVM garbage collectors are designed to manage memory using runtime information.</p><p>Unless profiling demonstrates a specific need, avoid introducing explicit garbage collection requests into frequently executed application paths.</p><p>Trust the JVM, but verify its behavior using GC logs and profiling tools.</p><h3>4. Small Costs Become Significant at Scale</h3><p>A 350-millisecond operation might appear insignificant.</p><p>But when repeated thousands of times, it can become a major performance bottleneck.</p><p>A useful principle to remember is:</p><p><strong>Total Cost = Cost Per Operation × Number of Executions</strong></p><p>Always evaluate performance in the context of execution frequency and workload size.</p><h3>Conclusion</h3><p>This investigation reinforced one of the most valuable lessons in backend performance engineering:</p><p><strong>Major performance improvements don’t always require complex algorithms or expensive infrastructure upgrades. Sometimes, they come from removing work that should never have been performed in the first place.</strong></p><p>Our application was processing nearly 3 million retailer and site records while building a Drools-based cache.</p><p>The initial assumption was that database performance was responsible for the delay.</p><p>However, profiling the complete execution pipeline revealed two deeper issues:</p><ul><li>Repeatedly fetching the same dataset across hierarchy-processing stages.</li><li>Requesting garbage collection after every batch.</li></ul><p>Individually, neither implementation decision looked particularly dangerous.</p><p>But at scale, their cumulative impact was significant.</p><p>By fetching data once, reusing it efficiently, and allowing the JVM to manage garbage collection, we reduced cache-loading time from approximately 40–50 minutes to just 10 minutes.</p><p>That’s an improvement of nearly 80%.</p><p>The biggest performance wins don’t always come from writing more code, introducing new technologies, or adding more servers.</p><p><strong>Sometimes, the best optimization is simply doing less unnecessary work.</strong></p><p><em>Have you encountered a similar performance bottleneck in a Java application? I’d love to hear what you discovered and how you solved it.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=918a7741a0bd" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>