<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Engineering Deep Dives - Medium]]></title>
        <description><![CDATA[Deep dives into distributed systems, infrastructure, and backend engineering; Real-world failure scenarios and how systems behave under load; Architecture explanations that go beyond surface-level diagrams; Tradeoffs, edge cases, and debugging techniques used in product - Medium]]></description>
        <link>https://medium.com/engineering-deep-dives?source=rss----177d7e26856f---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Engineering Deep Dives - Medium</title>
            <link>https://medium.com/engineering-deep-dives?source=rss----177d7e26856f---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 18:50:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/engineering-deep-dives" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Redis Hot Keys — Zipf Math & Early Detection]]></title>
            <link>https://medium.com/engineering-deep-dives/redis-hot-keys-zipf-math-early-detection-57a8faf76298?source=rss----177d7e26856f---4</link>
            <guid isPermaLink="false">https://medium.com/p/57a8faf76298</guid>
            <category><![CDATA[caching]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[redis]]></category>
            <category><![CDATA[performance]]></category>
            <dc:creator><![CDATA[Shriom Tripathi]]></dc:creator>
            <pubDate>Tue, 31 Mar 2026 01:53:29 GMT</pubDate>
            <atom:updated>2026-03-31T01:53:29.312Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Why Redis hot keys follow Zipf’s law, how they overload caches, and simple math + code to spot warm, hot, and extreme keys before they trigger incidents.</blockquote><p><strong>Series:</strong> Taming Redis Hot Keys <strong>Part 1 of 5</strong></p><p>At first glance, caching seems simple: store frequently accessed data in memory to avoid slow database queries. But there’s a hidden assumption in that statement: <em>frequently accessed</em>. How frequent? And how do different keys compare to each other?</p><p>Most engineers assume traffic distributes somewhat evenly across cache keys. Maybe some keys get 2x or 3x more traffic than others. Reality is far more extreme.</p><p>In production systems, the most popular cache key often receives <strong>100–1000x more traffic</strong> than the median key. This isn’t a bug or poor design — it’s a fundamental property of how humans and systems access data, governed by mathematical laws discovered nearly a century ago.</p><p>This is the story of hot keys, power laws, and why your cache’s worst enemy isn’t slow disks or network latency — it’s mathematics.</p><h3>The 80/20 Rule Doesn’t Go Far Enough</h3><p>You’ve heard of the Pareto Principle: 80% of effects come from 20% of causes. In caching, this manifests as:</p><pre>80% of cache hits come from 20% of keys</pre><p>Sounds manageable, right? You can optimize for the top 20% of keys and call it done.</p><p><strong>The reality is far more skewed:</strong></p><pre>50% of traffic → 1% of keys<br>80% of traffic → 5% of keys<br>99% of traffic → 20% of keys<br>99.9% of traffic → 50% of keys</pre><p>And within that top 1%? The distribution is <em>still</em> skewed. The #1 key might get 10x more traffic than the #10 key.</p><p>This isn’t Pareto. This is a <strong>power law distribution</strong>, and it breaks traditional caching assumptions.</p><h3>Zipf’s Law: The Mathematics of Inequality</h3><p>In 1949, Harvard linguist George Zipf discovered something remarkable while analyzing word frequency in English texts:</p><blockquote><strong><em>Zipf’s Law:</em></strong><em> The frequency of the kth most common word is approximately proportional to 1/k.</em></blockquote><p><strong>In plain English:</strong></p><ul><li>The 2nd most common word appears <strong>1/2</strong> as often as the 1st</li><li>The 10th most common word appears <strong>1/10</strong> as often as the 1st</li><li>The 100th most common word appears <strong>1/100</strong> as often as the 1st</li></ul><p>Here’s the exact formula:</p><pre>Frequency(k) = C / k</pre><pre>Where:<br>- k = rank (1st, 2nd, 3rd, etc.)<br>- C = normalization constant = 1/(1 + 1/2 + 1/3 + ... + 1/N)</pre><h3>Real-World Example: English Word Usage</h3><pre># Top 10 English words by frequency (simplified)<br>words = [<br>    (&quot;the&quot;, 1.00),   # Rank 1: normalized to 1.0<br>    (&quot;of&quot;, 0.50),    # Rank 2: ~1/2 as frequent<br>    (&quot;and&quot;, 0.33),   # Rank 3: ~1/3 as frequent<br>    (&quot;to&quot;, 0.25),    # Rank 4: ~1/4 as frequent<br>    (&quot;a&quot;, 0.20),     # Rank 5: ~1/5 as frequent<br>    (&quot;in&quot;, 0.17),    # Rank 6: ~1/6 as frequent<br>    (&quot;is&quot;, 0.14),    # Rank 7: ~1/7 as frequent<br>    (&quot;it&quot;, 0.13),    # Rank 8: ~1/8 as frequent<br>    (&quot;you&quot;, 0.11),   # Rank 9: ~1/9 as frequent<br>    (&quot;that&quot;, 0.10),  # Rank 10: ~1/10 as frequent<br>]</pre><pre># The word &quot;the&quot; appears 10x more often than &quot;that&quot;<br># It appears 100x more often than the 100th ranked word</pre><p><strong>Why this matters for caching:</strong></p><p>Zipf’s law doesn’t just apply to words. It applies to:</p><ul><li>Web page popularity</li><li>Product sales (Amazon best-sellers)</li><li>Social media followers</li><li><strong>Cache key access patterns</strong></li></ul><p>Your cache keys follow the same mathematical distribution as English word usage.</p><h3>Visualizing Power Laws: The Log-Log Plot</h3><p>Power law distributions have a distinctive signature: they appear as a <strong>straight line</strong> when plotted on a log-log scale.</p><pre>import matplotlib.pyplot as plt<br>import numpy as np<br># Generate Zipf distribution for 1000 keys<br>N = 1000<br>ranks = np.arange(1, N+1)<br># Zipf&#39;s law: frequency proportional to 1/rank<br>C = 1 / np.sum(1 / ranks)  # Normalization constant<br>frequencies = C / ranks<br># Regular plot: exponential decay (hard to read)<br>plt.figure(figsize=(12, 5))<br>plt.subplot(1, 2, 1)<br>plt.plot(ranks, frequencies)<br>plt.xlabel(&quot;Key Rank&quot;)<br>plt.ylabel(&quot;Access Frequency&quot;)<br>plt.title(&quot;Zipf Distribution (Linear Scale)&quot;)<br>plt.grid(True)<br># Log-log plot: straight line (easy to identify)<br>plt.subplot(1, 2, 2)<br>plt.loglog(ranks, frequencies)<br>plt.xlabel(&quot;Key Rank (log scale)&quot;)<br>plt.ylabel(&quot;Access Frequency (log scale)&quot;)<br>plt.title(&quot;Zipf Distribution (Log-Log Scale)&quot;)<br>plt.grid(True)<br>plt.tight_layout()<br>plt.show()</pre><p><strong>Output interpretation:</strong></p><ul><li><strong>Linear plot</strong>: Looks like steep exponential decay — hard to see structure</li><li><strong>Log-log plot</strong>: Perfect straight line — power law confirmed</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*S3PbVRvVBHUyxKHVqd9hNw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/1*jbJccxhEQDYXyzG-1unlkA.jpeg" /></figure><p>This is your diagnostic tool. If your cache access patterns show a straight line on a log-log plot, you have hot keys following Zipf’s law.</p><h3>Hot Keys vs Normal Keys: Defining the Threshold</h3><p>Not all popular keys are “hot keys.” There’s a difference between:</p><ul><li><strong>Popular key</strong>: Gets more traffic than average</li><li><strong>Hot key</strong>: Gets <strong>orders of magnitude</strong> more traffic</li></ul><p>Let’s define this mathematically:</p><pre>class KeyClassifier:<br>    def __init__(self, total_keys=10000, total_qps=100000):<br>        self.total_keys = total_keys<br>        self.total_qps = total_qps<br>        self.average_qps = total_qps / total_keys  # 10 QPS per key<br>    def classify_key(self, key_qps):<br>            &quot;&quot;&quot;Classify a key based on its traffic&quot;&quot;&quot;<br>            ratio = key_qps / self.average_qps<br>            if ratio &lt; 1:<br>                return &quot;cold&quot;, &quot;Below average traffic&quot;<br>            elif ratio &lt; 5:<br>                return &quot;normal&quot;, &quot;Expected variance&quot;<br>            elif ratio &lt; 50:<br>                return &quot;warm&quot;, &quot;Popular but manageable&quot;<br>            elif ratio &lt; 500:<br>                return &quot;hot&quot;, &quot;⚠️ Needs monitoring&quot;<br>            else:<br>                return &quot;extreme&quot;, &quot;🔥 Critical hot key&quot;<br># Example usage<br>classifier = KeyClassifier(total_keys=10000, total_qps=100000)<br><br># Test with real traffic patterns<br>test_keys = [<br>    (&quot;user_session:12345&quot;, 8),        # Normal<br>    (&quot;homepage_data&quot;, 450),            # Warm<br>    (&quot;trending_topic:covid&quot;, 4800),    # Hot<br>    (&quot;celebrity:taylorswift&quot;, 48000),  # Extreme (480x average!)<br>]<br><br>for key, qps in test_keys:<br>    category, description = classifier.classify_key(qps)<br>    ratio = qps / classifier.average_qps<br>    print(f&quot;{key:30} {qps:6} QPS  [{category:8}] {ratio:6.1f}x average - {description}&quot;)</pre><p><strong>Output:</strong></p><pre>user_session:12345                8 QPS  [normal  ]    0.8x average - Below average traffic<br>homepage_data                   450 QPS  [warm    ]   45.0x average - Popular but manageable<br>trending_topic:covid           4800 QPS  [hot     ]  480.0x average - ⚠️ Needs monitoring<br>celebrity:taylorswift         48000 QPS  [extreme ] 4800.0x average - 🔥 Critical hot key</pre><h3>The Celebrity Problem in Distributed Systems</h3><p>The “celebrity problem” is a famous computer science concept:</p><blockquote><em>In a social graph with N people, how do you efficiently find the celebrity (someone everyone knows, but who knows no one) ?</em></blockquote><p>In distributed caching, the celebrity problem manifests differently:</p><blockquote><em>When one cache key (the celebrity) receives traffic from everyone, how do you prevent it from becoming a bottleneck?</em></blockquote><p><strong>Traditional partitioning breaks down:</strong></p><pre># Standard consistent hashing<br>def get_partition(key, num_partitions=10):<br>    return hash(key) % num_partitions<br><br># Traffic distribution<br>keys_traffic = {<br>    &quot;celebrity:taylorswift&quot;: 48000,  # All goes to partition 8<br>    &quot;user:alice&quot;: 5,<br>    &quot;user:bob&quot;: 5,<br>    # ... 9,998 other normal keys ...<br>}<br><br># Result: Partition 8 gets 48,000 QPS while others get ~5,000 QPS each<br># One node is 10x overloaded!</pre><p>This is the fundamental challenge: <strong>Partitioning distributes keys evenly, but not traffic evenly.</strong></p><h3>Simulating Hot Key Traffic Patterns</h3><p>Let’s build a realistic traffic simulator using Zipf’s law:</p><pre>import random<br>import numpy as np<br>from collections import Counter<br><br>class ZipfTrafficSimulator:<br>    def __init__(self, num_keys=1000, alpha=1.0):<br>        &quot;&quot;&quot;<br>        Args:<br>            num_keys: Total number of unique keys<br>            alpha: Zipf parameter (1.0 = classic Zipf, higher = more skewed)<br>        &quot;&quot;&quot;<br>        self.num_keys = num_keys<br>        self.alpha = alpha<br><br>        # Generate key names<br>        self.keys = [f&quot;key:{i:04d}&quot; for i in range(num_keys)]<br><br>        # Calculate Zipf probabilities<br>        ranks = np.arange(1, num_keys + 1)<br>        probabilities = 1 / (ranks ** alpha)<br>        self.probabilities = probabilities / probabilities.sum()<br><br>    def simulate_requests(self, total_requests=100000):<br>        &quot;&quot;&quot;Simulate traffic and return key access counts&quot;&quot;&quot;<br>        # Sample keys according to Zipf distribution<br>        requests = np.random.choice(<br>            self.keys,<br>            size=total_requests,<br>            p=self.probabilities<br>        )<br><br>        return Counter(requests)<br><br>    def analyze_distribution(self, access_counts):<br>        &quot;&quot;&quot;Analyze how traffic is distributed&quot;&quot;&quot;<br>        total_requests = sum(access_counts.values())<br><br>        # Sort keys by traffic<br>        sorted_keys = sorted(<br>            access_counts.items(),<br>            key=lambda x: x[1],<br>            reverse=True<br>        )<br><br>        # Calculate cumulative percentages<br>        cumulative_traffic = 0<br>        results = []<br><br>        for i, (key, count) in enumerate(sorted_keys, 1):<br>            cumulative_traffic += count<br>            cumulative_pct = (cumulative_traffic / total_requests) * 100<br><br>            if i in [1, 10, 50, 100, 200, 500, 1000]:<br>                results.append({<br>                    &quot;top_k_keys&quot;: i,<br>                    &quot;cumulative_traffic_pct&quot;: cumulative_pct,<br>                    &quot;avg_qps_per_key&quot;: count if i == 1 else cumulative_traffic / i<br>                })<br><br>        return results<br><br># Run simulation<br>simulator = ZipfTrafficSimulator(num_keys=1000, alpha=1.0)<br>access_counts = simulator.simulate_requests(total_requests=100000)<br><br># Analyze results<br>results = simulator.analyze_distribution(access_counts)<br><br>print(&quot;Traffic Distribution Analysis (100,000 requests across 1,000 keys)&quot;)<br>print(&quot;=&quot; * 70)<br>for r in results:<br>    print(f&quot;Top {r[&#39;top_k_keys&#39;]:4d} keys: {r[&#39;cumulative_traffic_pct&#39;]:5.1f}% of traffic&quot;)<br><br># Show hottest keys<br>print(&quot;\n🔥 Hottest Keys:&quot;)<br>top_10 = access_counts.most_common(10)<br>for rank, (key, count) in enumerate(top_10, 1):<br>    pct = (count / 100000) * 100<br>    print(f&quot;  {rank:2d}. {key}  {count:5d} requests ({pct:4.1f}%)&quot;)</pre><p><strong>Typical Output:</strong></p><pre>Traffic Distribution Analysis (100,000 requests across 1,000 keys)<br>======================================================================<br>Top    1 keys:   9.2% of traffic  ← Single key gets 9,200 requests!<br>Top   10 keys:  35.7% of traffic<br>Top   50 keys:  62.1% of traffic<br>Top  100 keys:  75.4% of traffic<br>Top  200 keys:  84.9% of traffic<br>Top  500 keys:  95.2% of traffic<br>Top 1000 keys: 100.0% of traffic<br><br>🔥 Hottest Keys:<br>   1. key:0000   9182 requests (9.2%)  ← 92x average!<br>   2. key:0001   4621 requests (4.6%)<br>   3. key:0002   3079 requests (3.1%)<br>   4. key:0003   2314 requests (2.3%)<br>   5. key:0004   1851 requests (1.9%)</pre><p><strong>Key insight:</strong> The #1 key receives <strong>92x more traffic</strong> than the average key (100 req). This is a production-scale hot key problem.</p><h3>Read Hot Keys vs Write Hot Keys</h3><p>Not all hot keys are created equal. The mitigation strategy depends on whether the hot key is:</p><h3>1. Read Hot Keys (Most Common)</h3><pre># Scenario: Product page view counter<br>redis.incr(f&quot;views:product:{popular_product_id}&quot;)<br>redis.get(f&quot;views:product:{popular_product_id}&quot;)  # ← 1000s of reads</pre><p><strong>Characteristics:</strong></p><ul><li>High read QPS (1,000–100,000+ req/sec)</li><li>Rare writes (maybe once per second)</li><li>Can be cached locally</li><li><strong>Easier to mitigate</strong> (key splitting, local caching)</li></ul><h3>2. Write Hot Keys (More Challenging)</h3><pre># Scenario: Real-time like counter<br>redis.incr(f&quot;likes:post:{viral_post_id}&quot;)  # ← 1000s of writes</pre><p><strong>Characteristics:</strong></p><ul><li>High write QPS (1,000–10,000+ writes/sec)</li><li>Frequent mutations</li><li>Can’t cache locally (would miss updates)</li><li><strong>Harder to mitigate</strong> (need write sharding or buffering)</li></ul><h3>3. Read+Write Hot Keys (Most Difficult)</h3><pre># Scenario: Real-time inventory<br>redis.decr(f&quot;stock:product:{flash_sale_item}&quot;)  # Write<br>redis.get(f&quot;stock:product:{flash_sale_item}&quot;)   # Read</pre><p><strong>Characteristics:</strong></p><ul><li>High read AND write QPS</li><li>Strong consistency requirements</li><li><strong>Hardest to mitigate</strong> (need sophisticated strategies)</li></ul><h3>Hot Keys in the Wild: Real-World Examples</h3><h3>Example 1: Twitter Trending Topics</h3><pre># Every user checking trending topics hits the same key<br>key = &quot;trending:global:2024-11-28&quot;<br># Traffic: Millions of users × 1 request/minute = 100,000+ QPS</pre><h3>Example 2: Ecommerce Flash Sale</h3><pre># Limited stock item during sale<br>key = f&quot;inventory:product:{flash_sale_item_id}&quot;<br># Traffic: Thousands of concurrent buyers checking stock<br># Both reads (check availability) and writes (decrement stock)</pre><h3>Example 3: Live Sports Score</h3><pre># World Cup final score<br>key = &quot;score:worldcup:final&quot;<br># Traffic: Millions of fans refreshing every 10 seconds<br># Spike during goals</pre><h3>Example 4: Celebrity Social Media Profile</h3><pre># Taylor Swift announces album<br>key = f&quot;profile:user:{celebrity_id}&quot;<br># Traffic: Viral news → millions visit profile simultaneously</pre><h3>Detecting Power Law Distributions in Your Cache</h3><p>Here’s a production-ready analyzer to check if your cache follows Zipf’s law:</p><pre>from scipy import stats<br>import numpy as np<br><br>class PowerLawDetector:<br>    def __init__(self, access_counts):<br>        &quot;&quot;&quot;<br>        Args:<br>            access_counts: Dict of {key: request_count}<br>        &quot;&quot;&quot;<br>        self.access_counts = access_counts<br><br>    def test_power_law(self):<br>        &quot;&quot;&quot;Test if distribution follows power law using log-log regression&quot;&quot;&quot;<br>        # Sort by frequency (descending)<br>        frequencies = sorted(self.access_counts.values(), reverse=True)<br>        ranks = np.arange(1, len(frequencies) + 1)<br><br>        # Log-log regression: log(freq) = -alpha * log(rank) + C<br>        log_ranks = np.log10(ranks)<br>        log_freqs = np.log10(frequencies)<br><br>        # Linear regression in log space<br>        slope, intercept, r_value, p_value, std_err = stats.linregress(<br>            log_ranks, log_freqs<br>        )<br><br>        return {<br>            &quot;is_power_law&quot;: r_value ** 2 &gt; 0.95,  # R² &gt; 0.95 indicates strong power law<br>            &quot;alpha&quot;: -slope,  # Zipf exponent<br>            &quot;r_squared&quot;: r_value ** 2,<br>            &quot;interpretation&quot;: self._interpret(r_value ** 2, -slope)<br>        }<br><br>    def _interpret(self, r_squared, alpha):<br>        if r_squared &lt; 0.8:<br>            return &quot;Not a power law - traffic is more evenly distributed&quot;<br>        elif alpha &lt; 0.5:<br>            return &quot;Weak power law - some hot keys but manageable&quot;<br>        elif alpha &lt; 1.5:<br>            return &quot;Classic Zipf distribution - significant hot key risk&quot;<br>        else:<br>            return &quot;Extreme power law - severe hot key problem&quot;<br><br># Example usage with real cache data<br>cache_traffic = {<br>    f&quot;user:{i}&quot;: max(1, int(10000 / (i+1)**1.2))  # Simulated Zipf distribution<br>    for i in range(1000)<br>}<br><br>detector = PowerLawDetector(cache_traffic)<br>result = detector.test_power_law()<br><br>print(&quot;Power Law Analysis:&quot;)<br>print(f&quot;  R²: {result[&#39;r_squared&#39;]:.3f}&quot;)<br>print(f&quot;  Zipf exponent (α): {result[&#39;alpha&#39;]:.2f}&quot;)<br>print(f&quot;  Power law detected: {result[&#39;is_power_law&#39;]}&quot;)<br>print(f&quot;  {result[&#39;interpretation&#39;]}&quot;)</pre><h3>What This Means for Cache Design</h3><p>Traditional caching assumes roughly uniform access patterns. Zipf’s law proves this assumption is <strong>fundamentally wrong</strong>:</p><h3>❌ Wrong Assumption:</h3><pre>&quot;All keys will get similar traffic, give or take a factor of 2-3x&quot;</pre><h3>✅ Reality:</h3><pre>&quot;The top 1% of keys will get 100-1000x more traffic than average&quot;</pre><p><strong>Design implications:</strong></p><p><strong>You can’t ignore hot keys</strong></p><ul><li>They will emerge naturally (Zipf’s law)</li><li>They will break your system if unhandled</li></ul><p><strong>Partitioning alone doesn’t solve hot keys</strong></p><ul><li>Hash partitioning distributes keys evenly</li><li>But traffic is NOT evenly distributed across keys</li></ul><p><strong>Hot keys need special treatment</strong></p><ul><li>Different caching strategy (next parts of this series)</li><li>Dedicated monitoring</li><li>Fallback mechanisms</li></ul><h3>Key Takeaways</h3><ol><li><strong>Hot keys follow Zipf’s law</strong>: The kth most popular key gets approximately 1/k of the top key’s traffic</li><li><strong>Power law distributions are extreme</strong>: Top key can be 100–1000x more popular than average</li><li><strong>This isn’t a bug</strong>: It’s a mathematical property of how humans and systems access data</li><li><strong>You can detect power laws</strong>: Log-log plots show straight lines for Zipf distributions</li><li><strong>Design must account for hot keys</strong>: Traditional caching assumes uniform traffic — reality is Zipf</li></ol><p>The question isn’t <strong>if</strong> you have hot keys. If your system has real users, you <strong>definitely</strong> have hot keys. The question is: are you ready for them?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=57a8faf76298" width="1" height="1" alt=""><hr><p><a href="https://medium.com/engineering-deep-dives/redis-hot-keys-zipf-math-early-detection-57a8faf76298">Redis Hot Keys — Zipf Math &amp; Early Detection</a> was originally published in <a href="https://medium.com/engineering-deep-dives">Engineering Deep Dives</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When Your Docker Logs Fill the Disk and Kill Production]]></title>
            <link>https://medium.com/engineering-deep-dives/when-your-docker-logs-fill-the-disk-and-kill-production-161238859e18?source=rss----177d7e26856f---4</link>
            <guid isPermaLink="false">https://medium.com/p/161238859e18</guid>
            <category><![CDATA[logging]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[nginx]]></category>
            <dc:creator><![CDATA[Shriom Tripathi]]></dc:creator>
            <pubDate>Tue, 31 Mar 2026 01:52:18 GMT</pubDate>
            <atom:updated>2026-03-31T01:52:16.740Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>The default Docker logging driver — the one you get out of the box if you don’t configure anything — will happily fill your disk until your containers can’t write anymore and crash.</blockquote><h3>1. The Disk Space Incident</h3><p>Let’s say on a Saturday morning and your phone explodes with alerts. Not just one or two — fifteen alerts in rapid succession. All your Docker containers on production host prod-app-03 are dying and restarting in a loop.</p><p>You ssh into the server, groggy and confused. docker ps shows containers constantly restarting. df -h shows something terrifying:</p><pre>Filesystem      Size  Used Avail Use% Mounted on<br>/dev/sda1        50G   50G     0 100% /</pre><p>100% disk usage. Zero bytes available. Your heart sinks.</p><p>You dig deeper:</p><pre>du -sh /var/lib/docker/*</pre><p>And there it is:</p><pre>45G     /var/lib/docker/containers/<br>2G      /var/lib/docker/overlay2/<br>1G      /var/lib/docker/image/</pre><p>45 gigabytes of container logs. On a 50GB disk.</p><p>Now, here’s the part that made this incident so interesting (and so instructive for the rest of us): <strong>One single chatty microservice</strong> was writing debug logs at 5MB per minute. With Docker’s default json-file driver and no rotation configured, those logs had been accumulating for 6 days straight.</p><p>5 MB/min × 60 min/hour × 24 hours/day × 6 days = 43,200 MB ≈ 42 GB</p><p>It was <strong>Docker’s default logging behavior</strong>.</p><p>Let that sink in for a moment. The default Docker logging driver — the one you get out of the box if you don’t configure anything — will happily fill your disk until your containers can’t write anymore and crash.</p><p>This is what we call <strong>unbounded log growth</strong>. And if you’re running Docker in production without understanding logging drivers, this is one of those failure modes that will catch you at the worst possible time.</p><p>Let me walk you through how Docker’s logging actually works, why the defaults are dangerous, and — most importantly — how to configure logging that won’t surprise you at 3 AM.</p><h3>2. Let’s put focus on how Docker Logging Actually Works</h3><p>Before we dive into logging drivers and configuration, let’s make sure we understand how Docker captures logs in the first place. Because the architecture is more sophisticated than you might think.</p><p>Think about how a normal Linux process works. Your application writes to stdout (standard output) and stderr (standard error). In a terminal, you see those messages immediately. Simple, right?</p><p>But here’s what happens with Docker: When your container runs, Docker’s runtime (containerd) intercepts those stdout/stderr streams and redirects them to a <strong>logging driver</strong>.</p><p>Now, you might be thinking: “Okay, so it captures the output. What’s the big deal?”</p><p>Great question. Let’s think about what this means in practice.</p><p>Every time your application does this:</p><pre>print(&quot;User logged in: user_id=12345&quot;)  # Goes to stdout</pre><p>Docker’s runtime captures it and writes it somewhere — but <strong>where</strong> depends on the logging driver you’ve configured.</p><p>By default, Docker uses the json-file driver. Let me show you what actually happens:</p><pre>Your Application<br>    │<br>    │ print(&quot;Hello World&quot;)<br>    │<br>    ▼<br>stdout/stderr (file descriptors 1 &amp; 2)<br>    │<br>    │ containerd intercepts<br>    │<br>    ▼<br>Logging Driver (json-file by default)<br>    │<br>    │ writes to disk<br>    │<br>    ▼<br>/var/lib/docker/containers/&lt;container-id&gt;/&lt;container-id&gt;-json.log</pre><p>Each line of output gets wrapped in JSON with metadata:</p><pre>{&quot;log&quot;:&quot;Hello World\n&quot;,&quot;stream&quot;:&quot;stdout&quot;,&quot;time&quot;:&quot;2025-11-28T03:17:42.123456789Z&quot;}<br>{&quot;log&quot;:&quot;Error: connection timeout\n&quot;,&quot;stream&quot;:&quot;stderr&quot;,&quot;time&quot;:&quot;2025-11-28T03:17:43.987654321Z&quot;}</pre><p>Now here’s the critical part: <strong>By default, this file grows forever.</strong></p><p>No rotation. No compression. No size limit. Just infinite growth until your disk is full.</p><p>See the problem? Docker makes logging <strong>easy</strong> (just print to stdout!), but the default behavior is <strong>dangerous</strong> at scale.</p><h3>3. The Overflowing Bucket</h3><p>Here’s how I think about Docker logging drivers, and I promise this analogy will make the different drivers crystal clear.</p><p>Imagine your application is a water faucet, and logs are water flowing out.</p><h4>json-file Driver: The Bucket With No Drain</h4><p>With the default json-file driver, you’ve got a bucket sitting under the faucet. Every log line is a drop of water going into the bucket.</p><p>Now, here’s the key detail: <strong>The bucket has no drain, and it’s infinitely expandable</strong> (until you run out of floor space, which is your disk).</p><p>You’re probably thinking: “That bucket is going to overflow eventually.”</p><p>Exactly right! And when the floor (disk) is completely covered in water (logs), everything slips and crashes (containers can’t write and die).</p><h4>local Driver: The Bucket With Overflow Drain</h4><p>The local driver is like putting a drain at the top of the bucket. When the bucket fills to 10MB (configurable), the oldest water drains out (rotated log files get compressed and eventually deleted).</p><p>The bucket never overflows because there’s a maximum capacity.</p><h4>journald Driver: The Central Reservoir</h4><p>The journald driver is like piping the water to a central reservoir (systemd’s journal) instead of keeping it in individual buckets. All your containers’ water (logs) goes to one managed system that handles rotation, compression, and storage automatically.</p><h4>fluentd/syslog Drivers: The Water Treatment Plant</h4><p>These drivers are like piping the water directly to a treatment plant (centralized logging system). The water doesn’t even stay local — it gets sent somewhere else for processing and storage.</p><p>Think about what this means. If you’re running 50 containers, with json-file you have 50 buckets all independently filling up. With journald, you have one managed reservoir. With fluentd, the water goes off-site entirely.</p><p>In production terms:</p><ul><li>The “bucket” is your log file</li><li>“Water” is log volume</li><li>“Floor space” is disk capacity</li><li>“Drain” is rotation policy</li><li>“Treatment plant” is centralized logging (Fluentd, Splunk, etc.)</li></ul><p><strong>Choose your logging driver based on where you want the water to go and how you want to manage it.</strong></p><h3>4. The Real Scenario; Docker Drivers in Action</h3><p>Let’s look at how these logging drivers actually work with real code and configuration. I’ll show you the most common ones and when to use each.</p><h4>Scenario 1: json-file Driver (The Default Danger)</h4><p>Here’s what happens with default Docker settings:</p><pre># Run a container with default logging<br>docker run -d --name chatty-app nginx</pre><pre># Check where logs go<br>docker inspect chatty-app | grep LogPath<br># Output: &quot;/var/lib/docker/containers/abc123.../abc123...-json.log&quot;</pre><pre># Watch the log file grow<br>watch -n 1 &#39;ls -lh /var/lib/docker/containers/abc123*/*.log&#39;</pre><p>Now, think about what happens when this container logs 5MB per minute.</p><p>After 1 hour: 5 MB × 60 = 300 MB After 1 day: 5 MB × 1,440 = 7,200 MB = 7.2 GB After 1 week: 5 MB × 10,080 = 50,400 MB = 50 GB</p><p>On a 50GB disk, you’re dead in a week.</p><p><strong>The fix?</strong> Configure rotation:</p><pre># Method 1: Global daemon configuration<br># File: /etc/docker/daemon.json<br>{<br>  &quot;log-driver&quot;: &quot;json-file&quot;,<br>  &quot;log-opts&quot;: {<br>    &quot;max-size&quot;: &quot;10m&quot;,      // Rotate after 10MB<br>    &quot;max-file&quot;: &quot;3&quot;,        // Keep 3 rotated files<br>    &quot;compress&quot;: &quot;true&quot;      // Gzip old files<br>  }<br>}<br><br># Restart Docker daemon<br>sudo systemctl restart docker</pre><pre># Method 2: Per-container configuration<br>docker run -d \<br>  --name app \<br>  --log-driver json-file \<br>  --log-opt max-size=10m \<br>  --log-opt max-file=3 \<br>  --log-opt compress=true \<br>  nginx</pre><p>Now let’s do the math on disk usage:</p><pre>Max size per container:<br>= max-size × max-file<br>= 10 MB × 3<br>= 30 MB maximum (compressed even smaller)<br><br>50 containers:<br>= 50 × 30 MB<br>= 1.5 GB (vs 2.5 TB without rotation!)</pre><p>See how quickly this becomes manageable?</p><h4>Scenario 2: local Driver (Production-Ready Out of Box)</h4><p>This one’s even better. The local driver is essentially json-file with rotation enabled by default:</p><pre># /etc/docker/daemon.json<br>{<br>  &quot;log-driver&quot;: &quot;local&quot;,<br>  &quot;log-opts&quot;: {<br>    &quot;max-size&quot;: &quot;10m&quot;,<br>    &quot;max-file&quot;: &quot;5&quot;<br>  }<br>}</pre><p>What you get:</p><pre>/var/lib/docker/containers/&lt;id&gt;/local-logs/<br>├── container.log          # Current active log (growing)<br>├── container.log.1.gz     # Rotated log 1 (compressed)<br>├── container.log.2.gz     # Rotated log 2 (compressed)<br>├── container.log.3.gz     # Rotated log 3 (compressed)<br>└── container.log.4.gz     # Rotated log 4 (compressed)</pre><p><strong>Max size</strong>: 10 MB × 5 = 50 MB per container (compressed to ~10-15 MB typically)<strong><br>And it still works with </strong>docker logs<strong>!</strong></p><pre>docker logs &lt;container-id&gt;          # Works fine<br>docker logs -f &lt;container-id&gt;       # Follow works<br>docker logs --tail 100 &lt;container-id&gt;  # Tail works</pre><p>This is my recommended default for single-host Docker deployments.</p><h4>Scenario 3: journald Driver (SystemD Integration)</h4><p>For Linux systems using systemd, journald provides structured logging:</p><pre># /etc/docker/daemon.json<br>{<br>  &quot;log-driver&quot;: &quot;journald&quot;,<br>  &quot;log-opts&quot;: {<br>    &quot;tag&quot;: &quot;{{.Name}}/{{.ID}}&quot;<br>  }<br>}</pre><p>Now your logs go to the systemd journal:</p><pre># Query logs for specific container<br>journalctl CONTAINER_NAME=nginx<br><br># Follow logs in real-time<br>journalctl -f CONTAINER_NAME=nginx<br><br># Query with multiple filters<br>journalctl CONTAINER_NAME=nginx PRIORITY=3<br><br># Show as JSON (structured)<br>journalctl -o json-pretty CONTAINER_NAME=nginx</pre><p>What you get:</p><pre>{<br>  &quot;MESSAGE&quot;: &quot;GET /api/users HTTP/1.1 200&quot;,<br>  &quot;CONTAINER_ID&quot;: &quot;abc123def456&quot;,<br>  &quot;CONTAINER_NAME&quot;: &quot;nginx&quot;,<br>  &quot;PRIORITY&quot;: &quot;6&quot;,<br>  &quot;_HOSTNAME&quot;: &quot;prod-app-03&quot;,<br>  &quot;_SYSTEMD_UNIT&quot;: &quot;docker.service&quot;<br>}</pre><p><strong>The advantage?</strong> Container logs and system logs in one place. Query across all containers easily. Automatic rotation handled by journald.</p><p><strong>The tradeoff?</strong> docker logs doesn’t work (you must use journalctl). And you’re tied to systemd-based Linux distros.</p><p>See how each driver solves different problems?</p><h3>5. How Disk Filling Kills Containers</h3><p>Now that we understand the drivers, let’s trace exactly what happens when log files fill your disk. This is important because the failure cascades through multiple stages.</p><p>I’m going to walk you through the actual 3 AM incident, stage by stage, with specific numbers.</p><h4>Stage 1: Normal Operation (T+0 days)</h4><pre>Time: Day 0 - Deployment<br>Container: api-service (chatty debug logging enabled by mistake)<br>Log rate: 5 MB/minute<br><br>Disk usage:<br>├─ /var/lib/docker/containers/  1.2 GB (10 containers, rotating logs)<br>├─ /var/lib/docker/overlay2/     2.0 GB (image layers)<br>└─ Total: 3.2 GB / 50 GB (6% used)</pre><p>Everything looks healthy. Monitoring shows normal disk usage.</p><p>But here’s the critical detail: <strong>No one noticed the api-service was logging at 5 MB/min because it’s a new deployment.</strong></p><h4>Stage 2: Silent Growth (T+3 days)</h4><pre>Time: Day 3 - 72 hours later<br>Log accumulation: 5 MB/min × 60 × 24 × 3 = 21,600 MB = 21.6 GB<br><br>Disk usage:<br>├─ /var/lib/docker/containers/  22.8 GB (api-service: 21.6 GB!)<br>├─ /var/lib/docker/overlay2/     2.0 GB<br>└─ Total: 24.8 GB / 50 GB (49% used)</pre><p>Disk usage alerts trigger at 80%, so no one gets paged yet. The growth is steady but not alarming on monitoring graphs.</p><h4>Stage 3: Critical Threshold (T+5 days)</h4><pre>Time: Day 5 - 120 hours later<br>Log accumulation: 5 MB/min × 60 × 24 × 5 = 36,000 MB = 36 GB<br><br>Disk usage:<br>├─ /var/lib/docker/containers/  37.2 GB (api-service: 36 GB)<br>├─ /var/lib/docker/overlay2/     2.0 GB<br>└─ Total: 39.2 GB / 50 GB (78% used)</pre><p>Still below 80% alert threshold, but getting close.</p><h4>Stage 4: The Cascade Begins (T+6 days, 3:17 AM)</h4><pre>Time: Day 6, 3:17 AM - 144 hours later<br>Log accumulation: 5 MB/min × 60 × 24 × 6 = 43,200 MB = 43.2 GB<br><br>Disk usage:<br>├─ /var/lib/docker/containers/  44.4 GB<br>├─ /var/lib/docker/overlay2/     2.0 GB<br>├─ System logs:                  2.6 GB<br>└─ Total: 49.0 GB / 50 GB (98% used)<br>3:17:42 AM - API container tries to write log:<br>ERROR: No space left on device<br>3:17:43 AM - Container crashes (can&#39;t write logs)<br>3:17:45 AM - Docker restarts container<br>3:17:47 AM - Container tries to write startup logs<br>ERROR: No space left on device<br>3:17:48 AM - Container crashes again<br>3:17:50 AM - Restart loop begins</pre><p>This is the part people don’t always anticipate: <strong>When there’s no disk space, containers can’t even start</strong> because Docker can’t write startup logs.</p><p>But wait, it gets worse:</p><h4>Stage 5: Cluster-Wide Cascade (T+6 days, 3:20 AM)</h4><pre>3:18 AM - Other containers start failing:<br>  - postgres: Can&#39;t write WAL files (no space)<br>  - redis: Can&#39;t persist AOF (no space)<br>  - nginx: Can&#39;t write access logs (no space)<br><br>3:19 AM - Docker daemon starts having issues:<br>  - Can&#39;t write container metadata<br>  - Can&#39;t update container state<br>  - Can&#39;t create new containers<br><br>3:20 AM - Monitoring system floods with alerts:<br>  - 15 containers in restart loop<br>  - Disk at 100%<br>  - Services unreachable</pre><p>From “one chatty container” to “complete host failure” in 3 minutes.</p><h3>6. Why “Just Add More Disk” Doesn’t Work</h3><p>At this point, you might be thinking: “Okay, this is a disk space problem. Can’t we just give the server more disk?”</p><p>I love this question because it reveals a subtle but important misunderstanding about log management.</p><p>Let’s say you increase the disk from 50 GB to 500 GB. Problem solved, right?</p><p>But here’s what actually happens:</p><p>With 500 GB disk and the same 5 MB/min logging rate:</p><pre>Time to fill:<br>500 GB / 5 MB/min = 100,000 minutes<br>= 1,666 hours<br>= 69 days</pre><p>You’ve bought yourself 69 days instead of 6 days. But <strong>you haven’t solved the problem</strong> — you’ve just delayed it.</p><p>And worse, during those 69 days:</p><ul><li>Logs are growing silently</li><li>No one is investigating why logging is so high</li><li>Disk space is being wasted on duplicate, unrotated logs</li><li>When it finally fails, the blast radius is bigger (more logs lost when you clean up)</li></ul><p>This is why I tell people: <strong>Unbounded growth always catches you eventually. The question is when, not if.</strong></p><p>The real solution isn’t more disk. It’s:</p><ul><li><strong>Log rotation</strong> (automatic cleanup)</li><li><strong>Compression</strong> (reduce space usage)</li><li><strong>Rate limiting</strong> (don’t log at 5 MB/min in the first place)</li><li><strong>Centralized logging</strong> (ship logs off-host)</li></ul><p>Let me show you what actually works…</p><h3>7. Solution 1: Configure Rotation (json-file or local)</h3><p>This was the immediate fix for the 3 AM incident, and it’s my recommended baseline for any Docker deployment.</p><p>The idea: Set maximum log sizes and rotation counts so Docker automatically manages disk space.</p><p>Here’s what it looks like:</p><pre># /etc/docker/daemon.json<br>{<br>  &quot;log-driver&quot;: &quot;local&quot;,<br>  &quot;log-opts&quot;: {<br>    &quot;max-size&quot;: &quot;10m&quot;,<br>    &quot;max-file&quot;: &quot;3&quot;,<br>    &quot;compress&quot;: &quot;true&quot;<br>  }<br>}</pre><p>Let me walk you through what’s happening here:</p><p>When Docker <strong>writes a log</strong>, it checks: “Is the current log file over 10 MB?”</p><ul><li>No → Keep writing to current file</li><li>Yes → Rotate: rename current to .1.gz, create new file</li></ul><p>When Docker <strong>rotates</strong>, it checks: “Do I have more than 3 files?”</p><ul><li>No → Keep all files</li><li>Yes → Delete oldest file</li></ul><p>Now let’s think about the <strong>disk usage</strong>:</p><pre>Per container maximum:<br>= max-size × max-file<br>= 10 MB × 3<br>= 30 MB (uncompressed)<br>= ~10-15 MB (with gzip compression)<br><br>50 containers:<br>= 50 × 15 MB (compressed)<br>= 750 MB total<br><br>vs. unbounded:<br>= 50 containers × 5 MB/min × 10,080 min/week<br>= 2,520,000 MB<br>= 2.5 TB/week</pre><p>From 2.5 TB per week down to 750 MB total. That’s <strong>99.97% reduction</strong>.</p><p><strong>But wait</strong>, you might be thinking, <strong>“What if I need those old logs for debugging?”</strong></p><p>Exactly right! That’s the tradeoff. You’re trading <strong>disk space for log retention</strong>. With max-file=3, you only keep the most recent 30 MB of logs.</p><p>But here’s why it works for most use cases: If you need long-term log retention, you shouldn’t be keeping logs on the Docker host anyway — you should be shipping them to centralized logging (Fluentd, CloudWatch, Splunk, etc.). We’ll cover that in Part 2 of this series.</p><p>For local development and single-host deployments, rotation is sufficient.</p><p>This is what I mean when I say: <strong>Set rotation limits based on your debugging needs, not unlimited growth.</strong></p><p><strong>When to use this:</strong></p><ul><li>✅ Single-host Docker deployments</li><li>✅ Development environments</li><li>✅ Small-scale production (&lt; 20 containers)</li><li>✅ Want to keep docker logs working</li></ul><p><strong>When NOT to use this:</strong></p><ul><li>❌ Multi-host orchestration (Kubernetes, Swarm)</li><li>❌ Need to query logs across hosts</li><li>❌ Compliance requires long retention (&gt; 1 week)</li><li>❌ High log volume (&gt; 1 GB/day per container)</li></ul><h3>8. Solution 2: Use journald for Centralized Host Logs</h3><p>This is, in my opinion, the best approach for systemd-based Linux servers running Docker.</p><p>The idea: Let systemd’s journal handle all logging — container logs, system logs, everything — with one unified configuration.</p><p>Here’s what it looks like:</p><pre># /etc/docker/daemon.json<br>{<br>  &quot;log-driver&quot;: &quot;journald&quot;,<br>  &quot;log-opts&quot;: {<br>    &quot;tag&quot;: &quot;{{.Name}}/{{.ID}}&quot;,<br>    &quot;labels&quot;: &quot;app,env&quot;<br>  }<br>}</pre><pre># /etc/systemd/journald.conf<br>[Journal]<br>SystemMaxUse=2G          # Max 2GB for all logs<br>SystemMaxFileSize=100M   # 100MB per file<br>MaxRetentionSec=7days    # Keep logs for 1 week<br>Compress=yes             # Gzip compression</pre><p>Now all your container logs go into the systemd journal with structured metadata:</p><pre># Query logs for a specific container<br>journalctl CONTAINER_NAME=api-service<br># Query logs across all containers with label<br>journalctl CONTAINER_TAG=env=production<br># Follow logs in real-time<br>journalctl -f CONTAINER_NAME=api-service<br># Query with time range<br>journalctl --since &quot;2025-11-28 03:00:00&quot; --until &quot;2025-11-28 04:00:00&quot;<br># Export as JSON for processing<br>journalctl -o json CONTAINER_NAME=api-service | jq &#39;.MESSAGE&#39;</pre><p>Let’s think about the <strong>disk usage</strong>:</p><p>With SystemMaxUse=2G:</p><ul><li>All container logs + system logs: max 2 GB total</li><li>Automatic rotation when limit reached</li><li>Oldest logs automatically deleted</li><li>Compression reduces actual space used</li></ul><p>From potential terrabytes down to a fixed 2 GB cap.</p><p><strong>The tradeoff?</strong> You lose docker logs command. You must use journalctl instead.</p><p><strong>Is that acceptable?</strong> For production Linux servers, yes! journalctl is more powerful than docker logs:</p><pre># docker logs can&#39;t do this:<br>journalctl CONTAINER_NAME=api-service --since &quot;2 hours ago&quot; -p err<br># (All errors from api-service in last 2 hours)<br>journalctl _SYSTEMD_UNIT=docker.service PRIORITY=3<br># (All critical messages from Docker daemon)</pre><p><strong>When to use journald:</strong></p><ul><li>✅ Systemd-based Linux (Ubuntu 16.04+, RHEL 7+, etc.)</li><li>✅ Single or small cluster of hosts</li><li>✅ Want unified container + system logging</li><li>✅ Comfortable with journalctl instead of docker logs</li></ul><p><strong>When NOT to use journald:</strong></p><ul><li>❌ Non-systemd systems (Alpine, older Ubuntu, etc.)</li><li>❌ Large-scale multi-host (use centralized logging)</li><li>❌ Team strongly prefers docker logs command</li></ul><h3>9. Common Mistakes I’ve Seen (And Made)</h3><p>Let me share some pitfalls from real production experiences:</p><h4>Mistake 1: Setting max-file Too Low</h4><p>I did this once. “Let’s keep only 1 log file to save space!”</p><pre>{<br>  &quot;max-size&quot;: &quot;10m&quot;,<br>  &quot;max-file&quot;: &quot;1&quot;<br>}</pre><p>The problem: When debugging an issue, I only had the last 10 MB of logs. The error that triggered the incident had already rotated out.</p><p>And worse: During investigation, continuous logging kept rotating out the logs I was trying to analyze.</p><p><strong>Lesson:</strong> Set max-file to at least 3-5. You need enough retention to debug issues without logs rotating out during investigation.</p><h4>Mistake 2: Forgetting to Restart Docker After daemon.json Changes</h4><p>This one’s embarrassing. I edited /etc/docker/daemon.json, thought I was done, and went home. The next day, containers were still using old settings.</p><pre># I edited the file:<br>sudo vim /etc/docker/daemon.json<br><br># But forgot this:<br>sudo systemctl restart docker</pre><p><strong>The catch:</strong> Daemon config only applies to NEW containers. Existing containers keep their old settings.</p><p><strong>Lesson:</strong> After editing daemon.json:</p><ol><li>Restart Docker daemon</li><li>Recreate containers (or restart them to pick up new settings)</li><li>Verify with docker inspect &lt;container&gt; | grep -A5 LogConfig</li></ol><h4>Mistake 3: Not Monitoring Disk Usage</h4><p>We set rotation to max-size=100m and max-file=5. Should be fine, right?</p><p>But we had 200 containers on one host:</p><pre>200 containers × 100 MB × 5 files = 100 GB of logs</pre><p>On an 80 GB disk.</p><p>The math didn’t work out.</p><p><strong>Lesson:</strong> Monitor disk usage with alerts:</p><pre># Alert at 80% disk usage<br>df -h | awk &#39;$NF==&quot;/&quot;{printf &quot;%s&quot;, $5}&#39; | sed &#39;s/%//&#39;</pre><p>And calculate total max usage:</p><pre>(number of containers) × (max-size) × (max-file) &lt; (0.5 × disk size)</pre><p>Keep logs under 50% of disk capacity to leave room for images, volumes, and temp files.</p><h4>Mistake 4: Using json-file in Production Without Compression</h4><p>We set rotation but didn’t enable compression:</p><pre>{<br>  &quot;max-size&quot;: &quot;50m&quot;,<br>  &quot;max-file&quot;: &quot;10&quot;<br>  // Missing: &quot;compress&quot;: &quot;true&quot;<br>}</pre><p>Result: 50 MB × 10 files = 500 MB per container (uncompressed)</p><p>With compression: ~100 MB per container (80% reduction)</p><p><strong>Lesson:</strong> Always enable compression for rotated logs. It’s nearly free (minimal CPU) and saves 70-90% disk space.</p><h3>10. Try This: Test Log Rotation in Action</h3><p>Theory is great, but nothing beats seeing it work. Here’s a 10-minute exercise:</p><h4>Setup (2 minutes)</h4><pre># Configure Docker with rotation<br>sudo tee /etc/docker/daemon.json &gt; /dev/null &lt;&lt;EOF<br>{<br>  &quot;log-driver&quot;: &quot;local&quot;,<br>  &quot;log-opts&quot;: {<br>    &quot;max-size&quot;: &quot;1m&quot;,<br>    &quot;max-file&quot;: &quot;3&quot;,<br>    &quot;compress&quot;: &quot;true&quot;<br>  }<br>}<br>EOF<br><br># Restart Docker<br>sudo systemctl restart docker</pre><h4>Create a Chatty Container (2 minutes)</h4><pre># Run a container that logs rapidly<br>docker run -d --name log-test \<br>  bash -c &#39;while true; do echo &quot;Log message $(date +%s.%N): $(head -c 1000 /dev/urandom | base64)&quot;; sleep 0.01; done&#39;</pre><p>This generates ~100 KB per second of logs.</p><h4>Observe Rotation (3 minutes)</h4><pre># Watch log files<br>watch -n 1 &#39;ls -lh /var/lib/docker/containers/$(docker inspect log-test --format &quot;{{.Id}}&quot;)/local-logs/&#39;</pre><p><strong>What you’ll see:</strong></p><pre># First minute:<br>-rw-r----- container.log  (growing)<br><br># After 1 minute (1 MB reached):<br>-rw-r----- container.log      (new, growing)<br>-rw-r----- container.log.1.gz (1 MB compressed to ~200 KB)<br><br># After 2 minutes:<br>-rw-r----- container.log      (new, growing)<br>-rw-r----- container.log.1.gz (most recent rotated)<br>-rw-r----- container.log.2.gz (older rotated)<br><br># After 3 minutes:<br>-rw-r----- container.log      (new, growing)<br>-rw-r----- container.log.1.gz (most recent)<br>-rw-r----- container.log.2.gz (older)<br>-rw-r----- container.log.3.gz (oldest)<br><br># After 4 minutes:<br># Oldest file deleted, only 3 kept!</pre><h4>Test docker logs Still Works (2 minutes)</h4><pre># View logs (works with local driver!)<br>docker logs --tail 20 log-test<br><br># Follow logs<br>docker logs -f log-test<br><br># Cleanup<br>docker rm -f log-test</pre><p><strong>Expected results:</strong></p><ul><li>Logs automatically rotate at 1 MB</li><li>Maximum 3 compressed files kept (~600 KB total)</li><li>docker logs command still work</li><li>Oldest logs automatically deleted</li></ul><p>This 10-minute experiment shows exactly how rotation protects your disk.</p><h3>11. Logging Is a Resource, Not Free</h3><p>Let me zoom out and give you the core insight that changed how I think about Docker logging.</p><p><strong>Logging in Docker is not “just” stdout. It’s a resource that consumes disk, I/O, and can bring down your entire host.</strong></p><p>When you run a Docker container, you’re not just running your application. You’re also:</p><ul><li>Writing logs to disk (I/O)</li><li>Consuming disk space (storage)</li><li>Potentially blocking on log writes (performance)</li><li>Creating files that must be cleaned up (maintenance)</li></ul><p>The question isn’t: <em>“Should I configure logging?”</em></p><p>The question is: <em>“How much log storage am I willing to allocate per container, and what happens when it’s full?”</em></p><p>And the answer isn’t “infinite” or “I’ll deal with it later.” It’s:</p><ul><li><strong>Set explicit limits</strong> (max-size, max-file)</li><li><strong>Enable compression</strong> (compress: true)</li><li><strong>Choose the right driver</strong> (local for single host, journald for systemd, fluentd for centralized)</li><li><strong>Monitor disk usage</strong> (alert at 80%)</li><li><strong>Plan for growth</strong> (containers × max-size × max-file &lt; 0.5 × disk)</li></ul><p>When you think about it this way, the configuration becomes obvious.</p><h3>12. Bringing It All Together: The Quick-Start Checklist</h3><p>Here’s a practical checklist for Docker logging configuration:</p><h3>For Development Machines</h3><pre>{<br>  &quot;log-driver&quot;: &quot;json-file&quot;,<br>  &quot;log-opts&quot;: {<br>    &quot;max-size&quot;: &quot;10m&quot;,<br>    &quot;max-file&quot;: &quot;3&quot;<br>  }<br>}</pre><ul><li>Fast, simple, docker logs works</li><li>30 MB max per container</li></ul><h3>For Single Production Host (Non-Systemd)</h3><pre>{<br>  &quot;log-driver&quot;: &quot;local&quot;,<br>  &quot;log-opts&quot;: {<br>    &quot;max-size&quot;: &quot;10m&quot;,<br>    &quot;max-file&quot;: &quot;5&quot;,<br>    &quot;compress&quot;: &quot;true&quot;<br>  }<br>}</pre><ul><li>Automatic rotation and compression</li><li>50 MB max per container (~15 MB compressed)</li></ul><h3>For Production Systemd Host</h3><pre>{<br>  &quot;log-driver&quot;: &quot;journald&quot;,<br>  &quot;log-opts&quot;: {<br>    &quot;tag&quot;: &quot;{{.Name}}&quot;<br>  }<br>}</pre><pre># /etc/systemd/journald.conf<br>[Journal]<br>SystemMaxUse=2G<br>MaxRetentionSec=7days<br>Compress=yes</pre><ul><li>Centralized logging with structured queries</li><li>2 GB total for all logs</li></ul><h3>For Multi-Host Production (Preview for Part 2)</h3><pre>{<br>  &quot;log-driver&quot;: &quot;fluentd&quot;,<br>  &quot;log-opts&quot;: {<br>    &quot;fluentd-address&quot;: &quot;fluentd.example.com:24224&quot;,<br>    &quot;tag&quot;: &quot;{{.Name}}&quot;<br>  }<br>}</pre><ul><li>Logs shipped to central server</li><li>No local disk usage</li></ul><h3>13. What’s Next: Centralized Logging</h3><p>In this article, we covered Docker’s logging architecture and how to prevent disk-filling incidents with proper rotation.</p><p>But we only solved half the problem. With local logging (even with rotation), you still can’t:</p><ul><li><strong>Search across multiple hosts</strong>: Each server has its own logs</li><li><strong>Correlate between services</strong>: Logs are scattered</li><li><strong>Retain long-term</strong>: Rotation deletes old logs</li><li><strong>Visualize trends</strong>: No aggregation or dashboards</li></ul><p>That’s where centralized logging comes in.</p><p><strong>In Part 2</strong>, we’ll build a production-grade centralized logging system using Fluentd and Grafana Loki:</p><ul><li>Ship logs from all containers to central server</li><li>Search and filter across all services</li><li>Long-term retention in object storage (S3)</li><li>Beautiful Grafana dashboards</li></ul><p><strong>In Part 3</strong>, we’ll cover production patterns at scale:</p><ul><li>Log sampling to reduce volume</li><li>Structured logging with JSON</li><li>Performance tuning for high-throughput</li><li>Troubleshooting common logging issues</li></ul><p><em>If you found this helpful, I’d love to hear about your Docker logging incidents. What filled your disk? How did you fix it? What mistakes did you make along the way? These shared experiences make all of us better engineers.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=161238859e18" width="1" height="1" alt=""><hr><p><a href="https://medium.com/engineering-deep-dives/when-your-docker-logs-fill-the-disk-and-kill-production-161238859e18">When Your Docker Logs Fill the Disk and Kill Production</a> was originally published in <a href="https://medium.com/engineering-deep-dives">Engineering Deep Dives</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exactly-Once Processing is a Lie: Designing Idempotent Systems at Scale]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/engineering-deep-dives/exactly-once-processing-is-a-lie-designing-idempotent-systems-at-scale-9b48f8b0aba5?source=rss----177d7e26856f---4"><img src="https://cdn-images-1.medium.com/max/2600/1*KaJYL142_gONVH776j2pTw.png" width="2938"></a></p><p class="medium-feed-snippet">About This Article</p><p class="medium-feed-link"><a href="https://medium.com/engineering-deep-dives/exactly-once-processing-is-a-lie-designing-idempotent-systems-at-scale-9b48f8b0aba5?source=rss----177d7e26856f---4">Continue reading on Engineering Deep Dives »</a></p></div>]]></description>
            <link>https://medium.com/engineering-deep-dives/exactly-once-processing-is-a-lie-designing-idempotent-systems-at-scale-9b48f8b0aba5?source=rss----177d7e26856f---4</link>
            <guid isPermaLink="false">https://medium.com/p/9b48f8b0aba5</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[idempotency]]></category>
            <category><![CDATA[payment-processing]]></category>
            <dc:creator><![CDATA[Shriom Tripathi]]></dc:creator>
            <pubDate>Thu, 26 Mar 2026 11:35:39 GMT</pubDate>
            <atom:updated>2026-03-26T11:35:41.049Z</atom:updated>
        </item>
    </channel>
</rss>