<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Muhammad Mustufa Ahmed on Medium]]></title>
        <description><![CDATA[Stories by Muhammad Mustufa Ahmed on Medium]]></description>
        <link>https://medium.com/@mustufaahmed?source=rss-6e071c8bef88------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*JQJATgPPy7oct1UqfQjWFg.jpeg</url>
            <title>Stories by Muhammad Mustufa Ahmed on Medium</title>
            <link>https://medium.com/@mustufaahmed?source=rss-6e071c8bef88------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 04:07:50 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mustufaahmed/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Medium outrage? Investigating the November 5, 2026, 504 Gateway Timeout Error]]></title>
            <link>https://mustufaahmed.medium.com/medium-504-gateway-timeout-november-5-2026-85b6f435488e?source=rss-6e071c8bef88------2</link>
            <guid isPermaLink="false">https://medium.com/p/85b6f435488e</guid>
            <category><![CDATA[backend-development]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[medium]]></category>
            <dc:creator><![CDATA[Muhammad Mustufa Ahmed]]></dc:creator>
            <pubDate>Thu, 05 Nov 2026 09:08:05 GMT</pubDate>
            <atom:updated>2026-11-05T09:08:05.507Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Medium users encountering a 504 Gateway Timeout can learn what happens behind the scenes, why websites stop responding, and how engineers prevent these failures</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iXDziJ7l3wJan2CNRnyLRA.png" /></figure><h3>Introduction: When a Publishing Platform Stops Responding</h3><p>On November 5, 2026, I tried to access Medium and encountered an unexpected error:</p><p><strong>504 — Gateway Time-out</strong></p><p>The error page displayed the message:</p><blockquote><em>Oops! Something happened to Medium. Please try that again in a moment.</em></blockquote><p>It also showed a Cloudflare location of Jeddah and explained that the web server had reported a gateway timeout.</p><p>At the time of writing, Medium is still inaccessible to me.</p><p>But there is an interesting detail: when I checked Medium’s public status page and independent monitoring services, they reported that the platform was operational. That creates an important distinction between a confirmed platform-wide outage and a failure affecting a particular user, network, region, or request path.</p><p>So, is Medium down?</p><p>From my connection, it is not responding successfully. However, the available public status information does not establish that Medium is experiencing a global outage.</p><p>This incident gives us an opportunity to understand a common distributed-systems problem: <strong>the 504 Gateway Timeout</strong>.</p><p>What happens when a browser can reach a gateway, but that gateway cannot get a timely response from the service behind it? Why might one user experience an error while monitoring services report that everything is operational? And what can engineers do to prevent these failures?</p><p>Let’s investigate.</p><h3>What Happened on November 5, 2026?</h3><p>Here is what we can establish from the available evidence.</p><p>DetailObservationDateNovember 5, 2026PlatformMediumError504 Gateway TimeoutError pageMedium-branded error pageNetwork infrastructure shownCloudflareCloudflare location displayedJeddahUser experienceMedium remained inaccessiblePublic status informationReported operational when checked</p><p>There are two categories of evidence here.</p><p><strong>Directly observed:</strong> The error page displayed a 504 response, identified Cloudflare’s Jeddah location, and Medium remained inaccessible from my connection.</p><p><strong>Independently checked:</strong> Medium’s public status page and several third-party monitoring services reported that the platform was operational when checked.</p><p>These observations are not necessarily contradictory. A service can be reachable from some monitoring locations while individual users experience failures through other network paths.</p><p>We also need to be careful about what the screenshot proves. The Cloudflare location identifies the edge location shown on the error page. It does not, by itself, identify the root cause of the timeout or prove that Cloudflare was responsible.</p><p>Without Medium’s internal logs, infrastructure metrics, and incident report, we cannot identify the actual failing component.</p><p>That distinction matters throughout this investigation.</p><h3>What Is a 504 Gateway Timeout?</h3><p>A <strong>504 Gateway Timeout</strong> is an HTTP status code indicating that a server acting as a gateway or proxy did not receive a timely response from an upstream server.</p><p>Let’s make that easier to understand.</p><p>Imagine visiting a restaurant.</p><ol><li>You place an order with a waiter.</li><li>The waiter sends the order to the kitchen.</li><li>The kitchen takes too long to respond.</li><li>The waiter eventually tells you that the order could not be completed in time.</li></ol><p>The waiter is not necessarily the source of the problem. The kitchen might be overloaded, unavailable, or waiting for another dependency.</p><p>A gateway works in a similar way.</p><p>When you open a website, your request may travel through several infrastructure components before reaching the application that handles it.</p><p>A simplified request path looks like this:</p><pre>USER<br>               |<br>               | HTTPS request<br>               v<br>          DNS Resolution<br>               |<br>               v<br>        CDN / Edge Network<br>               |<br>               v<br>       Load Balancer / Proxy<br>               |<br>               v<br>        Application Server<br>               |<br>               v<br>           Database<br>               |<br>               v<br>          Response<br>               |<br>               v<br>             USER</pre><p>This is a representative architecture for a modern web platform. It is not a verified diagram of Medium’s internal infrastructure.</p><p>A 504 can happen when an intermediary waits for an upstream component and does not receive a response within its configured timeout.</p><p>For example:</p><pre>Browser<br>   |<br>   | Request<br>   v<br>CDN / Gateway<br>   |<br>   | Forward request<br>   v<br>Application Server<br>   |<br>   | Waiting for database<br>   v<br>Database<br>   |<br>   | Response takes too long<br>   |<br>   X Gateway timeout</pre><p>The gateway may return a 504 because the application server, database, or another upstream dependency did not respond quickly enough.</p><p>The important point is that <strong>a 504 tells us that a request timed out somewhere along an upstream communication path. It does not tell us exactly which component failed.</strong></p><h3>504 vs. 502 vs. 503: What Is the Difference?</h3><p>These three HTTP status codes are related, but they describe different situations.</p><p>Status codeMeaningExample502 Bad GatewayA gateway received an invalid response from an upstream serverAn upstream server returned a malformed response503 Service UnavailableA service is temporarily unable to handle the requestA service is overloaded or undergoing maintenance504 Gateway TimeoutA gateway did not receive a timely response from an upstream serverAn application server took too long to respond</p><p>Consider these examples:</p><p><strong>502 Bad Gateway</strong></p><pre>Gateway ---&gt; Upstream server</pre><pre>Upstream server responds incorrectly.</pre><pre>Gateway ---&gt; 502 Bad Gateway</pre><p><strong>503 Service Unavailable</strong></p><pre>Client ---&gt; Application server</pre><pre>Application server cannot currently accept requests.</pre><pre>Client &lt;--- 503 Service Unavailable</pre><p><strong>504 Gateway Timeout</strong></p><pre>Gateway ---&gt; Application server</pre><pre>Gateway waits for a response.<br>No response arrives before the timeout.</pre><pre>Gateway ---&gt; 504 Gateway Timeout</pre><p>These status codes help engineers narrow down the type of failure, but they are not complete diagnoses.</p><p>A 504 does not automatically mean the database is down. A 503 does not necessarily mean the entire platform is unavailable. Engineers need logs, metrics, traces, and infrastructure information to identify the actual cause.</p><h3>Why Can Medium Show as Operational While Some Users Get a 504?</h3><p>This is one of the most interesting parts of the November 5 incident.</p><p>If Medium’s public status page reports that the platform is operational, why might a user still be unable to access it?</p><p>Because a large web platform is not one server in one location.</p><p>It can be reached through different networks, edge locations, DNS resolvers, routing paths, and infrastructure components.</p><p>A simplified example:</p><pre>MEDIUM<br>                          |<br>                 Multiple service paths<br>                          |<br>          +---------------+---------------+<br>          |               |               |<br>          v               v               v<br>      Region A         Region B        Region C<br>          |               |               |<br>          v               v               v<br>       Healthy         Healthy          Timeout<br>          |               |               |<br>          v               v               v<br>      Requests         Requests         Some users<br>      succeed          succeed         experience 504</pre><p>This is an illustrative example, not a map of Medium’s actual regions.</p><p>Several situations can produce this kind of mismatch.</p><h3>1. Regional or edge-specific problems</h3><p>Content delivery networks use geographically distributed edge infrastructure to serve users closer to their locations.</p><p>A request may travel through an edge location before reaching the origin service.</p><p>If one edge location or its upstream network path experiences problems, users routed through that path may encounter errors while other users continue browsing normally.</p><p>The Jeddah location displayed in the screenshot is useful diagnostic information. It does not prove that the Jeddah edge location itself caused the failure.</p><h3>2. ISP or network routing problems</h3><p>A user connects to a website through an internet service provider and multiple network operators.</p><p>A routing problem somewhere along that path can prevent successful communication with the destination.</p><p>For example:</p><pre>User<br> |<br> v<br>Internet Service Provider<br> |<br> v<br>Network Route<br> |<br> v<br>CDN Edge<br> |<br> v<br>Origin Service</pre><p>If one part of this route becomes unreliable, the user may experience connection failures even when the website works through other networks.</p><h3>3. DNS resolution differences</h3><p>DNS translates a domain name into an IP address.</p><p>Different DNS resolvers may return different answers depending on the platform’s DNS configuration, geographic routing, and caching.</p><p>If one resolver returns an incorrect or outdated answer, some users may have trouble reaching the service.</p><p>DNS problems do not always produce a 504 specifically, but they are worth checking when investigating a regional or user-specific accessibility problem.</p><h3>4. Partial service failures</h3><p>A website may have several independent services.</p><p>For example:</p><ul><li>Homepage rendering</li><li>Authentication</li><li>Article publishing</li><li>Article reading</li><li>Image delivery</li><li>Search</li><li>Notifications</li></ul><p>A monitoring service that checks the homepage might report success even if another part of the platform is failing.</p><p>Likewise, an error affecting only some requests may not appear in a basic availability check.</p><h3>5. Monitoring coverage limitations</h3><p>A status page is useful, but it is not a guarantee that every request from every location is working.</p><p>Monitoring systems may check only selected locations, endpoints, and user journeys.</p><p>A simple health check might confirm that a server responds to a lightweight request. It may not test whether users can sign in, load an article, or publish a draft.</p><p>That is why engineers need monitoring that represents actual user experiences, not only server availability.</p><p><strong>The lesson:</strong> A green status page and a real user-facing error can coexist. They measure different things.</p><h3>What Could Have Caused the Medium 504 Error?</h3><p>We know the error code and the user-facing behavior. We do not know Medium’s internal root cause.</p><p>The following are possible explanations based on common web-system failure patterns. They are hypotheses, not confirmed findings about Medium.</p><h3>1. Slow or overloaded application servers</h3><p>An application server processes incoming requests.</p><p>If too many requests arrive at once, the server may run out of available resources or become unable to process requests quickly enough.</p><p>Common causes include:</p><ul><li>High CPU utilization</li><li>Memory pressure</li><li>Too many concurrent requests</li><li>Slow application code</li><li>Insufficient server capacity</li><li>Long-running operations</li></ul><p>For example:</p><pre>Incoming requests:</pre><pre>Request 1 ----\<br>Request 2 -----\<br>Request 3 ------&gt; Application Server<br>Request 4 -----/       |<br>Request 5 ----/        |<br>                       v<br>                 Limited capacity<br>                       |<br>                       v<br>                Requests wait<br>                       |<br>                       v<br>                 Gateway timeout</pre><p>A gateway may eventually stop waiting and return a 504.</p><h3>2. Database performance problems</h3><p>Most publishing platforms depend on databases to store and retrieve information.</p><p>A request to load an article might require several operations:</p><ol><li>Find the article.</li><li>Retrieve author information.</li><li>Load publication metadata.</li><li>Retrieve related content.</li><li>Check access permissions.</li></ol><p>If database queries become slow, application requests can take longer than expected.</p><p>Possible database issues include:</p><ul><li>Expensive queries</li><li>Missing or ineffective indexes</li><li>Database connection exhaustion</li><li>Lock contention</li><li>High storage latency</li><li>Replication delays</li></ul><p>For example:</p><pre>User Request<br>     |<br>     v<br>Application Server<br>     |<br>     v<br>Database Connection Pool<br>     |<br>     | No available connection<br>     v<br>Request waits<br>     |<br>     v<br>Gateway timeout</pre><p>Notice that the database does not necessarily need to be completely unavailable. A shortage of available connections can also make the application appear unresponsive.</p><h3>3. CDN or upstream connectivity problems</h3><p>A CDN or reverse proxy sits between users and the origin infrastructure.</p><p>If the edge cannot establish or maintain a successful connection to its upstream service, requests may fail.</p><p>Possible reasons include:</p><ul><li>Network packet loss</li><li>Upstream connection failures</li><li>Routing instability</li><li>TLS handshake problems</li><li>Firewall restrictions</li><li>Origin server unavailability</li></ul><p>The Cloudflare-branded error page in the screenshot makes the CDN and gateway path relevant to investigate, but it does not establish which of these conditions occurred.</p><h3>4. Traffic spikes</h3><p>A sudden increase in traffic can put pressure on application servers, databases, caches, and network infrastructure.</p><p>Imagine a publishing platform receiving significantly more requests than its normal capacity.</p><pre>Normal traffic<br>                  |<br>                  v<br>            Application<br>                  |<br>                  v<br>              Database<br></pre><pre>             Traffic spike<br>                  |<br>                  v<br>          Too many concurrent<br>               requests<br>                  |<br>                  v<br>          Application queues<br>                  |<br>                  v<br>          Dependencies become<br>               overloaded<br>                  |<br>                  v<br>             504 errors</pre><p>Traffic spikes can create cascading failures.</p><p>An overloaded application server sends more work to the database. The database becomes slower. Requests remain open for longer. More connections are occupied. New requests wait even longer.</p><p>A problem in one component can spread across the system.</p><h3>5. Deployment or configuration problems</h3><p>Infrastructure and application deployments can introduce unexpected behavior.</p><p>For example:</p><ul><li>Incorrect upstream configuration</li><li>An unhealthy application release</li><li>A changed timeout value</li><li>A broken service discovery record</li><li>An incorrect firewall rule</li><li>A deployment that temporarily reduces available capacity</li></ul><p>These are possible causes of gateway failures, but there is no evidence available here that a deployment caused the November 5 error.</p><h3>How Should Engineers Investigate a 504?</h3><p>When a 504 occurs, the first mistake is to assume that the component displaying the error is necessarily the component that failed.</p><p>A gateway might be healthy while an upstream application is slow. The application might be healthy while its database is struggling. Or the origin might be healthy while one network path is failing.</p><p>A reliable investigation follows the request path.</p><h3>Step 1: Confirm the scope of the failure</h3><p>Start by answering these questions:</p><ul><li>Is the failure affecting one user or many users?</li><li>Does it happen from different networks?</li><li>Is it limited to one geographic region?</li><li>Does it affect every endpoint or only certain operations?</li><li>Did the issue begin after a deployment or configuration change?</li></ul><p>Test the same website using:</p><ul><li>Your normal Wi-Fi connection</li><li>A mobile network</li><li>Another browser</li><li>A private browsing window</li><li>A different device, if available</li></ul><p>If the site works on mobile data but not Wi-Fi, the result suggests that the network path may be relevant.</p><p>It does not prove that the ISP is responsible, but it gives engineers a useful direction.</p><h3>Step 2: Check DNS resolution</h3><p>Use a terminal to inspect the DNS response.</p><pre>nslookup medium.com</pre><p>On systems with dig:</p><pre>dig medium.com</pre><p>These commands help you inspect the DNS response returned by your configured resolver.</p><p>Compare results from different resolvers if you are investigating a suspected DNS issue.</p><p>Remember: successful DNS resolution does not prove that the application is healthy. It only confirms that the domain name was resolved.</p><h3>Step 3: Inspect the HTTP response</h3><p>Use curl to inspect the response headers:</p><pre>curl -I https://medium.com</pre><p>For more connection details:</p><pre>curl -v https://medium.com</pre><p>These commands can help identify:</p><ul><li>HTTP status codes</li><li>Response headers</li><li>TLS connection information</li><li>Connection failures</li><li>Redirect behavior</li></ul><p>Do not repeatedly run aggressive tests against a production service. A few controlled checks are enough for basic diagnosis.</p><h3>Step 4: Inspect gateway and application logs</h3><p>For engineers operating the platform, logs are essential.</p><p>A gateway log might contain information such as:</p><pre>timestamp: 2026-11-05T...<br>request_path: /some-endpoint<br>status_code: 504<br>upstream_status: timeout<br>upstream_response_time: 30s</pre><p>This is an illustrative log format, not a real Medium log.</p><p>Application logs may reveal:</p><ul><li>Slow request handlers</li><li>Database connection failures</li><li>Exceptions</li><li>Dependency timeouts</li><li>Queue backlogs</li></ul><p>The goal is to correlate the user-facing error with what happened inside the system.</p><h3>Step 5: Follow the request through distributed tracing</h3><p>Logs tell us what individual components recorded. Metrics show patterns. Distributed traces help connect operations across services.</p><p>For example:</p><pre>Incoming HTTP Request                 32.0s<br>|<br>+-- API Gateway                         0.1s<br>|<br>+-- Application Service                31.7s<br>    |<br>    +-- Authentication                  0.2s<br>    |<br>    +-- Article Service                 0.3s<br>    |<br>    +-- Database Query                 30.9s</pre><p>This example suggests that the database query consumed most of the request time.</p><p>Without tracing, engineers might spend time investigating the gateway even though the slow operation is inside a database dependency.</p><p>Tools such as OpenTelemetry can help collect and correlate traces, metrics, and logs.</p><h3>How Can Engineers Prevent 504 Gateway Timeouts?</h3><p>No distributed system can guarantee that every request will always succeed.</p><p>Networks fail. Servers become overloaded. Databases slow down. Dependencies become unavailable.</p><p>The goal is not to eliminate every failure. It is to prevent a local problem from becoming a platform-wide incident and to recover gracefully when failures occur.</p><p>Here are some important engineering practices.</p><h3>1. Set appropriate timeouts</h3><p>Every network request should have a reasonable timeout.</p><p>Without a timeout, a service may wait indefinitely for an unresponsive dependency.</p><p>Consider this example using Node.js and TypeScript:</p><pre>async function fetchWithTimeout(<br>  url: string,<br>  timeoutMs: number<br>): Promise&lt;Response&gt; {<br>  const controller = new AbortController();</pre><pre>  const timeout = setTimeout(() =&gt; {<br>    controller.abort();<br>  }, timeoutMs);</pre><pre>  try {<br>    return await fetch(url, {<br>      signal: controller.signal,<br>    });<br>  } finally {<br>    clearTimeout(timeout);<br>  }<br>}</pre><p>Usage:</p><pre>const response = await fetchWithTimeout(<br>  &quot;https://api.example.com/articles&quot;,<br>  5000<br>);</pre><p>The request is aborted if it exceeds five seconds.</p><p>This is a simplified example. A production implementation should also handle cancellation from the caller, classify timeout errors, and consider how response-body consumption fits within the overall deadline.</p><p>A timeout should not be chosen randomly.</p><p>Engineers should consider:</p><ul><li>Expected dependency latency</li><li>User-facing latency requirements</li><li>The total request deadline</li><li>Downstream timeout settings</li><li>The time required for cleanup and fallback behavior</li></ul><p>A useful principle is to establish an overall deadline and make sure downstream operations cannot consume more time than that deadline allows.</p><h3>2. Use retries carefully</h3><p>A temporary network problem may disappear after a short delay.</p><p>Retries can help recover from transient failures, but uncontrolled retries can make an overloaded service even worse.</p><p>Consider this scenario:</p><pre>Service A<br>   |<br>   | Request fails<br>   v<br>Retry immediately<br>   |<br>   v<br>Service B is already overloaded<br>   |<br>   v<br>More requests arrive<br>   |<br>   v<br>Service B becomes even slower</pre><p>This is known as a retry storm.</p><p>A safer retry strategy uses:</p><ul><li>Limited retry attempts</li><li>Exponential backoff</li><li>Random jitter</li><li>A total request deadline</li><li>Error classification</li><li>Idempotency for operations that may be repeated</li></ul><p>For example, a request might retry after progressively longer delays rather than retrying immediately.</p><pre>Attempt 1: Immediate<br>Attempt 2: Wait 500 ms<br>Attempt 3: Wait 1 second<br>Attempt 4: Wait 2 seconds</pre><p>Random jitter helps prevent many clients from retrying at exactly the same time.</p><p>Retries should be limited to operations and errors where retrying is safe. A timed-out payment request, for example, may have succeeded even if the caller never received its response.</p><p>For more detail, see: <strong>Retries, Timeouts, and Exponential Backoff: Why Retrying Is Not Always Safe</strong>.</p><h3>3. Implement circuit breakers</h3><p>A circuit breaker helps stop repeated requests to a dependency that is already failing.</p><p>It generally has three states:</p><ul><li><strong>Closed:</strong> Requests are allowed.</li><li><strong>Open:</strong> Requests are rejected or redirected to a fallback.</li><li><strong>Half-open:</strong> A limited number of requests are allowed to test recovery.</li></ul><pre>CLOSED<br>                    |<br>            Failures exceed limit<br>                    |<br>                    v<br>                  OPEN<br>                    |<br>             Wait for cooldown<br>                    |<br>                    v<br>                HALF-OPEN<br>                 /      \<br>          Success        Failure<br>             |              |<br>             v              v<br>          CLOSED           OPEN</pre><p>For example, if an article recommendation service becomes unavailable, the main article-reading service might stop waiting for it and return the article without recommendations.</p><p>This protects the core user experience.</p><p>For more detail, see: <strong>Circuit Breakers and Retry Budgets: Preventing One Failure From Taking Down Your System</strong>.</p><h3>4. Use caching strategically</h3><p>Caching can reduce the number of requests reaching expensive dependencies.</p><p>A publishing platform may cache:</p><ul><li>Public article content</li><li>Static assets</li><li>Author profile information</li><li>Frequently accessed metadata</li><li>Popular read-only pages</li></ul><p>A simplified architecture:</p><pre>User Request<br>     |<br>     v<br>   CDN Cache<br>     |<br>     +---- Cache Hit ----&gt; Return Content<br>     |<br>     |<br>     +---- Cache Miss ---&gt; Origin Service<br>                                |<br>                                v<br>                             Database</pre><p>When content is available in the cache, the origin does not need to process every request.</p><p>Caching is not a universal solution. Engineers must account for freshness, invalidation, authorization, and private content. Serving stale data may be acceptable for some public pages but dangerous for sensitive or rapidly changing information.</p><h3>5. Scale application capacity</h3><p>Horizontal scaling allows a system to run more application instances when demand increases.</p><p>A load balancer can distribute incoming traffic across healthy instances.</p><pre>Incoming Requests<br>                         |<br>                         v<br>                   Load Balancer<br>                         |<br>              +----------+----------+<br>              |          |          |<br>              v          v          v<br>           Server A   Server B   Server C<br>              |          |          |<br>              +----------+----------+<br>                         |<br>                         v<br>                      Database</pre><p>Autoscaling can increase or decrease the number of instances based on metrics such as CPU utilization, request count, or queue depth.</p><p>However, adding application servers does not automatically solve every capacity problem.</p><p>If all application instances depend on one database with limited connection capacity, scaling the application layer may simply move the bottleneck to the database.</p><p>Capacity planning must consider the entire request path.</p><h3>6. Protect the system with rate limiting</h3><p>Rate limiting controls how many requests a user, client, or service can make within a defined period.</p><p>It can protect services from excessive traffic and help preserve resources for legitimate users.</p><p>For example:</p><pre>Client<br>  |<br>  v<br>Rate Limiter<br>  |<br>  +---- Within limit ----&gt; Application<br>  |<br>  +---- Over limit ------&gt; HTTP 429</pre><p>Rate limiting is especially important for expensive endpoints and public APIs.</p><p>A well-designed policy should consider endpoint cost, user identity, tenant boundaries, and legitimate traffic patterns.</p><p>For more detail, see: <strong>Rate Limiting in Distributed Systems: Protecting APIs Without Hurting Users</strong>.</p><h3>7. Design graceful degradation</h3><p>When a dependency fails, the entire application does not always need to fail.</p><p>Imagine an article page that depends on several services:</p><pre>Article Page<br>                      |<br>          +-----------+-----------+<br>          |           |           |<br>          v           v           v<br>        Article     Comments   Recommendations<br>        Service     Service      Service<br>          |           |           |<br>          v           X           v<br>       Available   Unavailable  Available</pre><p>If the comments service is unavailable, the platform may still display the article and recommendations.</p><p>This is graceful degradation.</p><p>The system preserves its most important functionality while temporarily reducing optional features.</p><p>It is often better to show an article without comments than to show a gateway timeout for the entire page.</p><h3>A Production-Ready Reliability Strategy</h3><p>The techniques above work best when they are designed together rather than implemented independently.</p><p>A simplified resilient request flow might look like this:</p><pre>USER REQUEST<br>                       |<br>                       v<br>                 CDN / EDGE<br>                       |<br>                       v<br>                LOAD BALANCER<br>                       |<br>                       v<br>                RATE LIMITER<br>                       |<br>                       v<br>               APPLICATION SERVICE<br>                       |<br>              +--------+--------+<br>              |                 |<br>              v                 v<br>         CACHE LAYER       DEPENDENCY CALL<br>              |                 |<br>              |                 v<br>              |           TIMEOUT CONTROL<br>              |                 |<br>              |                 v<br>              |           CIRCUIT BREAKER<br>              |                 |<br>              |                 v<br>              |              DATABASE<br>              |                 |<br>              +--------+--------+<br>                       |<br>                       v<br>                RESPONSE / FALLBACK</pre><p>A production system should also include:</p><ul><li>Health checks and readiness checks</li><li>Structured logs</li><li>Distributed tracing</li><li>Latency and error-rate metrics</li><li>Capacity monitoring</li><li>Deployment health checks</li><li>Alerting based on user-facing service objectives</li><li>Recovery procedures and incident communication</li></ul><p>These controls help engineers detect failures early, limit their impact, and understand what happened afterward.</p><h3>Why Health Checks and Monitoring Must Go Beyond “Server Is Up”</h3><p>A server can be running while the application is effectively unusable.</p><p>For example:</p><pre>Server Process:       Running<br>HTTP Health Endpoint: Responding<br>Database:             Unreachable<br>Article Requests:     Timing Out</pre><p>A basic health check might still report success if it only checks whether the application process is alive.</p><p>Engineers should distinguish between:</p><p><strong>Liveness checks:</strong> Is the process alive, or does it need to be restarted?</p><p><strong>Readiness checks:</strong> Is the instance ready to receive production traffic?</p><p><strong>Dependency health:</strong> Can the application communicate with critical dependencies?</p><p><strong>Synthetic monitoring:</strong> Can a simulated user complete important journeys from different locations?</p><p>For a publishing platform, useful synthetic checks might include:</p><ol><li>Open the homepage.</li><li>Load a public article.</li><li>Load an author profile.</li><li>Sign in through a test account.</li><li>Open the editor.</li><li>Verify that a controlled test publishing workflow works.</li></ol><p>These checks should be designed carefully so they do not create unwanted content or modify real user data.</p><p>Monitoring from multiple geographic locations is particularly useful for identifying regional failures that a single monitoring location might miss.</p><h3>What Should a User Do When Medium Is Not Working?</h3><p>If you encounter a similar 504 error, here are practical steps to narrow down the problem.</p><h3>Step 1: Check the official status page</h3><p>Visit:</p><p><a href="https://status.medium.com/">Medium System Status</a></p><p>A status page can reveal acknowledged incidents and maintenance activity.</p><p>However, an operational status does not guarantee that every region, network path, or feature is working for every user.</p><h3>Step 2: Try a different network</h3><p>Switch between Wi-Fi and mobile data.</p><p>If the result changes, record that information. It can help distinguish a network-specific problem from a broader accessibility issue.</p><h3>Step 3: Try another browser or private window</h3><p>This can help rule out certain browser extensions, cached state, or session-related issues.</p><p>It will not fix an upstream server timeout, but it is a useful diagnostic step.</p><h3>Step 4: Avoid repeatedly submitting important actions</h3><p>This is especially important when publishing an article or performing another action that changes server-side data.</p><p>A timeout does not always mean the server failed to complete the operation.</p><p>The server might have successfully saved the article, but the response may have been lost or delayed before reaching the browser.</p><p>Before submitting again:</p><ul><li>Check whether the article was published.</li><li>Check your drafts.</li><li>Verify whether the action completed.</li><li>Keep a separate copy of important writing.</li></ul><p>This is the same uncertainty that makes idempotency important in distributed systems.</p><p>For more detail, see: <strong>Idempotency: The Small Design Decision That Prevents Big Production Failures</strong>.</p><h3>Step 5: Record useful diagnostic information</h3><p>If you report the problem, include:</p><ul><li>Date and approximate time</li><li>HTTP error code</li><li>Screenshot of the error</li><li>Browser and device</li><li>Network type</li><li>Whether another network works</li><li>Whether the issue affects one page or the entire site</li></ul><p>Avoid sharing passwords, authentication tokens, or private account information.</p><h3>What This Incident Teaches Us About Distributed Systems</h3><p>The November 5 Medium error is a useful reminder that modern websites depend on many interconnected systems.</p><p>A user sees one page and expects one response.</p><p>Behind that page, however, there may be DNS infrastructure, CDN edge networks, gateways, load balancers, application servers, databases, caches, and external services.</p><p>A failure in one part of the request path can affect the entire user experience.</p><p>There are several important engineering lessons.</p><p><strong>1. Availability is not binary.</strong></p><p>A platform can be available from one region and inaccessible from another. A homepage can work while publishing fails. A service can respond to health checks while real user requests time out.</p><p><strong>2. A status code is a symptom, not a root-cause analysis.</strong></p><p>A 504 tells us that an upstream response was not received in time. It does not identify the exact failing component.</p><p><strong>3. Monitoring must represent real user journeys.</strong></p><p>Checking whether a server responds is not the same as checking whether users can complete important tasks.</p><p><strong>4. Timeouts, retries, and circuit breakers must work together.</strong></p><p>Timeouts limit waiting. Retries help with selected transient failures. Circuit breakers prevent repeated calls from overwhelming unhealthy dependencies.</p><p><strong>5. Systems should preserve core functionality during partial failures.</strong></p><p>Graceful degradation can keep important features available even when optional dependencies are failing.</p><p><strong>6. Engineers need evidence before declaring a root cause.</strong></p><p>A screenshot, a status page, and a set of monitoring checks provide useful evidence. They do not replace internal logs, metrics, traces, and incident investigation.</p><h3>Frequently Asked Questions</h3><h3>What does a 504 Gateway Timeout mean?</h3><p>A 504 Gateway Timeout means that a server acting as a gateway or proxy did not receive a timely response from an upstream server. It usually indicates a problem somewhere along the request path, but the status code alone does not identify the failing component.</p><h3>Why is Medium showing a 504 error?</h3><p>The November 5, 2026 screenshot shows a Medium-branded 504 error page with a Cloudflare location of Jeddah. Possible explanations include an upstream timeout, network routing issue, regional edge problem, application overload, or dependency latency. The available evidence does not establish Medium’s actual root cause.</p><h3>Is Medium down globally on November 5, 2026?</h3><p>The user experience described here confirms that Medium was inaccessible from the affected connection. However, the public status information and third-party checks available when this article was prepared did not confirm a widespread global outage. A regional or partial failure remains possible.</p><h3>Does Cloudflare cause a 504 error?</h3><p>Not necessarily. A CDN or gateway may display the error because it did not receive a timely response from an upstream service. The underlying cause could be in the origin application, a dependency, or the network path. The presence of a Cloudflare-branded error page does not prove that Cloudflare caused the failure.</p><h3>Can a website be operational while some users experience 504 errors?</h3><p>Yes. Monitoring may cover only selected locations, endpoints, and user journeys. A regional network problem, edge-specific issue, or partial service failure can affect some users while other checks continue to succeed.</p><h3>How can engineers prevent 504 errors?</h3><p>Engineers can reduce the likelihood and impact of 504 errors through sensible timeouts, controlled retries, circuit breakers, caching, rate limiting, autoscaling, database optimization, graceful degradation, distributed tracing, and multi-region monitoring.</p><p>These practices cannot eliminate every outage, but they can make systems more resilient and failures easier to diagnose.</p><h3>Final Thoughts</h3><p>The November 5, 2026 Medium 504 error demonstrates an important lesson about modern web infrastructure: a service can appear operational from one monitoring location while remaining inaccessible to a user on another network path.</p><p>We know that the affected user encountered a 504 Gateway Timeout and that the error page displayed Cloudflare’s Jeddah location. We also know that the public status information checked for this investigation did not confirm a widespread outage.</p><p>What we do not know is which internal component, if any, caused the problem.</p><p>That is why responsible incident analysis separates observations from hypotheses.</p><p>For engineers, the broader lesson is to design systems that can tolerate slow dependencies, recover from transient failures, preserve essential functionality, and expose enough telemetry to make root-cause investigation possible.</p><p>A reliable system is not one that never fails.</p><p>It is one that can detect failures, contain their impact, recover safely, and give engineers the evidence they need to understand what happened.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=85b6f435488e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Rate Limiting in Distributed Systems: Protecting APIs Without Hurting Users]]></title>
            <link>https://mustufaahmed.medium.com/rate-limiting-distributed-systems-002d94c4d3ef?source=rss-6e071c8bef88------2</link>
            <guid isPermaLink="false">https://medium.com/p/002d94c4d3ef</guid>
            <category><![CDATA[redis]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[api-design]]></category>
            <category><![CDATA[rate-limiting]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Muhammad Mustufa Ahmed]]></dc:creator>
            <pubDate>Tue, 03 Nov 2026 10:32:49 GMT</pubDate>
            <atom:updated>2026-11-03T10:32:49.094Z</atom:updated>
            <content:encoded><![CDATA[<p>Understand rate-limiting algorithms, Redis-based distributed enforcement, and practical strategies for protecting APIs under heavy traffic.</p><figure><img alt="Dark purple infographic illustrating rate limiting in distributed systems, showing an API gateway, Redis-backed rate limiter, allowed and rejected requests, and four rate-limiting algorithms." src="https://cdn-images-1.medium.com/max/1024/1*4zIQVXIWs7GwZKrqJnw1FQ.png" /></figure><p>Imagine you’re building an API for an e-commerce platform.</p><p>Everything works perfectly during normal traffic. Customers browse products, add items to their carts, and place orders.</p><p>Then a marketing campaign goes live.</p><p>Within a few minutes, thousands of users start sending requests. Some clients refresh pages repeatedly. A few automated scripts generate excessive traffic. Other integrations retry failed requests aggressively.</p><p>Your application begins consuming more CPU, database connections become exhausted, and response times increase.</p><p>Eventually, even legitimate customers struggle to place orders.</p><p><strong>Rate limiting is a mechanism that controls how frequently clients can access an API or resource within a defined policy. It protects system capacity, reduces excessive traffic, and helps ensure that one client cannot consume an unfair share of available resources.</strong></p><p>But implementing it correctly is more complicated than counting requests.</p><p>In a distributed system, multiple application instances must coordinate their decisions, algorithms have different trade-offs, and poorly designed limits can hurt legitimate users.</p><p>Let’s explore how to design and implement rate limiting for production systems.</p><h3>Why Do APIs Need Rate Limiting?</h3><p>An API has finite resources.</p><p>Every incoming request consumes some combination of:</p><ul><li>CPU and memory.</li><li>Database connections.</li><li>Network bandwidth.</li><li>Cache capacity.</li><li>External service quota.</li><li>Application worker capacity.</li></ul><p>When traffic exceeds available capacity, the system may experience increased latency, timeouts, connection exhaustion, or complete service degradation.</p><p>Consider an application without traffic controls:</p><pre>Incoming Requests<br>                         |<br>          +--------------+--------------+<br>          |              |              |<br>          v              v              v<br>       Client A       Client B       Client C<br>          |              |              |<br>          +--------------+--------------+<br>                         |<br>                         v<br>                    API Server<br>                         |<br>                         v<br>                      Database<br>                         |<br>                         v<br>                   Resource Exhaustion</pre><p>A single client generating excessive traffic can affect everyone sharing the same infrastructure.</p><p>Rate limiting introduces a policy before expensive resources are consumed.</p><pre>Incoming Requests<br>                         |<br>                         v<br>                   API Gateway<br>                         |<br>                         v<br>                   Rate Limiter<br>                         |<br>                +--------+--------+<br>                |                 |<br>             Allowed           Rejected<br>                |                 |<br>                v                 v<br>           API Service       HTTP 429<br>                |<br>                v<br>             Database</pre><p>The rate limiter decides whether a request is allowed to continue.</p><p>This provides several benefits:</p><p><strong>1. Resource protection</strong></p><p>Limits prevent individual clients from consuming unlimited application capacity.</p><p><strong>2. Fair usage</strong></p><p>One client should not be able to monopolize resources that are shared with other customers.</p><p><strong>3. Cost control</strong></p><p>Requests to paid external APIs, AI inference services, and other metered resources can be limited.</p><p><strong>4. Service stability</strong></p><p>Traffic controls can reduce the risk of overload during unexpected traffic spikes.</p><p><strong>5. Abuse prevention</strong></p><p>Rate limits can make automated abuse and excessive request generation more difficult.</p><p>However, rate limiting is not a complete security solution. It does not replace authentication, authorization, bot detection, DDoS protection, or infrastructure-level traffic filtering.</p><h3>What Is the Difference Between Rate Limiting and Throttling?</h3><p>These terms are related, but they are not always used in exactly the same way.</p><p>ConceptMeaningRate limitingDefines how many requests a client may make within a policyThrottlingControls or slows traffic when it approaches or exceeds capacityQuotaDefines an allowed usage amount over a longer periodConcurrency limitingRestricts how many operations can execute simultaneously</p><p>For example:</p><p>An API might allow a user to make 100 requests per minute.</p><p>That is a rate limit.</p><p>A service might also allow only 20 simultaneous expensive report-generation operations.</p><p>That is a concurrency limit.</p><p>A subscription might include 50,000 API requests per month.</p><p>That is a quota.</p><p>These mechanisms solve different problems and can be combined.</p><h3>Which Rate-Limiting Algorithm Should You Choose?</h3><p>There is no single algorithm that works best for every API.</p><p>Different algorithms provide different levels of precision, memory consumption, and traffic flexibility.</p><p>Let’s understand the most common approaches.</p><h3>1. Fixed Window Counter</h3><p>The fixed-window algorithm divides time into fixed intervals.</p><p>For example, an API might allow 100 requests per minute.</p><p>The system counts requests within each minute and resets the counter when the next window begins.</p><pre>Minute 1                  Minute 2<br>|-----------------------|-----------------------|<br>0                      60s                     120s</pre><pre>Requests: 100            Requests: 100<br>Allowed:  YES            Allowed: YES</pre><p>A simple implementation looks like this:</p><pre>type FixedWindowState = {<br>  count: number;<br>  expiresAt: number;<br>};</pre><pre>class FixedWindowLimiter {<br>  private requests = new Map&lt;string, FixedWindowState&gt;();</pre><pre>  constructor(<br>    private readonly limit: number,<br>    private readonly windowMs: number<br>  ) {}</pre><pre>  allowRequest(clientId: string): boolean {<br>    const now = Date.now();<br>    const current = this.requests.get(clientId);</pre><pre>    if (!current || now &gt;= current.expiresAt) {<br>      this.requests.set(clientId, {<br>        count: 1,<br>        expiresAt: now + this.windowMs<br>      });</pre><pre>      return true;<br>    }</pre><pre>    if (current.count &gt;= this.limit) {<br>      return false;<br>    }</pre><pre>    current.count++;</pre><pre>    return true;<br>  }<br>}</pre><p>This example demonstrates the basic logic, but it is an in-memory implementation. It does not coordinate multiple application instances, and it needs cleanup and concurrency protection in a real application.</p><p><strong>Advantages:</strong></p><ul><li>Simple to understand.</li><li>Low memory requirements.</li><li>Easy to implement.</li><li>Efficient counter operations.</li></ul><p><strong>Disadvantages:</strong></p><ul><li>Traffic can burst around window boundaries.</li><li>Users may effectively consume nearly twice the configured limit in a short period.</li></ul><p>For example:</p><pre>12:00:00 - 100 requests<br>12:00:59 - 100 requests</pre><pre>Total: 200 requests in approximately 59 seconds</pre><p>Both groups fall into different fixed windows.</p><p>This boundary effect makes fixed windows unsuitable when strict short-term traffic control is required.</p><h3>2. Sliding Window Log</h3><p>A sliding-window log tracks the timestamp of every request.</p><p>Instead of resetting a counter at a fixed boundary, it considers requests that occurred during the previous time interval.</p><p>Suppose the policy allows 100 requests in the last 60 seconds.</p><p>For every incoming request:</p><ol><li>Remove timestamps older than 60 seconds.</li><li>Count the remaining timestamps.</li><li>Reject the request if the count has reached 100.</li><li>Otherwise, record the new request.</li></ol><pre>Current time: 12:01:30</pre><pre>Previous 60 seconds:</pre><pre>12:00:35  Request<br>12:00:42  Request<br>12:00:55  Request<br>12:01:10  Request<br>12:01:25  Request<br>12:01:30  New request</pre><p>Only requests inside the rolling 60-second interval count toward the limit.</p><p><strong>Advantages:</strong></p><ul><li>Accurate rolling-window enforcement.</li><li>Avoids the fixed-window boundary problem.</li><li>Easy to reason about.</li></ul><p><strong>Disadvantages:</strong></p><ul><li>Stores individual request timestamps.</li><li>Memory consumption grows with request volume.</li><li>Requires cleanup of expired entries.</li></ul><p>This approach is useful when accurate rolling-window behavior matters and request volume is manageable.</p><h3>3. Sliding Window Counter</h3><p>A sliding-window counter approximates the previous window’s request count using two fixed-window counters.</p><p>It considers:</p><ul><li>The previous window’s request count.</li><li>The current window’s request count.</li><li>How much of the previous window overlaps the rolling interval.</li></ul><p>For example:</p><pre>Previous Window          Current Window<br>|-----------------------|-----------------------|<br>                       12:01:00</pre><pre>Previous count: 80<br>Current count:  30</pre><pre>Current position: 25% into the current window</pre><p>The estimated count is:</p><p>[ C = C_{current} + C_{previous}(1-p) ]</p><p>Where:</p><ul><li>(C_{current}) is the current window count.</li><li>(C_{previous}) is the previous window count.</li><li>(p) is the elapsed fraction of the current window.</li></ul><p>In this example:</p><p>[ C = 30 + 80(1–0.25) = 90 ]</p><p>The estimated rolling-window usage is 90 requests.</p><p><strong>Advantages:</strong></p><ul><li>Lower memory consumption than a full request log.</li><li>Smoother behavior than a fixed-window counter.</li><li>Suitable for high-volume APIs.</li></ul><p><strong>Disadvantages:</strong></p><ul><li>It is an approximation.</li><li>It does not know the exact distribution of requests inside the previous window.</li></ul><p>This is often a practical compromise between accuracy and resource usage.</p><h3>4. Token Bucket Algorithm</h3><p>The token bucket algorithm allows controlled bursts while maintaining an average request rate.</p><p>Imagine a bucket containing tokens.</p><p>Each incoming request consumes one token. Tokens are replenished at a configured rate.</p><p>For example:</p><ul><li>Bucket capacity: 10 tokens.</li><li>Refill rate: 2 tokens per second.</li><li>Each request costs 1 token.</li></ul><p>If the bucket contains 10 tokens, the client can immediately send 10 requests.</p><p>After that, the client must wait for tokens to become available.</p><pre>Token Bucket<br>          +------------------+<br>          |  ● ● ● ● ●       |<br>          |  ● ● ● ● ●       |<br>          +------------------+<br>                   |<br>                   v<br>             Incoming Request<br>                   |<br>                   v<br>             Token Available?<br>                   |<br>             +-----+-----+<br>             |           |<br>            YES          NO<br>             |           |<br>             v           v<br>        Consume Token   Reject<br>             |<br>             v<br>        Allow Request</pre><p>A simplified TypeScript implementation:</p><pre>class TokenBucket {<br>  private tokens: number;<br>  private lastRefill: number;</pre><pre>  constructor(<br>    private readonly capacity: number,<br>    private readonly refillRate: number<br>  ) {<br>    this.tokens = capacity;<br>    this.lastRefill = Date.now();<br>  }</pre><pre>  allowRequest(): boolean {<br>    const now = Date.now();</pre><pre>    const elapsedSeconds =<br>      (now - this.lastRefill) / 1000;</pre><pre>    const replenished =<br>      elapsedSeconds * this.refillRate;</pre><pre>    this.tokens = Math.min(<br>      this.capacity,<br>      this.tokens + replenished<br>    );</pre><pre>    this.lastRefill = now;</pre><pre>    if (this.tokens &lt; 1) {<br>      return false;<br>    }</pre><pre>    this.tokens -= 1;</pre><pre>    return true;<br>  }<br>}</pre><p>This implementation uses floating-point tokens and is intended to explain the algorithm. It is not safe for concurrent requests or multiple application instances without synchronization.</p><p><strong>Advantages:</strong></p><ul><li>Supports controlled bursts.</li><li>Maintains a configurable average request rate.</li><li>Efficient state representation.</li><li>Useful for API gateways and client-level limits.</li></ul><p><strong>Disadvantages:</strong></p><ul><li>Requires accurate refill calculations.</li><li>Concurrent updates need atomic coordination.</li><li>Burst capacity must be configured carefully.</li></ul><p>Token buckets are especially useful when legitimate clients occasionally need to send short bursts of requests.</p><h3>5. Leaky Bucket Algorithm</h3><p>The leaky bucket algorithm focuses on controlling the rate at which requests leave a system.</p><p>Imagine a bucket with a small hole.</p><p>Requests enter the bucket, but they leave at a controlled rate.</p><p>If requests arrive faster than the bucket can process them, the queue grows. Once capacity is reached, additional requests must be rejected.</p><pre>Incoming Requests<br>                  |<br>                  v<br>          +---------------+<br>          |     Queue     |<br>          |               |<br>          | Request A     |<br>          | Request B     |<br>          | Request C     |<br>          +---------------+<br>                  |<br>                  v<br>           Fixed Processing<br>                Rate<br>                  |<br>                  v<br>             API Service</pre><p><strong>Advantages:</strong></p><ul><li>Smooths traffic bursts.</li><li>Provides predictable processing rates.</li><li>Useful for downstream systems with strict capacity limits.</li></ul><p><strong>Disadvantages:</strong></p><ul><li>Queuing introduces additional latency.</li><li>Queue memory must be bounded.</li><li>Requests may expire before processing.</li><li>Requires careful overload handling.</li></ul><p>A useful distinction:</p><ul><li>Token bucket controls when requests may enter based on available tokens.</li><li>Leaky bucket controls the rate at which queued work is processed.</li></ul><p>Some implementations use the term “leaky bucket” differently, so always inspect the actual algorithm rather than relying on its name.</p><h3>How Do You Implement Distributed Rate Limiting?</h3><p>The previous examples work in a single process.</p><p>But production applications often run multiple instances behind a load balancer.</p><p>Consider this architecture:</p><pre>Clients<br>                     |<br>                     v<br>                Load Balancer<br>                     |<br>          +----------+----------+<br>          |          |          |<br>          v          v          v<br>       API-1      API-2      API-3<br>          |          |          |<br>          +----------+----------+<br>                     |<br>                     v<br>                Redis Store<br>                     |<br>                     v<br>              Shared Counters</pre><p>Suppose a user has a limit of 100 requests per minute.</p><p>If each API instance maintains its own counter, the user could potentially make 100 requests to each instance.</p><p>With three instances, that could become 300 requests.</p><p>The application has accidentally tripled the intended limit.</p><p>This is why distributed rate limiting needs shared state or another coordinated enforcement mechanism.</p><p>Redis is a common choice because it provides fast shared storage and atomic operations.</p><p>However, simply storing counters in Redis is not enough. The counter update and limit check must happen atomically.</p><h3>How Can Redis Implement a Sliding Window Rate Limiter?</h3><p>Let’s implement a sliding-window log using Redis sorted sets.</p><p>Each request is stored as a member with its timestamp as the score.</p><p>The key structure might look like:</p><pre>rate-limit:user:123</pre><pre>Sorted Set:</pre><pre>Request ID       Timestamp<br>req-1            1000<br>req-2            1200<br>req-3            1500<br>req-4            1800</pre><p>For every incoming request:</p><ol><li>Remove entries outside the rolling time window.</li><li>Count the remaining requests.</li><li>Check whether the configured limit has been reached.</li><li>If allowed, add the current request.</li><li>Set a key expiration to clean up inactive client state.</li></ol><p>These steps must execute atomically.</p><p>Otherwise, two application instances could both read the same count and both allow a request that exceeds the limit.</p><p>Redis Lua scripts allow these operations to execute as one atomic server-side operation.</p><h3>Step 1: Create the Redis Lua script</h3><pre>local key = KEYS[1]</pre><pre>local now = tonumber(ARGV[1])<br>local window = tonumber(ARGV[2])<br>local limit = tonumber(ARGV[3])<br>local requestId = ARGV[4]</pre><pre>local windowStart = now - window</pre><pre>-- Remove expired request entries.<br>redis.call(<br>  &quot;ZREMRANGEBYSCORE&quot;,<br>  key,<br>  &quot;-inf&quot;,<br>  windowStart<br>)</pre><pre>-- Count requests inside the rolling window.<br>local currentCount = redis.call(&quot;ZCARD&quot;, key)</pre><pre>if currentCount &gt;= limit then<br>  local oldest = redis.call(<br>    &quot;ZRANGE&quot;,<br>    key,<br>    0,<br>    0,<br>    &quot;WITHSCORES&quot;<br>  )</pre><pre>  local retryAfter = 0</pre><pre>  if oldest[2] then<br>    retryAfter = math.ceil(<br>      (tonumber(oldest[2]) + window - now) / 1000<br>    )<br>  end</pre><pre>  return {0, currentCount, retryAfter}<br>end</pre><pre>-- Record the accepted request.<br>redis.call(<br>  &quot;ZADD&quot;,<br>  key,<br>  now,<br>  requestId<br>)</pre><pre>-- Keep inactive keys from accumulating indefinitely.<br>redis.call(&quot;PEXPIRE&quot;, key, window)</pre><pre>return {1, currentCount + 1, 0}</pre><p>The script returns:</p><ul><li>1 when the request is allowed.</li><li>0 when the request is rejected.</li><li>The current request count.</li><li>An approximate number of seconds until a retry may become possible.</li></ul><p>The returned retry delay is based on the oldest recorded request. It is an estimate of when capacity may become available, not a guarantee that the next request will be accepted.</p><h3>Step 2: Call the script from Node.js</h3><p>We’ll use TypeScript and the ioredis package.</p><pre>npm install ioredis</pre><p>Example implementation:</p><pre>import Redis from &quot;ioredis&quot;;<br>import { randomUUID } from &quot;node:crypto&quot;;</pre><pre>const redis = new Redis(process.env.REDIS_URL!);</pre><pre>const slidingWindowScript = `<br>  local key = KEYS[1]</pre><pre>  local now = tonumber(ARGV[1])<br>  local window = tonumber(ARGV[2])<br>  local limit = tonumber(ARGV[3])<br>  local requestId = ARGV[4]</pre><pre>  local windowStart = now - window</pre><pre>  redis.call(<br>    &quot;ZREMRANGEBYSCORE&quot;,<br>    key,<br>    &quot;-inf&quot;,<br>    windowStart<br>  )</pre><pre>  local count = redis.call(&quot;ZCARD&quot;, key)</pre><pre>  if count &gt;= limit then<br>    local oldest = redis.call(<br>      &quot;ZRANGE&quot;,<br>      key,<br>      0,<br>      0,<br>      &quot;WITHSCORES&quot;<br>    )</pre><pre>    local retryAfter = 0</pre><pre>    if oldest[2] then<br>      retryAfter = math.ceil(<br>        (tonumber(oldest[2]) + window - now) / 1000<br>      )<br>    end</pre><pre>    return {0, count, retryAfter}<br>  end</pre><pre>  redis.call(<br>    &quot;ZADD&quot;,<br>    key,<br>    now,<br>    requestId<br>  )</pre><pre>  redis.call(&quot;PEXPIRE&quot;, key, window)</pre><pre>  return {1, count + 1, 0}<br>`;</pre><pre>type RateLimitResult = {<br>  allowed: boolean;<br>  currentCount: number;<br>  retryAfterSeconds: number;<br>};</pre><pre>async function checkRateLimit(<br>  clientId: string<br>): Promise&lt;RateLimitResult&gt; {<br>  const now = Date.now();</pre><pre>  const windowMs = 60_000;<br>  const limit = 100;</pre><pre>  const key = `rate-limit:${clientId}`;</pre><pre>  const result = await redis.eval(<br>    slidingWindowScript,<br>    1,<br>    key,<br>    now,<br>    windowMs,<br>    limit,<br>    randomUUID()<br>  ) as number[];</pre><pre>  return {<br>    allowed: result[0] === 1,<br>    currentCount: result[1],<br>    retryAfterSeconds: result[2]<br>  };<br>}</pre><p>This implementation uses one Redis sorted set per client and one Lua script to perform the check and update atomically.</p><p>It is a useful starting point, but production use requires additional decisions about key design, clock behavior, Redis failures, deployment topology, and request identity.</p><p>For example, this implementation uses the application server’s timestamp. In a multi-instance deployment, clock synchronization matters because different application instances could supply slightly different times.</p><p>For stricter consistency, you can obtain the current time from Redis inside the script using Redis’s TIME command.</p><h3>Step 3: Apply the limiter in an Express middleware</h3><pre>import { Request, Response, NextFunction } from &quot;express&quot;;</pre><pre>export async function rateLimitMiddleware(<br>  req: Request,<br>  res: Response,<br>  next: NextFunction<br>) {<br>  try {<br>    // Use an authenticated identity when available.<br>    const clientId = req.user?.id ?? req.ip;</pre><pre>    const result = await checkRateLimit(clientId);</pre><pre>    res.setHeader(&quot;RateLimit-Limit&quot;, &quot;100&quot;);<br>    res.setHeader(<br>      &quot;RateLimit-Remaining&quot;,<br>      String(<br>        Math.max(0, 100 - result.currentCount)<br>      )<br>    );</pre><pre>    if (!result.allowed) {<br>      res.setHeader(<br>        &quot;Retry-After&quot;,<br>        String(result.retryAfterSeconds)<br>      );</pre><pre>      return res.status(429).json({<br>        error: &quot;Too many requests&quot;,<br>        message: &quot;Please try again later.&quot;<br>      });<br>    }</pre><pre>    next();<br>  } catch (error) {<br>    next(error);<br>  }<br>}</pre><p>The middleware checks the client’s usage before allowing the request to continue.</p><p>A production implementation should also validate the trusted proxy configuration before relying on req.ip, and it should use authenticated identities for user-specific limits where possible.</p><p>The RateLimit-* headers shown here are illustrative. If you expose standardized rate-limit headers, follow the applicable HTTP specifications and ensure the values have the meanings your clients expect.</p><h3>Why Is HTTP 429 Important?</h3><p>When a client exceeds its allowed request rate, the standard HTTP response is:</p><pre>HTTP/1.1 429 Too Many Requests<br>Content-Type: application/json<br>Retry-After: 30</pre><p>Example response:</p><pre>{<br>  &quot;error&quot;: &quot;Too many requests&quot;,<br>  &quot;message&quot;: &quot;You have exceeded the allowed request rate.&quot;<br>}</pre><p>The Retry-After header tells the client how long it should wait before trying again.</p><p>This is useful because clients should not continuously retry requests that have been rejected due to rate limits.</p><p>A good API should communicate:</p><ul><li>That the request was rejected due to a limit.</li><li>When the client may try again.</li><li>Which limit was reached, where appropriate.</li><li>Whether the limit applies to a user, API key, endpoint, or account.</li></ul><p>Clear error responses help clients implement better retry behavior.</p><h3>How Should You Design Rate Limits for Different API Endpoints?</h3><p>Not all API operations consume the same amount of resources.</p><p>Consider these endpoints:</p><p>EndpointResource costExample policyGET /productsLowHigher request allowanceGET /profileLowModerate request allowancePOST /ordersMediumLower request allowancePOST /reportsHighStrict rate and concurrency limitsPOST /ai/generateVery highUsage-based limits and quotas</p><p>These are illustrative policies, not universal values.</p><p>An AI inference endpoint, for example, may consume substantially more compute than a simple profile lookup.</p><p>A single global limit is therefore rarely enough for a complex application.</p><p>A better design can apply multiple policies:</p><pre>Incoming Request<br>       |<br>       v<br>Identify Client<br>       |<br>       v<br>Check Global Limit<br>       |<br>       v<br>Check Account Limit<br>       |<br>       v<br>Check Endpoint Limit<br>       |<br>       v<br>Check Resource / Concurrency Limit<br>       |<br>       v<br>Allow or Reject</pre><p>The exact combination depends on the system.</p><p>For example, you might enforce:</p><ul><li>A global protection limit to defend the service.</li><li>A per-API-key limit for client fairness.</li><li>A per-user limit for authenticated usage.</li><li>A per-endpoint limit for expensive operations.</li><li>A monthly quota for subscription plans.</li></ul><p>Be careful not to create so many independent limits that clients cannot understand why requests are being rejected.</p><h3>What Are Common Rate-Limiting Mistakes in Production?</h3><p>A rate limiter can become a source of problems if its policies and failure behavior are not designed carefully.</p><h3>1. Using only IP addresses as client identity</h3><p>IP-based limiting is useful for anonymous traffic, but it has limitations.</p><p>Multiple legitimate users may share one public IP address through corporate networks, mobile carriers, or shared gateways.</p><p>On the other hand, one user may use multiple IP addresses.</p><p>For authenticated APIs, consider user IDs, API keys, or tenant IDs as the primary identity, while retaining IP-based protection for suspicious traffic.</p><h3>2. Keeping counters inside application memory</h3><p>A local counter works for a single process, but it does not provide a shared limit across multiple application instances.</p><p>Use shared state or an infrastructure-level rate limiter when the policy must apply across instances.</p><h3>3. Making every endpoint use the same limit</h3><p>A simple read request and an expensive report-generation request should not necessarily have the same allowance.</p><p>Design policies around resource consumption and business requirements.</p><h3>4. Forgetting concurrency limits</h3><p>A rate limit controls how frequently requests arrive.</p><p>It does not necessarily control how many long-running operations execute simultaneously.</p><p>A client could send requests slowly enough to stay under the rate limit while still maintaining hundreds of expensive operations in progress.</p><p>Use concurrency limits or bounded queues for expensive work.</p><h3>5. Ignoring Redis failures</h3><p>If Redis becomes unavailable, the rate limiter may not be able to make its decision.</p><p>You need an explicit failure policy.</p><ul><li><strong>Fail open:</strong> Allow traffic when the limiter cannot be reached.</li><li><strong>Fail closed:</strong> Reject traffic when the limiter cannot be reached.</li></ul><p>Fail open improves availability but may expose the service to excessive traffic.</p><p>Fail closed protects resources but may block legitimate users during a Redis outage.</p><p>A practical system can use different policies for different endpoints. A low-cost public read endpoint might tolerate fail-open behavior, while an expensive operation may require stricter protection.</p><h3>6. Trusting unverified proxy headers</h3><p>Using X-Forwarded-For directly without a trusted proxy configuration can allow clients to spoof their apparent IP address.</p><p>Configure trusted proxies correctly and derive client identity only from trusted infrastructure.</p><h3>7. Creating unbounded Redis keys</h3><p>If every arbitrary client identifier creates a new key, attackers can generate many unique identifiers and consume Redis memory.</p><p>Use authentication where appropriate, validate identifiers, apply key expiration, and monitor storage usage.</p><h3>8. Forgetting retry behavior</h3><p>When an API returns HTTP 429, clients should respect Retry-After when supplied.</p><p>Repeatedly retrying a rejected request immediately only increases traffic and makes the situation worse.</p><h3>How Should You Monitor a Distributed Rate Limiter?</h3><p>Rate limiting should be observable like any other critical infrastructure component.</p><p>Useful metrics include:</p><ul><li>Total allowed requests.</li><li>Total rejected requests.</li><li>Rejection rate by endpoint.</li><li>Rejection rate by client or tenant category.</li><li>Redis operation latency.</li><li>Redis errors and timeouts.</li><li>Rate-limit key count and memory consumption.</li><li>Number of requests reaching expensive downstream services.</li><li>Frequency of HTTP 429 responses.</li></ul><p>These metrics help answer important operational questions:</p><ul><li>Are legitimate customers being blocked too frequently?</li><li>Are some API clients generating unusually high traffic?</li><li>Are limits too restrictive?</li><li>Is Redis becoming a bottleneck?</li><li>Are expensive endpoints adequately protected?</li><li>Are clients respecting retry instructions?</li></ul><p>Avoid using raw user IDs, API keys, or request IDs as high-cardinality metric labels. Use aggregation and carefully controlled dimensions.</p><p>Also, alerting on every HTTP 429 is usually not useful. A rate limiter is expected to reject some traffic. Alert on unusual rejection patterns, infrastructure failures, or evidence that legitimate usage is being disrupted.</p><h3>Frequently Asked Questions</h3><h3>What is the best algorithm for API rate limiting?</h3><p>There is no universal best algorithm. Fixed windows are simple, sliding windows provide more precise rolling limits, and token buckets allow controlled bursts. Choose based on accuracy, traffic patterns, and implementation cost.</p><h3>Why is Redis commonly used for distributed rate limiting?</h3><p>Redis provides shared, low-latency state and supports atomic operations and Lua scripts. This makes it useful for coordinating rate-limit decisions across multiple application instances.</p><h3>Should rate limiting happen at the API gateway or application level?</h3><p>Both are valid. Gateways are useful for broad traffic protection, while application-level enforcement can apply business-specific policies based on users, tenants, subscriptions, and endpoints. Many production systems use both.</p><h3>Is rate limiting enough to prevent DDoS attacks?</h3><p>No. Application rate limiting helps control excessive requests, but large-scale DDoS protection requires infrastructure-level controls, traffic filtering, network protections, and specialized mitigation services.</p><h3>What happens when Redis is unavailable?</h3><p>The rate limiter cannot reliably access its shared state. The application must follow an explicit fail-open or fail-closed policy, depending on the resource cost and business risk of the endpoint.</p><h3>What is the difference between rate limiting and concurrency limiting?</h3><p>Rate limiting controls how many requests can arrive over time. Concurrency limiting controls how many operations can execute simultaneously. Both are important for protecting expensive resources.</p><h3>Final Thoughts</h3><p>Rate limiting is not simply about rejecting requests when a counter reaches a particular number.</p><p>It is about managing shared resources fairly and protecting the system from traffic patterns it cannot safely handle.</p><p>A fixed-window counter may be sufficient for a simple API. A token bucket may be better when controlled bursts are acceptable. A sliding-window approach may be more appropriate when accurate rolling limits matter.</p><p>For distributed applications, shared state and atomic enforcement are essential when a policy must apply consistently across multiple instances.</p><p>But the most important design decision is understanding what you’re protecting.</p><p>A good rate limiter should protect your database, application workers, external services, and infrastructure without unnecessarily blocking legitimate customers.</p><p><strong>The goal is not to accept as many requests as possible. The goal is to keep your system reliable while serving the requests that matter.</strong></p><p>In our next article, we’ll explore distributed locks and leases: how to coordinate work across multiple application instances without allowing concurrent workers to corrupt shared state.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=002d94c4d3ef" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Circuit Breakers and Retry Budgets: Preventing One Failure From Taking Down Your System]]></title>
            <link>https://mustufaahmed.medium.com/circuit-breakers-retry-budgets-75affadab4a4?source=rss-6e071c8bef88------2</link>
            <guid isPermaLink="false">https://medium.com/p/75affadab4a4</guid>
            <category><![CDATA[circuit-breaker]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[fault-tolerance]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <dc:creator><![CDATA[Muhammad Mustufa Ahmed]]></dc:creator>
            <pubDate>Sat, 31 Oct 2026 07:48:32 GMT</pubDate>
            <atom:updated>2026-10-31T08:27:25.819Z</atom:updated>
            <content:encoded><![CDATA[<p>Learn how circuit breakers, retry budgets, and exponential backoff work together to prevent cascading failures in distributed systems.</p><figure><img alt="Circuit breaker pattern showing closed, open, and half-open states in a distributed system" src="https://cdn-images-1.medium.com/max/1024/1*-fCu6NT9kql8bFrU-1Z_bA.png" /></figure><p>Imagine your Node.js API depends on a payment service.</p><p>Everything works normally until the payment provider becomes slow. Requests start taking several seconds. Your application keeps waiting, retries failed requests, and creates even more traffic.</p><p>Soon, your API is running out of available connections. Other endpoints begin slowing down, even though they do not directly depend on the payment provider.</p><p>One failing dependency has started affecting your entire system.</p><p><strong>Circuit breakers and retry budgets help prevent this situation. Circuit breakers temporarily stop requests to unhealthy dependencies, while retry budgets limit how much additional traffic retries can generate.</strong></p><p>They solve different problems, but together they form an important part of fault tolerance in distributed systems.</p><p>Let’s understand how they work and how to implement the ideas in a Node.js application.</p><h3>Why Do Distributed Systems Need Circuit Breakers?</h3><p>In a distributed system, services communicate over networks. Networks introduce uncertainty:</p><ul><li>A service may become unavailable.</li><li>A request may take longer than expected.</li><li>A dependency may return intermittent errors.</li><li>Network connections may fail.</li><li>A downstream service may become overloaded.</li></ul><p>Retries can help with temporary failures, but they can also make an existing failure worse.</p><p>Consider this example:</p><pre>API Gateway<br>    |<br>    v<br>Order Service<br>    |<br>    v<br>Payment Service<br>    |<br>    v<br>Payment Provider</pre><p>If the payment provider becomes slow, the Order Service may keep waiting for responses.</p><p>While waiting, it holds resources such as connections, memory, and request-processing capacity.</p><p>If the application retries every failed request, the provider receives even more traffic.</p><p>This can create a retry storm.</p><p>A <strong>circuit breaker</strong> helps by detecting repeated failures and temporarily stopping calls to an unhealthy dependency.</p><p>Instead of allowing every request to reach the failing service, the application fails fast or uses an appropriate fallback.</p><h3>How Does the Circuit Breaker Pattern Work?</h3><p>A circuit breaker has three primary states:</p><ol><li><strong>Closed:</strong> Requests are allowed through. The breaker monitors failures.</li><li><strong>Open:</strong> Requests are blocked temporarily because the dependency appears unhealthy.</li><li><strong>Half-open:</strong> A limited number of test requests are allowed to check whether the dependency has recovered.</li></ol><p>The state transitions look like this:</p><pre>Failures reach threshold<br>       +--------------------------------+<br>       |                                v<br>  +----------+                     +----------+<br>  |  CLOSED  |                     |   OPEN   |<br>  +----------+                     +----------+<br>       ^                                |<br>       |                                |<br>       |                         Recovery timeout<br>       |                                |<br>       |                                v<br>       |                          +-----------+<br>       +--------------------------| HALF-OPEN |<br>          Successful probe        +-----------+<br>                                        |<br>                                        |<br>                              Probe fails<br>                                        |<br>                                        v<br>                                  +----------+<br>                                  |   OPEN   |<br>                                  +----------+</pre><h3>1. Closed state</h3><p>The circuit is closed during normal operation.</p><p>Requests reach the downstream service, and the breaker records relevant failures.</p><p>For example:</p><pre>Request 1 -&gt; Success<br>Request 2 -&gt; Success<br>Request 3 -&gt; Failure<br>Request 4 -&gt; Success</pre><p>A single failure does not necessarily mean the dependency is unhealthy.</p><p>The breaker should use a defined failure threshold and an appropriate measurement window.</p><h3>2. Open state</h3><p>When failures exceed the configured threshold, the breaker opens.</p><p>New requests are rejected without calling the downstream service.</p><pre>Order Service<br>     |<br>     v<br>Circuit Breaker<br>     |<br>     | OPEN<br>     |<br>     v<br>Request rejected</pre><p>This prevents your application from continuously sending requests to a dependency that is already struggling.</p><p>The application may return a controlled error, use cached data, or apply another fallback if the operation supports one.</p><h3>3. Half-open state</h3><p>After a configured recovery period, the breaker allows a limited number of test requests.</p><p>These requests help determine whether the dependency has recovered.</p><ul><li>If the probes succeed, the breaker closes.</li><li>If they fail, the breaker opens again.</li></ul><p>The half-open state is important because immediately allowing all traffic back can overload a service that has only partially recovered.</p><h3>How Are Circuit Breakers Different From Retries?</h3><p>Retries and circuit breakers are related, but they solve different problems.</p><p>MechanismMain responsibilityRetryTry an operation again after a retryable failureExponential backoffIncrease the delay between retry attemptsCircuit breakerStop calls to a dependency that appears unhealthyRetry budgetLimit the total retry traffic generated</p><p>A retry policy asks:</p><p><em>Should I try this request again?</em></p><p>A circuit breaker asks:</p><p><em>Should I send requests to this dependency at all right now?</em></p><p>A retry budget asks:</p><p><em>How many additional attempts can this system afford?</em></p><p>These decisions should work together rather than independently.</p><h3>What Is a Retry Budget, and Why Does It Matter?</h3><p>A retry budget limits how many retries an application can generate within a defined period or relative to its original traffic.</p><p>Without a retry budget, every request may independently consume its maximum retry allowance.</p><p>Imagine an API receives 1,000 original requests. If each request can retry three times, the downstream service could receive up to 4,000 attempts:</p><pre>1,000 original requests<br>        |<br>        +-- 1,000 first attempts<br>        |<br>        +-- 1,000 retries<br>        |<br>        +-- 1,000 second retries<br>        |<br>        +-- 1,000 third retries<br>        |<br>        v<br>Up to 4,000 total attempts</pre><p>The original traffic has effectively multiplied.</p><p>A retry budget puts a limit on this additional load.</p><p>For example, a system might allow retries equal to 10% of its original request volume over a measurement window.</p><p>If the system processes 1,000 original requests, that policy would allow approximately 100 retries during the same window.</p><p>The exact limit should depend on the service’s capacity, latency objectives, and failure behavior.</p><p>The important principle is simple:</p><p><strong>Retries should help recover from temporary failures without overwhelming the dependency they are trying to reach.</strong></p><h3>How Do Exponential Backoff and Jitter Help Prevent Retry Storms?</h3><p>A retry budget limits the number of retries. Exponential backoff controls when those retries happen.</p><p>With exponential backoff, the delay increases after each failed attempt.</p><p>For example:</p><pre>Attempt 1: Immediate<br>Attempt 2: Wait 200 ms<br>Attempt 3: Wait 400 ms<br>Attempt 4: Wait 800 ms<br>Attempt 5: Wait 1600 ms</pre><p>The actual delays should be capped so that retries do not continue indefinitely.</p><p>However, exponential backoff alone does not solve every problem.</p><p>Imagine thousands of clients fail at the same time. If all of them use identical retry delays, they may retry together.</p><p>This creates synchronized traffic spikes.</p><p>Jitter adds randomness to retry delays:</p><pre>Client A: Retry after 350 ms<br>Client B: Retry after 520 ms<br>Client C: Retry after 410 ms<br>Client D: Retry after 680 ms</pre><p>The retries become more distributed over time.</p><p>This reduces the chance that clients repeatedly hit the same recovering service at exactly the same moment.</p><p>For a deeper explanation of retry safety, timeouts, and backoff, see our related article: <strong>Retries, Timeouts, and Exponential Backoff: Why Retrying Is Not Always Safe</strong>.</p><h3>How Can You Combine Circuit Breakers and Retry Budgets?</h3><p>The two mechanisms become more effective when they are part of the same request flow.</p><p>Consider this simplified flow:</p><pre>Incoming Request<br>       |<br>       v<br>Check Circuit Breaker<br>       |<br>       +---- OPEN ----&gt; Fail Fast / Fallback<br>       |<br>       v<br>Execute Request<br>       |<br>       +---- Success ----&gt; Return Response<br>       |<br>       v<br>Is Failure Retryable?<br>       |<br>       +---- NO ----&gt; Return Error<br>       |<br>       v<br>Check Retry Budget<br>       |<br>       +---- Exhausted ----&gt; Return Error<br>       |<br>       v<br>Wait Using Backoff + Jitter<br>       |<br>       v<br>Retry Request</pre><p>A sensible order is:</p><ol><li>Check whether the circuit breaker allows a request.</li><li>Execute the downstream operation.</li><li>Classify the result.</li><li>Retry only if the failure is eligible.</li><li>Check whether the retry budget allows another attempt.</li><li>Apply backoff and jitter before retrying.</li><li>Stop when the operation succeeds, the deadline expires, or the retry allowance is exhausted.</li></ol><p>The circuit breaker should also observe relevant dependency failures so it can detect when the downstream service is unhealthy.</p><p>Avoid counting every error as a dependency failure. For example, a validation error caused by an invalid user request should not normally open a circuit breaker.</p><h3>How Can You Implement a Simple Circuit Breaker in Node.js?</h3><p>The following TypeScript example demonstrates the core state transitions.</p><p>It is an educational implementation, not a complete production-ready circuit breaker.</p><pre>type CircuitState = &quot;CLOSED&quot; | &quot;OPEN&quot; | &quot;HALF_OPEN&quot;;</pre><pre>class CircuitBreaker {<br>  private state: CircuitState = &quot;CLOSED&quot;;<br>  private failures = 0;<br>  private openedAt = 0;<br>  private probeInFlight = false;</pre><pre>  constructor(<br>    private readonly failureThreshold = 5,<br>    private readonly resetTimeoutMs = 10_000<br>  ) {}</pre><pre>  async execute&lt;T&gt;(operation: () =&gt; Promise&lt;T&gt;): Promise&lt;T&gt; {<br>    if (this.state === &quot;OPEN&quot;) {<br>      const elapsed = Date.now() - this.openedAt;</pre><pre>      if (elapsed &lt; this.resetTimeoutMs) {<br>        throw new Error(&quot;Circuit is open&quot;);<br>      }</pre><pre>      this.state = &quot;HALF_OPEN&quot;;<br>    }</pre><pre>    const isProbe = this.state === &quot;HALF_OPEN&quot;;</pre><pre>    if (isProbe) {<br>      if (this.probeInFlight) {<br>        throw new Error(&quot;Half-open probe already running&quot;);<br>      }</pre><pre>      this.probeInFlight = true;<br>    }</pre><pre>    try {<br>      const result = await operation();</pre><pre>      this.failures = 0;<br>      this.state = &quot;CLOSED&quot;;</pre><pre>      return result;<br>    } catch (error) {<br>      this.failures++;</pre><pre>      if (<br>        isProbe ||<br>        this.failures &gt;= this.failureThreshold<br>      ) {<br>        this.state = &quot;OPEN&quot;;<br>        this.openedAt = Date.now();<br>      }</pre><pre>      throw error;<br>    } finally {<br>      if (isProbe) {<br>        this.probeInFlight = false;<br>      }<br>    }<br>  }<br>}</pre><p>Usage:</p><pre>const paymentBreaker = new CircuitBreaker(5, 10_000);</pre><pre>async function chargePayment() {<br>  return paymentBreaker.execute(async () =&gt; {<br>    const response = await fetch(<br>      &quot;https://payment-provider.example/charge&quot;<br>    );</pre><pre>    if (!response.ok) {<br>      throw new Error(&quot;Payment provider failed&quot;);<br>    }</pre><pre>    return response.json();<br>  });<br>}</pre><p>This example demonstrates the general idea, but it has important limitations:</p><ul><li>Its state exists only in one Node.js process.</li><li>Its failure counter is not based on a rolling time window.</li><li>It does not classify errors into retryable and non-retryable categories.</li><li>Its half-open behavior supports only a very simple probe model.</li><li>It does not include request deadlines, metrics, or fallback policies.</li></ul><p>A production implementation needs more careful concurrency handling, failure classification, rolling-window statistics, and operational observability. In most real systems, use a well-maintained circuit-breaker library or a thoroughly tested internal implementation rather than copying this example directly.</p><h3>How Can You Implement a Simple Retry Budget?</h3><p>A token bucket is one way to represent a retry budget.</p><p>The bucket starts with a limited number of tokens. A retry consumes one token. Tokens are replenished over time or according to an explicit traffic policy.</p><pre>class RetryBudget {<br>  private tokens: number;</pre><pre>  constructor(<br>    private readonly capacity: number,<br>    private readonly refillPerSecond: number<br>  ) {<br>    this.tokens = capacity;<br>  }</pre><pre>  tryConsume(): boolean {<br>    if (this.tokens &lt;= 0) {<br>      return false;<br>    }</pre><pre>    this.tokens--;<br>    return true;<br>  }</pre><pre>  refill(): void {<br>    this.tokens = Math.min(<br>      this.capacity,<br>      this.tokens + this.refillPerSecond<br>    );<br>  }<br>}</pre><p>Example:</p><pre>const retryBudget = new RetryBudget(100, 10);</pre><pre>async function retryOperation() {<br>  if (!retryBudget.tryConsume()) {<br>    throw new Error(&quot;Retry budget exhausted&quot;);<br>  }</pre><pre>  // Perform the retry here.<br>}</pre><p>This is deliberately simplified. The refill() method would need to be scheduled or integrated with a time-aware token calculation. The example also keeps the budget local to one process.</p><p>In a horizontally scaled application, each instance having its own budget can produce a much larger aggregate retry allowance. Decide whether the budget should be per process, per service, or centrally coordinated based on the protection you need.</p><h3>What Can Go Wrong When You Implement These Patterns?</h3><p>These patterns improve resilience, but incorrect configuration can create new problems.</p><p><strong>1. Retrying non-retryable errors</strong></p><p>A validation error or authorization failure usually will not become successful simply by repeating the request.</p><p>Classify failures before retrying.</p><p><strong>2. Opening the circuit too quickly</strong></p><p>A very low failure threshold can block traffic because of a small number of temporary errors.</p><p>Use an appropriate failure window and threshold.</p><p><strong>3. Allowing too many half-open requests</strong></p><p>When a dependency begins recovering, sending too many probes can overwhelm it again.</p><p>Limit the number of test requests.</p><p><strong>4. Using independent retry budgets on every instance without accounting for scale</strong></p><p>A per-process budget may be acceptable, but the total allowance grows with the number of instances.</p><p>Understand the aggregate effect.</p><p><strong>5. Ignoring request deadlines</strong></p><p>Retries must fit within the caller’s remaining time budget.</p><p>A request with a one-second deadline should not continue retrying for ten seconds.</p><p><strong>6. Retrying non-idempotent operations without protection</strong></p><p>A timeout does not prove that the downstream operation failed. The provider may have completed the operation even though your application never received the response.</p><p>For operations such as payments and order creation, use idempotency keys and a safe recovery strategy before retrying.</p><h3>How Should You Monitor Circuit Breakers and Retry Budgets?</h3><p>A pattern that cannot be observed is difficult to operate.</p><p>Useful metrics include:</p><ul><li>Circuit breaker state changes.</li><li>Number of requests rejected because the circuit is open.</li><li>Number of half-open probes.</li><li>Downstream success and failure rates.</li><li>Retry attempts per original request.</li><li>Retry budget consumption and exhaustion.</li><li>Downstream latency and timeout rates.</li><li>Fallback usage.</li></ul><p>These metrics help answer practical questions:</p><ul><li>Is a dependency actually unhealthy?</li><li>Are retries increasing during incidents?</li><li>Is the retry budget too restrictive?</li><li>Does the circuit recover too quickly or too slowly?</li><li>Are users receiving controlled failures or experiencing long waits?</li></ul><p>Track these metrics by dependency and operation where possible. Avoid high-cardinality labels such as unique request IDs in metric dimensions.</p><h3>Frequently Asked Questions</h3><h3>Do circuit breakers replace retries?</h3><p>No. Retries handle selected temporary failures, while circuit breakers stop requests from reaching a dependency that appears unhealthy. They complement each other.</p><h3>Are retry budgets necessary for every service?</h3><p>Not every service needs the same retry policy, but retry limits are important wherever retries can multiply traffic or increase downstream load. The budget should reflect the service’s capacity and reliability requirements.</p><h3>Should a circuit breaker be shared across all Node.js instances?</h3><p>It depends on the architecture. A local breaker is simple and fast, but each process makes its own decision. A shared breaker can coordinate decisions, but introduces distributed-state complexity and another dependency. Choose based on the failure mode you need to control.</p><h3>What is the difference between a circuit breaker and rate limiting?</h3><p>Rate limiting controls how much traffic a service accepts, often to protect capacity or enforce usage policies. A circuit breaker reacts to dependency health and temporarily blocks calls when failures indicate that the dependency may be unhealthy.</p><h3>Can circuit breakers prevent every cascading failure?</h3><p>No. They address one important failure path, but resilience also depends on timeouts, bulkheads, bounded queues, load shedding, graceful degradation, capacity planning, and careful retry behavior.</p><h3>Final Thoughts</h3><p>A retry policy can improve reliability, but uncontrolled retries can turn a small dependency failure into a system-wide incident.</p><p>Circuit breakers protect unhealthy dependencies from continued traffic. Retry budgets limit the additional load created by repeated attempts. Exponential backoff and jitter help distribute that traffic over time.</p><p>The goal is not to retry everything or to block every failing dependency immediately.</p><p>The goal is to make deliberate decisions about when to retry, when to stop, and how much additional load your system can safely generate.</p><p>When these mechanisms are designed together, your application becomes better prepared to handle partial failures without allowing one struggling service to take down everything around it.</p><p><strong>Next, we’ll explore how rate limiting works in distributed systems and how to protect APIs without unnecessarily hurting legitimate users.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=75affadab4a4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Retries, Timeouts, and Exponential Backoff: Why Retrying Is Not Always Safe]]></title>
            <link>https://mustufaahmed.medium.com/retries-timeouts-and-exponential-backoff-why-retrying-is-not-always-safe-ab081c0d42c9?source=rss-6e071c8bef88------2</link>
            <guid isPermaLink="false">https://medium.com/p/ab081c0d42c9</guid>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[backend-development]]></category>
            <category><![CDATA[reliability-engineering]]></category>
            <category><![CDATA[design-systems]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Muhammad Mustufa Ahmed]]></dc:creator>
            <pubDate>Thu, 29 Oct 2026 05:25:15 GMT</pubDate>
            <atom:updated>2026-10-31T08:21:16.124Z</atom:updated>
            <content:encoded><![CDATA[<p>A request times out. Your application retries it. The second attempt succeeds.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VLpQF9H6NX-XbH6dI3Kh0w.png" /></figure><p>That sounds like a resilient system.</p><p>But what if the first request already completed on the server? What if both attempts execute? What if hundreds of clients retry simultaneously and overwhelm the service you are trying to recover?</p><p>Retries can improve reliability, but they can also amplify failures.</p><p>Understanding retries requires more than adding a loop around an HTTP request. You need to understand timeouts, retryable errors, exponential backoff, jitter, deadlines, and idempotency.</p><p>Let’s break down how to design retries that improve system reliability instead of making an incident worse.</p><h3>1. Why Do We Need Retries?</h3><p>Distributed systems fail in ways that are often temporary.</p><p>A request might fail because of:</p><ul><li>A brief network interruption.</li><li>A temporary service overload.</li><li>An HTTP 429 Too Many Requests response.</li><li>An upstream service returning 502, 503, or 504.</li><li>A connection being reset during communication.</li><li>A dependency temporarily becoming unavailable.</li></ul><p>Some of these failures are transient. A later attempt may succeed without changing the request.</p><p>However, <strong>a failed request from the client’s perspective does not necessarily mean the operation failed on the server</strong>.</p><p>Consider a payment API.</p><p>The client sends a payment request. The server processes the payment and commits it to the database, but the response never reaches the client.</p><p>The client sees a timeout. It does not know whether the payment succeeded.</p><p>If it blindly retries, it might create a duplicate payment.</p><p>This is why retry logic must be designed together with operation semantics, not as an isolated networking feature.</p><h3>2. What Actually Happens During a Timeout?</h3><p>A timeout is a limit on how long a caller is willing to wait. It is not proof that the remote operation stopped.</p><p>Imagine this sequence:</p><pre>Client                  Payment API                 Database<br>  |                          |                           |<br>  |--- Create payment ------&gt;|                           |<br>  |                          |--- Insert payment -------&gt;|<br>  |                          |&lt;-- Commit successful -----|<br>  |                          |                           |<br>  |       Response lost      X                           |<br>  |                          |                           |<br>  |--- Timeout --------------|                           |<br>  |                          |                           |<br>  |--- Retry request -------&gt;|                           |</pre><p>The client timed out, but the payment may already exist.</p><p>This creates an important distinction:</p><ul><li><strong>Client timeout:</strong> The caller stopped waiting.</li><li><strong>Server completion:</strong> The server may still have completed the operation.</li><li><strong>Cancellation:</strong> A cancellation signal may ask work to stop, but it does not guarantee that all remote side effects have been undone.</li></ul><p>For read-only requests, retries are often relatively straightforward. For payments, order creation, email delivery, or other operations with side effects, retry safety needs additional guarantees.</p><h3>3. The Retry Decision Flow</h3><p>Not every failure deserves a retry.</p><p>A robust retry policy should consider whether the error is transient, whether attempts remain, and whether the request can still finish within its deadline.</p><pre>flowchart TD<br>    A[Send request] --&gt; B{Response received?}<br>    B --&gt;|Yes| C{Success?}<br>    C --&gt;|Yes| D[Return success]<br>    C --&gt;|No| E{Retryable error?}<br>    B --&gt;|No| E<br>    E --&gt;|No| F[Return error]<br>    E --&gt;|Yes| G{Attempts remaining?}<br>    G --&gt;|No| F<br>    G --&gt;|Yes| H[Wait with backoff and jitter]<br>    H --&gt; I{Deadline exceeded?}<br>    I --&gt;|Yes| F<br>    I --&gt;|No| A</pre><p>The key idea is that a retry is a <strong>decision</strong>, not an automatic reaction to every error.</p><h3>Which errors should we retry?</h3><p>FailureTypical approachReasonTemporary connection resetOften retryMay be a transient transport failureHTTP 429Often retryRespect Retry-After when providedHTTP 502, 503, 504Often retryMay indicate a temporary upstream failureHTTP 400Usually do not retry unchangedThe request is likely invalidHTTP 403Usually do not retry unchangedThe caller is not authorizedHTTP 401Refresh credentials if appropriateRepeating the same invalid credentials rarely helpsHTTP 404Depends on the operationThe resource may be permanently absent or not yet availableRequest timeoutDepends on operation semanticsThe server may have completed the operation</p><p>These are defaults, not universal rules. Your API contract and the meaning of the operation should determine the final policy.</p><p>For example, retrying a 404 might make sense when polling for a resource that is expected to appear shortly. It usually makes little sense when fetching a permanently nonexistent record.</p><h3>4. Exponential Backoff: Give the System Time to Recover</h3><p>A common mistake is retrying immediately.</p><p>Imagine 5,000 clients calling a service that is already overloaded. If every client retries immediately, the service receives another wave of requests before it has recovered.</p><p>This can turn a temporary issue into a prolonged outage.</p><p>Exponential backoff increases the delay between successive attempts.</p><p>A basic formula is:</p><p>[ d_n = \min(d_{\max}, d_0 \times 2^n) ]</p><p>Where:</p><ul><li>(d_n) is the delay before a retry.</li><li>(d_0) is the initial delay.</li><li>(d_{\max}) is the maximum delay.</li><li>(n) is the retry index, starting at zero.</li></ul><p>For an initial delay of 200 milliseconds and a maximum of 2 seconds:</p><pre>Attempt 1 fails<br>    |<br>    +--&gt; Wait 200 ms<br>              |<br>              v<br>Attempt 2 fails<br>    |<br>    +--&gt; Wait 400 ms<br>              |<br>              v<br>Attempt 3 fails<br>    |<br>    +--&gt; Wait 800 ms<br>              |<br>              v<br>Attempt 4</pre><p>The delay increases after each failure, giving the dependency more time to recover.</p><p>But exponential backoff has another problem: many clients can still retry at roughly the same time.</p><p>That is where jitter comes in.</p><h3>5. Why Jitter Matters</h3><p>Suppose thousands of clients experience a failure at the same moment. They all use the same backoff schedule:</p><pre>Time 0 ms:       Thousands of requests fail<br>Time 200 ms:     Thousands of retries<br>Time 600 ms:     Thousands of retries<br>Time 1400 ms:    Thousands of retries</pre><p>This synchronized behavior is known as a retry storm.</p><p><strong>Jitter adds randomness to retry delays</strong>, spreading retries over time rather than letting all clients retry together.</p><p>One common strategy is full jitter:</p><p>[ d_n = U(0, \min(d_{\max}, d_0 \times 2^n)) ]</p><p>Here, (U(0,x)) means a random delay between zero and (x).</p><p>Instead of every client waiting exactly 400 milliseconds, clients choose different delays within the permitted range.</p><pre>Retry window: 0–400 ms</pre><pre>Client A:  73 ms<br>Client B: 211 ms<br>Client C: 356 ms<br>Client D: 129 ms<br>Client E: 294 ms</pre><p>This reduces synchronized retry bursts.</p><p>For HTTP 429 or 503 responses, the server may also provide a Retry-After header. Your retry policy should respect that guidance where possible, while still enforcing the caller&#39;s overall deadline.</p><h3>6. Timeouts, Attempt Limits, and Overall Deadlines</h3><p>A retry policy needs more than a maximum attempt count.</p><p>Consider three settings:</p><ul><li><strong>Per-attempt timeout:</strong> How long one request attempt may take.</li><li><strong>Maximum attempts:</strong> How many total attempts are allowed, including the initial attempt.</li><li><strong>Overall deadline:</strong> How long the caller is willing to spend on the entire operation, including requests and waiting between attempts.</li></ul><p>For example:</p><pre>Overall deadline: 5 seconds</pre><pre>Attempt 1: Request takes 1.2 seconds → timeout<br>Backoff:   Wait 200 ms<br>Attempt 2: Request takes 1.3 seconds → timeout<br>Backoff:   Wait 300 ms<br>Attempt 3: Remaining time is limited</pre><p>Even if more attempts are configured, the client should not keep retrying after its overall deadline expires.</p><p>This is particularly important in service-to-service calls. If Service A has a 2-second deadline, it should not start a new downstream request when there is insufficient time for that request to finish usefully.</p><p>A downstream timeout should generally fit inside the remaining upstream deadline. Otherwise, work may continue after the original caller has already given up.</p><h3>7. TypeScript and Node.js: A Bounded Retry Helper</h3><p>Here is an illustrative implementation using the built-in fetch API available in modern Node.js.</p><p>It demonstrates:</p><ul><li>A maximum attempt count.</li><li>Per-attempt timeouts.</li><li>An overall deadline.</li><li>Exponential backoff with full jitter.</li><li>Retryable HTTP status codes.</li><li>Basic support for Retry-After.</li></ul><pre>type RetryOptions = {<br>  maxAttempts?: number;<br>  totalTimeoutMs?: number;<br>  perAttemptTimeoutMs?: number;<br>  baseDelayMs?: number;<br>  maxDelayMs?: number;<br>};</pre><pre>const RETRYABLE_STATUSES = new Set([<br>  429,<br>  502,<br>  503,<br>  504,<br>]);</pre><pre>function sleep(ms: number): Promise&lt;void&gt; {<br>  return new Promise((resolve) =&gt; setTimeout(resolve, ms));<br>}</pre><pre>function getRetryAfterMs(<br>  response: Response<br>): number | undefined {<br>  const value = response.headers.get(&quot;retry-after&quot;);</pre><pre>  if (!value) {<br>    return undefined;<br>  }</pre><pre>  // Retry-After may be a number of seconds.<br>  const seconds = Number(value);</pre><pre>  if (Number.isFinite(seconds) &amp;&amp; seconds &gt;= 0) {<br>    return seconds * 1000;<br>  }</pre><pre>  // Or it may be an HTTP date.<br>  const retryAt = Date.parse(value);</pre><pre>  if (!Number.isNaN(retryAt)) {<br>    return Math.max(0, retryAt - Date.now());<br>  }</pre><pre>  return undefined;<br>}</pre><pre>export async function fetchWithRetry(<br>  url: string,<br>  init: RequestInit = {},<br>  options: RetryOptions = {}<br>): Promise&lt;Response&gt; {<br>  const {<br>    maxAttempts = 3,<br>    totalTimeoutMs = 5_000,<br>    perAttemptTimeoutMs = 1_500,<br>    baseDelayMs = 200,<br>    maxDelayMs = 2_000,<br>  } = options;</pre><pre>  if (<br>    !Number.isInteger(maxAttempts) ||<br>    maxAttempts &lt; 1 ||<br>    totalTimeoutMs &lt;= 0 ||<br>    perAttemptTimeoutMs &lt;= 0 ||<br>    baseDelayMs &lt; 0 ||<br>    maxDelayMs &lt; 0<br>  ) {<br>    throw new Error(&quot;Invalid retry configuration&quot;);<br>  }</pre><pre>  const deadline = Date.now() + totalTimeoutMs;<br>  let lastError: unknown;</pre><pre>  for (<br>    let attempt = 0;<br>    attempt &lt; maxAttempts;<br>    attempt++<br>  ) {<br>    const remainingMs = deadline - Date.now();</pre><pre>    if (remainingMs &lt;= 0) {<br>      throw new Error(&quot;Overall request deadline exceeded&quot;);<br>    }</pre><pre>    const attemptTimeoutMs = Math.min(<br>      perAttemptTimeoutMs,<br>      remainingMs<br>    );</pre><pre>    const controller = new AbortController();</pre><pre>    // Illustrative: this helper does not combine a caller&#39;s<br>    // AbortSignal with the per-attempt controller.<br>    const timeout = setTimeout(<br>      () =&gt; controller.abort(),<br>      attemptTimeoutMs<br>    );</pre><pre>    let retryAfterMs: number | undefined;</pre><pre>    try {<br>      const response = await fetch(url, {<br>        ...init,<br>        signal: controller.signal,<br>      });</pre><pre>      if (!RETRYABLE_STATUSES.has(response.status)) {<br>        return response;<br>      }</pre><pre>      retryAfterMs = getRetryAfterMs(response);</pre><pre>      // Do not leave an unused response body open.<br>      await response.body?.cancel();</pre><pre>      if (attempt === maxAttempts - 1) {<br>        return new Response(null, {<br>          status: response.status,<br>          statusText: response.statusText,<br>          headers: response.headers,<br>        });<br>      }<br>    } catch (error) {<br>      lastError = error;</pre><pre>      if (attempt === maxAttempts - 1) {<br>        throw error;<br>      }<br>    } finally {<br>      clearTimeout(timeout);<br>    }</pre><pre>    const remainingAfterAttempt = deadline - Date.now();</pre><pre>    if (remainingAfterAttempt &lt;= 0) {<br>      throw new Error(&quot;Overall request deadline exceeded&quot;);<br>    }</pre><pre>    const exponentialCap = Math.min(<br>      maxDelayMs,<br>      baseDelayMs * 2 ** attempt<br>    );</pre><pre>    // Full jitter: random delay between 0 and the cap.<br>    const jitterDelayMs =<br>      Math.random() * exponentialCap;</pre><pre>    const requestedDelayMs =<br>      retryAfterMs === undefined<br>        ? jitterDelayMs<br>        : Math.max(jitterDelayMs, retryAfterMs);</pre><pre>    if (requestedDelayMs &gt;= remainingAfterAttempt) {<br>      throw new Error(&quot;Insufficient time for another retry&quot;);<br>    }</pre><pre>    await sleep(requestedDelayMs);<br>  }</pre><pre>  throw lastError ?? new Error(&quot;Request failed&quot;);<br>}</pre><h3>Important production caveats</h3><p>This example is intended to explain the mechanics, not to serve as a drop-in production library.</p><p>Before using a retry helper in production, review these concerns:</p><ol><li><strong>Caller cancellation:</strong> Combine the caller’s AbortSignal with the per-attempt timeout so the caller can cancel the whole operation.</li><li><strong>Request body replay:</strong> A consumed stream or other non-repeatable body may not be reusable for a second attempt. Build a fresh request body for every attempt when necessary.</li><li><strong>Transport error classification:</strong> The example retries any thrown error. Production code should distinguish retryable network failures from programming errors, invalid URLs, and other permanent failures.</li><li><strong>Response-body deadlines:</strong> fetch() resolving does not necessarily mean the entire response body has been read. If body consumption is part of the operation, its time must be bounded too.</li><li><strong>Overall cancellation:</strong> The sleep and active request should respond to cancellation and the remaining overall deadline.</li><li><strong>Retry-After limits:</strong> A server-provided delay can exceed the caller&#39;s deadline. Do not sleep past the deadline or retry sooner than required by your policy.</li><li><strong>HTTP semantics:</strong> A retryable status does not automatically make the request safe to repeat.</li></ol><p>Also note that this helper returns the final retryable HTTP response when attempts are exhausted, while transport errors are thrown. Callers should handle both outcomes explicitly.</p><p>The example has not been presented as tested production code; validate it with your runtime, request types, and failure scenarios before adopting it.</p><h3>8. Retries Do Not Make POST Requests Safe</h3><p>This is where retry logic connects directly to idempotency.</p><p>Suppose a client sends:</p><pre>POST /payments<br>Idempotency-Key: payment-123<br>Content-Type: application/json</pre><p>The payment service creates the payment, but the response is lost.</p><p>The client retries the request using the <strong>same idempotency key</strong>.</p><p>The service can recognize the logical operation and return the original result instead of creating another payment.</p><p>The sequence looks like this:</p><pre>sequenceDiagram<br>    participant C as Client<br>    participant A as Payment API<br>    participant D as Database</pre><pre>    C-&gt;&gt;A: POST /payments (key: payment-123)<br>    A-&gt;&gt;D: Create payment and save operation result<br>    D--&gt;&gt;A: Transaction committed<br>    A--xC: Response lost or times out</pre><pre>    C-&gt;&gt;A: Retry with the same key<br>    A-&gt;&gt;D: Look up idempotency record<br>    D--&gt;&gt;A: Existing operation and result<br>    A--&gt;&gt;C: Return original result</pre><p>A few rules matter:</p><ul><li>Reuse the same idempotency key for retries of the same logical operation.</li><li>Do not generate a new key for every attempt.</li><li>Reject reuse of the same key with a different normalized request payload.</li><li>Persist the operation result reliably.</li><li>For external payment providers, use their idempotency support when available and reconcile uncertain outcomes.</li></ul><p>An idempotency key does not mean your handler executes exactly once. It means your system can ensure repeated equivalent requests have the intended effect without duplicating the logical operation.</p><p>For more detail, see the related article on idempotency and production bugs.</p><h3>9. Beware of Nested Retries</h3><p>Retries become especially dangerous when multiple layers implement them independently.</p><p>Imagine this call chain:</p><pre>API Gateway<br>    |<br>    v<br>Order Service<br>    |<br>    v<br>Payment Service<br>    |<br>    v<br>External Payment Provider</pre><p>Suppose each layer makes up to three attempts.</p><p>If each layer independently retries every downstream call, the number of requests can multiply. Under a simplified model with three attempts at each of four layers, the worst-case number of provider calls can reach:</p><p>[ 3⁴ = 81 ]</p><p>The exact count depends on where retries occur and how failures propagate, but the risk is clear: independent retry policies can amplify load dramatically.</p><p>Better practices include:</p><ul><li>Define which layer owns retries for each dependency.</li><li>Set explicit attempt limits and an overall deadline.</li><li>Avoid retrying the same operation at every layer without coordination.</li><li>Use retry budgets to limit additional traffic.</li><li>Consider circuit breakers to stop repeatedly calling an unhealthy dependency.</li><li>Use bulkheads to prevent one failing dependency from consuming all available resources.</li></ul><p>Retries help when failures are transient. They hurt when they keep adding traffic to a system that cannot recover.</p><h3>10. What About Retry-After, Rate Limits, and Backpressure?</h3><p>A service may explicitly tell clients when to try again.</p><p>For example:</p><pre>HTTP/1.1 429 Too Many Requests<br>Retry-After: 2</pre><p>The server is asking the client to wait approximately two seconds before retrying.</p><p>Respecting this signal helps clients cooperate with server-side rate limiting.</p><p>However, clients must still consider their overall deadlines. If the requested delay exceeds the remaining time, the correct behavior may be to return a failure to the caller rather than retrying too late.</p><p>At scale, retry policies should work alongside:</p><ul><li>Rate limiting.</li><li>Queue-based load leveling.</li><li>Concurrency limits.</li><li>Circuit breakers.</li><li>Backpressure.</li><li>Observability and alerting.</li></ul><p>Retries are one reliability mechanism, not a substitute for capacity management or overload protection.</p><h3>11. Testing Retry Logic Properly</h3><p>A retry helper can look correct while failing under concurrency, cancellation, or unexpected response timing.</p><p>Test at least the following cases:</p><ul><li>The first attempt succeeds and no retry occurs.</li><li>A transient error is followed by a successful attempt.</li><li>A non-retryable response is returned immediately.</li><li>The maximum attempt count is respected.</li><li>Backoff never exceeds its configured cap.</li><li>Retry-After is handled correctly.</li><li>The overall deadline prevents additional attempts.</li><li>Caller cancellation stops pending work.</li><li>A timeout does not result in unsafe duplicate side effects.</li><li>The same idempotency key is reused across attempts.</li><li>Exhausted retries produce a predictable outcome.</li></ul><p>For idempotent business operations, also test the ambiguous case where the server commits the operation but the client never receives the response.</p><p>That is the failure scenario where retry logic and idempotency must work together.</p><h3>12. A Practical Reliability Checklist</h3><p>Before enabling retries for a dependency, ask:</p><ul><li>Is the error genuinely transient?</li><li>Is the operation safe to repeat, or protected by idempotency?</li><li>What is the per-attempt timeout?</li><li>What is the overall deadline?</li><li>How many total attempts are allowed?</li><li>Does the policy use exponential backoff and jitter?</li><li>Does it respect Retry-After?</li><li>Can the request body be replayed?</li><li>Which layer owns the retry policy?</li><li>Are retries observable through metrics and logs?</li><li>Can retries amplify an overload incident?</li><li>What happens when the operation succeeds but the response is lost?</li></ul><p>If these questions do not have clear answers, adding a retry loop may make the system less reliable rather than more reliable.</p><h3>Conclusion</h3><p>Retries are a powerful reliability tool, but they are not free.</p><p>A well-designed retry policy combines:</p><ul><li>Timeouts that bound individual attempts.</li><li>Overall deadlines that bound the entire operation.</li><li>Exponential backoff that gives dependencies time to recover.</li><li>Jitter that prevents synchronized retry storms.</li><li>Error classification that avoids repeating permanent failures.</li><li>Idempotency that protects operations with side effects.</li><li>Retry budgets and backpressure that prevent overload amplification.</li></ul><p>The most important principle is this:</p><p><strong>Retrying a request is easy. Knowing when it is safe to retry is the real engineering problem.</strong></p><p>In the previous article, we explored how idempotency prevents duplicate effects. This article builds on that idea: retries help a client recover from uncertainty, while idempotency helps the server handle repeated requests safely.</p><p>Together, they form a foundation for building more resilient distributed systems.</p><p>That article can build on this one by exploring what happens when a dependency remains unhealthy even after retries and backoff.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab081c0d42c9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Idempotency: The Small Design Decision That Prevents Big Production Failures]]></title>
            <link>https://mustufaahmed.medium.com/idempotency-the-small-design-decision-that-prevents-big-production-failures-0790bfbca4d6?source=rss-6e071c8bef88------2</link>
            <guid isPermaLink="false">https://medium.com/p/0790bfbca4d6</guid>
            <category><![CDATA[backend-engineering]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[fintech]]></category>
            <category><![CDATA[design-systems]]></category>
            <dc:creator><![CDATA[Muhammad Mustufa Ahmed]]></dc:creator>
            <pubDate>Wed, 28 Oct 2026 06:10:18 GMT</pubDate>
            <atom:updated>2026-10-31T08:25:25.171Z</atom:updated>
            <content:encoded><![CDATA[<p><em>An architectural deep dive into retries, duplicate requests, PostgreSQL transactions, concurrency, and reliable API design.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_JaqVQRrhTTEVQzFvo0_Zw.png" /></figure><p>Imagine a customer clicks <strong>Pay</strong>.</p><p>The backend processes the request and charges the customer. But before the response reaches the browser, the network connection drops.</p><p>The customer sees a timeout.</p><p>They click Pay again.</p><p>The backend receives another request.</p><p>What should happen now?</p><p>If the system creates another payment, the customer might be charged twice. If the backend blindly rejects every repeated request, the customer might see an error even though their original payment succeeded.</p><p>This is not an unusual edge case. Network failures, client retries, load balancers, mobile connectivity, and process crashes make repeated requests a normal condition in distributed systems.</p><p>The architectural question is not how to guarantee that a request arrives exactly once.</p><p>It is how to make repeated attempts safe.</p><p>That is where <strong>idempotency</strong> comes in.</p><h3>1. What is idempotency?</h3><p>An operation is idempotent when repeating it produces the same intended effect as performing it once.</p><p>In mathematical terms:</p><p>[ f(f(x)) = f(x) ]</p><p>For an API, the practical interpretation is that repeating the same logical operation must not create additional business effects.</p><p>Consider these examples:</p><p>Operation expected behavior when repeated set a user’s email preference to disabled. The preference remains disabled set an order’s delivery address to a specified address. The address remains at that value create a payment without duplicate protection May create multiple payment attempts or charges create a payment with properly implemented idempotency repeated attempts resolve to the same logical payment operation</p><p>There is an important distinction here: idempotency does not necessarily mean the server executes the handler only once. It means repeated equivalent operations have the intended same effect.</p><p>An API might return a stored response on the second request. It might return a different HTTP status for a conflicting request. Those are implementation details around the business guarantee.</p><h3>HTTP idempotency is not the whole story</h3><p>HTTP defines PUT and DELETE as idempotent methods in terms of their intended effect. POST is not generally idempotent by default.</p><p>But an application can implement idempotency for a POST endpoint using an idempotency key.</p><p>This matters because HTTP method semantics alone do not protect a payment workflow from duplicate business operations.</p><h3>2. Why retries create architectural problems</h3><p>Suppose a client sends:</p><pre>POST /payments<br>Content-Type: application/json</pre><pre>{<br>  &quot;orderId&quot;: &quot;ord_123&quot;<br>}</pre><p>The server processes the payment, but the response is lost.</p><p>The client cannot tell whether:</p><ul><li>The server never received the request.</li><li>The server received it but failed before processing.</li><li>The server processed it successfully but failed to return the response.</li><li>The payment provider completed the charge, but the application failed before recording the outcome.</li></ul><p>From the client’s perspective, these situations can look identical.</p><p>This is a fundamental distributed-systems problem: <strong>a timeout does not prove that an operation failed</strong>.</p><p>If the client retries without a mechanism to recognize the original operation, the server may create a duplicate.</p><p>Disabling retries is not a solution. Retries are essential for recovering from transient failures. We need to design the operation so that retries are safe.</p><h3>3. Introduce the idempotency key</h3><p>The client generates a unique key for a logical operation and reuses that same key for every retry of that operation.</p><p>For example:</p><pre>POST /payments<br>Idempotency-Key: 7d9f0a1e-4a21-4a6b-9d0a-example<br>Content-Type: application/json</pre><pre>{<br>  &quot;orderId&quot;: &quot;ord_123&quot;<br>}</pre><p>If the request times out, the client sends the same request with the same key.</p><p>The server can now distinguish between:</p><ul><li>A retry of an existing logical operation.</li><li>A new operation with a different key.</li><li>An invalid attempt to reuse a key with a different request.</li></ul><p>A key is not an authentication credential. It must not replace authentication, authorization, or validation.</p><h3>What should an idempotency key identify?</h3><p>A key should identify one logical operation within a defined scope.</p><p>For example, a server might scope keys by:</p><ul><li>Tenant or authenticated account.</li><li>Operation or endpoint.</li><li>Idempotency key.</li></ul><p>The database uniqueness constraint then becomes:</p><p>(tenant_id, operation, idempotency_key)</p><p>The authenticated tenant must come from trusted server-side identity context, not from an untrusted request field.</p><p>The client must also preserve the key across retries. Generating a new key for every attempt defeats the purpose.</p><h3>4. The architecture behind idempotency</h3><p>A reliable implementation needs more than an HTTP header.</p><p>It needs a durable record of the logical operation, a way to detect conflicting requests, and a safe mechanism for storing the business result.</p><p>The following architecture uses TypeScript, Node.js, Express.js, and PostgreSQL.</p><pre>flowchart TD<br>    A[Client] --&gt;|Request + idempotency key| B[Express API]<br>    B --&gt; C[Authenticate and validate]<br>    C --&gt; D[PostgreSQL transaction]<br>    D --&gt; E{Existing scoped key?}<br>    E --&gt;|Yes| F{Same request hash?}<br>    F --&gt;|Yes| G[Return stored result]<br>    F --&gt;|No| H[Return conflict]<br>    E --&gt;|No| I[Validate payable order]<br>    I --&gt; J[Create payment record]<br>    J --&gt; K[Store response]<br>    K --&gt; L[Commit transaction]<br>    L --&gt; M[Return response]</pre><p>The important components are:</p><p><strong>1. Idempotency record:</strong> stores the key, request fingerprint, and resulting response.</p><p><strong>2. Request fingerprint:</strong> detects whether a key is being reused with different input.</p><p><strong>3. Business record:</strong> represents the actual operation, such as a payment.</p><p><strong>4. Database constraints:</strong> protect against competing requests, even when they arrive concurrently.</p><p><strong>5. Transaction boundary:</strong> ensures that the local payment record and its completed idempotency result are committed together.</p><p>This design assumes the business mutation can be performed within the same database transaction. External payment providers require an additional layer of coordination, which we will cover later.</p><h3>5. Design the PostgreSQL schema</h3><p>Let’s use a simplified order-payment domain.</p><p>An order is payable once, and each order has a trusted amount and currency. The client sends the order identifier; the server obtains the amount from the database instead of trusting the client to supply it.</p><p>First, the orders table:</p><pre>CREATE TABLE orders (<br>    id UUID PRIMARY KEY,<br>    tenant_id UUID NOT NULL,<br>    amount_minor BIGINT NOT NULL<br>        CHECK (amount_minor &gt; 0),<br>    currency CHAR(3) NOT NULL,<br>    status TEXT NOT NULL<br>        CHECK (status IN (&#39;awaiting_payment&#39;, &#39;paid&#39;))<br>);</pre><p>The amount is stored in minor currency units. For example, USD 49.99 is represented as 4999.</p><p>Next, the payments table:</p><pre>CREATE TABLE payments (<br>    id UUID PRIMARY KEY,<br>    order_id UUID NOT NULL UNIQUE REFERENCES orders(id),<br>    tenant_id UUID NOT NULL,<br>    amount_minor BIGINT NOT NULL,<br>    currency CHAR(3) NOT NULL,<br>    status TEXT NOT NULL<br>        CHECK (status IN (&#39;pending&#39;, &#39;succeeded&#39;, &#39;failed&#39;)),<br>    created_at TIMESTAMPTZ NOT NULL DEFAULT now()<br>);</pre><p>The unique constraint on order_id represents a business rule for this example: one payment record per order. This is separate from idempotency protection. It prevents two different idempotency keys from creating two payment records for the same order.</p><p>Finally, the idempotency table:</p><pre>CREATE TABLE idempotency_records (<br>    id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,<br>    tenant_id UUID NOT NULL,<br>    operation TEXT NOT NULL,<br>    idempotency_key TEXT NOT NULL,<br>    request_hash TEXT NOT NULL,<br>    payment_id UUID REFERENCES payments(id),<br>    response_status INTEGER,<br>    response_body JSONB,<br>    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),</pre><pre>    UNIQUE (tenant_id, operation, idempotency_key)<br>);</pre><p>The response fields are temporarily nullable inside the transaction while the operation is being constructed. In this short, fully transactional implementation, the application must never commit an incomplete idempotency record.</p><p>For a production implementation, enforce that invariant through appropriate database constraints or a carefully designed state model, and define an explicit retention policy.</p><h3>Why store the request hash?</h3><p>Imagine a client accidentally reuses the same key for two different orders.</p><p>The key alone cannot tell us whether the request is equivalent to the original request.</p><p>The request hash solves this problem.</p><p>For this example, the normalized request consists of the order identifier. The server computes a hash of that validated value. More complex APIs should hash a well-defined canonical representation of the inputs that affect the operation’s meaning.</p><p>If the same key arrives with a different request hash, the server returns a conflict instead of silently returning an unrelated result.</p><h3>6. Implement idempotency with TypeScript and Express</h3><p>The following example illustrates the transaction logic. It assumes Express authentication middleware has already authenticated the request and attached a trusted tenantId to the request context.</p><p>Install the dependencies:</p><pre>npm install express pg zod<br>npm install -D typescript tsx @types/express @types/node @types/pg</pre><p>We’ll use pg for PostgreSQL access and zod for request validation.</p><h3>The endpoint</h3><pre>import express, {<br>  Request,<br>  Response,<br>  NextFunction,<br>} from &quot;express&quot;;<br>import { Pool } from &quot;pg&quot;;<br>import { createHash, randomUUID } from &quot;node:crypto&quot;;<br>import { z } from &quot;zod&quot;;</pre><pre>const app = express();<br>const pool = new Pool({<br>  connectionString: process.env.DATABASE_URL,<br>});</pre><pre>app.use(express.json());</pre><pre>const paymentRequestSchema = z.object({<br>  orderId: z.string().uuid(),<br>}).strict();</pre><pre>const OPERATION = &quot;create-payment:v1&quot;;</pre><pre>type AuthenticatedRequest = Request &amp; {<br>  auth: {<br>    tenantId: string;<br>  };<br>};</pre><pre>function hashRequest(orderId: string): string {<br>  const canonicalRequest = JSON.stringify({ orderId });</pre><pre>  return createHash(&quot;sha256&quot;)<br>    .update(canonicalRequest)<br>    .digest(&quot;hex&quot;);<br>}</pre><pre>app.post(<br>  &quot;/payments&quot;,<br>  async (<br>    req: AuthenticatedRequest,<br>    res: Response,<br>    next: NextFunction,<br>  ) =&gt; {<br>    const parsed = paymentRequestSchema.safeParse(req.body);<br>    const key = req.get(&quot;Idempotency-Key&quot;);</pre><pre>    if (!parsed.success || !key || key.length &gt; 200) {<br>      return res.status(400).json({<br>        error: &quot;invalid_request_or_idempotency_key&quot;,<br>      });<br>    }</pre><pre>    const { orderId } = parsed.data;<br>    const tenantId = req.auth.tenantId;<br>    const requestHash = hashRequest(orderId);</pre><pre>    const client = await pool.connect();<br>    let inTransaction = false;</pre><pre>    try {<br>      await client.query(&quot;BEGIN&quot;);<br>      inTransaction = true;</pre><pre>      const inserted = await client.query(<br>        `INSERT INTO idempotency_records<br>          (tenant_id, operation, idempotency_key, request_hash)<br>         VALUES ($1, $2, $3, $4)<br>         ON CONFLICT<br>           (tenant_id, operation, idempotency_key)<br>         DO NOTHING<br>         RETURNING id`,<br>        [tenantId, OPERATION, key, requestHash],<br>      );</pre><pre>      // A conflicting insert may wait for another transaction.<br>      // If that transaction commits, read its completed result.<br>      if (inserted.rowCount === 0) {<br>        const existing = await client.query(<br>          `SELECT request_hash, response_status, response_body<br>           FROM idempotency_records<br>           WHERE tenant_id = $1<br>             AND operation = $2<br>             AND idempotency_key = $3`,<br>          [tenantId, OPERATION, key],<br>        );</pre><pre>        const record = existing.rows[0];</pre><pre>        if (!record) {<br>          throw new Error(&quot;Idempotency record not found&quot;);<br>        }</pre><pre>        if (record.request_hash !== requestHash) {<br>          await client.query(&quot;ROLLBACK&quot;);<br>          inTransaction = false;</pre><pre>          return res.status(409).json({<br>            error: &quot;idempotency_key_reused&quot;,<br>          });<br>        }</pre><pre>        if (<br>          record.response_status == null ||<br>          record.response_body == null<br>        ) {<br>          await client.query(&quot;ROLLBACK&quot;);<br>          inTransaction = false;</pre><pre>          return res.status(503).json({<br>            error: &quot;operation_result_unavailable&quot;,<br>          });<br>        }</pre><pre>        await client.query(&quot;COMMIT&quot;);<br>        inTransaction = false;</pre><pre>        return res<br>          .status(record.response_status)<br>          .json(record.response_body);<br>      }</pre><pre>      // Load trusted payment data from the database.<br>      const orderResult = await client.query(<br>        `SELECT id, amount_minor, currency<br>         FROM orders<br>         WHERE id = $1<br>           AND tenant_id = $2<br>           AND status = &#39;awaiting_payment&#39;<br>         FOR UPDATE`,<br>        [orderId, tenantId],<br>      );</pre><pre>      if (orderResult.rowCount === 0) {<br>        await client.query(&quot;ROLLBACK&quot;);<br>        inTransaction = false;</pre><pre>        return res.status(404).json({<br>          error: &quot;payable_order_not_found&quot;,<br>        });<br>      }</pre><pre>      const order = orderResult.rows[0];<br>      const paymentId = randomUUID();</pre><pre>      await client.query(<br>        `INSERT INTO payments<br>          (id, order_id, tenant_id, amount_minor, currency, status)<br>         VALUES ($1, $2, $3, $4, $5, &#39;pending&#39;)`,<br>        [<br>          paymentId,<br>          order.id,<br>          tenantId,<br>          order.amount_minor,<br>          order.currency,<br>        ],<br>      );</pre><pre>      await client.query(<br>        `UPDATE orders<br>         SET status = &#39;paid&#39;<br>         WHERE id = $1 AND tenant_id = $2`,<br>        [orderId, tenantId],<br>      );</pre><pre>      const responseBody = {<br>        paymentId,<br>        orderId,<br>        amountMinor: Number(order.amount_minor),<br>        currency: order.currency.trim(),<br>        status: &quot;pending&quot;,<br>      };</pre><pre>      await client.query(<br>        `UPDATE idempotency_records<br>         SET payment_id = $1,<br>             response_status = $2,<br>             response_body = $3::jsonb<br>         WHERE tenant_id = $4<br>           AND operation = $5<br>           AND idempotency_key = $6`,<br>        [<br>          paymentId,<br>          201,<br>          JSON.stringify(responseBody),<br>          tenantId,<br>          OPERATION,<br>          key,<br>        ],<br>      );</pre><pre>      await client.query(&quot;COMMIT&quot;);<br>      inTransaction = false;</pre><pre>      return res.status(201).json(responseBody);<br>    } catch (error) {<br>      if (inTransaction) {<br>        try {<br>          await client.query(&quot;ROLLBACK&quot;);<br>        } catch {<br>          // The connection may already be unavailable.<br>        }<br>      }</pre><pre>      return next(error);<br>    } finally {<br>      client.release();<br>    }<br>  },<br>);</pre><p>This endpoint is a teaching example, not a complete payment-processing system. Production use requires additional work around database error classification, authentication middleware, order-state transitions, input limits, response schemas, monitoring, and external payment execution.</p><p>One especially important detail: the example records a local payment in the pending state. It does not claim that money has been charged. The real charge must be confirmed by a payment provider or another trusted payment-processing component.</p><h3>What the implementation does</h3><p>The important steps are:</p><ol><li>Validate the request and identify the authenticated tenant.</li><li>Compute a request hash from normalized input.</li><li>Attempt to claim the idempotency key using a database-enforced unique constraint.</li><li>If the key already exists, compare hashes and replay the saved result.</li><li>Otherwise, create the local payment and store its response inside the same transaction.</li><li>Commit before sending the response.</li></ol><p>Under PostgreSQL’s usual READ COMMITTED isolation level, INSERT ... ON CONFLICT DO NOTHING can wait for a competing transaction. The subsequent SELECT runs as a new statement and can observe the row committed by that competing transaction.</p><p>The implementation assumes the idempotency record and business mutation are always committed together. The 503 branch is defensive; in the intended transaction model, another transaction should not observe an incomplete committed record.</p><h3>7. What happens when two requests arrive simultaneously?</h3><p>Consider two identical requests with the same tenant, operation, and idempotency key.</p><p><strong>Request A</strong> inserts the idempotency record first.</p><p><strong>Request B</strong> attempts to insert the same unique key. PostgreSQL coordinates the competing insert.</p><p>If A commits successfully, B’s insert does nothing. B then reads A’s completed result, verifies that the request hash matches, and returns the stored response.</p><p>Only one local payment record is created.</p><p>If A rolls back, its idempotency record and payment changes disappear together. B can then claim the key and proceed.</p><p>The unique constraint is essential. A simple application-level check such as “look up the key, then insert it if absent” is not sufficient because both requests could observe that the key is absent before either inserts it.</p><p><strong>Architectural lesson:</strong> concurrency guarantees must be enforced by the system that owns the durable state, not only by application-level checks.</p><h3>8. Failure scenarios every architect should understand</h3><p>ScenarioExpected resultThe client retries after a successful commitThe saved result is returned; no second local payment is created.Two identical requests arrive concurrentlyThe unique constraint coordinates them.The same key is used for a different orderThe server returns 409 Conflict.The process crashes before the transaction commitsPostgreSQL rolls back the uncommitted transaction.The process crashes after commit but before respondingA retry can discover the completed operation.The database connection fails during commitThe application may not know whether the transaction committed; retrying with the same key resolves the ambiguity.The client generates a fresh key on every retryThe server sees a different logical operation; idempotency cannot identify it as the original request.The idempotency record expires too earlyA late retry may no longer be recognized; retention must reflect business risk.</p><p>Notice that the payment row and idempotency result are committed together. That is what allows the application to recover safely after a lost response.</p><p>The broader principle is that <strong>an uncertain response does not imply an uncertain business outcome forever</strong>. A durable operation identifier lets the system discover the outcome.</p><h3>9. The difficult part: external payment providers</h3><p>The previous example deliberately avoids making a network call to a payment provider inside the database transaction.</p><p>Why?</p><p>Because PostgreSQL cannot atomically commit a transaction together with an independent provider’s system.</p><p>Suppose we instead:</p><ol><li>Begin a database transaction.</li><li>Call a payment provider.</li><li>Receive a successful charge response.</li><li>Crash before committing the local transaction.</li></ol><p>The provider may have charged the customer even though the local database rolled back.</p><p>Simply retrying the provider call could create another charge unless the provider supports and correctly applies its own idempotency mechanism.</p><p>A more robust payment architecture separates local persistence from external execution.</p><pre>flowchart TD<br>    A[Client request] --&gt; B[API and idempotency check]<br>    B --&gt; C[Database transaction]<br>    C --&gt; D[Persist payment intent]<br>    C --&gt; E[Persist durable work item]<br>    D --&gt; F[Commit]<br>    E --&gt; F<br>    F --&gt; G[Payment worker]<br>    G --&gt; H[Provider request with stable operation key]<br>    H --&gt; I{Outcome known?}<br>    I --&gt;|Yes| J[Persist provider outcome]<br>    I --&gt;|Unknown| K[Reconcile with provider]<br>    K --&gt; J<br>    J --&gt; L[Expose final payment status]</pre><p>In this design:</p><ul><li>The API creates a durable payment intent and a durable work item in one local transaction.</li><li>A worker invokes the provider using a stable operation identifier.</li><li>If the outcome is ambiguous, the worker reconciles with the provider rather than blindly creating a new charge.</li><li>The system records the outcome and exposes a status to the client.</li></ul><p>The durable work item can be implemented using a transactional outbox or another persistent workflow mechanism.</p><p>Provider-side idempotency is especially valuable, but its key scope, retention window, and behavior for concurrent or mismatched requests must be understood. Reconciliation is still necessary when an outcome remains unknown.</p><h3>10. Idempotency versus exactly-once processing</h3><p>These concepts are related, but they are not interchangeable.</p><p><strong>Idempotency</strong> controls the effect of repeated equivalent operations.</p><p><strong>At-least-once delivery</strong> means a message or request may be delivered more than once.</p><p><strong>Exactly-once processing</strong> is often used too loosely. A broker might offer specific processing guarantees within its own boundaries, but those guarantees do not automatically make an external payment, email, or database mutation happen exactly once across an entire workflow.</p><p>A practical architecture often combines:</p><ul><li>At-least-once delivery.</li><li>Idempotent consumers.</li><li>Database uniqueness constraints.</li><li>Transactional outbox patterns.</li><li>Stable operation identifiers.</li><li>Reconciliation of ambiguous external outcomes.</li></ul><p>The goal is not to pretend failures never cause repetition. It is to make repetition safe and recovery observable.</p><h3>11. Testing the implementation</h3><p>A few happy-path unit tests are not enough. The concurrency and recovery behavior needs to be validated against PostgreSQL.</p><p>At minimum, test the following:</p><ol><li>A first request creates one payment and stores a response.</li><li>A sequential retry returns the original result.</li><li>Two simultaneous requests with the same key create only one payment.</li><li>Reusing a key with a different order returns a conflict.</li><li>A transaction rollback leaves no partial payment or idempotency record.</li><li>A simulated lost response can be recovered through a retry.</li><li>Different keys cannot bypass the business rule that an order is payable only once.</li></ol><p>The concurrent test should issue multiple requests simultaneously and then verify the durable database state. It should not merely assert that all HTTP requests returned successful responses.</p><p>Useful operational metrics include:</p><ul><li>Idempotency replay count.</li><li>Key-reuse conflicts.</li><li>Duplicate-key contention and database wait time.</li><li>Operations stuck in progress, where applicable.</li><li>Payment outcomes requiring reconciliation.</li><li>Age of the oldest unreconciled operation.</li></ul><p>Do not log access tokens, payment credentials, or sensitive customer data merely to make debugging easier.</p><h3>12. Production checklist</h3><p>Before introducing idempotency to a real API, an architect should answer these questions:</p><ul><li>Is the key scoped to the authenticated tenant and operation?</li><li>Does the client reuse the same key for retries?</li><li>Is the request validated and normalized before hashing?</li><li>Does the database enforce uniqueness for competing requests?</li><li>Are the idempotency record and local business mutation committed atomically?</li><li>Is key reuse with a different request rejected?</li><li>Is the behavior for concurrent requests explicitly defined?</li><li>Are timeouts and ambiguous commit outcomes handled safely?</li><li>Is retention long enough for realistic retry and recovery windows?</li><li>Are external side effects protected by their own idempotency or reconciliation mechanisms?</li><li>Are business invariants enforced independently of the idempotency key?</li><li>Can the team observe and recover failed or ambiguous operations?</li></ul><h3>13. What founders should take away</h3><p>Idempotency is a small implementation detail with a potentially large business impact.</p><p>For a startup, duplicate payments, repeated order creation, and duplicated background jobs can produce customer complaints, financial losses, support costs, and difficult reconciliation work.</p><p>The answer is not to introduce a complex distributed architecture everywhere. A well-designed database transaction and a unique constraint can solve the local problem for many applications.</p><p>More elaborate mechanisms become appropriate when the workflow crosses external systems, takes a long time, or requires durable asynchronous execution.</p><p>The architectural principle is:</p><p><strong>Start with the smallest design that guarantees the business invariant, then introduce additional coordination only where the system boundary requires it.</strong></p><p>That is how we build reliable systems without overengineering them.</p><h3>Final thoughts</h3><p>Retries are unavoidable. Network failures are unavoidable. Processes will crash at inconvenient times.</p><p>A reliable architecture accepts these conditions and makes them part of the design.</p><p>Idempotency is one of the small building blocks that makes this possible. It connects API design, database constraints, concurrency control, distributed systems, and business reliability.</p><p>And it provides a useful reminder for every architect: a successful request is not just one that works under ideal conditions. It is one whose behavior remains correct when the same operation is attempted again.</p><h3>Next article in this series</h3><p><strong>Retries, Timeouts, and Exponential Backoff: Why Retrying Is Not Always Safe</strong></p><p>We’ll explore how retry policies interact with idempotency, why uncontrolled retries can amplify an outage, and how to design resilient clients and services.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0790bfbca4d6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[NVIDIA GPUs: Powering the Future of Web3 and Blockchain Innovation]]></title>
            <link>https://mustufaahmed.medium.com/nvidia-gpus-powering-the-future-of-web3-and-blockchain-innovation-b3d33db06073?source=rss-6e071c8bef88------2</link>
            <guid isPermaLink="false">https://medium.com/p/b3d33db06073</guid>
            <category><![CDATA[future]]></category>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[dapps]]></category>
            <category><![CDATA[web3]]></category>
            <category><![CDATA[nvidia]]></category>
            <dc:creator><![CDATA[Muhammad Mustufa Ahmed]]></dc:creator>
            <pubDate>Tue, 07 Jan 2025 07:52:38 GMT</pubDate>
            <atom:updated>2025-01-07T07:52:38.916Z</atom:updated>
            <content:encoded><![CDATA[<p>In recent years, the blockchain industry has experienced rapid growth, evolving from a niche concept into a multi-billion-dollar ecosystem. Central to this transformation has been the adoption of Web3, a decentralized internet model powered by blockchain technology. At the heart of many advancements in Web3 is NVIDIA — a global leader in high-performance computing and graphics processing units (GPUs). NVIDIA’s cutting-edge technologies are not only accelerating blockchain computations but are also unlocking new possibilities for scalability, energy efficiency, and innovation within the Web3 space.</p><h3>The Power of NVIDIA GPUs</h3><p>NVIDIA’s GPUs are renowned for their unparalleled performance in computational tasks. Originally designed for rendering complex graphics in gaming and professional visualization, these GPUs have become indispensable in high-performance computing, artificial intelligence (AI), and blockchain.</p><p>Key features of NVIDIA GPUs that make them ideal for blockchain applications include:</p><ol><li><strong>Parallel Processing Power:</strong> NVIDIA’s CUDA architecture enables thousands of cores to process multiple tasks simultaneously, drastically improving computational efficiency.</li><li><strong>Energy Efficiency:</strong> The company’s GPUs are optimized for power consumption, which is crucial for blockchain networks aiming to reduce their environmental impact.</li><li><strong>Flexibility with CUDA:</strong> The CUDA platform provides developers with the tools to design customized applications for blockchain, AI, and data analytics.</li><li><strong>Advanced Hardware:</strong> With support for ray tracing, tensor cores, and AI accelerators, NVIDIA’s RTX and A100 series are redefining what is possible in data-intensive applications.</li></ol><h3>GPU-Accelerated Blockchain Applications</h3><p>Blockchain networks, especially those running Proof-of-Work (PoW) or Proof-of-Stake (PoS) mechanisms, require significant computational power. NVIDIA GPUs are helping in the following ways:</p><h4>1. Mining Efficiency</h4><p>In PoW blockchains like Bitcoin and Ethereum (pre-merge), miners compete to solve complex cryptographic puzzles. NVIDIA’s GPUs have been the hardware of choice for miners due to their efficiency and raw computational power. For example:</p><ul><li>NVIDIA’s GeForce RTX 3090 and 3080 GPUs offer high hash rates, enabling miners to validate transactions faster.</li><li>Their power efficiency reduces operational costs, addressing one of the key criticisms of PoW — energy consumption.</li></ul><h4>2. Smart Contract Execution</h4><p>Web3 relies heavily on smart contracts, self-executing codes running on decentralized platforms like Ethereum, Solana, and Binance Smart Chain. NVIDIA’s GPUs speed up the processing of these smart contracts by:</p><ul><li>Enabling faster simulation and testing during development.</li><li>Reducing latency in decentralized applications (dApps) that require complex computations.</li></ul><h4>3. Scaling Solutions</h4><p>Layer 2 scaling solutions, such as Optimistic Rollups and zk-Rollups, aggregate transactions off-chain to reduce the load on the main blockchain. These solutions benefit significantly from GPU acceleration, as NVIDIA GPUs can:</p><ul><li>Perform cryptographic proofs like zk-SNARKs and zk-STARKs at high speed.</li><li>Optimize the compression and aggregation of data for efficient on-chain validation.</li></ul><h4>4. AI-Powered Blockchain Analytics</h4><p>NVIDIA’s GPUs are also integral in blockchain analytics, where AI is used to:</p><ul><li>Detect fraudulent activities and anomalies in real time.</li><li>Predict trends in tokenomics and user behaviour.</li><li>Enhance decision-making for decentralized finance (DeFi) protocols and NFT marketplaces.</li></ul><h3>NVIDIA Omniverse and Web3</h3><p>One of NVIDIA’s groundbreaking contributions to the Web3 ecosystem is its Omniverse platform — a real-time 3D simulation and collaboration platform. Omniverse is playing a pivotal role in the development of:</p><ul><li><strong>Metaverse Projects:</strong> NVIDIA Omniverse provides tools for building immersive virtual worlds powered by blockchain. These worlds integrate digital ownership through NFTs and decentralized economies.</li><li><strong>Digital Twins:</strong> By simulating real-world environments on the blockchain, industries can optimize operations, logistics, and supply chains using Omniverse.</li><li><strong>Cross-Chain Collaboration:</strong> Omniverse enables developers to seamlessly integrate blockchain functionalities into metaverse projects, ensuring interoperability across networks.</li></ul><h3>Advancing Decentralized AI and Edge Computing</h3><p>With the rise of decentralized AI and edge computing, NVIDIA’s GPUs are facilitating:</p><ol><li><strong>AI Training on Blockchain:</strong> Decentralized AI platforms leverage NVIDIA GPUs to train machine learning models collaboratively while ensuring data privacy.</li><li><strong>Edge Computing for IoT Devices:</strong> NVIDIA’s Jetson platform brings GPU acceleration to edge devices, enabling blockchain applications in real-time IoT environments.</li></ol><h3>Challenges and NVIDIA’s Role in Addressing Them</h3><p>While blockchain and Web3 promise immense potential, they face challenges such as scalability, energy consumption, and interoperability. NVIDIA is tackling these challenges by:</p><ul><li><strong>Energy-Efficient GPUs:</strong> The latest GPUs, like those built on the Ada Lovelace architecture, are designed to deliver maximum performance with minimal energy consumption.</li><li><strong>Research and Development:</strong> NVIDIA’s ongoing investments in AI, quantum computing, and blockchain-specific optimizations are paving the way for the next generation of Web3 applications.</li><li><strong>Collaboration with Developers:</strong> Through programs like NVIDIA Inception, the company supports startups and developers in creating innovative blockchain solutions.</li></ul><h3>The Future: NVIDIA and the Evolution of Web3</h3><p>As Web3 continues to expand, NVIDIA’s role will likely grow even more significant. From enabling high-speed transaction processing to powering the metaverse and decentralized AI, NVIDIA GPUs are the backbone of many transformative innovations in the blockchain space.</p><p>The synergy between NVIDIA’s technology and Web3’s decentralized vision is setting the stage for a more scalable, efficient, and inclusive internet — a future where blockchain can reach its full potential with the help of high-performance computing.</p><h3>Conclusion</h3><p>NVIDIA’s high-performance GPUs have emerged as a cornerstone of the Web3 and blockchain revolution. By accelerating computational processes, enhancing energy efficiency, and enabling innovative use cases, NVIDIA is empowering developers and businesses to push the boundaries of what’s possible in the decentralized internet. As blockchain technology evolves, NVIDIA’s contributions will continue to be instrumental in shaping the next era of digital innovation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b3d33db06073" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Blockchain Decentralization]]></title>
            <link>https://mustufaahmed.medium.com/blockchain-decentralization-baaa3c054741?source=rss-6e071c8bef88------2</link>
            <guid isPermaLink="false">https://medium.com/p/baaa3c054741</guid>
            <category><![CDATA[peer-to-peer]]></category>
            <category><![CDATA[finance]]></category>
            <category><![CDATA[crypto]]></category>
            <category><![CDATA[decentralization]]></category>
            <category><![CDATA[decentralized-finance]]></category>
            <dc:creator><![CDATA[Muhammad Mustufa Ahmed]]></dc:creator>
            <pubDate>Fri, 03 Jan 2025 09:35:47 GMT</pubDate>
            <atom:updated>2025-01-03T09:35:47.157Z</atom:updated>
            <content:encoded><![CDATA[<p>In the blockchain space, decentralization means shifting control and decision-making from a single centralized authority (like an individual or organization) to a distributed network of participants. These decentralized systems aim to minimize the need for trust between users and prevent any single entity from dominating or compromising the network’s functionality.</p><h3>Understanding Blockchain Decentralization</h3><p>In its simplest form, decentralization refers to the redistribution of control and decision-making from a central authority to a distributed network. In a blockchain ecosystem, this involves participants — or nodes — sharing equal responsibility for maintaining the system, processing transactions, and ensuring its integrity. Unlike centralized systems, where a single authority oversees operations, decentralized systems operate on consensus among participants.</p><p>The key advantage of blockchain decentralization lies in its ability to reduce reliance on trust. Traditional systems demand trust in central entities, which are prone to inefficiencies, biases, and corruption. Blockchain shifts this paradigm by employing cryptographic protocols and distributed ledgers to ensure that participants can interact transparently and securely.</p><h3>The Role of Peer-to-Peer Networks</h3><p>A blockchain’s decentralized nature is underpinned by its peer-to-peer (P2P) network architecture. In a P2P network:</p><ol><li><strong>Direct Interaction:</strong> Nodes communicate directly without intermediaries, facilitating faster and more secure data exchanges.</li><li><strong>Redundancy:</strong> Each node maintains a copy of the blockchain, ensuring data integrity even if some nodes go offline.</li><li><strong>Consensus Mechanisms:</strong> Algorithms like <strong>Proof of Work</strong> (PoW) or <strong>Proof of Stake</strong> (PoS) enable nodes to agree on the validity of transactions without central oversight.</li><li><strong>Resilience:</strong> Decentralization ensures no single point of failure, making the network robust against attacks and outages.</li></ol><h3>Centralized Systems: How Banks Operate</h3><p>Traditional banking systems exemplify centralized architectures:</p><ol><li><strong>Single Authority:</strong> A central bank or institution controls monetary policies, data, and transactions.</li><li><strong>High Dependency:</strong> Users rely entirely on the institution’s trustworthiness, which can be compromised through mismanagement, corruption, or hacking.</li><li><strong>Bottlenecks:</strong> Centralized systems often experience delays, especially during high transaction volumes, as all operations funnel through a single entity.</li><li><strong>Limited Transparency:</strong> Users typically lack visibility into decision-making processes and financial records.</li></ol><h3>Comparing Centralized, Distributed, and Decentralized Systems</h3><p>In <strong>centralized systems</strong>, control is entirely held by a single authority, which manages all operations and decision-making. While these systems are straightforward and scalable, they are prone to single points of failure, making them less resilient. Transparency in such systems is often limited, with participants having minimal visibility into operations. Security relies heavily on the integrity of the central authority, leaving it vulnerable to insider threats and cyberattacks.</p><p><strong>Distributed systems</strong>, on the other hand, involve multiple authorities managing data flow centrally. These systems provide improved resilience compared to centralized systems but still maintain a degree of hierarchy that limits their transparency and security. They offer partial scalability with some bottlenecks during coordination.</p><p><strong>Decentralized systems</strong> break away from these constraints by eliminating central authority. Control is shared among multiple nodes, creating a highly resilient and transparent system. Data is immutable, and security is enhanced through cryptographic validation. However, scalability in decentralized systems often depends on the efficiency of their consensus mechanisms.</p><h3>Decentralization in Blockchain: Transforming Finance</h3><p>Blockchain decentralization is a game-changer in the financial world, offering several significant benefits:</p><ol><li><strong>Enhanced Accessibility:</strong> Anyone with internet access can participate in decentralized financial (DeFi) systems, bypassing traditional barriers like geographic location or credit history.</li><li><strong>Reduced Costs:</strong> By eliminating intermediaries, transaction fees are minimized, making financial services more affordable.</li><li><strong>Global Reach:</strong> Blockchain allows instant cross-border transactions without reliance on centralized banks.</li><li><strong>Transparency and Trust:</strong> All transactions are recorded on an immutable ledger, fostering trust and reducing fraud.</li></ol><h3>Impact of Decentralization on Security</h3><p>Security is one of the most compelling advantages of blockchain decentralization. Here’s how it strengthens systems:</p><ol><li><strong>No Single Point of Failure:</strong> Unlike centralized systems, where a breach can compromise the entire network, blockchain’s distributed nature ensures redundancy.</li><li><strong>Immutable Records:</strong> Transactions, once validated, are cryptographically sealed and cannot be altered, preventing fraud and unauthorized changes.</li><li><strong>Resilience Against Attacks:</strong> Decentralized systems are more resistant to Distributed Denial of Service (DDoS) attacks and other disruptions.</li><li><strong>Privacy-Enhancing:</strong> Blockchain uses advanced encryption techniques to protect user data while maintaining transparency in transaction history.</li></ol><h3>Market Growth Through Decentralization</h3><p>Decentralization has significantly influenced the growth and adoption of blockchain technologies across industries. Key impacts include:</p><ol><li><strong>Increased Investor Confidence:</strong> Transparent operations and tamper-proof records attract investors to blockchain-based projects.</li><li><strong>Innovation in Financial Products:</strong> Decentralization drives the development of decentralized applications (dApps), DeFi platforms, and smart contracts.</li><li><strong>Global Market Expansion:</strong> The borderless nature of decentralized systems enables businesses to tap into new markets without traditional restrictions.</li><li><strong>Empowerment of Individuals:</strong> Users gain more control over their assets, reducing dependency on centralized institutions.</li></ol><h3>Conclusion</h3><p>Blockchain decentralization represents a paradigm shift, challenging traditional systems and paving the way for more transparent, secure, and efficient operations. Its impact on the financial world is profound, democratizing access, reducing costs, and fostering trust. Furthermore, its inherent security and resilience make it a cornerstone for the future of technology and finance.</p><p>As markets continue to evolve, decentralization will play an increasingly pivotal role, driving innovation and redefining how we interact with digital and financial ecosystems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=baaa3c054741" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Overview of Smart Contract Characteristics]]></title>
            <link>https://mustufaahmed.medium.com/overview-of-smart-contract-characteristics-3c5b00383b55?source=rss-6e071c8bef88------2</link>
            <guid isPermaLink="false">https://medium.com/p/3c5b00383b55</guid>
            <category><![CDATA[smart-contracts]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[solidity-tutorial]]></category>
            <category><![CDATA[web3]]></category>
            <category><![CDATA[crypto]]></category>
            <dc:creator><![CDATA[Muhammad Mustufa Ahmed]]></dc:creator>
            <pubDate>Fri, 27 Dec 2024 10:02:10 GMT</pubDate>
            <atom:updated>2024-12-27T10:02:10.280Z</atom:updated>
            <content:encoded><![CDATA[<p>In today’s digital world, technology is transforming various industries, and one of the most revolutionary advancements is blockchain technology. A key feature of blockchain is the <strong>smart contract</strong>, which automates and secures transactions between parties without the need for intermediaries.</p><p>In this article, we’ll explore the <strong>characteristics of smart contracts</strong>, breaking them down in simple terms to help you understand how they function and why they’re gaining significant attention.</p><h3>Table of Contents</h3><ol><li>What is a Smart Contract?</li><li>Characteristics of Smart Contracts<br>1. Self-Execution<br>2. Automation<br>3. Transparency<br>4. Security<br>5. Decentralization<br>6. Immutability<br>7. Cost-Effectiveness<br>8. Efficiency<br>9. Accuracy</li><li>Applications of Smart Contracts</li><li>Challenges of Smart Contracts</li><li>Conclusion</li></ol><h3>1. What is a Smart Contract?</h3><p>A <strong>smart contract</strong> is a program or script that runs on a blockchain platform, like <strong>Ethereum</strong> or <strong>Solana</strong>, and automatically enforces the terms of an agreement. The contract is deployed on the blockchain, which ensures that it cannot be altered once created and that all actions taken by the contract are traceable and transparent.</p><h3>Key Features:</h3><ul><li><strong>Self-executing</strong>: Once conditions are met, the contract automatically triggers actions, such as releasing funds or transferring assets.</li><li><strong>Blockchain-based</strong>: Operates on decentralized, tamper-resistant networks, making it secure and transparent.</li></ul><h3>2. Characteristics of Smart Contracts</h3><p>Smart contracts are defined by several key characteristics that make them distinct and revolutionary. Let’s explore each of these in more detail.</p><h4>2.1. Self-Execution</h4><p>The hallmark of a smart contract is its ability to self-execute when predefined conditions are met, without needing manual intervention. Once a contract is programmed, it automatically performs the specified actions. For example, if two parties agree to a payment upon delivery, the smart contract will automatically release payment once the product is confirmed as delivered.</p><ul><li><strong>Example</strong>: In a supply chain scenario, a smart contract could automatically release payment when the goods are verified as delivered at the destination, removing the need for intermediaries like <strong>banks or escrow services</strong>.</li><li><strong>Benefit</strong>: This self-execution feature eliminates human delays and reduces the potential for disputes.</li></ul><h4>2.2. Automation</h4><p>Smart contracts can automate a wide variety of processes. When the contract conditions are met, actions are triggered without the need for human input. This automation improves efficiency by streamlining operations that would normally require multiple steps and manual verification.</p><ul><li><strong>Example</strong>: In insurance, a smart contract could automate claims processing by verifying conditions (e.g., flight delay, accident) and automatically triggering payments without human involvement.</li><li><strong>Benefit</strong>: Increased speed and reduced administrative workload, ensuring faster and error-free executions.</li></ul><h4>2.3. Transparency</h4><p>All smart contract transactions are recorded on the <strong>blockchain</strong>, making them visible and auditable by any participant. This transparency means that once a smart contract is deployed, anyone with access to the blockchain can verify the contract’s details, including its terms and execution history.</p><ul><li><strong>Example</strong>: In a voting system, every vote cast through a smart contract can be tracked on the blockchain, ensuring transparency and reducing the potential for tampering or fraud.</li><li><strong>Benefit</strong>: Enhances trust among participants, as all contract-related data is publicly available and verifiable.</li></ul><h4>2.4. Security</h4><p>Smart contracts leverage blockchain’s security model, which is designed to be tamper-resistant. Blockchain networks use cryptographic techniques to ensure that once data is recorded, it cannot be altered, making the contract execution both secure and resistant to fraud.</p><ul><li><strong>Example</strong>: In real estate transactions, the details of property ownership and transaction history can be securely encoded in a smart contract, reducing the risk of fraud.</li><li><strong>Benefit</strong>: Smart contracts are highly secure, reducing risks of errors, fraud, and manipulation.</li></ul><h4>2.5. Decentralization</h4><p>Smart contracts operate on <strong>decentralized blockchain networks</strong>, meaning no central authority controls the contract’s execution. This decentralization removes the need for intermediaries such as banks, lawyers, or notaries, which not only reduces costs but also eliminates the risk of a single point of failure.</p><ul><li><strong>Example</strong>: In decentralized finance (DeFi), smart contracts allow users to lend, borrow, or trade assets without the need for traditional financial institutions.</li><li><strong>Benefit</strong>: Reduces the dependency on intermediaries, leading to lower transaction costs and greater resilience against centralized failures.</li></ul><h4>2.6. Immutability</h4><p>Once a smart contract is deployed on the blockchain, it <strong>cannot be changed</strong>. This feature ensures that the terms agreed upon by the parties are final and cannot be altered or manipulated after the contract is initiated.</p><ul><li><strong>Example</strong>: Once a smart contract is used for a cryptocurrency transaction, its details (such as the amount, recipient, and timestamp) are fixed and cannot be altered, ensuring fairness and consistency.</li><li><strong>Benefit</strong>: Guarantees that the terms of the contract will be upheld as written, preventing post-agreement changes or disputes.</li></ul><h4>2.7. Cost-Effectiveness</h4><p>By removing intermediaries and automating processes, smart contracts can significantly reduce the costs associated with traditional contract management. There are fewer administrative expenses, such as legal fees or transaction fees for intermediaries like banks.</p><ul><li><strong>Example</strong>: In cross-border payments, a smart contract could automatically convert currency and complete the transaction, avoiding costly fees charged by banks and payment processors.</li><li><strong>Benefit</strong>: Helps businesses and individuals save money by eliminating middlemen and minimizing processing fees.</li></ul><h4>2.8. Efficiency</h4><p>Smart contracts enhance the efficiency of processes by speeding up transactions and automating actions. Since they operate on a blockchain, which allows for <strong>real-time verification</strong> of contract conditions, there is no need to wait for manual approval or paperwork.</p><ul><li><strong>Example</strong>: In an e-commerce scenario, a smart contract could instantly verify the purchase and dispatch of goods, allowing for immediate delivery confirmation and payment settlement.</li><li><strong>Benefit</strong>: Transactions are completed faster, improving overall workflow and enabling time-sensitive processes to run smoothly.</li></ul><h4>2.9. Accuracy</h4><p>With smart contracts, the chance of <strong>human error</strong> is greatly reduced. Since the contract is executed based on precise coding logic, the terms are implemented exactly as specified, ensuring accuracy in execution.</p><ul><li><strong>Example</strong>: In supply chain management, a smart contract can accurately track and record the exact quantities of goods delivered, ensuring that the right amounts are paid or received.</li><li><strong>Benefit</strong>: Reduces errors that may occur in manual execution, such as incorrect payment amounts or missed deadlines.</li></ul><h3>3. Applications of Smart Contracts</h3><p>Smart contracts are finding applications across various sectors due to their efficiency and security:</p><ul><li><strong>Finance and DeFi</strong>: Automating loans, insurance claims, and decentralized exchanges.</li><li><strong>Supply Chain</strong>: Tracking goods from source to destination, ensuring payments and deliveries are accurate.</li><li><strong>Healthcare</strong>: Managing medical records and automating healthcare insurance claims.</li><li><strong>Real Estate</strong>: Automating property transfers, reducing paperwork, and ensuring secure transactions.</li><li><strong>Voting Systems</strong>: Enabling transparent and secure voting mechanisms.</li></ul><h3>4. Challenges of Smart Contracts</h3><p>While smart contracts offer numerous advantages, there are some challenges:</p><ul><li><strong>Complexity</strong>: Writing and auditing smart contracts requires specialized knowledge of blockchain programming.</li><li><strong>Legal Uncertainty</strong>: In some jurisdictions, smart contracts may not yet be recognized as legally binding.</li><li><strong>Security Risks</strong>: If the code is flawed, it can be exploited. Smart contract vulnerabilities have led to high-profile hacks.</li><li><strong>Scalability</strong>: Blockchain networks can become congested, leading to delays and higher costs for contract execution.</li></ul><h3>5. Conclusion</h3><p>Smart contracts are a transformative technology with significant potential across various industries. Their key characteristics — such as automation, transparency, decentralization, and immutability — make them a reliable and efficient tool for executing agreements without the need for intermediaries. However, challenges related to scalability, security, and legal recognition still need to be addressed. As technology continues to evolve, smart contracts will likely play an increasingly important role in the digital economy.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c5b00383b55" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating an ERC-20 Token Using OpenZeppelin: A Detailed Guide]]></title>
            <link>https://blog.blockmagnates.com/creating-an-erc-20-token-using-openzeppelin-a-detailed-guide-da7e130e0867?source=rss-6e071c8bef88------2</link>
            <guid isPermaLink="false">https://medium.com/p/da7e130e0867</guid>
            <category><![CDATA[crypto-token-development]]></category>
            <category><![CDATA[ethereum-blockchain]]></category>
            <category><![CDATA[cryptocurrency]]></category>
            <category><![CDATA[solidity]]></category>
            <category><![CDATA[erc20]]></category>
            <dc:creator><![CDATA[Muhammad Mustufa Ahmed]]></dc:creator>
            <pubDate>Thu, 26 Dec 2024 14:32:31 GMT</pubDate>
            <atom:updated>2024-12-30T12:07:23.085Z</atom:updated>
            <content:encoded><![CDATA[<p>OpenZeppelin is a popular open-source framework for building secure and modular smart contracts. It provides pre-built, well-tested smart contracts for creating ERC-20 tokens, which saves you time ensures your token is secure and follows industry standards.</p><p>In this guide, we will walk through the process of creating an ERC-20 token using the OpenZeppelin library.</p><h3>Table of Contents:</h3><ol><li><strong>What is OpenZeppelin?</strong></li><li><strong>Why Use OpenZeppelin for ERC-20 Tokens?</strong></li><li><strong>Prerequisites</strong></li><li><strong>Step 1: Setting Up the Development Environment</strong></li><li><strong>Step 2: Installing OpenZeppelin Contracts</strong></li><li><strong>Step 3: Writing the ERC-20 Token Contract</strong></li><li><strong>Step 4: Deploying the Token on the Ethereum Testnet</strong></li><li><strong>Step 5: Deploying the Token on Ethereum Mainnet</strong></li><li><strong>Conclusion</strong></li></ol><h3>1. What is OpenZeppelin?</h3><p>OpenZeppelin is a framework for building secure smart contracts. It provides a set of libraries and tools that allow developers to implement commonly used functionalities, such as token standards (ERC-20, ERC-721), access control mechanisms, and upgradable contracts. The contracts provided by OpenZeppelin are community-reviewed and widely used, ensuring reliability and security.</p><h3>2. Why Use OpenZeppelin for ERC-20 Tokens?</h3><p>Using OpenZeppelin to create an ERC-20 token has several benefits:</p><ul><li><strong>Security</strong>: OpenZeppelin contracts are thoroughly audited and widely used in the industry, reducing the chances of vulnerabilities in your contract.</li><li><strong>Efficiency</strong>: OpenZeppelin’s contracts are well-documented and easy to use, allowing you to focus on your specific business logic.</li><li><strong>Customization</strong>: You can extend OpenZeppelin’s contracts with custom logic if needed, while still relying on their secure foundation.</li></ul><h3>3. Prerequisites</h3><p>Before you start, ensure you have the following tools installed and set up:</p><ul><li><strong>Node.js</strong>: A JavaScript runtime to run the development environment.</li><li><strong>npm (Node Package Manager)</strong>: To install packages.</li><li><strong>Hardhat</strong>: Development framework for deploying smart contracts.</li><li><strong>MetaMask</strong>: Ethereum wallet to interact with the Ethereum network.</li><li><strong>Infura or Alchemy</strong>: To interact with the Ethereum network via API.</li><li><strong>Solidity</strong>: The programming language for Ethereum smart contracts.</li></ul><h3>4. Step 1: Setting Up the Development Environment</h3><p>We will use <strong>Hardhat</strong> as the development framework for this tutorial, but <strong>Truffle</strong> can be used in the same way.</p><h4>Installing Hardhat</h4><p>Open a terminal and create a new directory for your project. Then, install Hardhat:</p><pre>mkdir MyToken<br>cd MyToken<br>npm init -y<br>npm install --save-dev hardhat</pre><h4>Initialize Hardhat Project</h4><p>Once Hardhat is installed, initialize a new Hardhat project:</p><pre>npx hardhat</pre><p>Follow the prompts to create a basic project setup. You can choose the <strong>“Create a basic sample project”</strong> option, which will automatically create the necessary files.</p><h3>5. Step 2: Installing OpenZeppelin Contracts</h3><p>Now that the project is set up, let’s install the OpenZeppelin contracts library, which includes the ERC-20 implementation.</p><p>In your project directory, run:</p><pre>npm install @openzeppelin/contracts</pre><p>This command installs the OpenZeppelin contracts, including ERC-20 functionality.</p><h3>6. Step 3: Writing the ERC-20 Token Contract</h3><p>Now, let’s write the ERC-20 token contract using OpenZeppelin’s pre-built ERC20 contract. Create a new file in the contracts directory named MyToken.sol.</p><p>Here’s a simple implementation:</p><pre>// SPDX-License-Identifier: MIT<br>pragma solidity ^0.8.0;<br><br>import &quot;@openzeppelin/contracts/token/ERC20/ERC20.sol&quot;;<br><br>contract MyToken is ERC20 {<br><br>    // Constructor to set the name and symbol for the token<br>    constructor(uint256 initialSupply) ERC20(&quot;MyToken&quot;, &quot;MTK&quot;) {<br>        _mint(msg.sender, initialSupply * 10 ** decimals());<br>    }<br>}</pre><h4>Explanation:</h4><ul><li><strong>import &quot;@openzeppelin/contracts/token/ERC20/ERC20.sol&quot;;</strong>: This imports the ERC20 contract from OpenZeppelin, which already contains all the standard functionality for an ERC-20 token.</li><li><strong>constructor(uint256 initialSupply)</strong>: The constructor takes an initialSupply as a parameter and mints the specified amount of tokens to the address that deploys the contract.</li><li><strong>ERC20(&quot;MyToken&quot;, &quot;MTK&quot;)</strong>: This calls the parent ERC20 constructor to set the token’s name (&quot;MyToken&quot;) and symbol (&quot;MTK&quot;).</li><li><strong>_mint(msg.sender, initialSupply * 10 ** decimals());</strong>: This mints the specified number of tokens to the deployer&#39;s address. The 10 ** decimals() ensures the correct number of tokens are minted (taking decimals into account).</li></ul><h3>7. Step 4: Deploying the Token on the Ethereum Testnet</h3><p>Now that we’ve written the token contract, let’s deploy it on the Ethereum testnet (Sepolia, etc.) to test it.</p><h4>Writing the Deployment Script</h4><p>In the scripts directory, create a new file called deploy.js to handle the deployment:</p><pre>async function main() {<br>    const [deployer] = await ethers.getSigners();<br><br>    console.log(&quot;Deploying contracts with the account:&quot;, deployer.address);<br><br>    const Token = await ethers.getContractFactory(&quot;MyToken&quot;);<br>    const token = await Token.deploy(1000000);  // Mint 1 million tokens<br>    console.log(&quot;Token deployed to:&quot;, token.address);<br>}<br><br>main()<br>    .then(() =&gt; process.exit(0))<br>    .catch((error) =&gt; {<br>        console.error(error);<br>        process.exit(1);<br>    });</pre><h4>Deploying the Contract</h4><p>To deploy the contract, you need to configure your network settings in hardhat.config.js. Add the following configuration for the testnet (you can replace it with any testnet like Goerli):</p><pre>module.exports = {<br>  solidity: &quot;0.8.0&quot;,<br>  networks: {<br>    sepolia: {<br>      url: `https://sepolia.infura.io/v3/YOUR_INFURA_PROJECT_ID`,<br>      accounts: [`0x${YOUR_PRIVATE_KEY}`]<br>    }<br>  }<br>};</pre><p>Replace YOUR_INFURA_PROJECT_ID with your Infura Project ID and YOUR_PRIVATE_KEY with the private key of your Ethereum wallet.</p><p>Finally, deploy the contract using the following command:</p><pre>npx hardhat run scripts/deploy.js --network localhost</pre><p>This will deploy the contract to the Rinkeby testnet. After the deployment is complete, you’ll see the contract address printed in the terminal.</p><h3>8. Step 5: Deploying the Token on Ethereum Mainnet</h3><p>Once you’ve tested your token on the testnet and are confident it works as expected, you can deploy it on the Ethereum mainnet.</p><h4>Mainnet Deployment</h4><p>To deploy the token on the Ethereum mainnet, modify the hardhat.config.js to include the Ethereum mainnet configuration. You will need to have some ETH in your wallet to pay for the gas fees. Here&#39;s an example configuration for the Ethereum mainnet:</p><pre>module.exports = {<br>  solidity: &quot;0.8.0&quot;,<br>  networks: {<br>    mainnet: {<br>      url: `https://mainnet.infura.io/v3/YOUR_INFURA_PROJECT_ID`,<br>      accounts: [`0x${YOUR_PRIVATE_KEY}`]<br>    }<br>  }<br>};</pre><p>Once the configuration is updated, deploy your contract to the mainnet:</p><pre>npx hardhat run scripts/deploy.js --network mainnet</pre><h3>9. Conclusion</h3><p>Using OpenZeppelin to create and deploy an ERC-20 token is a simple, secure, and efficient way to get started with blockchain development. By leveraging their well-tested contracts, you avoid the complexities and risks associated with writing your own ERC-20 implementation from scratch.</p><p>With the token deployed to the Ethereum testnet and mainnet, you can now interact with it through wallets, decentralized exchanges, and other Ethereum-based platforms.</p><p>This guide walked you through creating an ERC-20 token using OpenZeppelin, from setting up your development environment to deploying it on the Ethereum network. <strong>Happy coding!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da7e130e0867" width="1" height="1" alt=""><hr><p><a href="https://blog.blockmagnates.com/creating-an-erc-20-token-using-openzeppelin-a-detailed-guide-da7e130e0867">Creating an ERC-20 Token Using OpenZeppelin: A Detailed Guide</a> was originally published in <a href="https://blog.blockmagnates.com/">Block Magnates</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Step-by-Step Guide to Building a Local DApp Development Environment Hardhat]]></title>
            <link>https://mustufaahmed.medium.com/step-by-step-guide-to-building-a-local-dapp-development-environment-hardhat-0b01d696e506?source=rss-6e071c8bef88------2</link>
            <guid isPermaLink="false">https://medium.com/p/0b01d696e506</guid>
            <category><![CDATA[crypto]]></category>
            <category><![CDATA[solidity-development]]></category>
            <category><![CDATA[solidity-tutorial]]></category>
            <category><![CDATA[ethereum]]></category>
            <category><![CDATA[dapps]]></category>
            <dc:creator><![CDATA[Muhammad Mustufa Ahmed]]></dc:creator>
            <pubDate>Thu, 26 Dec 2024 10:04:47 GMT</pubDate>
            <atom:updated>2024-12-26T10:12:59.222Z</atom:updated>
            <content:encoded><![CDATA[<p>This blog will walk us through the steps to create a local development environment for building Decentralized Applications (DApps) using Hardhat. Hardhat is one of the most popular development frameworks for Ethereum, and it provides a robust environment for writing, testing, and deploying smart contracts locally</p><h3>Why Hardhat for DApp Development</h3><p>Before diving into the setup, let’s briefly touch on why Hardhat is an excellent choice for developing smart contracts:</p><ul><li><strong>Local Ethereum Network</strong>: Hardhat allows you to spin up a local Ethereum network for development, eliminating the need for testnets.</li><li><strong>Advanced Debugging</strong>: Hardhat provides powerful tools like stack traces and Solidity stack traces to help with error resolution.</li><li><strong>Plugins and Ecosystem</strong>: Hardhat offers a plugin-based architecture that integrates with other tools and services like Ethers.js, Waffle, and more.</li><li><strong>Solidity Support</strong>: Full compatibility with Solidity, and seamless integration with the latest Ethereum updates.</li></ul><h4><strong>Step 1: Prerequisites</strong></h4><p>Before we dive into the actual setup, make sure you have the following tools installed on your machine:</p><ol><li><strong>Node.js</strong>: Hardhat requires Node.js. You can download it from <a href="https://nodejs.org/">Node.js official website</a>.</li><li><strong>npm</strong>: npm is the package manager for Node.js and comes bundled with Node.js. You can check its installation by running npm -v in the terminal.</li></ol><h4><strong>Step 2: Setting Up a New Project</strong></h4><p>To start with, we need to create a new directory for our project.</p><ol><li>Create a new directory and navigate into it:</li></ol><pre>mkdir my-dapp<br>cd my-dapp</pre><p>2. Initialize the project with npm:</p><p>Run the following command to initialize a new Node.js project:</p><pre>npm init -y</pre><p>This creates a package.json file that will hold your project’s metadata.</p><h4>Step 3: Installing Hardhat</h4><p>Next, we will install Hardhat in our project directory.</p><ol><li>Install Hardhat and other required dependencies:<br>Run the following command to install Hardhat:</li></ol><pre>npm install --save-dev hardhat</pre><p>2. Create a Hardhat project:</p><p>After installation, run the following command to set up a Hardhat project:</p><pre>npx hardhat</pre><p>This will prompt you to choose an option. For the sake of simplicity, you can choose <strong>“Create a basic sample project.”</strong></p><p>3. Install additional dependencies (optional but recommended):</p><p>If you plan on using libraries like Ethers.js for testing for interacting with your smart contracts, install them with the following:</p><pre>npm install --save-dev @nomicfoundation/hardhat-ethers ethers</pre><h4>Step 4: Writing Your First Smart Contract</h4><p>Now that the environment is set up, let’s write a simple smart contract.</p><p><strong>Create a new Solidity contract</strong>:</p><p>Inside the contracts folder, create a new file called HelloWorld.sol:</p><pre>// SPDX-License-Identifier: MIT<br>pragma solidity ^0.8.0;<br><br>contract HelloWorld {<br>    <br>    // A state variable to store the greeting message<br>    string public greeting;<br><br>    // Constructor to initialize the greeting message<br>    constructor() {<br>        greeting = &quot;Hello, World!&quot;;<br>    }<br><br>    // Function to get the greeting message<br>    function getGreeting() public view returns (string memory) {<br>        return greeting;<br>    }<br><br>    // Function to set a custom greeting message<br>    function setGreeting(string memory newGreeting) public {<br>        greeting = newGreeting;<br>    }<br>}</pre><h4>Step 5: Running a Local Ethereum Network</h4><p>Hardhat has a built-in local Ethereum network, which can be used for development and testing.</p><ol><li>Run the local Hardhat Network:</li></ol><p>Start the local network by running:</p><pre>npx hardhat node</pre><p>This will spin up a local Ethereum network at http://localhost:8545 with some pre-funded accounts.</p><p>2. Interact with the network:</p><p>Once the Hardhat network is running, you can interact with it through Hardhat’s console or by writing scripts to deploy and interact with your contracts.</p><p>To open the Hardhat console, use:</p><pre>npx hardhat console --network localhost</pre><h4>Step 6: Deploying Your Smart Contract</h4><p>Now, let’s deploy the smart contract to the local network.</p><ol><li>Create a deployment script:<br>Inside the scripts folder, create a new file called deploy.js.</li></ol><pre>async function main() {<br>    const [deployer] = await hre.ethers.getSigners();<br><br>    console.log(&quot;Deploying contracts with the account:&quot;, deployer.address);<br><br>    const Contract = await hre.ethers.getContractFactory(&quot;HelloWorld&quot;);<br>    const contract = await Contract.deploy(42);<br><br>    console.log(&quot;MyContract deployed to:&quot;, contract.address);<br>}<br><br>main()<br>    .then(() =&gt; process.exit(0))<br>    .catch((error) =&gt; {<br>        console.error(error);<br>        process.exit(1);<br>    });</pre><p>2. Deploy the contract:</p><p>Run the deployment script with:</p><pre>npx hardhat run scripts/deploy.js --network localhost</pre><h3>Conclusion</h3><p>By following these steps, you’ve successfully set up a local environment for DApp development using Solidity and Hardhat. You can now write and deploy smart contracts in a controlled local environment before moving to more complex setups like testnets or the Ethereum mainnet.</p><p>Hardhat makes it easy to integrate additional features, plugins, and testing tools into your development process, so feel free to explore and build your DApp further!</p><h4>Call to Action</h4><p>If you enjoyed this tutorial, don’t forget to follow my blog for more insights on Ethereum development and DApp building. Share your thoughts or questions in the comments!</p><p>This structure should provide a clear, step-by-step guide for your readers to follow while also explaining key concepts and tools. Good luck with your blog!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0b01d696e506" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>