Circuit Breaker Pattern

Preventing Cascade Failures in Distributed Systems

Introduction

Imagine a light switch in your house. When everything’s normal, the switch stays ON and electricity flows freely. But if there’s a short circuit, the circuit breaker OPENS to stop the flow and prevent a fire.

In distributed systems, we have the same problem. When one service fails, other services keep hammering it with requests, making things worse. The Circuit Breaker pattern is our digital version of that electrical circuit breaker — it stops the bleeding before cascade failure destroys your entire system.

In this post, I’ll show you exactly how it works, why it matters, and how to implement it with real Python code.

What is the Circuit Breaker Pattern?

The Circuit Breaker pattern monitors calls to external services and stops sending requests when those services are failing. Instead of letting requests fail over and over (wasting resources), it opens the circuit and rejects requests immediately.

Key insight: It’s not about retrying — it’s about knowing when to stop trying.

The 3 States

CLOSED  → Normal operation, requests pass through
    ↓
OPEN    → Service is failing, reject requests immediately
    ↓
HALF_OPEN → Testing if service recovered
    ↓
CLOSED  → Service is back, resume normal operation

Let’s visualize this:

Why Do You Need Circuit Breaker?

Problem: Cascade Failure

Without circuit breaker:

1. Service B calls Service A (which is down)
2. Request fails but B keeps retrying
3. B’s resources get exhausted (threads, memory, connections)
4. B slows down, then crashes
5. Service C calls B, gets timeout, crashes too
6. Entire system fails 💥

Solution: Circuit Breaker

With circuit breaker:

1. Service B calls Service A (down)
2. After 5 failures → Circuit OPENS
3. Subsequent requests rejected immediately (no retry)
4. B’s resources preserved
5. System stays partially functional
6. When A recovers → Circuit slowly reopens ✅

Visual comparison:

The 3 States Explained

1. CLOSED State ✅

• Everything is working
• Requests pass through normally
• Failures are counted
• Once failures reach threshold → OPEN

if state == CLOSED:
    if failures >= threshold:
        state = OPEN
    else:
        allow_request()

2. OPEN State 🚫

• Service is failing
• All requests are rejected immediately
• No timeout waste
• After timeout period → HALF_OPEN

if state == OPEN:
    if time_since_last_failure > timeout:
        state = HALF_OPEN
    else:
        reject_request()  # Fail fast!

3. HALF_OPEN State 🔄

• Testing if service recovered
• Allow one test request to pass through
• If success → CLOSED
• If fail → OPEN again

if state == HALF_OPEN:
    allow_test_request()
    if success:
        state = CLOSED
    else:
        state = OPEN

Real-World Scenario: App B Calling App A

Let’s use our webhook example:

Get nithesh kumar’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Setup:

• App B (sender) calls App A (receiver)
• Circuit breaker monitors the connection
• Threshold: 5 failures
• Timeout: 10 seconds

Scenario: What happens when App A goes down?

Step-by-step breakdown:

Step State Action Result 1–2 CLOSED Requests succeed No change 3–7 CLOSED 5 requests fail Failure count = 5 7 CLOSED→OPEN Threshold reached Circuit opens 8–10 OPEN Requests arrive Rejected immediately ⚡ After 10s OPEN→HALF_OPEN Timeout expires Test request allowed 11 HALF_OPEN Test request succeeds State = CLOSED 12+ CLOSED Normal operation Failure count reset

Implementation Overview

The Circuit Breaker implementation involves:

1. State tracking: Monitor failures and switch between CLOSED, OPEN, and HALF_OPEN states based on configurable thresholds.

2. Failure counting: Increment counter on each failed request. When it reaches the threshold (e.g., 5 failures), open the circuit.

3. Timeout mechanism: When OPEN, wait for a timeout period (e.g., 10 seconds) before allowing the circuit to transition to HALF_OPEN.

4. Test requests: In HALF_OPEN state, allow one test request. If it succeeds, reset to CLOSED. If it fails, return to OPEN.

Key snippet — Check before executing:

if not circuit_breaker.can_execute():
    reject_request()  # Fail fast!
else:
    try_request()

Key snippet — Record outcome:

if request_succeeds():
    circuit_breaker.record_success()
else:
    circuit_breaker.record_failure()

Pros & Cons of Circuit Breaker

✅ Pros

Benefit Explanation Prevents cascade failure Stops one service from crashing the entire system Saves resources No wasted retries when service is down Fail fast Rejects requests immediately (OPEN state) Auto-recovery HALF_OPEN allows graceful recovery without manual intervention Better UX Users get fast rejection instead of hanging timeout Resource protection Preserves threads, memory, connections

❌ Cons

Limitation Explanation Adds complexity More state management to track Configuration tuning Need to tune threshold & timeout values Temporary unavailability Requests rejected during OPEN state Not ideal for all failures Better for infrastructure failures, not app bugs Monitoring required Need alerts when circuit opens Cold start problem When HALF_OPEN, first test request might fail

When to Use Circuit Breaker

Use Circuit Breaker when:

• ✅ Calling external APIs (Payment gateways, 3rd party services)
• ✅ Database connections (Might be temporarily down)
• ✅ Microservice communication (Network unreliable)
• ✅ High-traffic systems (Prevent cascade failure)

Don’t use when:

• ❌ Simple synchronous calls with built-in timeout
• ❌ Rare external calls (Overhead not worth it)
• ❌ Application logic errors (Won’t help with bugs)

Key Takeaways

1. Circuit Breaker = Digital Light Switch — Opens when things go wrong, closes when recovered
2. 3 States Matter:

• CLOSED = Normal operation
• OPEN = Service down, reject requests
• HALF_OPEN = Testing recovery

1. Prevents cascade failure — One service down doesn’t crash the system
2. Fail fast philosophy — Reject immediately instead of retrying uselessly
3. Auto-recovery — HALF_OPEN state allows graceful comeback
4. Monitor & tune — Need alerts and threshold tuning for production

Conclusion

The Circuit Breaker pattern is essential for building resilient distributed systems. Instead of hoping services stay up, it actively monitors them and gracefully degrades when they fail.

By implementing circuit breaker in your webhook system (or any microservice), you transform a cascade failure scenario into a controlled degradation — keeping your system partially functional while the service recovers.

Start simple (like our example), monitor the metrics, tune thresholds based on your system’s behavior, and you’ll have a robust, fault-tolerant architecture.

Your services will thank you. 🔌✅