Extreme Programming — Part 1

1. What is “Extreme Programming”?

Extreme Programming is not about how much software we build. It’s about how much scale the software can absorb — without falling apart.

The real engineering challenge starts when the numbers stop looking normal. Millions of users arrive at once. Traffic suddenly jumps 10x.
A downstream service slows down. A database reaches its limit.
A queue starts building. Networks become saturated. Retries multiply the traffic.

And yet the business still expects one thing. The platform to keep working. That’s what this series is about. Not simply scaling infrastructure. Not simply adding more pods. But designing systems that can absorb extreme workloads, contain failures, degrade intelligently and continue serving the critical journey.

Think about the systems behind a major ticket release, a global shopping event, a financial market spike or a streaming event. The interesting engineering isn’t that they have thousands of servers.

It’s what happens when those servers, databases, networks and services are suddenly under pressure — and failure in one part doesn’t become failure everywhere.

That is Extreme Programming.

Push the workload to the extreme. Design the system so it doesn’t become the failure.

2. Three parts. One resilience problem.

We’ll explore that idea through three different parts of the platform.

Part 1 — Users & Sessions

50K → 10M users/hour

What happens at the front door?

Authentication, sessions, concurrency, edge protection, DDoS, rate limiting, bursts and the ability to absorb millions of users without turning a traffic spike into an outage.

Part 2 — APIs, Cache & Database

20 TPS → 50K → 100K TPS

Once users start doing things, the pressure moves deeper into the platform.

APIs, services, cache, connections, databases, contention and throughput.

The question becomes:

Can the platform keep serving requests when the workload reaches an entirely different order of magnitude?

Part 3 — Backend Processing & Data

The API response isn’t always the end of the work.

Transactions trigger events. Events trigger processing. Processing creates database activity. Workers fall behind. Queues grow.

Now the question becomes:

“Can the platform keep up with the work it creates — and what happens when it can’t? “

These three parts are connected. A backend bottleneck can overload a database. The database can slow APIs. Slow APIs can trigger retries. Retries can increase traffic. Traffic can put pressure back on sessions and authentication. So this isn’t really a three-layer architecture story.

It’s a resilience story as shown in the picture below:

And that’s the question running through the entire series:

«How far can we push the system before it breaks — and how do we design it so that it doesn’t? »

Extreme Programming — Part 1

The Front Door: From Internet to Identity

Before a user reaches the business application, a lot has already happened.

A request arrives from the internet. It passes through the edge, security controls, traffic management and eventually reaches the identity layer. The user may then authenticate, receive a token, establish a session and go through authorization before the application finally allows the business operation to happen.

We tend to think of this simply as “login.” At scale, it is much more than that. That’s where Extreme Programming begins.

In Part 1, we will start with the normal journey and then progressively push each part of the front door towards the extreme.

From Internet to API

1. The Front Door

What happens when traffic first arrives?

We will look at the edge, CDN, DDoS protection, WAF, bot protection, rate limiting and load balancing — and why the application shouldn’t be the first line of defense.

2. The Identity Choice

Do we build our own identity platform, buy a CIAM capability, or compose one?

We will look at the architectural choices around platforms such as Ping, Okta, Microsoft Entra External ID, Amazon Cognito and similar capabilities, and more importantly, understand what responsibility remains with the application.

3. The Login Journey

What actually happens when a user logs in?

Credentials, MFA, risk evaluation, authentication policies, token issuance and federation all form part of the journey. What looks like a single login button can actually trigger a surprisingly complex distributed flow.

4. After Login

The user is authenticated. Now what?

We will look at access tokens, refresh tokens, application sessions and how subsequent API requests establish that the user is still allowed to access the application.

5. Where Does State Live?

Should the application maintain a server-side session? Should it rely on token-based validation? Or should it use a hybrid model?

Every choice moves state and responsibility somewhere else. The question is not whether state exists, but where it lives and what happens when that state becomes unavailable.

6. What Can Scale Independently?

Once the journey is understood, we can break the front door into individual scaling boundaries: client traffic, edge, security, identity, session infrastructure, API gateways and the application tier.

This is where we start asking a different question:

Which parts of the platform can scale independently, and which parts are still coupled?

7. Now Push It to the Extreme

Only then do we introduce the scale challenge.

What changes when an application moves from 50,000 users an hour to 10 million users an hour?

The number of users is only the starting point. We need to understand authentication rate, concurrent sessions, token traffic, API requests, bursts, refresh storms and the workload generated by each authenticated user.

8. What Breaks First?

Extreme scale exposes the assumptions hidden inside the architecture.

What happens when the identity service slows down? The session store reaches capacity? Traffic arrives faster than autoscaling can react? A downstream dependency times out and clients start retrying?

This is where capacity engineering becomes resilience engineering.

9. Security Is Capacity Engineering

Not every request arriving at the front door is necessarily legitimate.

DDoS, bots, credential stuffing, scraping and abusive clients can consume the same capacity required by legitimate users. Security therefore isn’t something sitting beside the scaling architecture.

Security is part of the scaling architecture.

10. Failure Containment

Finally, we look at what happens when the platform reaches its limits.

Rate limiting, backpressure, circuit breakers, load shedding, graceful degradation and prioritization determine whether the platform simply becomes unavailable or continues to protect the critical customer journey.

Let’s discuss each of these layers now:

1. The Front Door

What actually sits at the front door?

Internet Traffic Protection

Each layer has a different job.

Edge/CDN is about getting traffic as close to the user as possible and keeping unnecessary traffic away from the origin.

DDoS protection is about absorbing traffic that should never reach the platform in the first place.

WAF and bot protection examine whether the request looks legitimate and whether it is behaving within expected patterns.

Rate limiting controls how much traffic a client, identity, IP, API or other dimension is allowed to generate.

Load balancing distributes legitimate traffic across available capacity.

Only then should the request reach the application or identity layer. Why not just send everything to the application?

Because the application is usually the most expensive place to make that decision. If a malicious bot sends 100,000 requests per second, we don’t want 100,000 requests travelling through the entire stack before discovering that they should have been rejected.

The same principle applies to legitimate traffic during a sudden burst.

If the application can process 20,000 requests per second but 100,000 arrive, something has to decide what happens to the remaining 80,000. That decision should not be left to the application accidentally.

The first engineering principle

Don’t make the deepest layer responsible for protecting itself from the workload arriving at its front door.

Push protection and traffic management towards the edge wherever possible.

This is where security becomes capacity engineering.

A DDoS attack, a bot storm, a credential-stuffing attack and a genuine customer surge may all look like “traffic” from the perspective of the infrastructure underneath.

The architecture has to distinguish between them before they consume expensive downstream capacity.

And this is where scale starts to get interesting

At 50,000 users an hour, the front door may appear almost invisible.

At 10 million users an hour, it becomes a completely different engineering problem. And even 10 million users/hour is not the real number we need to design for.

We need to ask:

• How many users arrive in the peak minute?
• How many authenticate simultaneously?
• How many requests does each user generate?
• How much traffic is cacheable?
• How much is dynamic?
• What happens when traffic arrives in a burst?
• How much traffic is legitimate?
• What happens when a downstream service slows down?
• What happens when clients start retrying?

That leads to the next important idea:

Users are not the unit of load. The workload generated by those users is.

And once we understand that — we can move from the front door into the next boundary: identity.

2. Identity — Choosing the Capability

Once the request has passed through the front door, we reach the identity boundary.

The first architectural question is not how does login work? It is:

Where and how do we provide customer identity?

There are three broad choices:

The choice affects more than functionality and licensing. At high traffic, we need to understand who owns the capacity and what happens when that capacity is exhausted. Key considerations include:

• Authentication throughput and peak concurrency
• Token issuance capacity
• MFA and risk-service capacity
• Identity-store performance
• Provider quotas and rate limits
• Multi-region and failover capability
• External dependencies
• Burst behaviour and latency
• Retry behaviour
• Abuse and credential-stuffing protection
• Availability during partial provider failure

Once that capability is in place, we can look at what happens when a user actually tries to sign in.

3. The Login Journey — Using the Identity Capability

The login journey is the runtime interaction with the identity capability.

At a simplified level:

CIAM Journey

The important point is that not every login follows exactly the same path.

A user may:

• Authenticate directly against the CIAM platform
• Be redirected to a federated identity provider
• Be challenged for MFA
• Undergo additional risk/policy checks

Eventually, successful authentication results in credentials — typically an access token, potentially accompanied by a refresh token or an application session. This is where the architecture starts to get interesting.

Login is an event. It should not become the mechanism by which every subsequent request proves identity. Once the user is logged in, every API call needs some way to establish:

Who is this user, and is this request still allowed?

That takes us to the next section:

4. After Login — Tokens, Sessions and Validation (OIDC)

A successful login doesn’t mean the application needs to call the identity provider for every request. The purpose of modern identity protocols such as OpenID Connect (OIDC) is to establish the user’s identity and then provide the application with credentials it can use during the rest of the journey.

A simplified OIDC flow looks like this:

OIDC Flow

The important architectural point is that the ID token and access token have different jobs.

ID Token

The ID token is primarily about the authentication event. It tells the client application that the identity provider authenticated a particular user and provides identity claims about that user. It is generally not the token that an API should use to authorize access to its resources.

Access Token

The access token represents permission to access a particular resource/API. It may contain claims such as:

• Subject
• Issuer
• Audience
• Scopes
• Expiry
• roles or other claims

The API uses those claims to determine whether the request can be accepted.

Refresh Token

A refresh token is used to obtain a new access token without forcing the user through the complete authentication journey again.

This is particularly relevant to traffic.

If access tokens have short lifetimes, millions of users can generate a substantial token-refresh workload even when they aren’t actively logging in.

The scaling question

Now consider 10 million users.

If every API request required a round trip to the identity provider:

User
↓
Application
↓
Identity Provider
↓
Application
↓
API

the identity platform could quickly become a central bottleneck.

Instead, access tokens — commonly JWTs — can often be validated locally using the issuer’s published keys:

User
↓
Application / API
↓
Validate Access Token Locally
↓
Authorize Request

The identity provider isn’t necessarily sitting synchronously in the path of every API request.

That is a massive architectural difference at scale.

But local validation isn’t free of trade-offs

If validation is entirely local, the API may not immediately know that something has changed centrally.

For example:

• User has been disabled
• Entitlement has changed
• Token has been revoked
• Security policy has changed

So we have a fundamental trade-off:

Local validation

→ lower latency
→ much lower dependency on the identity platform
→ easier horizontal scaling
→ but potentially less immediate awareness of central state changes

Remote introspection / session validation

→ more central control
→ potentially fresher state
→ but introduces network dependency and additional identity traffic

Hybrid approaches

→ local validation for normal requests
→ central checks for sensitive journeys or specific events

And this is where traffic becomes interesting. The user count alone isn’t the workload.

Suppose:

10M users/hour × 20 API requests/user

That’s:

200M API requests/hour ≈ 55,000 requests/second average.

If every request performs remote token/session validation, the identity layer could now be exposed to tens of thousands of additional requests per second.

Get Nitinagg’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

And that’s before considering:

• Login bursts
• MFA
• Token issuance
• Refresh‑token traffic
• Retries
• Multiple APIs per user journey
• Peak rather than average traffic

So — an important principle emerges:

Identity should authenticate the user without becoming a synchronous dependency for every piece of work that user subsequently performs.

That leads naturally into the next question:

Where does the state live?

Because once we start talking about tokens, sessions, revocation and validation, we’re really deciding where the system keeps the information needed to know whether a user is still allowed to continue.

5. Session State (Storage)

After authentication, the system needs a way to maintain continuity across requests. A user doesn’t authenticate from scratch on every API call.

Instead, the browser or application carries some form of credential or session reference that allows the platform to recognize the ongoing session. But a cookie is not the same thing as a session.

Cookies

A cookie is simply a mechanism for storing and sending data between the browser and a website.

Common categories include:

• Session cookies — typically live for the browser session
• Persistent cookies — survive beyond the immediate browser session
• Secure cookies — sent only over HTTPS
• HttpOnly cookies — inaccessible to browser JavaScript, reducing exposure to certain client‑side attacks
• SameSite cookies — control when cookies are sent in cross‑site requests and are an important CSRF defense mechanism

There can also be functional, preference and analytics/marketing cookies, but those are different from the authentication/session cookie we’re concerned with here.

Session ID

A common pattern is to put an opaque session ID in the cookie:

Browser
│
│ Cookie: session_id=abc123
▼
Application
│
│ lookup
▼
Session Store
│
└── user=123
roles=…
state=…
expiry=…

The browser doesn’t need to carry the complete session state. The session ID is simply a reference to state held elsewhere.

This gives us central control over things such as:

• Session expiry
• Logout
• Revocation
• Concurrent sessions
• User/device information

But it also creates a dependency on the session store.

At extreme traffic, that matters.

Stateless sessions

Instead of storing session state centrally, we can carry relevant claims in a signed token. The API can validate the token without looking up a session record for every request.

That removes a high-volume dependency:

Stateful

Request → Session Store → Application

Stateless

Request → Token Validation → Application

But we haven’t magically eliminated state.

We have moved some of the state into the credential itself, while dynamic information such as revocation, entitlement changes or account status may still need another mechanism.

Hybrid

Many real-world architectures end up somewhere in between.

For example:

Browser
│
└── Secure / HttpOnly Session Cookie
│
▼
Application / BFF
│
└── Access Token
│
▼
API

The browser maintains an application session, while the backend uses OAuth/OIDC tokens to communicate with APIs. This can provide a useful separation between browser session management and API authorization. The scaling question Now the architecture starts to reveal another set of capacity boundaries:

• How many sessions can the session store hold?
• How many session lookups per second can it handle?
• Does every API call require a session lookup?
• Can sessions be distributed across regions?
• What happens when the session store is unavailable?
• How quickly do sessions expire?
• Can a large population suddenly create sessions?
• What happens during a login or session‑refresh storm?
• Can the application scale independently from the session store?
• How do we revoke sessions without creating a central bottleneck?

And there is a particularly important distinction…

A session cookie is a transport mechanism. A session ID is an identifier. Session state is the information associated with that identifier. A token is a credential. They are related, but they are not the same thing.

Once we’ve established where session and authentication state live, we’re ready to look at the architecture from a capacity perspective:

6. What Can Scale Independently?

So far, we’ve followed the journey from the internet to the application:

Front Door → Identity → Login → Token/Session → Application

The important realization is that these are not one scaling unit. Each boundary can have a different workload, capacity limit and scaling strategy.

For example, adding application pods doesn’t help if the identity provider is already at its authentication limit.

Increasing identity capacity doesn’t help if the session store is saturated.

And scaling the API gateway doesn’t help if the downstream application or database cannot consume the traffic.

This is why “Can the platform scale?” is not really one question.

We need to ask:

• Can the edge absorb the traffic?
• Can security inspect and filter it?
• Can identity handle authentication bursts?
• Can the session/token mechanism handle concurrent users?
• Can the gateway accept and control the request volume?
• Can the application scale independently?
• What happens when one of these reaches its limit?

The architecture therefore becomes a chain of independently scalable boundaries.

And now we can finally push the workload.

7. Now Push It to the Extreme

Let’s take a relatively modest starting point:

50,000 users/hour.

Now imagine the business says:

“We need to support 10 million users per hour.”

At first glance, that sounds like a simple 200× scaling problem.

But it isn’t.

10 million users/hour is approximately:

2,778 users/second average.

And users aren’t requests.

Suppose each user generates just 20 API requests during their journey:

10M × 20 = 200M requests/hour

That’s approximately:

55,000 requests/second average.

And averages can be dangerously reassuring.

Now the questions become much more interesting:

• What does the peak minute look like?
• How many users arrive simultaneously?
• How many actually need to authenticate?
• How many MFA challenges occur?
• How many tokens need to be issued?
• How many tokens are refreshed?
• How many API calls does each user generate?
• How much traffic can the edge absorb?
• How much can be rejected or served from cache?
• How quickly can application capacity scale?
• What happens if a downstream service is already saturated?

And this is where the original 50K → 10M story becomes useful.

We’re no longer asking:

“How many servers do I need?”

We’re asking:

“Which boundary becomes the constraint first, and what does the architecture do when it reaches that constraint?”

That is the real beginning of Extreme Programming.

Because at extreme scale, capacity planning and resilience engineering become the same conversation.

8. What Breaks First?

Once we push the workload from 50K to 10M users/hour, the interesting question is no longer can we scale?

It is:

Which boundary reaches its limit first?

It could be identity. It could be the session store. It could be the gateway. It could be application compute. Or it could be something that wasn’t obvious in the original architecture. The dangerous part is that failure can amplify the original workload.

• A downstream service becomes slow
• Requests start timing out
• Clients retry
• The retries create more traffic
• More traffic creates more contention
• Latency increases further
• Capacity problem becomes a cascading failure
• Adding more compute isn’t always the answer

Mechanisms to contain failure:

• Timeouts — don’t allow a slow dependency to hold resources indefinitely
• Retry with exponential backoff and jitter — avoid synchronized retry storms
• Circuit breakers — stop repeatedly calling an unhealthy dependency
• Concurrency limits — prevent a service from accepting more work than it can safely process
• Load shedding — deliberately reject lower‑priority work when capacity is exhausted
• Backpressure — make upstream systems slow down rather than overwhelming downstream systems
• Graceful degradation — preserve the critical journey while removing non‑essential functionality

The objective isn’t to design a system that never reaches a limit.

The objective is to know what happens when it does.

9. Security Is Capacity Engineering

At extreme scale, we also need to challenge one fundamental assumption:

Not every request is a customer request.

A platform may be receiving legitimate users alongside:

• DDoS traffic
• Bots
• Credential‑stuffing attempts
• Automated scraping
• Abusive clients
• Malformed requests
• Unexpectedly aggressive consumers

From the perspective of the infrastructure underneath, all of this consumes capacity.

That makes security controls part of the traffic-management architecture, not something bolted on after capacity planning. The earlier we can identify traffic that shouldn’t consume application capacity, the more capacity we preserve for legitimate users.

But security controls themselves need to scale.

A WAF that becomes the bottleneck isn’t solving the problem. A rate limiter that cannot operate at the required throughput isn’t helping. An identity platform overwhelmed by credential-stuffing traffic can become unavailable to genuine users as well.

So we need to think about:

• DDoS absorption capacity
• WAF throughput
• Bot detection at scale
• Credential‑stuffing protection
• Per‑user / per‑client / per‑IP rate limits
• Distributed rate limiting
• MFA abuse protection
• Regional traffic controls
• Security‑event processing
• Fail‑open vs fail‑closed behaviors

And this introduces an important architectural trade-off:

The more security processing we put in the request path, the more capacity that path needs.

The answer isn’t to remove those controls.

It’s to place them at the right boundary, make them scalable, and ensure that expensive processing happens only after cheaper controls have filtered the traffic.

That brings us to the final part of Part 1: Failure Containment — what the platform should actually do when the traffic exceeds the capacity we’ve designed for.

10. Failure Containment

At some point, every system reaches a limit. The engineering question is what happens next. A resilient platform doesn’t allow one saturated component to consume everything around it. It contains the pressure.

The controls are familiar:

• Rate limiting — controls how much traffic is admitted
• Backpressure — prevents downstream systems from being overwhelmed
• Circuit breakers — stop repeatedly calling unhealthy dependencies
• Load shedding — deliberately rejects lower‑priority work
• Graceful degradation — keeps the important journey working while non‑essential capabilities are reduced
• Prioritization — ensures not all traffic is treated equally when capacity becomes constrained

This changes the objective of scaling. We’re no longer trying to guarantee infinite capacity.

We’re designing the system to fail in a controlled way when capacity is exhausted.

A platform that rejects 10% of non-critical traffic while protecting 90% of the critical journey may be behaving exactly as designed. A platform that lets those 10% consume resources until everything fails is not.

Where Does the Pressure Go Next?

And this is where Part 1 ends.

We’ve dealt primarily with the user and session boundary. But getting the user successfully through the front door is only the beginning. Once authenticated, those users start generating requests.

A single user might generate dozens of API calls. Millions of users can therefore turn into tens of thousands of requests per second.

Those requests now hit a completely different set of engineering boundaries:

APIs → Services → Cache → Connections → Database

And the questions change.

• Can the APIs handle the request rate?
• Can the cache absorb the read workload?
• Can we maintain enough database connections?
• What happens when connection pools fill?
• What happens when the database reaches its throughput limit?

And most importantly:

When adding more application pods no longer increases throughput, where does the bottleneck move?

That’s the story for Part 2 — APIs, Cache & Database.

See you soon!