System Design Building Blocks: Load Balancer vs API Gateway

Abstract

In modern distributed systems, both load balancers and API gateways are essential building blocks. They often appear near the front of a system architecture, and both handle network traffic before requests reach backend services. Because of this, they are sometimes confused with each other. However, their responsibilities are different. A load balancer mainly distributes traffic across multiple servers to improve availability, scalability, and performance. An API gateway mainly manages API traffic by providing routing, authentication, authorization, rate limiting, protocol translation, monitoring, and request transformation. This paper explains the differences between load balancers and API gateways, their roles in system design, and how they work together in real-world architectures.

1. Introduction

As applications grow from a single server into distributed systems, traffic management becomes a major design concern. A simple web application may initially run on one backend server. But as user traffic increases, one server may no longer be enough. The system needs multiple instances, better fault tolerance, and a reliable way to route requests.

Two common components used to solve these problems are the load balancer and the API gateway.

At a high level:

A load balancer answers the question:

Which backend server should handle this request?

An API gateway answers a broader question:

How should this API request be controlled, secured, routed, transformed, and observed before reaching a service?

Both components sit between clients and backend systems, but they operate at different levels of responsibility.

2. What Is a Load Balancer?

A load balancer is a system component that distributes incoming traffic across multiple backend servers. Its main goal is to prevent any single server from becoming overloaded.

For example, suppose an application has three backend servers:

Client → Load Balancer → Server A
                      → Server B
                      → Server C

When a request arrives, the load balancer selects one of the available servers and forwards the request to it.

3. Main Responsibilities of a Load Balancer

The core responsibilities of a load balancer include:

3.1 Traffic Distribution

The load balancer spreads traffic across multiple backend instances. Common algorithms include:

• Round robin
• Least connections
• Weighted round robin
• IP hash
• Random selection

For example, with round robin, requests may be sent like this:

Request 1 → Server A
Request 2 → Server B
Request 3 → Server C
Request 4 → Server A

This helps the system use resources more evenly.

3.2 High Availability

If one server fails, the load balancer can stop sending traffic to it. This improves system availability because the client does not need to know which backend servers are healthy.

Server A: Healthy
Server B: Failed
Server C: Healthy

In this case, the load balancer sends traffic only to Server A and Server C.

3.3 Health Checks

Load balancers usually perform health checks. They periodically call a backend endpoint, such as:

GET /health

If a server does not respond correctly, it is removed from the active pool.

3.4 SSL/TLS Termination

A load balancer may also terminate HTTPS connections. This means the client connects securely to the load balancer, and the load balancer forwards traffic to backend servers.

Client --HTTPS--> Load Balancer --HTTP/HTTPS--> Backend

This simplifies certificate management.

4. What Is an API Gateway?

An API gateway is a higher-level entry point for API traffic. It is commonly used in microservice architectures where many backend services exist.

For example:

Client → API Gateway → User Service
                    → Order Service
                    → Payment Service
                    → Notification Service

Instead of clients calling many services directly, clients call the API gateway. The gateway then routes requests to the correct internal service.

5. Main Responsibilities of an API Gateway

An API gateway usually provides more application-level features than a load balancer.

5.1 API Routing

The gateway routes requests based on paths, methods, headers, or versions.

Example:

GET /users/123      → User Service
POST /orders        → Order Service
POST /payments      → Payment Service
GET /api/v2/items   → Inventory Service v2

This is more advanced than simple server-level traffic distribution.

5.2 Authentication and Authorization

API gateways often verify user identity before forwarding requests.

For example, the gateway may check:

Authorization: Bearer <JWT token>

If the token is invalid, the gateway rejects the request before it reaches backend services.

This reduces duplicated authentication logic across services.

5.3 Rate Limiting

The API gateway can limit how many requests a client can make.

Example:

User A: 100 requests per minute
User B: 1,000 requests per minute

Rate limiting helps prevent abuse, protects backend services, and supports different customer tiers.

5.4 Request and Response Transformation

An API gateway can modify requests or responses.

For example:

External request:
GET /profile

Internal request:
GET /users/current/details

The gateway can also add headers, remove sensitive fields, or convert response formats.

5.5 Protocol Translation

An API gateway can translate between different protocols.

Get Beyond51’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Example:

Client uses REST
Gateway calls backend using gRPC

This allows external APIs to remain stable while internal services use different communication styles.

5.6 Monitoring and Logging

The gateway is a natural place to collect API metrics:

• Request count
• Latency
• Error rate
• Client identity
• Endpoint usage
• Status codes

This makes it easier to observe system behavior.

6. Key Differences Between Load Balancer and API Gateway

Although both components route traffic, their focus is different.

AreaLoad BalancerAPI GatewayMain purposeDistribute traffic across serversManage and control API requestsLayerUsually lower-level network or transport layer, sometimes application layerApplication/API layerRouting styleServer-based routingAPI path, method, header, version, or business-based routingAuthenticationUsually limited or not responsibleCommonly handles authentication and authorizationRate limitingSometimes supportedCommonly supportedRequest transformationUsually limitedCommonly supportedHealth checksCore featureMay support service health awarenessBest forScaling and high availabilityAPI management and microservice access control

A simple way to remember the difference is:

A load balancer manages where traffic goes.
An API gateway manages how API traffic is handled.

7. Example Architecture

A common production architecture uses both a load balancer and an API gateway:

Client
  ↓
Load Balancer
  ↓
API Gateway
  ↓
Microservices
  ↓
Databases / Caches / Message Queues

In this design, the load balancer distributes incoming traffic across multiple API gateway instances.

Client → Load Balancer → API Gateway 1
                      → API Gateway 2
                      → API Gateway 3

Then each API gateway instance handles authentication, API routing, rate limiting, and request forwarding.

API Gateway → User Service
            → Order Service
            → Payment Service

This design combines the strengths of both components.

8. Why Not Use Only a Load Balancer?

A load balancer is excellent for distributing traffic, but it usually does not provide full API management.

For example, a load balancer may route traffic like this:

/api/users → User Service
/api/orders → Order Service

But it may not be the best place to handle complex API concerns such as:

• JWT validation
• User-level rate limiting
• API versioning
• Request transformation
• Developer API keys
• Centralized API analytics
• Per-endpoint authorization policies

These are API gateway responsibilities.

9. Why Not Use Only an API Gateway?

An API gateway can route traffic, but it still needs to be highly available. In production, there are usually multiple API gateway instances.

If clients connect to only one gateway instance, that gateway becomes a single point of failure.

Therefore, a load balancer is often placed in front of the API gateway:

Client → Load Balancer → Multiple API Gateway Instances

This allows the API gateway layer itself to scale horizontally.

10. Real-World Analogy

A load balancer is like a receptionist who sends customers to the next available service desk.

An API gateway is like a security officer, receptionist, translator, and traffic controller combined. It checks identity, applies rules, decides where the request should go, and may modify the message before passing it on.

The load balancer focuses on distribution. The API gateway focuses on control.

11. Design Considerations

When designing a system, the decision is usually not “load balancer or API gateway.” In many systems, the answer is both.

Use a load balancer when you need:

• Multiple backend instances
• High availability
• Traffic distribution
• Health checks
• Horizontal scaling

Use an API gateway when you need:

• Centralized API entry point
• Authentication and authorization
• Rate limiting
• API versioning
• Request transformation
• Protocol translation
• API monitoring

For a small monolithic application, a load balancer may be enough. For a microservice-based platform, an API gateway is usually necessary.

12. Common Misunderstanding

A common misunderstanding is that an API gateway is just a smarter load balancer. This is not fully accurate.

An API gateway may include some load balancing behavior, and a modern load balancer may include some application-level routing features. However, their design goals are different.

The load balancer is mainly infrastructure-oriented.

The API gateway is mainly API-management-oriented.

In large systems, the two components are complementary rather than competing.

13. Quantity Relationship

This picture explains how different system design components scale in a distributed architecture. Usually, the number of load balancers is the smallest, often only one or two logical entry points. Behind them, there are multiple API gateway instances to handle authentication, routing, rate limiting, and policy enforcement. A system may then contain many different services, such as User Service, Order Service, and Payment Service. Each service can run multiple service instances or pods, making total service instances the largest layer.

The key relationship is:

Load Balancers < API Gateway Instances < Services < Total Service Instances

For example, a system may have 2 load balancers, 4 API gateway instances, 12 services, and 3 instances per service, resulting in 36 total service instances.

14. Conclusion

Load balancers and API gateways are both important system design building blocks, but they solve different problems.

A load balancer improves scalability and availability by distributing traffic across multiple servers. It protects the system from overload and allows failed instances to be removed from traffic automatically.

An API gateway manages API traffic at a higher level. It provides routing, authentication, authorization, rate limiting, transformation, protocol translation, and monitoring. It is especially useful in microservice architectures where many services need to be exposed through a unified interface.

In real production systems, the most common pattern is to use them together:

Client → Load Balancer → API Gateway → Backend Services

This architecture provides both infrastructure-level reliability and application-level API control.

The simplest summary is:

Use a load balancer to scale servers.
Use an API gateway to manage APIs.
Use both when building scalable, secure, and maintainable distributed systems.