Understanding BGP: The Backbone of Multi-Cloud Networking
Introduction
As organizations embrace multi-cloud strategies — running workloads across AWS, Azure, GCP, and other providers — one question becomes critical: How do you make these disparate cloud networks talk to each other efficiently?
The answer lies in a 30-year-old protocol that powers the entire internet: Border Gateway Protocol (BGP). In this post, I’ll explain what BGP is, why it matters for multi-cloud environments, and how to implement it effectively.
What is BGP?
Border Gateway Protocol (BGP) is the routing protocol that makes the internet work. Think of it as a sophisticated GPS system for data packets that need to travel across different networks.
The Simple Analogy
Imagine you’re planning a road trip from New York to Los Angeles:
- Static routing is like following a pre-printed map with one fixed route
- BGP is like using a smart navigation system that constantly monitors traffic, road conditions, and construction, then picks the best path in real-time
Key Concepts
1. Autonomous Systems (AS)
- An AS is a collection of IP networks under a single organization’s control
- Each cloud provider (AWS, Azure, GCP) operates its own AS
- Your organization’s network is also an AS
- AS Numbers (ASNs) uniquely identify each autonomous system
2. BGP Peers (Neighbors)
- BGP routers establish connections with each other called “peering sessions”
- They exchange routing information about which networks they can reach
- Two types: eBGP (external, between different organizations) and iBGP (internal, within the same organization)
3. Route Advertisement
- BGP routers announce (advertise) the IP prefixes they can reach
- Each announcement includes a path of AS numbers the route traverses
- Routers evaluate multiple paths and choose the best one based on policies
Why BGP Matters for Multi-Cloud
Traditional Problem
Without BGP, multi-cloud networking looks like this:
AWS VPC ──static route──> Transit Gateway ──static route──> Azure VNet
│
static route
│
GCP VPCProblems:
- Manual configuration for every new network
- No automatic failover if a path goes down
- Difficult to implement traffic engineering
- Scaling becomes a nightmare
The BGP Solution
With BGP, you get:
AWS VPC ←──BGP──→ Transit Gateway ←──BGP──→ Azure VNet
↕ BGP
GCP VPCBenefits:
- Dynamic route learning: New networks automatically propagate
- Automatic failover: If a path fails, traffic reroutes instantly
- Traffic engineering: Control which paths are preferred
- Scalability: Add hundreds of networks without manual updates
Implementing BGP in Multi-Cloud Environments
Architecture Overview
Here’s a practical multi-cloud BGP architecture:
┌─────────────────────────────────────────────────────────────┐
│ Transit Hub (Your AS) │
│ ┌────────────────────────────────────────────────────────┐ │
│ │ BGP Router (Transit Gateway) │ │
│ │ - AS: 65000 │ │
│ │ - Route Reflector for scalability │ │
│ └────────────────────────────────────────────────────────┘ │
└──────────┬────────────────┬────────────────┬────────────────┘
│ │ │
eBGP │ eBGP │ eBGP │
│ │ │
┌──────▼─────┐ ┌──────▼─────┐ ┌──────▼─────┐
│ AWS │ │ Azure │ │ GCP │
│ AS 64512 │ │ AS 64513 │ │ AS 64514 │
│ │ │ │ │ │
│ 10.1.0.0/16│ │ 10.2.0.0/16│ │ 10.3.0.0/16│
└────────────┘ └────────────┘ └────────────┘Implementation Steps
Step 1: Design Your AS Architecture
Choose your ASN strategy:
- Private ASNs (64512–65534): For internal multi-cloud networks
- Public ASN: If you need internet BGP peering
Example:
Transit Hub: AS 65000
AWS Region: AS 64512
Azure Region: AS 64513
GCP Region: AS 64514Step 2: Configure BGP on Transit Gateways
Each transit gateway needs:
Basic BGP Configuration:
# Pseudocode for BGP configuration
transit_gateway_bgp = {
'local_asn': 65000,
'router_id': '192.168.0.1', # Unique identifier
'networks': ['10.0.0.0/8'], # Networks to advertise
'neighbors': []
}Add BGP Neighbors (Peers):
# Peer with AWS
aws_neighbor = {
'peer_ip': '169.254.10.1',
'peer_asn': 64512,
'multihop': False, # Direct connection
'auth_password': 'secure_password_here'
}# Peer with Azure
azure_neighbor = {
'peer_ip': '169.254.20.1',
'peer_asn': 64513,
'multihop': False
}Step 3: Implement Route Filtering
Don’t accept or advertise all routes blindly:
# Route filter example
route_policy = {
'inbound': {
# Only accept specific prefixes from AWS
'allow': ['10.1.0.0/16', '10.1.1.0/24'],
'deny': ['0.0.0.0/0'] # Deny default route
},
'outbound': {
# Only advertise your internal networks
'allow': ['10.0.0.0/16'],
'deny': ['0.0.0.0/0']
}
}Step 4: Configure BGP Attributes for Traffic Engineering
Use BGP attributes to control path selection:
AS Path Prepending (make a path less preferred):
# Make Azure path less preferred for certain traffic
azure_route = {
'prefix': '10.2.0.0/16',
'as_path_prepend': '65000 65000 65000', # Add your ASN multiple times
}Local Preference (prefer certain paths):
# Prefer AWS for 10.1.0.0/16
aws_route = {
'prefix': '10.1.0.0/16',
'local_preference': 200 # Higher = more preferred (default: 100)
}MED (Multi-Exit Discriminator) (hint to neighbors):
# Tell AWS to prefer one path over another
route_advertisement = {
'prefix': '10.0.0.0/16',
'med': 50 # Lower = more preferred
}Step 5: Implement High Availability
For production environments, use BGP for automatic failover:
Primary Path (Active)
AWS VPC ═══════════════════════> Transit Hub
│ ║
│ Secondary Path (Standby) ║
└──────────────────────────────╢Configuration:
# Primary path - higher local preference
primary_connection = {
'peer_ip': '169.254.10.1',
'local_preference': 200,
'prepend': '' # No prepending
}# Secondary path - lower local preference
secondary_connection = {
'peer_ip': '169.254.10.2',
'local_preference': 100,
'prepend': '65000 65000' # Make it less preferred
}Step 6: Monitoring and Verification
Essential BGP monitoring metrics:
bgp_health_checks = {
'peer_state': 'Established', # Must be 'Established'
'routes_received': 150, # Number of routes learned
'routes_advertised': 75, # Number of routes sent
'uptime': '7d 12h 30m', # How long peer has been up
'prefixes_active': 145 # Routes actually being used
}Key commands for verification:
# Check BGP neighbor status
show bgp neighbor 169.254.10.1# View BGP routing table
show bgp ipv4 unicast# Check specific route
show bgp ipv4 unicast 10.1.0.0/16# View BGP summary
show bgp summary
End-User Flow: How Netflix Uses DNS and BGP
Let’s walk through what happens when you click play on a Netflix video, showing how DNS and BGP work together behind the scenes.
The Complete Journey: From Click to Stream
You (User) → DNS Resolution → BGP Routing → Content Delivery
│ │ │ │
│ │ │ │
│ Finds closest Routes through Streams video
│ data center optimal path at 4K qualityStep-by-Step Flow
Phase 1: DNS Resolution (Finding the Right IP Address)
Step 1: User Action
User clicks "Play" on "Stranger Things"
URL: https://www.netflix.com/watch/80057281Step 2: DNS Lookup Begins Your device doesn’t know Netflix’s IP address yet, so it asks DNS:
Your Device: "What's the IP address for www.netflix.com?"
│
▼
Local DNS Resolver (ISP: Comcast)
│
▼
Root DNS Server: "Ask the .com servers"
│
▼
.com DNS Server: "Ask Netflix's authoritative DNS"
│
▼
Netflix Authoritative DNS Server (with GeoDNS)Step 3: Smart DNS Response (GeoDNS in Action)
Netflix’s DNS server sees your request and analyzes:
- Your location: Los Angeles, CA (based on source IP)
- Your ISP: Comcast
- Current load: Server health across data centers
- Network proximity: Latency to various sites
# Netflix DNS Logic (simplified)
def get_optimal_server(user_ip, user_isp):
user_location = geolocate(user_ip) # Los Angeles, CA
candidates = [
{
'location': 'AWS us-west-2 (Oregon)',
'ip': '52.10.20.30',
'latency': '15ms',
'load': '60%',
'bgp_as': 'AS16509' # AWS
},
{
'location': 'Open Connect Cache (LA - Comcast)',
'ip': '198.45.10.50',
'latency': '2ms',
'load': '45%',
'bgp_as': 'AS7922' # Comcast
},
{
'location': 'GCP us-west1 (Oregon)',
'ip': '35.185.20.40',
'latency': '18ms',
'load': '55%',
'bgp_as': 'AS15169' # Google
}
]
# Choose closest cache with capacity
return candidates[1] # Open Connect in LAStep 4: DNS Response
Netflix DNS → Your Device
"www.netflix.com = 198.45.10.50"
TTL: 60 seconds (short TTL for quick failover)Phase 2: BGP Routing (Getting Packets to the Right Place)
Now your device has the IP address (198.45.10.50), but how do packets actually get there? This is where BGP takes over.
Step 5: Your Device Sends the First Packet
Source: Your IP (73.162.45.123)
Destination: Netflix Server (198.45.10.50)
Payload: HTTPS Request (TLS handshake)Step 6: Packet Hits Your Home Router
Home Router:
├─ Checks routing table
├─ 198.45.10.50 not in local network
└─ Forwards to Default Gateway (ISP)Step 7: ISP’s Border Router (BGP Decision Point)
Get Abhishek Gorisaria’s stories in your inbox
Join Medium for free to get updates from this writer.
This is where BGP becomes critical! Your ISP (Comcast) has multiple paths to reach Netflix:
Comcast Border Router (AS7922)
┌─────────────────────────────────────────────────────┐
│ Routing Table for 198.45.10.0/24 │
├─────────────────────────────────────────────────────┤
│ Path 1: Direct peering with Netflix AS2906 │
│ • AS Path: 7922 → 2906 │
│ • Cost: Local (free peering) │
│ • Latency: 2ms │
│ • Local Preference: 200 ✓ BEST │
│ │
│ Path 2: Via AWS AS16509 │
│ • AS Path: 7922 → 16509 → 2906 │
│ • Cost: Transit fees │
│ • Latency: 15ms │
│ • Local Preference: 100 │
│ │
│ Path 3: Via Cogent AS174 (Tier-1) │
│ • AS Path: 7922 → 174 → 2906 │
│ • Cost: Transit fees │
│ • Latency: 25ms │
│ • Local Preference: 50 │
└─────────────────────────────────────────────────────┘Decision: Use Path 1 (Direct Peering) ✓How BGP Made This Decision:
# BGP Path Selection Process (in order)
def select_best_bgp_path(paths):
# 1. Highest Weight (Cisco-specific, local to router)
paths = filter_highest_weight(paths)
# 2. Highest Local Preference (prefer certain paths)
paths = filter_highest_local_pref(paths) # Path 1 wins: 200
if len(paths) == 1:
return paths[0]
# 3. Shortest AS Path
paths = filter_shortest_as_path(paths)
# 4. Lowest Origin Type (IGP > EGP > Incomplete)
paths = filter_origin_type(paths)
# 5. Lowest MED (Multi-Exit Discriminator)
paths = filter_lowest_med(paths)
# 6. eBGP over iBGP
paths = filter_ebgp_over_ibgp(paths)
# 7. Lowest IGP cost to next hop
paths = filter_lowest_igp_cost(paths)
# 8. Oldest route (for stability)
return oldest_route(paths)Step 8: Packet Travels Through Comcast Network
Your Home (LA)
↓ (2ms)
Comcast Regional Router (LA)
↓ (0.5ms)
Comcast Open Connect Cache (LA) ← Netflix server located here!Step 9: Reverse Path (Response Comes Back)
Netflix server responds, but the return path might be different!
Netflix Server (198.45.10.50) decides return path via BGP:┌──────────────────────────────────────────────────┐
│ Return Routing for 73.162.45.0/24 (Your ISP) │
├──────────────────────────────────────────────────┤
│ Option 1: Direct to Comcast AS7922 │
│ • AS Path: 2906 → 7922 │
│ • Local Preference: 200 ✓ BEST │
│ │
│ Option 2: Via Level3 AS3356 │
│ • AS Path: 2906 → 3356 → 7922 │
│ • Local Preference: 100 │
└──────────────────────────────────────────────────┘Return packets take direct path back to Comcast
Phase 3: What Happens During Failures (BGP Auto-Healing)
Let’s say the Open Connect cache in LA crashes:
Failure Scenario:
Time: 14:32:00 - Open Connect LA goes down
Time: 14:32:03 - BGP detects failure (via BFD)
Time: 14:32:04 - BGP withdraws routes to 198.45.10.50
Time: 14:32:05 - New best path selected automaticallyAutomatic Failover:
Step 1: BGP Route Withdrawal
Comcast Router receives BGP UPDATE message:
"WITHDRAW route: 198.45.10.0/24 via direct peering"Step 2: BGP Reconverges
Comcast Router recalculates:
├─ Path 1 (Direct): WITHDRAWN ✗
├─ Path 2 (AWS): NOW BEST ✓
└─ Path 3 (Cogent): AvailableStep 3: New Path Activated
Traffic now flows:
Your Home → Comcast → AWS us-west-2 → Netflix Origin
Latency increases from 2ms → 15ms (acceptable)Step 4: DNS Cache Updates
After 60 seconds (TTL expired):
New DNS lookup returns different IP:
www.netflix.com = 52.10.20.30 (AWS instance)
User Experience:
14:32:00 - Streaming "Stranger Things" - perfect 4K
14:32:03 - Brief buffer (1-2 seconds) ← BGP reconverging
14:32:05 - Streaming resumes from AWS - still 4K
(You barely noticed!)Real-World Example: Multi-Cloud E-Commerce Application
Let’s trace a more complex scenario: An e-commerce site using AWS, Azure, and GCP.
Application: ShopFast.com (Multi-Cloud Architecture)
┌─────────────────────────────────────────────────────────────┐
│ ShopFast.com Multi-Cloud Architecture │
├─────────────────────────────────────────────────────────────┤
│ • Frontend: AWS CloudFront + S3 │
│ • API Gateway: AWS us-east-1 + Azure eastus (active) │
│ • Product Search: GCP us-central1 (Elasticsearch) │
│ • Database: AWS RDS Primary + Azure Replica │
│ • Payment: Azure (PCI compliance) │
│ • Analytics: GCP BigQuery │
└─────────────────────────────────────────────────────────────┘User Flow: Buying a Laptop
1. User Searches for “Gaming Laptop”
User (New York) → DNS Lookup
└─ api.shopfast.com = 18.210.45.60 (AWS us-east-1)
Request hits AWS API Gateway
↓
│ Internal Request to Search Service
├─ Destination: search.internal.shopfast.com
├─ DNS Resolution (Internal): 10.200.15.30
└─ BGP routing needed: AWS VPC → GCP VPCBGP Routing: AWS to GCP
┌────────────────────────────────────────────────────────┐
│ ShopFast Transit Hub (AS 65000) │
│ Routes between clouds │
└───────┬────────────────────────────┬───────────────────┘
│ │
eBGP│(AS 65001) eBGP│(AS 65002)
│ │
┌───────▼────────┐ ┌────────▼───────┐
│ AWS us-east-1 │ │ GCP us-central1│
│ API Gateway │──BGP────→│ Search Engine │
│ 10.100.0.0/16 │ │ 10.200.0.0/16 │
└────────────────┘ └────────────────┘BGP Decision:
• AS Path: 65001 → 65000 → 65002
• Route: AWS Transit Gateway → Direct Connect
→ ShopFast DC → Interconnect → GCP
• Latency: 8ms
• Backup Path: Via Internet (if Direct Connect fails)2. User Views Product Details
Request Flow:
User → CloudFront → AWS API Gateway
↓
Query Product Database
↓
BGP Route: AWS us-east-1 → Azure eastus (read replica)
↓
Why Azure?
• Closer to AWS us-east-1 than AWS us-west-2
• BGP shows lower latency (3ms vs 70ms)
• Load balancing: Database primary is busyBGP Path Selection:
# Real-time BGP decision for database query
database_routes = [
{
'endpoint': 'AWS RDS Primary (us-west-2)',
'ip': '10.100.50.10',
'as_path': [65001, 65000, 65001], # AWS → Hub → AWS
'latency_ms': 70,
'load': 85, # High load
'preference': 100
},
{
'endpoint': 'Azure Read Replica (eastus)',
'ip': '10.150.50.10',
'as_path': [65001, 65000, 65003], # AWS → Hub → Azure
'latency_ms': 3,
'load': 45, # Low load
'preference': 150 # Prefer for reads
}
]# BGP + Application Logic
selected = azure_replica # Best path3. User Clicks “Buy Now” — Payment Processing
Payment Request Flow:
User → AWS API Gateway → Payment Service (Azure)
↓
BGP Route: AWS → Transit Hub → Azure
↓
Why Azure for payments?
• PCI DSS compliance zone
• Dedicated network path
• Higher security requirementsBGP Security for Payment Path:
┌──────────────────────────────────────────────────────┐
│ Secure Payment Path (BGP + IPsec) │
├──────────────────────────────────────────────────────┤
│ 1. BGP advertises payment subnet: 10.150.100.0/24 │
│ 2. Route filter: ONLY accept from trusted ASNs │
│ 3. IPsec tunnel: Encrypted over BGP-routed path │
│ 4. BFD: 300ms failover to backup path │
│ │
│ Primary: AWS → Azure Direct Peering │
│ Backup: AWS → Transit Hub → Azure │
│ Emergency: AWS → Internet → Azure (last resort) │
└──────────────────────────────────────────────────────┘4. Real-Time Inventory Check (Multi-Cloud Query)
Inventory Check requires data from all clouds:User Request: "Is this laptop in stock near me?"
↓
┌──────────────────────────────────────────────────┐
│ Parallel Queries (BGP routes all) │
├──────────────────────────────────────────────────┤
│ Query 1: AWS → Warehouse DB (AWS RDS) │
│ • BGP Path: Local (same VPC) │
│ • Latency: 1ms │
│ │
│ Query 2: Azure → Store Locations (Azure SQL) │
│ • BGP Path: AWS → Hub → Azure │
│ • Latency: 3ms │
│ │
│ Query 3: GCP → Real-time Analytics (BigQuery) │
│ • BGP Path: AWS → Hub → GCP │
│ • Latency: 8ms │
└──────────────────────────────────────────────────┘
↓
All responses aggregate in 10ms total
↓
User sees: "3 available nearby, ships today"The Magic: How BGP Enables This Seamlessly
Without BGP:
❌ Manual static routes between every cloud
❌ No automatic failover
❌ Can't optimize for latency dynamically
❌ Network team manually updates routes weekly
❌ Downtime during failuresWith BGP:
✅ Routes automatically propagated
✅ Sub-second failover (with BFD)
✅ Always uses optimal path
✅ Self-healing network
✅ Scales to thousands of routes
✅ Traffic engineering for complianceTimeline: Complete User Experience
Time Action BGP Role
─────────────────────────────────────────────────────────────
0.00s User types shopfast.com None (DNS only)
0.10s DNS returns 18.210.45.60 GeoDNS chose closest
0.12s Browser sends HTTPS request
0.13s ISP uses BGP to route packet BGP finds best AS path
0.15s Packet arrives at AWS
0.16s API Gateway queries search
0.17s BGP routes AWS → GCP BGP path selection
0.25s Search results from GCP
0.26s BGP routes GCP → AWS Return path via BGP
0.27s API Gateway returns response
0.28s BGP routes AWS → ISP → User BGP guides packet home
0.40s User sees search results Total: 400msKey Takeaway
DNS gets you to the right data center. BGP gets your packets there efficiently.
- DNS: “Who should I talk to?” (IP address selection)
- BGP: “What’s the best path to talk to them?” (Route selection)
Together, they create the fast, reliable internet experience we take for granted!
Real-World Multi-Cloud BGP Scenarios
Scenario 1: Active-Active Multi-Region
Goal: Run workloads in AWS US-East and Azure US-West simultaneously
Application Traffic
│
┌───▼────┐
│ DNS │ (GeoDNS routing)
└───┬────┘
│
┌──┴──┐
│ │
┌───▼──┐ ┌▼────┐
│ AWS │ │Azure│
│US-East│ │US-W │
└───┬──┘ └┬────┘
│ │
└──┬──┘
┌───▼────┐
│Transit │
│ Hub │ ← BGP keeps routes synchronized
└────────┘BGP Configuration:
- Equal-cost multipath (ECMP) for load balancing
- Fast convergence (< 3 seconds) on failure
- Route health monitoring via BFD (Bidirectional Forwarding Detection)
Scenario 2: Cloud Bursting
Goal: Run primary workload in on-premises datacenter, burst to cloud during peak
bgp_config = {
'on_prem_preference': 200, # Prefer on-premises
'cloud_preference': 100, # Use cloud as backup
'bandwidth_threshold': 80, # Trigger at 80% utilization
}# When on-prem bandwidth exceeds threshold:
# 1. Modify BGP local preference
# 2. New traffic flows to cloud
# 3. Existing connections stay on-premScenario 3: Multi-Cloud Data Replication
Goal: Replicate data across clouds with optimal routing
┌─────────┐ BGP advertises ┌──────────┐
│ AWS │ shortest path │ Azure │
│Primary │◄───────────────►│Replica │
│Database │ │ │
└────┬────┘ └─────┬────┘
│ │
│ ┌────────┐ │
└───────►│Transit │◄─────────┘
│ Hub │
└────┬───┘
│
┌────▼───┐
│ GCP │
│Replica │
└────────┘BGP automatically:
- Routes replication traffic via lowest latency path
- Fails over to alternate path if primary fails
- Balances traffic across multiple paths
Best Practices for Multi-Cloud BGP
1. Security
bgp_security = {
'md5_authentication': True, # Always use BGP MD5
'ttl_security': True, # GTSM (Generalized TTL Security)
'max_prefixes': 1000, # Prevent route table overflow
'prefix_filtering': 'strict', # Only accept expected routes
}2. Route Summarization
# Don't advertise: 10.1.1.0/24, 10.1.2.0/24, 10.1.3.0/24
# Instead, summarize to: 10.1.0.0/22
summary_route = '10.1.0.0/22'3. Use Private ASNs Correctly
- Use unique ASN per cloud region
- Document your ASN allocation
- Reserve ASNs for future expansion
4. Implement BFD for Fast Failover
bfd_config = {
'min_tx_interval': 300, # ms between BFD packets
'min_rx_interval': 300, # ms between received packets
'multiplier': 3, # Declare down after 3 missed packets
# Total failover time: 900ms (300ms × 3)
}5. Route Dampening
Prevent flapping routes from destabilizing your network:
dampening_config = {
'half_life': 15, # minutes to reduce penalty by half
'suppress_threshold': 3000,
'reuse_threshold': 750,
'max_suppress_time': 60 # minutes
}Common Pitfalls to Avoid
1. ❌ Not Using Route Filters
Problem: Accept all routes, including default route (0.0.0.0/0) Solution: Always implement strict prefix filtering
2. ❌ Forgetting AS Path Loops
Problem: Your ASN appears in the path, causing route rejection Solution: Use allowas-in carefully or fix architecture
3. ❌ Ignoring BGP Timers
Problem: Default timers (60s keepalive, 180s hold) = slow failover Solution: Tune for faster convergence (10s/30s or use BFD)
4. ❌ Not Monitoring BGP State
Problem: BGP peer goes down, no one notices until users complain Solution: Monitor peer state, route counts, and flap history
5. ❌ Symmetric Routing Assumptions
Problem: Traffic goes AWS→Azure via path A, returns via path B Solution: Use consistent BGP policies on both sides
Conclusion
BGP transforms multi-cloud networking from a manual, error-prone process into a dynamic, self-healing system. By implementing BGP correctly, you get:
✅ Automatic failover in seconds, not minutes
✅ Dynamic route learning without manual updates
✅ Traffic engineering for optimal performance
✅ Scalability to hundreds of cloud networks
✅ High availability with minimal downtime
The key is to start simple:
- Connect two clouds with basic BGP
- Add route filtering for security
- Implement monitoring and alerting
- Gradually add advanced features (BFD, traffic engineering)
- Document everything
Multi-cloud networking is complex, but BGP gives you the tools to manage that complexity effectively. Whether you’re connecting AWS and Azure, implementing hybrid cloud, or building a global multi-cloud network, BGP is the foundation that makes it all work.






