Design Instagram — The System Design Series
How would you design a social media platform like Instagram that serves billions of users, supports millions of daily uploads, and delivers personalized feeds in milliseconds?
Instagram is much more than an application for sharing photos and videos. Behind its simple interface lies a distributed system responsible for media storage, social relationships, feed generation, real-time notifications, messaging, and recommendation algorithms.
In this article, we will design a scalable version of Instagram from scratch, covering functional requirements, high-level architecture, database design, feed generation, media processing, caching, and system design trade-offs.
1. Understanding the Requirements
Before choosing databases or designing services, we need to understand what the system must accomplish.
Functional Requirements.
The platform should support:
- User registration, login, and profile management.
- Uploading photos and videos with captions.
- Following and unfollowing other users.
- Generating a personalized home feed.
- Liking and commenting on posts.
- Stories that disappear after a defined period.
- Notifications for likes, comments, and new followers.
- Searching for users, hashtags, and content.
- Private accounts and access control
Non-Functional Requirements
The system should provide:
- High availability: Users should be able to browse content even if some services fail.
- Scalability: The architecture must handle increasing users, uploads, and traffic.
- Low latency: Feeds and images should load quickly.
- Durability: Uploaded media should not be lost.
- Consistency: Privacy settings and authorization checks must be enforced reliably.
- Security: Authentication, rate limiting, and abuse prevention must be built in.
2. Back-of-the-Envelope Estimation
Let’s make some illustrative assumptions for a large social media platform.
- Daily active users: 100 million
- Average posts uploaded per user per day: 0.2
- Average original media size: 3 MB
- Average feed reads per user per day: 50
Upload estimation
Daily uploads:
100 million × 0.2 = 20 million posts/day
Average upload rate:
20 million ÷ 86,400 ≈ 231 posts/second
Assuming an average media size of 3 MB:
20 million × 3 MB = 60 TB/day
This represents approximately 60 TB of original media per day, before replication, thumbnails, video transcoding, and derived versions.
Peak traffic could be several times higher than the average.
Key takeaway: Storing every image and video directly in a relational database would be impractical. We need object storage, a CDN, asynchronous processing, and distributed metadata storage.
3. High-Level System Architecture
A simplified architecture looks like this:
Mobile / Web Clients
|
v
DNS + CDN Edge
|
v
Load Balancer
|
v
API Gateway
|
+---------------+----------------+
| | |
v v v
User Service Post Service Feed Service
| | |
v v v
User Database Post Metadata Feed Cache
|
v
Message Queue
|
+------------+------------+
| | |
v v v
Media Worker Feed Worker Notification
| | Worker
v v v
Object Storage Feed Store Notification DB
|
v
CDN
|
v
Photos / VideosThis architecture separates user-facing requests from expensive background operations.
For example, uploading a photo should not require the application to synchronously generate every thumbnail, transcode every video variant, and update every follower’s feed before responding.
Instead, the application accepts the upload, stores the media, records the post, and delegates processing to asynchronous workers.
4. Core Components Explained
4.1 API Gateway
The API Gateway is the entry point for application requests.
Its responsibilities include:
- Routing requests to the appropriate service.
- Validating authentication tokens.
- Applying rate limits.
- Enforcing request-size limits.
- Collecting request metrics and traces.
For example:
POST /api/v1/posts
GET /api/v1/feed
POST /api/v1/posts/{postId}/likes
GET /api/v1/users/{userId}/profileFor large media uploads, clients should generally upload directly to object storage using short-lived, authorized presigned URLs rather than sending large files through the application server.
4.2 User Service
The User Service manages profiles, account settings, follow relationships, and privacy preferences.
It may store:
- User ID
- Username
- Profile photo URL
- Bio
- Privacy settings
- Account creation timestamp
A relational database is a strong choice for core user records because transactions, uniqueness constraints, and consistent updates are valuable.
Username uniqueness should be enforced by a database constraint, not just by checking availability in application code.
4.3 Post Service
The Post Service handles post creation, captions, metadata, and deletion.
A post record might look like this:
{
"post_id": "post_123",
"user_id": "user_456",
"caption": "Sunset by the beach",
"media_urls": [
"https://cdn.example.com/media/abc123"
],
"created_at": "2026-10-27T18:30:00Z",
"visibility": "followers"
}The actual media files live in object storage. The database stores references and metadata.
This separation keeps database records relatively small and allows media to be served efficiently through a CDN.
5. Media Upload and Processing Pipeline
Photos and videos are among the most expensive resources in a social media application.
A reliable upload flow could work as follows:
1. Client requests an upload URL.
|
v
2. API validates user and upload permissions.
|
v
3. Client uploads media to object storage.
|
v
4. Client confirms upload with the Post Service.
|
v
5. Post metadata is persisted.
|
v
6. Processing event enters a message queue.
|
v
7. Workers scan and process the media.
|
v
8. Generate thumbnails and video variants.
|
v
9. Publish media-ready event.
|
v
10. CDN serves optimized media to viewers.Why asynchronous processing?
Video transcoding and image transformations can take significant time. Performing all of this in the request path increases latency and makes uploads more vulnerable to timeouts.
A queue such as Kafka, Amazon SQS, or RabbitMQ can decouple uploads from processing.
Workers should be idempotent, meaning that processing the same event more than once does not create duplicate posts or corrupt state. Retries, dead-letter queues, and monitoring help recover from failures.
6. Designing the Home Feed
The home feed is one of the most interesting challenges in Instagram’s system design.
When a user opens the application, they expect to see relevant posts from accounts they follow, ranked in a useful order.
There are two major feed-generation strategies.
Strategy A: Fan-out on Write
When a user publishes a post, the system distributes a reference to that post into the feeds of their followers.
User A publishes a post
|
v
Post Service
|
v
Event Queue
|
v
Feed Workers
|
v
Followers' Feed StoresAdvantages:
- Fast feed reads.
- Feed entries can be prepared before users open the application.
- Suitable for users who follow a moderate number of accounts.
Disadvantages:
- A post from a user with millions of followers can trigger enormous write amplification.
- Feed updates consume background-processing capacity.
- Newly created posts may take time to propagate.
Strategy B: Fan-out on Read
Instead of precomputing feeds, the system fetches recent posts from followed accounts when the user requests their feed, then merges and ranks the results.
Advantages:
- Publishing a post does not require writing to every follower’s feed.
- Efficient for accounts with enormous audiences.
- Less work is performed for followers who rarely open the application.
Disadvantages:
- Feed requests require more computation and data retrieval.
- Users following many accounts can experience slower feed generation.
- Caching and ranking become more important.
Strategy C: Hybrid Fan-out
For a large social platform, a hybrid approach is often a strong design.
- Regular accounts use fan-out on write.
- Celebrity or high-follower accounts use fan-out on read.
- The Feed Service merges precomputed feed entries with recent posts from high-follower accounts.
- A ranking service orders candidates using recency, relationships, and predicted relevance.
This balances read latency against write amplification.
Get Mohit Salvi’s stories in your inbox
Join Medium for free to get updates from this writer.
The exact follower threshold should be determined through workload measurements rather than treated as a universal constant.
7. Database Design
Different parts of Instagram have different access patterns. A single database is unlikely to be optimal for every workload.
A possible starting point is:
Users and account settings
-> PostgreSQL / MySQL
Post metadata
-> Distributed SQL or a horizontally scalable NoSQL store
Follow relationships
-> Relational tables or a distributed key-value model
Feed entries
-> Redis for hot entries + durable distributed storage
Photos and videos
-> Object storage
Search
-> Elasticsearch / OpenSearch
Events and asynchronous work
-> Kafka / Amazon SQS / RabbitMQThese are design options, not a claim about Instagram’s exact internal production architecture.
Example relational schema
CREATE TABLE users (
user_id BIGINT PRIMARY KEY,
username VARCHAR(50) UNIQUE NOT NULL,
bio TEXT,
is_private BOOLEAN DEFAULT FALSE,
created_at TIMESTAMP NOT NULL
);
CREATE TABLE posts (
post_id BIGINT PRIMARY KEY,
user_id BIGINT NOT NULL,
caption TEXT,
media_key TEXT NOT NULL,
created_at TIMESTAMP NOT NULL,
FOREIGN KEY (user_id) REFERENCES users(user_id)
);
CREATE TABLE follows (
follower_id BIGINT NOT NULL,
followee_id BIGINT NOT NULL,
created_at TIMESTAMP NOT NULL,
PRIMARY KEY (follower_id, followee_id)
);
CREATE TABLE comments (
comment_id BIGINT PRIMARY KEY,
post_id BIGINT NOT NULL,
user_id BIGINT NOT NULL,
comment_text TEXT NOT NULL,
created_at TIMESTAMP NOT NULL
);In a production system, these schemas would require additional indexes, partitioning, deletion handling, moderation fields, and potentially denormalized representations.
For example, feed queries frequently retrieve posts by user and creation time, so an index on (user_id, created_at) can help.
8. Handling Likes and Comments at Scale
A viral post can receive a large number of likes and comments in a short time.
If every like updates the same database row, that row can become a contention hotspot.
A better design stores individual like relationships and processes counter updates separately.
User likes a post
|
v
Like Service
|
v
Validate request and deduplicate
|
v
Persist like relationship
|
v
Publish like event
|
v
Counter aggregation workers
|
v
Update cached / aggregated countThe like relationship should have a uniqueness constraint on (user_id, post_id) so retries do not create duplicate likes.
Counters may be updated asynchronously or through sharded counters for high-volume posts. This means displayed counts can be briefly stale.
For most social experiences, eventual consistency in a displayed like count is an acceptable trade-off, provided the underlying action is handled reliably.
Comments require additional controls, including pagination, abuse detection, moderation, and rate limiting.
9. Caching Strategy
Caching reduces database load and improves response times.
Useful caching layers include:
- CDN: Caches public or appropriately authorized media near users.
- Redis: Stores hot feed entries, profile data, and frequently accessed metadata.
- Application cache: Reuses frequently needed configuration and short-lived results.
- Database cache: Uses indexes and query optimization to reduce expensive scans.
A simple feed cache key could be:
feed:{user_id}:page:{cursor}However, cache design needs careful invalidation. If a user makes their account private, blocks someone, or deletes a post, cached data must not bypass authorization.
Private media should use access-controlled delivery mechanisms, such as short-lived signed URLs or an authorized media proxy. CDN caching must respect visibility and access rules.
10. Stories and Expiring Content
Stories introduce time-based visibility and potentially high read traffic.
A story record might contain:
story_id
user_id
media_key
created_at
expires_atThe application should check whether the story has expired when retrieving it. A background job or database TTL mechanism can clean up expired metadata, but cleanup should not be the only enforcement mechanism.
Media deletion and retention also need to account for derived thumbnails, caches, and backup policies.
The important principle is that expiration is a correctness rule, while physical cleanup is a storage-management task.
11. Notifications and Real-Time Updates
Notifications include likes, comments, follows, mentions, and other relevant activity.
A scalable design uses events rather than making every user action synchronously call the Notification Service.
Like / Comment / Follow
|
v
Domain Event
|
v
Message Queue
|
v
Notification Service
|
+----------------+
| |
v v
Notification DB Push Provider
| |
v v
In-app Activity Mobile PushThe Notification Service can batch similar events, suppress duplicates, and prioritize delivery.
Push notifications are not guaranteed to arrive immediately. The application should maintain a durable in-app notification history and support retries for transient failures.
WebSockets or server-sent events can be useful for live in-app updates, while mobile push notifications handle background delivery.
12. Sharding and Horizontal Scalability
As the platform grows, some datasets become too large or too busy for a single database instance.
Sharding distributes data across multiple database partitions.
Possible strategies include:
- Users: Partition by user ID or a hash of user ID.
- Posts: Partition by author ID, post ID, or a composite strategy.
- Feed entries: Partition by recipient user ID.
- Media: Use object-storage keys and buckets rather than database-style media sharding.
Hash-based partitioning can distribute writes more evenly, but it may make some range queries harder.
Partitioning posts only by author can create hot partitions for unusually active users. A production design may need composite partition keys or additional distribution layers.
The key is to choose partitioning based on actual access patterns and to plan for rebalancing, hot keys, and cross-partition queries.
13. Reliability, Security, and Observability
A scalable application must also be resilient and secure.
Reliability
- Use retries with exponential backoff for transient failures.
- Add dead-letter queues for repeatedly failing events.
- Make asynchronous handlers idempotent.
- Apply timeouts and circuit breakers to downstream calls.
- Replicate durable data and test disaster recovery.
- Degrade gracefully when recommendation or notification services are unavailable.
Security and privacy
- Enforce authentication and authorization on every relevant operation.
- Verify follower permissions for private accounts.
- Use rate limiting against spam and automated abuse.
- Validate uploaded files and scan for malicious content.
- Protect signed upload URLs and limit their scope and lifetime.
- Support account blocking, content reporting, and deletion workflows.
Observability
Monitor:
- Feed API latency and error rates.
- Media upload success rate.
- Queue depth and processing delays.
- Cache hit ratio.
- Database latency and connection saturation.
- Notification delivery failures.
- CDN cache hit ratio and media load time.
Distributed tracing can help identify whether slow feed requests are caused by the API, database, cache, ranking service, or another dependency.
14. Important System Design Trade-offs
The following table is formatted as code so it can be copied directly into a Medium article.
+----------------------+--------------------------+----------------------------+
| Design Decision | Option A | Option B |
+----------------------+--------------------------+----------------------------+
| Feed Generation | Fan-out on write | Fan-out on read |
| Read Performance | Fast precomputed feeds | More work per request |
| Write Amplification | High for popular users | Lower at publish time |
| Media Storage | Database blobs | Object storage + CDN |
| Processing | Synchronous | Asynchronous workers |
| Counter Updates | Immediate exact updates | Aggregated eventual counts |
| Data Storage | Single database | Partitioned distributed DB |
| Caching | Minimal caching | Multi-layer caching |
+----------------------+--------------------------+----------------------------+A practical design combines these approaches rather than choosing one strategy for the entire system.
15. Putting Everything Together
A typical request flow looks like this:
When a user uploads a post:
- The client requests an authorized upload URL.
- Media is uploaded to object storage.
- The Post Service stores metadata.
- An event triggers media processing and feed distribution.
- Workers generate media variants and update feed entries.
- The CDN serves optimized media to viewers.
When a user opens the home feed:
- The API Gateway authenticates the request.
- The Feed Service retrieves precomputed feed entries.
- Recent content from selected high-follower accounts is fetched separately.
- Candidate posts are filtered for visibility and ranked.
- Post metadata is retrieved, with caching where appropriate.
- The client receives a paginated response.
- Media loads independently from the CDN.
When a user likes a post:
- The Like Service validates the action.
- The like relationship is persisted without duplication.
- A domain event triggers counter aggregation and notification processing.
- Cached counts and in-app activity are updated asynchronously.
Conclusion
Designing Instagram is fundamentally a problem of balancing read performance, write amplification, media delivery, data consistency, and operational complexity.
The most important architectural lessons are:
- Separate media storage from post metadata.
- Use object storage and a CDN for large media files.
- Use asynchronous queues for expensive background tasks.
- Choose a hybrid feed-generation strategy to handle both regular and extremely popular accounts.
- Apply caching without compromising authorization or privacy.
- Partition data according to access patterns and monitor hot spots.
- Accept eventual consistency where appropriate, but protect correctness for actions and access control.
- Design for retries, failures, observability, and recovery from the beginning.
A system that works for thousands of users is not automatically suitable for millions. The real engineering challenge is identifying bottlenecks, understanding trade-offs, and evolving the architecture as traffic grows.
That is the essence of system design: building systems that remain fast, reliable, and maintainable as scale increases.
