Cracking AI Engineer Interviews: System Design Questions You’ll Actually Face

5 real-world architecture challenges with production-ready answers

System design interviews for AI engineering roles have evolved dramatically. Gone are the days when “throw it in a Jupyter notebook” was an acceptable answer. Today’s interviewers want to know if you can build scalable, cost-effective, production-grade AI systems — not just train models.

This guide walks through five system design questions I’ve encountered across FAANG, startups, and AI-first companies, with detailed architectural answers that demonstrate real-world thinking.

Question 1: Design a Production-Ready RAG (Retrieval Augmented Generation) System

Scenario: Design a system that allows users to ask questions about a company’s internal documentation (10,000+ documents, constantly updated). Responses should be accurate, cite sources, and handle 1,000 concurrent users with <2s latency.

Answer

High-Level Architecture

User Query → API Gateway → Query Service → 
  ├─ Embedding Model (vectorize query)
  ├─ Vector Store (similarity search)
  └─ LLM (generate answer with context)
     └─ Response with citations

Components Breakdown

1. Document Ingestion Pipeline

• Chunk Strategy: Use recursive text splitting (500 tokens, 100 overlap) to preserve context
• Embedding Model: text-embedding-3-small (OpenAI) or all-MiniLM-L6-v2 (open-source, 384 dim)
• Vector Store: Pinecone or pgvector (PostgreSQL extension)
• Metadata: Store document ID, title, section, timestamp alongside vectors
• Update Strategy: Incremental updates with document versioning; reindex nightly for major changes

2. Query Processing

• API Layer: FastAPI or Spring Boot with rate limiting (per-user quotas)
• Query Rewriting: Use a small LLM to expand abbreviations, fix typos, rephrase ambiguous queries
• Hybrid Search: Combine vector similarity (semantic) + BM25 (keyword) for better recall
• Retrieval: Top-k=10 chunks, re-rank with cross-encoder model, return top-3

3. Generation

• LLM: GPT-4o-mini (cost-effective) or Claude 3.5 Sonnet (complex reasoning)
• Prompt Template:
• Context: {retrieved_chunks} User Question: {query} Instructions: Answer using ONLY the provided context. Cite sources using [Doc Title, Section]. If unsure, say "I don't have enough information."
• Streaming: Stream tokens to frontend for perceived speed

4. Caching & Optimization

• Semantic Cache: Store embeddings of past queries; if new query is >0.95 cosine similarity, return cached response
• Redis: Cache LLM responses (TTL: 1 hour)
• Batch Processing: Group queries when possible to optimize throughput

5. Monitoring & Evaluation

• Latency Tracking: P50, P95, P99 for each component (embedding, retrieval, LLM)
• Relevance Metrics: NDCG@3, MRR for retrieval quality
• User Feedback: Thumbs up/down on answers; retrain retriever quarterly
• Cost Tracking: Tokens/day, embedding API calls, vector store queries

Scaling Considerations

• Horizontal Scaling: Stateless query service behind load balancer
• Vector Store Sharding: Partition by document category or date
• LLM Fallback: Primary (GPT-4o-mini) → Fallback (local Llama 3 8B) if API down
• CDN for Static Assets: Embedding model weights cached at edge

Trade-offs

Approach Pros Cons Cloud LLM (OpenAI/Anthropic) High quality, low ops burden Expensive at scale, data privacy concerns Self-hosted (Llama 3) Cost-effective, full control Requires GPU infra, lower quality Hybrid Search Better recall Added complexity Large chunks (1000 tokens) More context Less precise retrieval

Estimated Cost (1,000 concurrent users, 10k docs):

• Embeddings: ~$50/month (assuming 1M queries)
• LLM (GPT-4o-mini): ~$500–1,000/month
• Vector DB (Pinecone): ~$70/month (1M vectors)
• Infrastructure: ~$200/month (compute, Redis)
• Total: ~$1,000–1,500/month

Question 2: Design a Real-Time Personalized Recommendation System with LLMs

Scenario: Build a system that recommends products to users based on browsing history, using LLMs to generate natural language explanations. Must handle 100k DAU with <500ms recommendation latency.

Answer

Architecture

User Event Stream → Kafka → 
  ├─ Feature Store (user/item embeddings)
  ├─ Candidate Generation (vector similarity)
  ├─ Ranking Model (click-through prediction)
  └─ LLM (explanation generation)
     └─ Recommendations + Explanations

Components

1. Feature Engineering

• User Embeddings:
• Real-time: Last 10 interactions (lightweight neural network, 128-dim)
• Batch: Historical preferences (trained daily, 256-dim)
• Merge: Weighted average (0.3 real-time + 0.7 batch)
• Item Embeddings: Product descriptions → CLIP/BERT embeddings
• Storage: Redis (hot) + DynamoDB (cold)

2. Candidate Generation (Two-Tower Model)

• Fast Retrieval: ANN (Approximate Nearest Neighbor) search using FAISS
• Candidate Pool: Top-100 items via dot product similarity
• Filtering: Remove already purchased, out-of-stock, policy violations

3. Ranking

• Model: LightGBM or small transformer (DistilBERT) predicting CTR
• Features:
• User: age, location, purchase history, session time
• Item: category, price, popularity
• Context: time of day, device, current session behavior
• Inference: Batch score top-100 candidates → return top-10
• Serving: TorchServe or TensorFlow Serving (GPU instances)

4. LLM Explanation Generation

• When to Use: Only for top-3 recommendations (cost control)
• Prompt:
• User recently viewed: [items] Recommending: {product_name} Write a 1-sentence personalized reason (friendly, concise). Example: "Since you loved the running shoes, you might enjoy these moisture-wicking socks!"
• Model: GPT-4o-mini or fine-tuned Llama 3 8B
• Caching: Cache explanations per (user_segment, product) pair (TTL: 24h)
• Fallback: Template-based explanations if LLM times out

5. Online Learning

• Feedback Loop: Clicks, purchases, dwell time → retrain ranking model weekly
• A/B Testing: Shadow mode for new models, gradual rollout (5% → 20% → 100%)

Latency Budget (500ms target)

• Feature retrieval: 50ms
• Candidate generation: 100ms
• Ranking: 150ms
• LLM explanation: 200ms (parallelized, cached)

Scaling Strategy

• Read Replicas: Redis Cluster for feature store
• Model Sharding: Split ranking model by user cohort (geography)
• Edge Inference: Deploy lightweight models to CDN edge (CloudFlare Workers AI)

Cost Optimization

• Pre-compute Embeddings: Nightly batch job for item embeddings
• LLM Batching: Group explanation requests (micro-batches of 10)
• Progressive Enhancement: Serve recommendations immediately, stream explanations after

Question 3: Design a Multi-Modal Document Processing Pipeline

Scenario: Extract structured data from invoices (PDFs with text, tables, images). Handle 1M invoices/month with varying formats. Output: JSON with vendor, amount, line items, confidence scores.

Answer

Architecture

Invoice Upload → S3 → SQS → 
  ├─ OCR (Textract / Tesseract)
  ├─ Layout Analysis (LayoutLM)
  ├─ Entity Extraction (LLM + NER)
  └─ Validation & Storage
     └─ PostgreSQL + Audit Trail

Pipeline Stages

1. Preprocessing

• Image Enhancement: Deskew, denoise, contrast adjustment (OpenCV)
• Format Detection: PDF type (native text vs scanned image)
• Page Splitting: Separate multi-page invoices

2. OCR Layer

• Native PDF: PyMuPDF / pdfplumber (direct text extraction)
• Scanned PDF:
• AWS Textract (managed, high accuracy) OR
• Tesseract 5 + EasyOCR (open-source, self-hosted)
• Table Detection: Textract Tables API or CascadeTabNet
• Output: Bounding boxes + text per region

3. Document Understanding

• Model: LayoutLMv3 or Donut (end-to-end transformer)
• Fine-Tuning: Train on 5,000 labeled invoices (vendor, date, total, line items)
• Entity Extraction:
• Invoice Number (regex + NER)
• Vendor Name (fuzzy matching against known vendor DB)
• Total Amount (regex + currency normalization)
• Line Items (table extraction → structured JSON)

4. LLM-Powered Validation

• Prompt (GPT-4o-mini):
• OCR Output: {text_blocks} Extracted Data: {preliminary_json} Task: Verify and correct: 1. Does total match sum of line items? 2. Is vendor name standardized? 3. Are dates in ISO format? Return corrected JSON with confidence scores (0-1).
• Confidence Thresholds:
• 0.9: Auto-approve
• 0.7–0.9: Human review queue
• <0.7: Reject with error message

5. Human-in-the-Loop

• Review UI: Show original PDF + extracted JSON side-by-side
• Active Learning: Retrain LayoutLM quarterly with corrected labels
• SLA: 95% auto-processed, 5% human review (4-hour turnaround)

Scalability

• Asynchronous Processing: SQS queue → Lambda (OCR) → Step Functions (orchestration)
• GPU Instances: EC2 G5 for LayoutLM inference (batched)
• Spot Instances: 70% cost savings for non-critical workloads
• Parallel Processing: 100 workers processing simultaneously

Error Handling

• Retry Logic: Exponential backoff for transient failures
• Dead Letter Queue: Failed invoices → manual investigation
• Monitoring: CloudWatch alarms for processing lag >15 minutes

Cost Breakdown (1M invoices/month)

• AWS Textract: ~$3,000 (assuming 3 pages avg)
• GPU instances: ~$1,200 (G5.xlarge spot)
• LLM validation: ~$200 (10% require LLM check)
• Storage (S3 + RDS): ~$500
• Total: ~$5,000/month (~$0.005/invoice)

Question 4: Design a Model Serving Infrastructure with A/B Testing

Scenario: Deploy 3 versions of a sentiment analysis model for customer reviews. Route traffic dynamically, measure accuracy/latency, auto-rollback on regression. Support 10k RPS.

Answer

Architecture

Client Request → API Gateway → 
  ├─ Experiment Service (A/B assignment)
  ├─ Model Router → 
  │   ├─ Model A (v1.0 - baseline)
  │   ├─ Model B (v2.0 - fine-tuned)
  │   └─ Model C (v2.1 - distilled)
  ├─ Feature Store
  └─ Monitoring & Feedback Loop

Core Components

1. Experiment Management

• Framework: Custom service or LaunchDarkly / Optimizely
• Traffic Split:
• Model A (baseline): 70%
• Model B (challenger): 20%
• Model C (canary): 10%
• User Bucketing: Consistent hashing on user_id (ensures same user always sees same model)
• Overrides: Force-assign internal users to specific model for testing

2. Model Serving

• Serving Framework: TorchServe, TensorFlow Serving, or KServe (Kubernetes)
• Deployment Strategy:
• Model A: 6 replicas (CPU, cost-optimized)
• Model B: 2 replicas (GPU, T4)
• Model C: 1 replica (CPU, lightweight)
• Auto-Scaling: HPA (Horizontal Pod Autoscaler) based on queue depth + latency
• Health Checks: /health endpoint (model loaded + sample inference)

3. Feature Store

• Real-Time Features: Last 5 reviews, user tier (Redis)
• Batch Features: Review history, product category (BigQuery → cached)
• Versioning: Feature schemas versioned alongside models

4. Inference Pipeline

def predict(review_text, user_id):
    # 1. Assign experiment variant
    model_version = experiment_service.get_variant(user_id)
    
    # 2. Fetch features
    features = feature_store.get(user_id)
    
    # 3. Route to model
    prediction = model_router.predict(
        model=model_version,
        text=review_text,
        features=features
    )
    
    # 4. Log for analysis
    analytics.log(user_id, model_version, prediction, latency)
    
    return prediction

5. Metrics & Monitoring

Get Niket Lekariya’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Metric Tool Alert Threshold Latency (P95) Prometheus + Grafana >200ms Error Rate Datadog >1% Accuracy (online) Custom dashboard <baseline — 2% Throughput Load balancer metrics <8k RPS Cost/1k requests CloudWatch >$0.10

6. Automated Rollback

• Trigger Conditions:
• Error rate >3% for 5 minutes
• P95 latency >300ms for 10 minutes
• Accuracy drop >5% (validated via human labels on sample)
• Action:

1. Shift traffic to baseline (Model A)
2. Send Slack alert to ML team
3. Preserve logs for post-mortem

7. Evaluation Strategy

• Online Metrics: CTR, user engagement (indirect)
• Offline Metrics:
• Holdout test set accuracy (daily batch)
• Human evaluation (500 samples/week)
• Statistical Significance: Chi-squared test, minimum 10k samples per variant
• Decision Criteria:
• Accuracy improved >2% AND latency delta <10% → Promote to baseline
• Else → Reject variant

Infrastructure

• Kubernetes: GKE or EKS (managed)
• Service Mesh: Istio (traffic splitting, retries, circuit breaking)
• Load Balancer: NGINX or AWS ALB
• GPU Nodes: T4 (cost-effective for inference)

Handling 10k RPS

• Batching: Collect requests for 10ms → batch inference (throughput ↑)
• Connection Pooling: Reuse HTTP connections
• Caching: Cache predictions for identical inputs (TTL: 1 hour)
• Rate Limiting: Per-user quotas (prevent abuse)

Question 5: Design a Conversational AI System with Context Management

Scenario: Build a customer support chatbot that maintains conversation history, escalates to humans when needed, supports 50k daily active conversations. Must handle multi-turn reasoning and integrate with CRM.

Answer

Architecture

User Message → WebSocket → Chat Service →
  ├─ Context Manager (conversation history)
  ├─ Intent Classifier (routing)
  ├─ LLM (response generation)
  ├─ Tool Calling (CRM, knowledge base)
  └─ Escalation Logic
     └─ Human Agent Handoff

System Components

1. Context Management

Storage Strategy:

• Active Conversations (last 1 hour): Redis
• Key: conversation:{user_id}
• Value: JSON array of messages
• TTL: 1 hour (sliding window)
• Historical Conversations: PostgreSQL
• Indexed by user_id, timestamp
• Retention: 90 days (compliance)

Context Window:

{
  "conversation_id": "uuid",
  "user_id": "12345",
  "messages": [
    {"role": "user", "content": "Track my order", "timestamp": "..."},
    {"role": "assistant", "content": "Order #789 shipped yesterday", "timestamp": "..."},
    {"role": "user", "content": "When will it arrive?", "timestamp": "..."}
  ],
  "metadata": {
    "user_tier": "premium",
    "sentiment": "neutral",
    "intent_history": ["track_order"]
  }
}

2. Intent Classification

Approach: Lightweight classifier (DistilBERT, 20ms latency) routing to:

• Simple FAQ → Template response (no LLM needed)
• Transactional (order status, refund) → Tool calling + LLM
• Complex reasoning → Advanced LLM (GPT-4)
• Frustrated user → Immediate escalation

Sample Intents:

• track_order, request_refund, product_question, technical_support, escalation_request

3. LLM Response Generation

Primary Model: GPT-4o-mini (cost-effective, fast)

Prompt Template:

System: You are a helpful customer support agent for Acme Corp.
Access to: Order history, product catalog, refund policy.

Context:
{conversation_history}
User tier: {user_tier}
Current sentiment: {sentiment}
Tools available:
- get_order_status(order_id)
- create_refund(order_id, reason)
- search_knowledge_base(query)
User: {latest_message}
Instructions:
1. Be empathetic and concise
2. Use tools when needed
3. If unsure, escalate to human
4. Never make up order details

4. Tool Calling (Function Calling)

CRM Integration:

• Order Status: REST API to order management system
• User Profile: GraphQL to CRM (Salesforce, HubSpot)
• Refund Creation: Transactional API with idempotency keys

Implementation:

tools = [
    {
        "name": "get_order_status",
        "description": "Fetch current status of an order",
        "parameters": {
            "order_id": {"type": "string", "required": True}
        }
    },
    {
        "name": "create_refund",
        "description": "Initiate refund process",
        "parameters": {
            "order_id": {"type": "string"},
            "reason": {"type": "string"}
        }
    }
]

# LLM decides which tool to call
response = llm.chat(messages, tools=tools)
if response.tool_calls:
    results = execute_tools(response.tool_calls)
    # Feed results back to LLM for final answer
    final_response = llm.chat(messages + [results])

5. Escalation Logic

Trigger Conditions:

• User explicitly requests human (“speak to agent”)
• Sentiment drops to “very negative” (via sentiment model)
• LLM confidence <0.6 (model returns low probability)
• 5 turns without resolution
• High-value customer (premium tier) prefers humans

Handoff Process:

1. Notify available agents (Slack, internal dashboard)
2. Transfer full conversation context
3. LLM generates summary for agent: “User frustrated about delayed order #789, already offered refund but declined”
4. Agent accepts → WebSocket switches to human mode
5. Log handoff time, resolution time, outcome

6. Monitoring & Quality Control

Real-Time Metrics:

• Avg response time: <2s
• Resolution rate: >70% (no human escalation)
• User satisfaction (post-chat survey): >4.2/5
• Escalation rate: <15%

Offline Evaluation:

• Weekly human review: 100 random conversations
• Accuracy of tool calls: >95%
• Hallucination rate: <2%

7. Scaling for 50k Daily Conversations

Peak Load: ~35 concurrent conversations (assuming 10-min avg conversation)

Infrastructure:

• WebSocket Servers: 4 instances (Uvicorn/FastAPI), auto-scale up to 10
• Redis Cluster: 3 nodes (HA)
• LLM Requests: ~200k/day → batching, caching
• Database: PostgreSQL with read replicas

Optimizations:

• Semantic Cache: Similar questions → cached responses
• Streaming Responses: User sees tokens as generated (perceived speed ↑)
• Background Jobs: Sentiment analysis, intent logging (asynchronous)

Cost Estimate (50k daily conversations)

• LLM API (GPT-4o-mini): $500/day (15k conversations use LLM)
• Infrastructure (WebSocket, Redis, DB): ~$200/day
• CRM API calls: ~$50/day
• Total: $750/day ($22,500/month)

ROI: If each conversation saves 10 minutes of human agent time ($0.50/min), savings = ~$250k/month

Key Takeaways for Interview Success

When answering system design questions:

1. Start with Requirements Clarification

• Scale (users, requests, data volume)
• Latency/throughput targets
• Budget constraints
• Compliance (data privacy, retention)

1. Draw the Architecture First

• High-level boxes and arrows
• Data flow (synchronous vs asynchronous)
• External dependencies (APIs, databases)

1. Discuss Trade-offs

• Cloud LLM vs self-hosted
• Real-time vs batch processing
• Accuracy vs cost vs latency

1. Show Production Thinking

• Monitoring and alerting
• Failure modes and fallbacks
• Cost estimation
• A/B testing and rollout strategy

1. Know the Tools

• Vector databases: Pinecone, Weaviate, pgvector
• Model serving: TorchServe, Triton, KServe
• Orchestration: Airflow, Temporal, Step Functions
• Feature stores: Feast, Tecton

1. Cite Real Examples

• “At my last company, we used X for Y and learned Z”
• “I’ve read about how Netflix/Uber/Spotify does X”

Bonus: Red Flags Interviewers Watch For

❌ “Just use the most powerful model” (ignoring cost/latency)
❌ No error handling or monitoring
❌ Over-engineering (blockchain for a simple FAQ bot)
❌ Under-engineering (“just call OpenAI API” for 1M RPS)
❌ Ignoring data quality (garbage in = garbage out)
❌ No human-in-the-loop for high-stakes decisions

✅ Good answers:

• Start simple, add complexity incrementally
• Justify every component
• Acknowledge unknowns (“I’d need to benchmark X vs Y”)
• Ask clarifying questions throughout

Practice Resources

• Mock Interviews: Pramp, Interviewing.io
• System Design Primers:
• System Design Primer (GitHub)
• Machine Learning System Design (Chip Huyen)
• Real Architectures:
• Uber’s Michelangelo (ML platform)
• Netflix’s recommendation system
• Airbnb’s ML infrastructure blog posts

Final tip: The best system design answers tell a story of how you’d build it incrementally — MVP → scale → optimize → measure. Interviewers want to see you think like an engineer shipping to production, not a researcher writing papers.

Good luck! 🚀