The key insight? There was no single magic optimization. Instead, OpenAI systematically applied multiple safeguards that progressively addressed common pain points. Let’s explore their approach and what we can learn from it.

The Challenge: Explosive Growth Without a full Architecture rewrite

After launching ChatGPT, OpenAI experienced unprecedented traffic growth. Within a year, their database load increased around 10 times. Despite this massive growth, they maintained a single-primary architecture with one Azure PostgreSQL Flexible Server primary instance and nearly 50 read replicas across different regions.

Rather than migrating to a different database system, they took a methodical approach:

• systematically reduce peak load
• isolate risks
• quickly cut off problematic patterns

The rest of this article will show you their specific techniques, the historical context behind these approaches, and the associated risks.

1. Reducing Primary Load: Less Reading, Less Writing

The Problem

In a single-primary architecture, when you have one writer for the entire production system, it works well for read-heavy workloads, but struggles with write spikes. One bad deployment or backfill operation can instantly overload the primary. As OpenAI states directly: “heavy write spikes can quickly overload the primary.”

Their Solution

Offload reads from the primary

• Moved as many read queries as possible from the writer node to replicas
• Isolated the primary for critical write operations only

Separate write-heavy, shardable workloads

• Migrated horizontally partitionable workloads to separate databases like Azure Cosmos DB
• Particularly effective for event logs, metrics, and high-volume update streams

Application-level optimizations

• Eliminated duplicate or unnecessary writes
• Implemented lazy writes where business logic permitted, smoothing out write spikes
• Applied strict rate limits to backfill operations to prevent overwhelming the primary

Architectural constraint

• Established a rule: no new tables in the current PostgreSQL instance
• New workloads default to sharded systems

Industry Context

This is a standard strategy in high-scale systems. Keep PostgreSQL for the transactional core, and move write-heavy data (events, logs, high-frequency updates) to:

• Sharded/partitioned NoSQL databases
• Event stores
• Separate PostgreSQL clusters with partitioning

Lazy writes are equally common — deferring writes or batching them when business requirements allow.

Risks to Consider

• Hot partitions: Incorrect partition keys simply move the pain from relational to non-relational databases
• Consistency challenges: With state split between PostgreSQL and NoSQL, you need clear ownership of truth and reconciliation strategies
• Semantic differences: OLTP relies on transactions and joins; sharded systems don’t, requiring changes to data access patterns
• Operational complexity: Multiple databases mean multiple failure modes and incident types
• Product expectations: Lazy writes can break assumptions about data visibility

2. Query Optimization: Tackling OLTP Anti-Patterns

The Problem

OpenAI discovered that several expensive queries were consuming CPU and degrading API performance during traffic spikes. One particularly problematic query joined 12 tables, and spikes in this query’s execution led to high-severity incidents.

They also identified a “silent” problem: long idle transactions blocking autovacuum and causing table bloat.

Their Solution

SQL optimization

• Simplified complex multi-table JOINs where possible
• Split necessary JOINs into multiple queries, moving some logic to the application layer
• Specifically reviewed ORM-generated SQL, as this is often where performance disasters originate

Idle transaction management

• Configured idle_in_transaction_session_timeout to prevent idle transactions from blocking autovacuum

Industry Context

This is standard practice for high-QPS OLTP systems. While PostgreSQL excels in many scenarios, large multi-table joins can become problematic at production scale. ORMs simplify development but can transform simple queries into N+1 problems.

The “idle in transaction” issue is particularly insidious — PostgreSQL can keep transactions open, blocking vacuum or interfering with other sessions. The idle_in_transaction_session_timeout parameter does exactly what its name suggests: it terminates sessions stuck inside transactions.

Risks to Consider

• Timeout impact on legitimate use cases: Overly aggressive timeouts can break migrations, long-running admin jobs, or report generation. These require separate roles and connections.
• Application-level joins: Moving joins to the application can introduce N+1 queries, increase application load, and create race condition opportunities
• Profiling is essential: Fixing symptoms without proper profiling will lead you back to the same problems within months

3. High Availability: Surviving Primary Failures

The Problem

In a single-primary setup, the writer is the most critical point of failure. Losing a read replica is manageable — other replicas handle the load. But if the primary fails, writes stop completely, and sometimes even read queries fail.

Their Solution

Read resilience

• Moved nearly all critical read operations to replicas
• This ensures that even if the primary fails, the service can continue reading
• Writes fail, but this is no longer a SEV0 incident since the product remains partially functional

High availability configuration

• Deployed the primary in HA mode with a hot standby read replica
• The standby remains constantly synchronized and ready for promotion during failures or planned maintenance
• Expensive, but highly reliable

Regional redundancy

• Deployed multiple replicas in each region
• Single replica failure doesn’t cause regional outages

Industry Context

This is the fundamental pattern of graceful degradation. If you can make your product read-only temporarily instead of completely unavailable, that’s a significant win. HA with hot standby is standard for managed PostgreSQL services.

Risks to Consider

• Eventual consistency: Reading from replicas requires managing replication lag
• Failover complexity: Potential split-brain scenarios, Unexpected latency or connection issues after promotion, Need for clear writer identification
• Cost: Capacity buffers are expensive, but the alternative (total outages) is worse

4. Connection Pooling: Managing Thousands of Connections

The Problem

PostgreSQL doesn’t handle thousands of direct connections efficiently. Each connection consumes resources, and without connection management, applications can exhaust available connections during traffic spikes.

Their Solution

PgBouncer deployment

• Implemented connection pooling at scale
• Dramatically reduced the number of actual database connections
• Enabled connection reuse across application requests

Session vs. transaction pooling

• Carefully chose pooling modes based on workload characteristics
• Transaction pooling for stateless operations
• Session pooling where connection state matters

Industry Context

PgBouncer is the de facto standard for PostgreSQL connection pooling. It sits between applications and the database, multiplexing many application connections into fewer database connections. This is critical for applications with many short-lived queries.

Risks to Consider

• Mode selection is critical: Transaction pooling breaks session-based features (temp tables, prepared statements, advisory locks)
• Configuration complexity: Pool sizes, timeouts, and connection limits require careful tuning
• Monitoring overhead: An additional layer to monitor and troubleshoot

5. Workload Isolation: Separating Critical from Batch Operations

The Problem

When high-priority user-facing queries share resources with low-priority batch jobs or analytics, traffic spikes in batch operations can degrade user experience.

Their Solution

Dedicated infrastructure

• Separate read replicas for different workload types
• High-priority replicas exclusively serve user-facing queries
• Low-priority replicas handle analytics and batch operations

Resource allocation

• Clear boundaries prevent batch jobs from impacting critical paths
• Enables independent scaling of different workload types

Industry Context

Workload isolation is standard in mature systems. You separate:

• OLTP from OLAP
• User-facing from internal/admin
• Real-time from batch

This can be done through separate instances, replicas, schemas, or even separate databases.

Risks to Consider

• Increased cost: More infrastructure to maintain and pay for
• Operational complexity: More moving parts to monitor and manage
• Data freshness trade-offs: Analytics replicas may lag behind the primary

6. Cache Strategy: Reducing Database Load

The Problem

Without effective caching, every request hits the database. This creates unnecessary load on both read replicas and the primary, especially for frequently accessed but rarely changing data.

Their Solution

Multi-layer caching

• Application-level caches for hot data
• Careful cache invalidation strategies
• Cache warming for critical data paths

Request coalescing

• Implemented request deduplication for identical concurrent queries
• A single query serves multiple waiting requests
• Particularly effective for expensive queries or hot keys

Industry Context

Caching is fundamental to scalability, but implementation varies widely:

• CDNs for static content
• Redis/Memcached for application-level caching
• Application in-memory caches
• Query result caching

Request coalescing (sometimes called “request collapsing”) prevents stampedes when cache entries expire.

Risks to Consider

• Cache consistency: Stale data can break business logic
• Thundering herd: Cache misses on hot keys can overwhelm the database
• Complexity: More code paths to test and debug
• Hot key bottlenecks: Extremely popular keys can create queuing issues even with caching

7. Cascading Replication: Scaling Read Capacity Globally

The Problem

OpenAI maintains nearly 50 read replicas worldwide to reduce latency. However, in this architecture, the primary must stream WAL logs to every replica. More replicas mean increased network and CPU load on the primary, and greater replica lag. As they note: “we can’t add replicas indefinitely.”

Their Solution

Cascading replication architecture

• Testing with Azure PostgreSQL team
• Intermediate replicas retransmit WAL logs down the chain
• Enables scaling to hundreds of replicas without overloading the primary

Operational considerations

• Adds operational complexity, especially for failover management
• Still in the testing phase due to these complexities

Industry Context

Cascading replication is used when:

• You need many replicas
• Cross-region traffic is expensive
• The primary lacks network capacity to serve all replicas directly

PostgreSQL documentation explicitly describes how cascading replication reduces direct connections to the primary and can save cross-site bandwidth.

Risks to Consider

• Failover complexity: Who’s upstream, who’s downstream, how to rebuild the tree
• Intermediate replica failures: Affect entire subtrees, requiring clear failover matrices
• Lag propagation: Replication lag can compound across the chain
• Debugging challenges: More complex to diagnose issues in multi-tier replication

8. Rate Limiting and Query Blocking: Preventing Cascading Failures

The Problem

OpenAI describes several SEV incident scenarios:

1. Sudden endpoint traffic spikes
2. Expensive query spikes
3. Retry storms exhausting CPU, I/O, and connections
4. Complete system degradation

Their Solution

Multi-layer rate limiting

• Application level
• Connection pooler level
• Proxy level
• Query level

Retry policy control

• Careful management of retry intervals
• Prevention of retry storms

Targeted load shedding

• ORM-level rate limiting support
• Ability to completely block specific query digests
• Quickly shed the worst traffic to enable system recovery

Industry Context

In mature large-scale systems, rate limiting and load shedding exist at multiple layers:

• API gateways
• Services
• Workers
• Databases

The distinction between rate limiting and load shedding is important — rate limiting prevents problems; load shedding responds to them.

Risks to Consider

• Collateral damage: Can accidentally block business-critical traffic
• Incomplete coverage: Limits at entry points don’t prevent retry storms from workers
• Query digest blocking: A powerful lever that requires careful processes to avoid blocking legitimate traffic

9. Schema Management: Lightweight DDL Only

The Problem

Even small schema changes (like ALTER COLUMN TYPE) can trigger full table rewrites. On large tables, this means hours of I/O, replication lag, long locks, and degradation of the production database.

Their Solution

Strict DDL policies

• Only lightweight schema changes permitted
• 5-second timeout on all schema changes (DDL must not “hang”)
• CREATE/DROP INDEX CONCURRENTLY allowed (reduces write blocking)

Table creation freeze

• Schema changes only on existing tables
• New tables for new features go to sharded systems (CosmosDB)

Controlled backfills

• It can take over a week
• Strictly rate-limited to prevent write spikes

Industry Context

In high-load OLTP systems, DDL is extremely dangerous. To prevent incidents, teams either:

• Implement zero-downtime migrations (expand/contract, shadow columns)
• Enforce strict rules like OpenAI’s

Risks to Consider

• Lock escalation: Even “lightweight” DDL often takes strong locks by default (PostgreSQL docs explicitly mention ACCESS EXCLUSIVE for many operations)
• Architectural constraints: Refusing new tables in PostgreSQL is a strong limitation that forces careful domain boundary thinking
• Human error: Someone will run a seemingly innocent ALTER that triggers a rewrite
• Automation is essential: Rules must be automatically verified

Understanding Write Amplification in PostgreSQL

OpenAI explicitly addresses why heavy writes are painful in PostgreSQL: MVCC (Multi-Version Concurrency Control). When updating even a single field, PostgreSQL copies the entire row. This creates:

• Write amplification
• Dead tuples
• Increased read amplification

For deeper analysis, they reference “The Part of PostgreSQL We Hate the Most” by Bohan Zhang and Andy Pavlo.

The Results

After one year:

• PostgreSQL load increased over 10x
• Single primary + ~50 read replicas maintained
• Near-zero replication lag
• Low-latency reads across regions
• Low double-digit milliseconds for p99 client-side latency
• Five-nines availability
• Only one SEV-0 incident in a year (during ImageGen viral launch with 10x write traffic spike and 100M+ new users in a week)

This isn’t a story of finding one trick that magically solved everything. It’s a story of progressively and deliberately addressing nine different problem classes.

Practical Takeaways: What You Can Apply Today

Quick Wins (Can Implement Tomorrow)

• Enable and tune idle_in_transaction_session_timeout. Separate migrations and admin jobs into different roles/connections
• SQL review practice for ORM-generated queries. Start with top-N queries by CPU and latency
• Rate limit your most problematic endpoints
• Enforce rate limits on all backfill operations

Medium-Term Improvements (Requires Maturity & Discipline)

• Deploy PgBouncer for connection pooling
• Implement workload isolation
• High/low priority separation
• Dedicated instances/replicas for different workload types

Advanced Optimizations (Difficult But High ROI at Scale)

Schema change policy

• Only lightweight DDL
• Timeouts enforced
• CONCURRENTLY for indexes
• Automated checks for table rewrites

Migrate write-heavy workloads

• Move to sharded systems or separate clusters
• Eliminate redundant writes
• Implement lazy writes where appropriate

Cascading replication

• For deployments with many replicas where primary WAL streaming becomes a bottleneck

Conclusion

OpenAI’s journey with PostgreSQL demonstrates that scaling doesn’t always require radical architecture changes. Through systematic application of proven techniques — each addressing a specific pain point. They achieved remarkable results while maintaining a single primary architecture.

The lesson isn’t that PostgreSQL is perfect for all use cases, but that disciplined engineering practices, careful monitoring, and incremental improvements can take you surprisingly far. Every technique they employed is well-documented and battle-tested; the innovation was in the systematic application and operational discipline.

Whether you’re managing 800 million users or 800 thousand, these patterns provide a roadmap for scaling relational databases effectively.

Background

Apache Airflow has long been the go-to standard for building complex workflows, especially in data engineering and machine learning. The 2.x series brought stability, DAG-level parallelism, task mapping, and efficient scheduling.

However, with the rise of distributed, event-driven, and polyglot architectures, limitations began to emerge. The monolithic architecture of version 2.x struggled to meet modern requirements.

Airflow 3.0 is not just a version upgrade — it’s a rethinking of architecture, UI, and even the orchestration philosophy. Let’s explore each part in detail.

A Completely Redesigned UI

This is arguably the most visible and delightful improvement. While Airflow already had a functional UI, it was outdated and long overdue for an overhaul.

The new architecture ensures a much smoother user experience, especially in environments with hundreds or thousands of DAGs. Pages load faster, UI elements are more intuitive, and the navigation model is consistent and responsive across all sections.

Key UI Improvements:

• Improved Grid View: Better timeline navigation, filtering, and searching. Users can now more easily inspect DAG run statuses and diagnose failed or stuck tasks.
• Enhanced Graph View: Improved zooming, scrolling, and task metadata interactions make understanding complex DAGs at a glance much easier.
• Asset Panel: A new interface specifically for data-centric workflows. It allows users to visualize assets, their producers and consumers, and track DAG interactions.
• Built-in Dark Mode: Although dark mode existed in version 2.x, this release introduces a fully polished, design-first dark theme. This was one of the most requested features from the community.
• Dedicated DAG Runs Tab: Now with full customization. You can enable only the columns you need and sort by almost any column — including duration (available in v3.1) and conf. This dramatically improves usability.
• FastAPI Instead of Flask-AppBuilder: A major performance win. It brings async support, better type validation via Pydantic, and overall improved performance for the Airflow API.

Dark mode

DAG Runs page

Assets panel

DAG Versioning

One of the most critical features introduced in Airflow 3.0 is full support for DAG versioning.

In previous versions, DAGs were read directly from code files, and the latest version was always applied, even to previously executed runs. This led to several well-known issues:

• Execution Drift: If a DAG was updated during execution, different tasks could run with different versions.
• No Auditing: Airflow lacked built-in tools to trace which version of a DAG was used in a specific run.
• Historical Inaccuracy: The UI always showed the latest DAG version, even for older runs, making the execution history unreliable or misleading.

With DAG versioning:

• You can rerun a DAG in the exact version it originally ran.
• Errors can be tied to specific code versions.
• In regulated environments, auditing is now possible.
• CI/CD workflows can now integrate commit hashes directly with Airflow DAG versions.

Airflow 3.0 introduces:

• Automatic DAG versioning on every code change or deployment.
• Historical DAG runs are preserved with their specific version.
• UI lets you select and rerun specific DAG versions.

I find this feature particularly appealing, as in practice it’s nearly impossible to create a DAG that never requires changes. Business logic evolves, bugs emerge, or optimizations become necessary, and built-in versioning makes this process transparent and manageable. The screenshot below shows the new DAG orchestration interface along with a dropdown list of version history.

DAGs versioning

Asset-Driven DAGs

The cornerstone of smart scheduling in Airflow 3.0 is the concept of Assets. Inspired by datasets introduced in earlier versions, the new @asset decorator enables pipelines that react to data states.

An Asset in Airflow is a data unit or result created, transformed, or consumed by a DAG.

Assets can be:

• Database tables
• Files in cloud storage
• Trained ML models
• API responses

Building DAG-to-DAG dependencies using an intermediate asset.

To create this example, you only need to define 2 DAGs with the following code.

# producer DAG
from airflow.sdk import Asset, dag, task

@dag
def my_producer_dag():
    @task(outlets=[Asset("my_asset")])
    def my_producer_task():
        pass

    my_producer_task()


my_producer_dag()

# consumer DAG
from airflow.sdk import Asset, dag
from airflow.providers.standard.operators.empty import EmptyOperator

@dag(schedule=[Asset("my_asset")],)
def my_consumer_dag():
    EmptyOperator(task_id="empty_task")

my_consumer_dag()

Now, when the producer is triggered, data will be written to our asset, and afterward, the consumer will automatically be triggered with the data in the asset. You can use the “&” (and) and “|” (or) operators to create more complex dependencies between assets. Something like this:

from airflow.sdk import Asset, dag

@dag(
    schedule=(Asset("asset1")  | Asset("asset2") | Asset("asset3") | Asset("asset4")),
)
def downstream1_on_any():
    pass
downstream1_on_any()

This allows the next DAG to be triggered even if only one asset changes. It will appear in the DAG console like this:

Conditional asset scheduling

Assets also support:

• Structured Metadata: Including name, URI, group IDs, and custom metadata fields for easier debugging and search.
• Composable Design: Assets can refer to each other for modular, interdependent pipelines.
• Watchers: Currently supporting AWS SQS, assets can be monitored for external signals that trigger DAG runs. More integrations are promised soon.

This turns Airflow into a “run when the data is ready” scheduler rather than a purely time-based one.

Architectural Changes

Airflow 3.0 focuses on modularity, scalability, and deployment flexibility — fitting the needs of hybrid and multi-cloud data platforms.

Comparison of architecture of Airflow 2 and Airflow 3

Task Execution API and SDK

One of the most powerful changes is the new Task Execution API and Task SDK, allowing task execution independent of Airflow’s core engine.

Benefits:

• Multi-language Support: Tasks can now be written in Java, Go, R, etc.
• Isolated Environments: Run tasks in containers or remote environments, avoiding dependency conflicts.
• Simplified Dev Workflow: Standardized interfaces make testing, deploying, and onboarding easier.

This architecture is a breakthrough that makes Airflow more flexible and cloud-native. It aligns with the broader trends in data engineering toward language diversity, containerization, and task-level reproducibility.

Edge Executor

The Edge Executor allows event-driven and geo-distributed task execution. Tasks can now run close to the data source — whether that’s a remote data center or an edge device.

Advantages:

• Geographic flexibility: Tasks can be executed in remote or regional clusters, data centers, or even on edge devices.
• Instant responsiveness: Ideal for low-latency, event-driven use cases — such as IoT, financial streams, or real-time monitoring.
• Reduced network latency: Computation happens near the data source, lowering the cost of transferring large data volumes to a central cluster.
• Event-driven pipelines: The Edge Executor fully supports the new scheduling model introduced in Airflow 3.0.

ML and AI Workflow Support

Airflow 3.0 becomes a first-class orchestration tool for modern ML/AI pipelines.

Interval-Free DAGs

Older Airflow versions relied on execution_date and fixed scheduling intervals. This didn’t work well for ML workflows.

Now, DAGs allow you:

• Model experimentation: Run the same DAG multiple times with different model configurations or data subsets, without conflicts caused by execution_date.
• Hyperparameter tuning: Execute multiple DAGs in parallel with different parameter grids or search strategies — each tracked independently.
• Inference DAGs: Invoke DAGs repeatedly to process new data in real time or for batch predictions triggered by external events or user actions. Ideal for flexible scenarios like inference, data quality checks, or user-triggered workflows.
• Event-driven DAGs: DAGs can now be triggered by external events, such as a new file appearing in cloud storage or a Kafka message. This enables true real-time reactive orchestration.

This decouples execution logic from time-based scheduling, making Airflow more flexible for ML engineers.

Developer Experience Improvements

CLI Split: airflow and airflowctl

Airflow 3.0 splits the CLI into two tools:

• airflow — for local dev and testing
• airflowctl — for managing remote production environments

This separation improves workflows for both developers and platform teams.

Migration Guide: 2.x → 3.0

Minimum Requirements:

• Python 3.9+

Steps:

1. Upgrade to latest 2.x (2.7.3+ recommended)
2. Check DAGs for deprecated features
3. Backup your DB
4. Install 3.0 in a separate environment
5. Test DAGs
6. Integrate airflowctl into production

Airflow 2.x vs 3.0: Comparison Table

Should You Upgrade?

Yes, if you:

• Need secure, remote task execution
• Want DAG versioning
• Build real-time, event- or asset-driven orchestration
• Use multi-language pipelines

Wait, if you:

• Have deep Flask customizations
• Don’t use event-based scheduling or Assets
• Still use Python < 3.9 or have legacy dependencies

Final Thoughts

Airflow 3.0 is a major release with timely and impactful improvements. Support for event-driven and inference pipelines opens doors for many ML/AI projects. The reworked architecture makes Airflow more modular and decentralized. DAG versioning and asset orchestration elevate reliability and clarity.

This release is particularly exciting to me because I personally contributed to some of its new features. 🙂

Hey everyone!
I’m Valentyn, and today I’d like to share my experience with the Machine Learning Specialization on Coursera. It’s the updated version of Andrew Ng’s course — yep, the same Andrew Ng who launched the original ML course over 10 years ago. Back when it wasn’t mainstream yet 🙂

What’s inside?

The specialization includes three courses:

1. Supervised Machine Learning
Explains what ML is, how regression, classification, and neural networks work. A lot of focus is on intuition — why a model behaves a certain way, which metrics to choose, how to detect overfitting, etc.

2. Advanced Learning Algorithms
Covers deeper models, optimization, regularization, and multilayer neural networks. You’ll start using TensorFlow, and it already feels like real ML work.

3. Unsupervised Learning, Recommenders, Reinforcement Learning
In my opinion, this one was the easiest. It covers clustering (k-means), PCA, recommendation systems, and a short intro to reinforcement learning. Feels more like an overview than a deep dive.

What models and algorithms are covered?

• Linear and logistic regression
• Softmax, decision trees
• Neural networks (single and multilayer)
• K-means, PCA
• Recommendation systems (collaborative filtering)
• Q-learning (intro-level)

What tools are used?

Everything’s done in Python.
Core libraries:

• NumPy, Pandas, Matplotlib
• TensorFlow (via Keras API)

The best part — you don’t need to install anything. All the exercises run in Jupyter Notebooks, right in your browser.

How long does it take?

It took me over two months, to study about 6–8 hours per week. The material is explained clearly and accessibly. Even if your math is a little rusty — don’t worry, you’ll be fine.

Pros:

• Super clear and accessible explanations
• Lots of hands-on practice right after each module
• The difficulty grows gradually
• You genuinely understand what ML is and how it works

Cons:

• I wish there were more real-world tasks (like Kaggle competitions or industry use cases)
• The reinforcement learning part feels a bit shallow — more of a preview than a full course

Who is this for?

• Anyone looking to get started in AI/ML
• Analysts, QA engineers, and developers thinking of switching fields

Final thoughts

I liked this specialization. It’s not just theory — you learn how to build models.
If you’re starting from scratch (or almost), and want to understand what machine learning is and how to use it — this is a great place to start.

Next, you’ll want to dive into more practical, real-world tasks — and Kaggle is perfect for that.