<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Tise Product &amp; Tech - Medium]]></title>
        <description><![CDATA[Tise is a Nordic, fast-growing software company. We do everything from coding to design and business development in-house. Our marketplace-app, Tise, was introduced to the world in 2016, with the aim of making it more easy and inspiring to buy and sell second hand. - Medium]]></description>
        <link>https://medium.com/tise-product-tech?source=rss----2e523a5f1218---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Tise Product &amp;amp; Tech - Medium</title>
            <link>https://medium.com/tise-product-tech?source=rss----2e523a5f1218---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 13:06:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/tise-product-tech" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Inside Our Strategy for Zero-Downtime PostgreSQL Upgrades on AWS RDS]]></title>
            <link>https://medium.com/tise-product-tech/inside-our-strategy-for-zero-downtime-postgresql-upgrades-on-aws-rds-fd5a77b7da1d?source=rss----2e523a5f1218---4</link>
            <guid isPermaLink="false">https://medium.com/p/fd5a77b7da1d</guid>
            <category><![CDATA[aws-rds]]></category>
            <category><![CDATA[postgres]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[postgresql]]></category>
            <dc:creator><![CDATA[Prasad Ostwal]]></dc:creator>
            <pubDate>Fri, 10 Oct 2025 07:02:06 GMT</pubDate>
            <atom:updated>2025-10-10T07:02:05.539Z</atom:updated>
            <content:encoded><![CDATA[<p>👋 We’re <a href="http://tise.com/">Tise.com</a>, a leading fashion marketplace in the Nordics. In this post, we’ll share how we upgraded our three production PostgreSQL databases.</p><blockquote><em>✅ </em><strong><em>TL;DR:</em></strong><em> We upgraded three production PostgreSQL databases on AWS RDS from v13 to v17 using RDS Blue/Green Deployments — and we did it with no downtime. It took weeks of preparation, dry runs, and careful planning, but the final switch was smooth and repeatable. In this post, we share how we approached it, what worked, and what we learned from the process. The notes that we captured in Notion during the process became the backbone for both this article and our final runbook.</em></blockquote><h3>Why we had to upgrade</h3><p>Running on PostgreSQL 13 had served us well, but it was reaching the end of its support lifecycle, and earlier upgrades would have been risky and time-consuming. AWS RDS encourages timely upgrades, and with Postgres 17 available, skipping directly to 17 made sense from a support and performance perspective.</p><p>This wasn’t just a regular quick and easy routine bump. Our setup today looks like this:</p><ul><li>Three production databases with different workloads</li><li>Zero downtime tolerance and millions of users who rely on us</li><li>High stakes for data safety, so even small inconsistencies can cascade into major user-facing issues</li></ul><p>We wanted to perform this upgrade in a way that was safe, repeatable, and stress-tested, so that we can ensure that future major version jumps will be low-risk and seamless as well.</p><p>We evaluated the following options for the upgrade:</p><ul><li><strong>Simple Dump/Restore</strong> — Too much downtime for hundreds of gigabytes of data.</li><li><strong>In‑place major version upgrade</strong> — RDS does support a direct 13 to17 jump, but the maintenance window and potential hours of unavailability didn’t fit our no downtime policy.</li><li><strong>AWS DMS</strong> — We felt it’s too black‑box for our use case.</li><li><strong>Manual blue-green upgrade with replication</strong> — More control and visibility, but also more prone to errors.</li><li><strong>RDS Blue/Green</strong> — <em>chosen</em>. It allows us to provision a Green cluster on PG 17, replicate from Blue (PG 13), then swap endpoints when we’re satisfied. Under the hood this is logical replication, with RDS managing slots and the cutover. It struck the balance between visibility, safety, and minimal downtime.</li></ul><h3>Preparing for the upgrades</h3><p>Before we touched a single production database for migration, we invested significant time in preparation. The goals were simple: minimize risk, avoid downtime, and surface potential surprises early. Here’s how we approached the migration, step by step:</p><ol><li><strong>Read release notes</strong>: Reviewed Postgres 14–17 changelogs to spot breaking changes and query behavior shifts.</li><li><strong>Schema dump &amp; AI check</strong>: Took schema-only dumps and even ran them through an LLM against changelogs to highlight potential incompatibilities.</li><li><strong>Extension compatibility</strong>: Verified installed extensions (e.g. PostGIS) against supported versions with compatibility matrix and planned upgrades where needed.</li><li><strong>Replication requirements</strong>: Ensured every table had a primary key, and checked WAL/replication slot settings for Blue/Green.</li><li><strong>Table &amp; index analysis</strong>: Measured table sizes, stats freshness, and index usage; trimmed unused indexes ahead of replication.</li><li><strong>Analyze strategy</strong>: Planned to run ANALYZE immediately on green (stats aren’t transferred to the green database)</li><li><strong>PgBouncer validation</strong>: Confirmed PgBouncer compatibility with PG17 and tested connection handling. On our very first cutover we discovered some of our older PgBouncers didn’t actually support PG17, more about that later.</li><li><strong>Parameter groups</strong>: Created new PG17 parameter groups and carried over observability settings. Verified if the newer version needs any special parameters.</li><li><strong>Dry runs</strong>: Practiced a lot with restored snapshots in development environment with Blue/Green to watch replication lag and test monitoring queries.</li><li><strong>Preflight checklist</strong>: Final “go/no-go” included: fresh snapshots, freeze schema changes, verify extensions/params, and record baseline query timings.</li><li><strong>Keep handy queries:</strong> We used pg_stat_statements to identify the heaviest queries on our existing databases. These queries were noted down so we could benchmark them later on both blue and green, ensuring performance was not degraded after cutover.</li></ol><p>For us, this phase was about turning unknowns into knowns. Every risk we could discover in practice was one less surprise waiting on production day.</p><h3>How we de-risked the upgrade</h3><p>Running a zero-downtime upgrade on three production databases meant assuming failure modes upfront, and asking ourselves the following questions: “<em>What if replication lags? What if ANALYZE isn’t done in time? What if Pgbouncers fail? What if an extension breaks?”</em> By playing out each of these scenarios beforehand, we built guardrails for safety. Also, rather than going directly into production, we began by testing with the following:</p><ul><li>Practiced on snapshots to break things safely.</li><li>Repeated the process in development to validate app behavior.</li><li>Rehearsed again on production snapshots for all possible scenarios.</li></ul><p>Only after three full dry runs did we attempt the real cutover. Each rehearsal gave us new insights, so by D-day, little was unknown.</p><p>We also prepared for the worst-case scenario — a failure right after cutover. That could have meant data loss and a scramble to restore writes on the old cluster. To ensure we were fully prepared, we had a rollback plan ready:</p><ol><li>Stop traffic to green immediately.</li><li>Re-enable writes on blue</li><li>Redirect applications back.</li></ol><p>Fortunately, this scenario never occurred, but knowing the fallback existed reduced pressure and made the cutover far less stressful.</p><h3>The practical runbook</h3><p>This runbook was the heart of our upgrade process. It distilled preparation, dry runs, and live note-taking into a sequence of repeatable steps we could trust. Each database followed the same playbook, and the consistency gave us confidence when it was time to flip the switch.</p><h3>1. Preflight</h3><p>We began by taking a fresh manual snapshot of the source (blue) database. At the same time, we double-checked parameter groups, extension compatibility, and logical replication requirements. To establish a baseline for performance comparisons, we also ran a set of heavy queries and recorded their timings.</p><h3>2. Create the Blue/Green deployment</h3><p>The next step was to create the Blue/Green deployment in the AWS console. During this process, RDS automatically adjusted some parameters, such as max_wal_senders, which occasionally triggered a reboot of the green cluster. We also made sure that networking, security groups, and IAM roles were consistent between blue and green to avoid surprises later.</p><h3>3. While replication catches up</h3><p>Once the deployment was running, we closely monitored replication progress. The AWS console often showed misleading values, so instead we relied on SQL queries like pg_wal_lsn_diff and CloudWatch metrics such as OldestLogicalReplicationSlotLag to measure lag accurately. This reassured us that green was staying in sync.</p><h3>4. Analyze statistics on green</h3><p>As soon as green was available, we ran ANALYZE across all tables. Since optimizer statistics do not transfer across major versions, this step was essential to avoid poor query plans. Using pg_stat_progress_analyze, we monitored its progress and confirmed completion before moving forward.</p><h3>5. Validate queries and plans</h3><p>With fresh statistics in place, we replayed the heaviest queries gathered from pg_stat_statements on both blue and green. We compared execution plans with EXPLAIN ANALYZE to check for regressions and verified that extensions such as PostGIS continued to behave correctly under the new version.</p><h3>6. Sanity-check replication health and data</h3><p>Next, we validated replication health. We confirmed that replication slots were advancing normally and that WAL files were not piling up. Spot checks of row counts between the two clusters gave us further reassurance that the data matched.</p><h3>7. Point replicas to green for read traffic</h3><p>Before the final switch, we redirected our read replicas to the green cluster. This step allowed us to test real production read traffic on green while blue was still handling writes. It gave us valuable insight into how query performance held up under actual workload patterns and connection pooling, not just synthetic benchmarks. The results showed stable performance, which reinforced our readiness for the cutover.</p><h3>8. Cutover</h3><p>When replication lag reached zero and all checks passed, we triggered the switchover, making green the new writer. The cutover itself took only seconds, but we kept a close eye on application logs, connection pools, and error dashboards to catch any anomalies.</p><h3>9. Post-cutover cleanup</h3><p>After the switch, we verified that all read and write traffic was flowing to green correctly. We then removed the Blue/Green deployment object from AWS to prevent accidental writes to the old cluster and ran a series of business-level health checks to confirm everything was working end-to-end.</p><h3>10. Post-cutover monitoring</h3><p>The final phase was ongoing monitoring. We kept dashboards open for CPU usage, WAL growth, connection counts, and slow query logs, and we tailed the Postgres logs for unexpected behavior. We also re-ran the validation queries we had benchmarked earlier, ensuring that performance remained consistent after the transition.</p><p><em>Runbook summarised:</em></p><figure><img alt="runbook summary" src="https://cdn-images-1.medium.com/max/999/1*09lPuqcyr4wNvMxZ-UTs1A.png" /></figure><h3>Unexpected hiccups we faced</h3><p>Despite weeks of preparation and several dry runs, production still had a couple of surprises for us. We ran into two main issues — one with PgBouncer versions and another involving Blue/Green WAL receiver behavior.</p><p><strong>PgBouncer version mismatch</strong></p><p>For the very first database we cut over, connections suddenly failed. It turned out our PgBouncer instances were still running on old EC2 hosts that didn’t support Postgres 17, since we hadn’t upgraded them in years. We had to scramble and upgrade PgBouncer before traffic could flow again. Luckily, the fix was quick. We eventually scrapped those EC2-based PgBouncers entirely and migrated to a proper ECS service with the latest version.</p><p><strong>Blue/Green quirks with WAL receiver</strong></p><p>On one of the larger databases, our first Blue/Green deployment failed when the WAL receiver on the green side unexpectedly stopped. We had to recreate the deployment three times before replication worked as expected. While we had anticipated these RDS Blue/Green quirks, there wasn’t much we could do since most of the underlying process is hidden behind a black box.</p><p>These hiccups reinforced our belief that rehearsals and rollback plans aren’t optional. Because we had already imagined failure scenarios, both incidents were handled smoothly and minimal downtime.</p><h3>Lessons from upgrade</h3><p>What started as a daunting project turned into a repeatable process we now trust. The success didn’t come from a clever shortcut, but from careful preparation, rehearsals, and documentation. Along the way, we learned:</p><ul><li>Preparation pays off more than execution. The actual switchover took seconds and it was mostly uneventful; the weeks of planning, testing, and rehearsals were what made it smooth. If you feel bored of your checklist, that’s a good sign.</li><li>Replication metrics are subtle. Cloud consoles often present a simplified picture. To really know if replication was safe to cut over, we had to use SQL queries and cross-check multiple metrics.</li><li>Operational surprises are normal. Even managed services like RDS have quirks — parameters get auto-tuned, nodes reboot at inconvenient times. And sometimes, the issue isn’t RDS at all — our old PgBouncers didn’t support Postgres 17 and blocked connections right after cutover.</li><li>Incremental validation beats a big bang test. We didn’t wait until the final cutover to test the new cluster. Exercising it with read traffic, replaying heavy queries, and checking extensions gave us incremental confidence along the way.</li></ul><p>If you’re planning your own upgrade, we hope this playbook gives you a solid starting point — and makes your next major version upgrade smoother and less intimidating.</p><p>We’re always excited to chat about tech, engineering, and the stuff we’re passionate about. Get in touch!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fd5a77b7da1d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tise-product-tech/inside-our-strategy-for-zero-downtime-postgresql-upgrades-on-aws-rds-fd5a77b7da1d">Inside Our Strategy for Zero-Downtime PostgreSQL Upgrades on AWS RDS</a> was originally published in <a href="https://medium.com/tise-product-tech">Tise Product &amp; Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>